Transformation in the Air: A Review of the FAA's Certification Research Plan (2015)

2

Specific Shortfalls in the February 2014 Research Plan

There are a number of issues that the February 2014 Research Plan¹ addresses only superficially and are in need of more extensive discussion in any future research plan. Many of these issues are both highly complex and interrelated due to the nature and scope of NextGen. NextGen is a large, complex system made up of systems, each of which is a “system of systems.” NextGen also exists within a global context, meaning that it has to operate with other air traffic control systems around the world, such as those under development in the European Union Single European Skies Air Traffic Management Research program (SESAR), as well as Canada’s system (briefly discussed in Chapter 3). These complex issues can be divided into general categories that interact and overlap in complicated ways.

SYSTEMS ENGINEERING

Systems engineering is used to manage complexity and is most valuable when applied to address the entire system throughout its life cycle. Systems engineering principles are necessary to manage the integration obstacles that must be met in order for the NextGen National Airspace System to run smoothly. These challenges, with the complexity introduced by systems maturing and also transitioning into operation on a timeline spread over more than two decades, create a strain on engineering and integration. The FAA’s approach to grappling with this is explained in the FAA Systems Engineering Manual (SEM).² The SEM demonstrates the FAA’s understanding of the principles and processes necessary to support integration and achieve lifecycle management. Unfortunately, systems engineering does not receive detailed discussion in the February 2014 Research Plan, and the plan does not even cite the SEM as a reference.

____________________

¹ FAA, Research Plan: Methods and Procedures to Improve Confidence in and Timeliness of Certification of New Technologies Into the National Airspace System, Final, Office of NextGen, Washington, D.C., February 2014; reprinted in Appendix A.

² FAA, Systems Engineering Manual, Version 1.0.1, June 19, 2014, https://nasea.faa.gov/publications/main.

REQUIREMENTS DEVELOPMENT

Requirements development is an important and fundamental component of the systems engineering process. Requirements development must be devolved from and linked to the system strategic plan in terms of objectives and schedule. It is by necessity an integrated team activity involving operational components and technology developers with strong human factors influence in the case of a system such as NextGen.

The FAA Systems Engineering Manual (SEM) and Acquisition Management System (AMS) lay out clear descriptions of the importance and role of requirements analysis to include requirements development and management, and it is in the best interests of technology program managers to rigorously follow the guidelines in the AMS. The committee agrees with the governing premise laid out by the February 2014 Research Plan:³

Unfortunately, the research plan contradicts this stated assumption by focusing on the design approval process—a programmatic concern that takes place very early in the acquisition process prior to initial investment decision. The bulk of the plan’s proposed outputs⁴ address design approval. It is the only approval area of the three shown in Figure 1.1 that the plan indicated would be addressed and provides no indication of how the Acquisition Management System procedures are to be followed.

The majority of technologies needed to deliver near- and mid-term NextGen capabilities are at or approaching maturity. What the committee would expect in the plan is that it would focus on research for the approvals/ certifications that are required to transition NextGen to the field and into the hands of the users.

The plan also lacks a description of a coherent concept for the research and how elements of the plan relate to the overall NextGen implementation plan. Also missing is any stated intent to conduct a shortfall analysis or identify, prioritize, and extract stakeholder needs and develop relevant requirements for the product of the research. Without defined requirements, the plan cannot state objectives, propose tasks, develop milestones, submit a budget, or define how it will measure success.

HUMAN FACTORS

Human factors may be defined as the scientific discipline concerned with the understanding of interactions among humans and other elements of a system. The field of human factors also includes the engineering profession that applies theory, principles, data, and methods with the goal of improving the human operator’s contribution to system performance through improved technologies, work environments, work practices and procedures, and training. For NextGen, the key human operators include not only air traffic controllers and pilots, but also those with whom they interact, including traffic flow management, cabin crew, and airline operations. Key operators may also need to include the technicians and ground handlers who service and maintain the technologies.

The human’s contribution to system performance is provided in several key ways that should be considered in certification, specifically the approval of the operational capability of the technology and its ultimate implementation by National Airspace System users. One important contribution is when humans operate and interact with technology, both as users of information and as the controllers or supervisors of automation; here, human-computer interaction, human-automation interaction, and human supervisory control of automation are important fields to consider.

Human operators also contribute to system performance in several other important ways that must also be accounted for in certification. This includes the following:

____________________

³ FAA, Research Plan, 2014; reprinted in Appendix A, p. 4.

⁴ FAA, Research Plan, 2014; reprinted in Appendix A, p. 5.

• Human operators communicate and interact with each other. Examples relevant to NextGen include Collaborative Decision Making, and trajectory negotiations conducted before and during flight that may involve traffic flow management, air traffic controllers, pilots, dispatchers, and airline operations managers. Historically, new developments directed at one operator have had side-effects in how those operators interact with others, such as changes in air traffic control required to allow more fuel-efficient descent profiles. Air and ground operators cannot be certified independently with confidence in tightly-coupled operations.
• Human operators integrate multiple different tasks and systems. Thus, procedures and human-integrated-systems that are developed and certified as correct independently may not function together. For example, different prototype air traffic systems have proposed different content and information for the controller’s data block. Each alone may be fine for their purpose, but their combination may be confusing or overwhelming. Likewise, different air traffic procedures may each be manageable on their own, but their combination may create workload spikes or other operational blocks to effective human contributions to system performance.
• Human operators adapt their behavior to meet goals and procedures. Unlike technologies with fixed, deterministic functioning, human operators can vary their behavior to meet goals and procedures. However, this strength also implies that they can use the same system differently in two contexts, and thus the context of use must also be examined in certification, including operational procedures and best practices.

SAFETY AND PERFORMANCE ASSURANCE—SOFTWARE ASSURANCE AND SOFTWARE CERTIFICATION

Software assurance addresses the confidence that the software functionality performs as intended without additional side effects. The FAA defines software assurance as “the level of confidence that software is free from vulnerabilities, either intentionally designed into the software or accidentally inserted at any time during its life cycle, and that the software functions in the intended manner.”⁵

The February 2014 Research Plan identifies software assurance and software certification as areas of technologies to be further investigated. The plan specifically identifies researching current and future processes of air traffic control to derive measures. However, it falls short at identifying source motivators for improving the software assurance, an assessment of the current software assurance practices, and the desired level or attributes to target in its software assurance program.

Relevant software assurance issues include the following:

• Software complexity. Software has become and will continue to be prevalent in implementing existing and future functionality across the FAA enterprise, including the ground, aircraft, and associated communications. These software implementations range in size, complexity, and cohesion, highlighting the importance of addressing software assurance as a means to enhance confidence of intended and only intended behavior.
• Staged software development and integration. Software implementations often involve integration of modules developed over time. This is particularly true for functionality integrated across the aircraft and ground systems. The staged software development and integration may result in limited validation, analysis, and testing of the overall capability.

____________________

⁵ FAA Order 1370.109.

• Increased usage of open source and commercial off-the-shelf software. The increased use of software from different sources introduces many challenges, including integrating software with different pedigrees with varying levels of trust. Other categories of software that may have varying pedigrees include legacy software and reusable software. The varying sources and type of software for analysis range from having detailed source code to executables.
• Software integration into overall system. Software is one of many components comprising future capabilities. As such, it is important that the software analysis be fully integrated in certification, verification, and validation activities. Software assurance needs to be considered in this context to improve assurance as part of integrated and timely processes and increase confidence.
• Integrated test beds. Test beds provide an opportunity to perform enhanced analyses targeting software assurance in a low-risk environment. Test results need to be appropriately interpreted and incorporated into the testing, verification, and validation activities.
• Data privacy. A system such as the current ability of a user to block FAA release of flight information for a particular aircraft has been mentioned in various FAA documents and is important as automatic dependent surveillance-broadcast (ADS-B) is implemented.
• Resilience to cyberattack. Software assurance activities need to be included as a dimension of the overall cybersecurity strategy and implementation targeting continued system operations during cyberattack. Systematic approaches to cybersecurity are a fundamental part of effective and improved future technologies and capabilities. Cybersecurity spans the development and operational activities including failure/anomaly analysis and integrity of the software development tools and environment.

There are increasing indicators of cyberattacks targeting and exploiting various programs, technologies, and stakeholders. Consequently, cybersecurity should be addressed as an integral part of delivering services to ensure that the threat environment be considered as part of the architecture, design, development, systems engineering, and operational activities.

SAFETY AND PERFORMANCE ASSURANCE—VERIFICATION, VALIDATION, AND TESTING

The processes of validation and verification generally are applied to software, but in the context of certification and implementation for NextGen, they are integral parts of software, processes, and hardware development, procurement, and application. To be effective, these have to be conducted across the entire FAA organization.

Verification is generally defined as the detailed process of review and evaluation of products through the completion of the development phase to assure that it functions properly and meets the specified requirements for which it was intended. Validation examines and determines if the product satisfies specified business or user requirements. The objective is to determine if the product meets the user’s needs; it may also address whether the specifications were correct in the first place. Together, verification and validation of the overall system demonstrate that the product meets the need when placed in the user environment.

The management of the validation and verification processes extends to the procurement phase, which may include multiple contractors with differing verification and validation approaches. The management also has to deal with the rate of advancements in technology, specifically software and the changing operational needs and the management complexities within the organization. These factors highlight the need for a strong policy on software development, deployment, and maintenance.

In addition to policy needs, the committee notes the importance of maintaining a strong and capable staff to address all phases of validation and verification. This is particularly true in the area of software development and

procurement. The rapid evolution of technology in this area points to the need for training in addition to a capable team of staff and management personnel.

The implementation of new technology involves user equipage, and the En Route Automation Modernization system (ERAM) is a good example. Implementation took years, and operational testing was done incrementally in a series of “builds”: keyboard entries, hand offs, flight plan amendments, altitude changes, etc. Each step along the way had to be tested to ensure correct interfacing with legacy systems while adhering to a requirement to continue services without interruption.

Introducing a new operational capability that requires user equipage is much more complex than implementing procedural changes and requires user investment and time. The benefits do not accrue until users equip, and users will not equip unless they are confident in the ability of the FAA to deliver, not only on new technology, but also new operational benefits that have an economic payoff on a reliable schedule.

Whereas avionics improvements happen at a relatively fast pace, it takes much longer to verify, validate, and test their operational performance. The FAA was charged to develop a plan to increase user confidence that the FAA would deliver promised operational capabilities with an economic payoff on a stable schedule. But the relative lack of attention to the challenges associated with verification, validation, and testing makes it difficult to see how users could gain confidence without knowing how this will be done.

TRAINING

The training for and the maintenance of proficiency of operational personnel in the proper use of systems is essential to the successful integration of new technologies and procedures into the National Airspace System. The processes for FAA approval (certification) of training programs are often time consuming and generally complex. A regulation is usually issued mandating the training. This step is followed by the creation of an advisory circular offering an example of a compliant training program. In turn, that step is followed by written “guidance” to the FAA local certificate management offices (CMOs) delineating standards for acceptance of an individual airline training program. Using these documents, airlines then develop individual training programs for submission to their CMO for approval. Typically, the CMO submits the draft training program to the FAA in Washington, D.C., for the concluding review before a final letter of approval is issued. A similar process is followed for air traffic controllers.

The February 2014 Research Plan does not specifically address training issues. These issues are only discussed vaguely, and the plan only addresses research on the approval process for the implementation of new ground-based technology, processes, and procedures, as opposed to a fully integrated system and its components. The plan could be more robust and useful if it addressed the steps required in the development and approval of training programs using process mapping and redundancy elimination tools. The number of times stakeholder groups are required to comment in the approval process is an issue that requires close attention. Although training is not inherently a “research” issue, NextGen creates challenges for pilots and ground controllers. It is possible for the FAA to conduct research on better training methods, and there may also be ways to streamline and expedite the certification and training processes as an integral part of the National Airspace System.

OPERATIONAL TEST AND EVALUATION

The FAA has a long-standing operational test and evaluation capability that supports its development and deployment of equipment and procedures. As both ground and airborne equipment is developed and ultimately certified, the operational evaluation plan is used to ensure that requirements are accurate, beneficial, and able to be implemented.

One of the most critical aspects is evaluating the new capability and its smooth integration into the existing technology baseline. Safety and performance requirements for the air traffic system are extremely high. Overall, the operational evaluation program is designed to ensure there is no degradation in performance as new capabilities are integrated into the National Airspace System. To the extent feasible, the intent is to validate that new capabilities add to system safety and performance. The FAA’s operational evaluations take a multi-disciplinary approach and incorporate participants from multiple offices and lines of business. External vendors and the frontline workforce representatives are also actively involved. The teams are established to operate with independence, to ensure a lack of bias, and to focus on safety outcomes.

The February 2014 Research Plan mentions how new technologies intended to benefit the users will need to be evaluated and that specific outputs for timeliness and confidence in the areas of verification and validation, testing, and operational evaluation will be part of the program. The project schedule, milestones, and budgeting table⁶ lists a gap analysis on the long-term needs, and an analysis of new processes, procedures, and technologies. However, the committee could not determine if these activities, as briefly described, would adequately address the requirement for detailed and continuing operational evaluation. Operational test and evaluation can possibly be streamlined to take fuller advantage of the new and emerging technologies and capabilities.

CONCLUDING REMARKS

Despite the complexity of the task, the committee notes that the FAA has demonstrated the ability to do successful integrated projects as part of NextGen, and these have already had substantial impacts on improving operations within the National Airspace System. Several examples of these are addressed in Chapter 3, along with some relevant lessons from other organizations.

_________________

⁶ FAA, Research Plan, 2014; reprinted in Appendix A, p. 7-9.