Transformation in the Air: A Review of the FAA's Certification Research Plan (2015)

Summary

The Federal Aviation Administration (FAA) is currently undertaking a broad program known as NextGen (Next Generation Air Transportation System) to develop, introduce, and certify new technologies into the National Airspace System. NextGen is a fundamentally transformative change that is being implemented incrementally over a period of many years.¹ Its implementation is analogous to the introduction of radar during the 1950s; radar was also a transformative change in domestic air travel, but it took many years to be fully implemented and for users to adapt and take advantage of its capabilities. Currently, the FAA is putting into place the foundation that provides support for the future building blocks of a fully operational NextGen.

NextGen is a challenging undertaking that includes ground systems, avionics installed in a wide range of aircraft, and procedures to take advantage of the new technology. Not all of these systems or procedures are under the FAA’s control, a reality that determines implementation and therefore requires close coordination among the FAA and various stakeholders.² NextGen is also best understood as a continuing evolution of systems and procedures, not simply technology systems that have to be developed and deployed. Like all complex systems of systems, the implementation has experienced problems, but has also made important advances, some of which are highlighted in Chapter 3 of this report.

The FAA has continuously updated the National Airspace System for decades, although what is generally referred to as NextGen had its origins in the late 1980s but was really only formalized by the late 1990s. Since that time the conditions of, and justification for, this enormous undertaking have changed. The biggest of these changes are the following:

• The overcrowding of the airspace that resulted in air travel gridlock almost a decade ago has faded with reduced demand for air travel during the economic downturn. Airlines have also responded to high fuel costs by operating fewer aircraft with more passengers on board. As a result, the need for greater airspace capacity in the short term has diminished.

____________________

¹ This committee was not asked to evaluate NextGen. For a broader perspective on NextGen, see the 2015 National Research Council (NRC) report A Review of the Next Generation Air Transportation System: Implications and Importance of System Architecture, The National Academies Press, Washington, D.C.

² The term “stakeholders” in the National Airspace System is a broad one, covering many entities. The stakeholders include the airlines, the general aviation community, the U.S. military, manufacturers, and the traveling public. Thus, the FAA’s research and its certification processes can affect many different entities in different ways.

• Other concerns such as efficiency, environmental, and noise impacts of aviation have increased during this same period of time.
• The anticipated significant increase in general aviation traffic has not occurred.
• The current safety record of the National Airspace System is at an all-time high. Safety has not deteriorated but has actually improved statistically within the last few years to unprecedented levels. Paradoxically, this high safety rate has resulted in little incentive by those required to invest in the system to meet NextGen’s future safety goals while increasing the number of operations in the National Airspace System.

The operators, such as airlines, support an improved system, but the benefits for them are not as clear today as they seemed years earlier, particularly when measured against the significant investments they are asked to make. Thus, there is less incentive for faster implementation of new technologies into the National Airspace System than there was over a decade ago.³

While the FAA can be a capable program manager and direct public capital investment, it does not control investment in and implementation by a broad and diverse operator community in necessary technology, training, and other elements required in an integrated plan. This diverse stakeholder community seeks a broad set of differing operational benefits from the FAA. All stakeholders would benefit substantially from an explanation of the end-to-end processes necessary to certify, approve, and implement advanced NextGen capabilities beyond the mid-term (i.e., 5-7 years).

The FAA Modernization and Reform Act of 2012 required the Federal Aviation Administration to develop a research plan for the certification of new technologies into the National Airspace System and to have the National Research Council (NRC) review that plan.⁴ The NRC’s Committee to Review the Federal Aviation Administration Research Plan on Certification of New Technologies into the National Airspace System reviewed the February 2014 research plan of the FAA Office of NextGen, Research Plan: Methods and Procedures to Improve Confidence in and Timeliness of Certification of New Technologies Into the National Airspace System,⁵ and determined that the plan lacks detail and specificity and does not provide an effective guide to FAA research over the 5-year term required by the act. The committee concluded that the plan does not meet the requirements of the authorizing legislation. Whereas the plan restates the language from the FAA Modernization and Reform Act of 2012, it lacks the specificity required to generate actionable objectives.⁶ It is more of a high-level task plan for incrementally developing over the next 5 years the detailed research plan that the FAA will actually need.

In particular, the February 2014 Research Plan does not demonstrate how the integration of aircraft, ground systems, and procedures will occur. Successfully demonstrating this will create confidence in implementation and attract stakeholder and operator investment, which is vital for success. The committee also was concerned with the lack of detail in the plan on the transition of research into applications through a structured certification approach that results in an approved operational capability. The plan contains a large amount of content on background and scope and assumptions, but very little on product schedule, milestones, and budgeting. In addition, the committee

____________________

³ The FAA’s Acquisition Management System (AMS), which is discussed in Chapter 1, requires consideration of costs and benefits.

⁴ Regarding research on design for certification, the act stated the following:

Research plan—Not later than 6 months after the date of enactment of the FAA Modernization and Reform Act of 2012, the Administrator shall develop a plan for the research under paragraph (1) that contains objectives, proposed tasks, milestones, and a 5-year budgetary profile.

Review—The Administrator shall enter into an arrangement with the National Research Council to conduct an independent review of the plan developed under paragraph (2) and shall provide the results of that review to the Committee on Science, Space, and Technology of the House of Representatives and the Committee on Commerce, Science, and Transportation of the Senate not later than 18 months after the date of enactment of the FAA Modernization and Reform Act of 2012.

⁵ FAA, Research Plan: Methods and Procedures to Improve Confidence in and Timeliness of Certification of New Technologies Into the National Airspace System, Final, Office of NextGen, Washington, D.C., February 2014.

⁶ The February 2014 Research Plan is reprinted in Appendix A of this report. The plan includes a timeline listing the key points in development and approval of the plan. Finalized in February 2014 and approved by FAA senior management in April 2014, it was made available to the NRC in summer 2014, at which point the NRC created a committee to review it and hold meetings to gather additional information. NRC committee meetings occurred in November 2014 and January and March 2015.

believes that one of the missed opportunities of the plan was that it did not explain how much progress has already been made or what is left to be done and does not reference or provide material from the NextGen Strategic Plan⁷ or the National Aviation Research Plan.⁸

The plan also uses the term “certification” inconsistently, using the term “approval” interchangeably with “certification” in some cases. A more meaningful milestone than certification of the new technology is the approval of the operational capability of that technology for implementation by National Airspace System users. The committee used this broader view of the term certification in its assessment.

Assessing the timeliness of the research plan requires understanding the stakeholders, benefits, and control over what gets done in order to certify new technologies for introduction into the National Airspace System, and who does it. The FAA is the central focus of the certification of new technologies, but the FAA does not have control over the stakeholders or over the derivation of the benefits to the stakeholders.

The committee has concluded that it is in the best interests of the FAA and its stakeholders for the FAA to describe and carefully explain the steps that the FAA and aviation stakeholders are taking to expedite the realization of the NextGen capabilities. Thus, there is value to the FAA’s producing a comprehensive research plan that explains its research goals and plans for integrating and certifying technology into the National Airspace System. The committee concluded that future FAA research plans, when properly assembled and executed, can play a valuable role in guiding the FAA and stakeholders and explaining progress in certifying new technologies into the National Airspace System. But although a research plan can help, without goals and operational performance-based metrics such as fuel burn, capacity, delays, cancellations, carbon emissions, and other relevant factors, a plan by itself cannot control the pace of implementation of capabilities or the realization of stakeholder operational benefits. These kinds of metrics are found in other FAA documents, but are not present in the February 2014 Research Plan.

Because the February 2014 Research Plan does not include air systems or procedures, it is unlikely that the plan by itself would address all the elements necessary to improve the timeliness and effectiveness of the certification and implementation of technologies into the National Airspace System. The National Airspace System is tightly integrated, and air and ground capabilities are closely linked with each other and with operational procedures. This requires that all three segments be addressed by a research plan. Omitting air systems and procedures from the research plan makes it unresponsive to the request from Congress, in the committee’s opinion. In addition, the plan does not reference other agency reports, plans, and resources that inform and frame research to improve confidence in and timeliness of certification of new technologies.

During the course of this study, the committee concluded that Congress and the FAA were not effectively communicating and there was some misunderstanding on both sides. Congress wrote legislation calling for a “research plan,” but according to discussions with congressional staff, Congress actually wanted a detailed description of processes, plans, and capabilities to certify new technologies in a timely manner, of which a research plan is only one key part. The FAA responded by producing a research plan that is too narrow in scope and failed to explain all of the other relevant factors in the timely certification of new technologies and progress in implementing them. Better communication between the FAA and Congress prior to production of the February 2014 Research Plan by the FAA would have ameliorated this confusion. However, even taking this communication failure into account,

____________________

⁷ FAA, NextGen Strategic Plan.

⁸ FAA, National Aviation Research Plan, September 2013, https://www.faa.gov/about/office_org/headquarters_offices/ang/offices/tc/about/campus/faa_host/rdm/media/pdf/2013%20NARP.pdf. The February 2014 Research Plan in Appendix A makes a single cursory reference to the NARP (see p. 34).

the February 2014 Research Plan is still too narrow in scope (for instance, omitting the aircraft segment) to satisfy the requirements of the 2012 act. The committee believes that future improved communication between congressional staff and FAA staff could alleviate concerns and be highly productive.

OBSERVATIONS ON FAA RESEARCH PLAN RESPONSIBILITIES AND IMPLEMENTATION

The complexity of the process of certifying new technologies also leads to unrealistic expectations by the stakeholders. This complexity begins with the significant relationships between initial hardware or process certification and subsequent broad implementation and approval of new operations in the National Airspace System. While it is easy to believe that once a system is certified the job is finished, the reality is that certification of a particular system component is only an important milestone in a larger process of implementation. This implementation diffuses the control not only to within different organizations within the FAA but also to stakeholders, who must also make investments and implement changes within their organizations. Thus, the FAA is not in a position to single-handedly enact or regulate the intended benefits of NextGen implementation, and their regulations may impose costs.

The implementation effort creates issues affecting aircraft modifications, ground-based system upgrades, training at both the FAA and stakeholders, development of new procedures, revision of operational regulations, and so on. During this time of change, human factors considerations are critical to ensure safe operations within a system in flux, and their inclusion in a research plan will be a key part of its success.

It is obvious from this discussion that the challenge is to create a single fully integrated system from an amorphous mass of projects and stakeholders. The February 2014 Research Plan does not provide a plan to research certification and implementation of such a system.

There are many components to an effective research plan. The committee in this report sought to identify several that it believes deserve particular attention, in part because they are often overlooked or underemphasized and in part because they are gaining increasing importance. A well-planned systems engineering approach is essential to the success of the integration and implementation of a complex multi-faceted system of systems like NextGen. Currently, the FAA is almost entirely reliant upon vendors for software assurance, but the plan does not describe how the FAA is conducting a robust software assurance program. Fundamental to these requirements is the development of an enterprise architecture and National Airspace System-level system architecture. These tools are critical for setting the context for all levels of research and for determining which subjects should be investigated as priority and which topics could offer the most benefits in the nearer term.

Cybersecurity, including the important issue of data privacy and verification and validation of new systems to ensure that they work, are also vital aspects to the success of FAA systems. Cybersecurity is a critical component of the National Airspace System that needs to be addressed early, continually, and comprehensively across the systems, segments, and procedures. Similarly, verification and validation is a vital but easily overlooked issue, and the committee determined that it deserves specific attention in any research plan because it can create significant implementation delays.

Although the FAA has a research budget and research facilities, a substantial amount of research on new air traffic control systems, procedures, and related technologies is performed by NASA. The February 2014 Research Plan does not refer to research conducted by NASA or other organizations that perform relevant research (for instance, the development of new air traffic control technologies or regarding the certification of new technologies) or engage in analogous activities, some of which is discussed in Chapter 3.

Chapter 1 of this report addresses the committee’s review of the February 2014 Research Plan, explains the components of an effective research plan, and formulates findings and recommendations. Chapter 2 of this report examines specific shortfalls in the plan and offers suggestions on issues to be included in any future plan. Chapter 3 of this report addresses examples of successful NextGen projects as well as the work of other organizations such as the U.S. Air Force and NavCanada with analogous experience. Chapter 4 of this report addresses the committee’s information gathering from the FAA and interpretation of activities related to the research plan. Chapter 4 also discusses some of the extensive work the FAA has done to date to implement the incremental NextGen approach.