A Review of the Next Generation Air Transportation System: Implications and Importance of System Architecture (2015)

4

Minimize Cultural and Organizational Barriers

External Next Generation Air Transportation System (NextGen) stakeholders have a variety of interests, demands, and constraints. Agreement, or at least rough consensus, on requirements is important (albeit challenging to achieve). Significant effort is required to ensure that these various interests do not prevent the realization of the public benefits for which the nation has invested. Some NextGen programs and components will undoubtedly have implications for the workforce, especially controllers and pilots. The capacity, skill sets, size, and expectations of the associated workforces must be taken into account when developing and deploying new or changed capabilities.¹

The Federal Aviation Administration (FAA) has made numerous efforts toward stakeholder engagement. The committee believes that the architecture, if moved in directions as described in early chapters, can be a vital communications tool among all stakeholders. The system architecture should be expressed (documented) in a form that readily facilitates communication, inspection, and debate with stakeholders and advisers. The architecture leadership community that the committee recommends can help ensure that any documentation that is produced is at appropriate levels of abstraction to enable productive discussions among all stakeholders.

_______________

¹ The 2007 National Research Council (NRC) report Human-System Integration in the System Development Process: A New Look (The National Academies Press, Washington, D.C.) explores iterative development processes suitable for systems that have intensive human interaction and with humans having functional roles within the system (e.g., pilots, controllers, and so on.)

In addition to the challenges of meeting the needs of a large and diverse stakeholder community, the FAA must contend with internal organizational, cultural, and structural barriers. The rest of this chapter offers the committee’s recommendations on a number of related issues: human factors, a look at costs and benefits of NextGen, the challenge of being a system integrator, and the need for sustained support for operations and maintenance.

HUMAN FACTORS

In any complex system, human factors should be incorporated in design, technical, engineering, and architectural discussions as early as possible.² Human factors should not only be addressed after the design is complete (e.g., to check on or tune the design), but much earlier in the process. For the FAA, this is both an organizational challenge—it may not have sufficient human factors personnel to integrate contractors’ work with system design—as well as a technical and engineering challenge—to determine how requirements and constraints flow to early-stage technical requirements so that human factors perspectives can contribute to early design work. When human factors are not included at the outset, products and services need to be modified subsequently to meet the human factors requirements, which then delays the release of products and services and significantly increases cost.

NextGen depends on existing technology and future development and on creating processes for both to work together. Consequently, human factors efforts will be retroactive in some cases while simultaneously looking to the future. The National Airspace System (NAS) has many moving parts that are being upgraded and updated. Thus, a challenge of human factors is the introduction of concepts while the system is being maintained, upgraded with existing technology, and then being further updated with newer technology. This complexity reinforces the need for human factors input at all phases.

Although human factors expertise exists within a research group in the NextGen organization³ and in the safety organization, there seems to

_______________

² The use of the term human factors in this report is meant to encompass more modern terms such as “human-system interaction” (HSI) and “user experience design” (UX). Attention to the entirety of the interaction, including protocol, robustness (against human error, human inattention, etc.), visualization, affordances, and so on, is needed. The business of “human factors” goes well beyond response latencies and interfaces “on the screen” such as pixels and fonts.

³ The Human Factors Research and Engineering Division (ANG-C1) is located within the NextGen Advanced Concepts and Development Office (FAA, February 2015 Organizational Chart, http://www.faa.gov/about/office_org/headquarters_offices/ang/offices/media/orgChartANG.pdf).

be no human factors representation at higher levels of NextGen management to participate in and sign off on designs and to track and ensure contractor inclusion of appropriate human factors considerations in implementation. This lack has apparently existed at NextGen from the beginning. Human factors has played an insufficiently substantive role in the design integration of NextGen systems and procedures. Human factors seems to be viewed as research and as relevant to testing and integration but not as a significant part of system design—and certainly not as a core activity. And given the traditionally late integration of human factors expertise, this is understandable. But NextGen poses a major human factors system design problem—namely, developing confidence through participation in design and simulation, especially for off-nominal scenarios that enable controllers and pilots to understand the capabilities and limits of NextGen capabilities and be able to perform without undue workload, delay, or error.

The human factors research group operates a high-quality laboratory at the Atlantic City Tech Center. That laboratory conducts human-in-the-loop simulations (HITLS) on selected early-stage concepts. In addition, it sponsors some university research and publishes reports that are made available to whomever would like to read them. In addition, NASA Ames Research Center conducts human factors and simulation research (including HITLS) in its Aeronautical Research Division. (A recent example focused on terminal sequencing and spacing.⁴) However, research is not system design, and individuals involved in those projects are too few to participate in any meaningful way in the many stove-piped programs charged with generating design specifications or contracting for and testing of component subsystems of NextGen. The committee’s understanding is that ANG-C1 human factors experts are not invited to reviews, and those in the programs often do not know that experts are available. The human factors research group is occasionally asked to help review such designs, but without any formal sign-off responsibility. When human factors experts are not involved in the very early stages of technology and procedure design, a likely result is that many systems (displays, controls, procedures, and subsystems) will not be subject to human factor critiques until contractors take over, if then.

Changes in equipment and procedures beget human errors. The proven means to uncover unintended and unexpected events in human-

_______________

⁴ See, for instance, J. Thipphavong, J. Jung, H. Swenson, L. Martin, M. Lin, and J. Nguyen, “Evaluation of the Terminal Sequencing and Spacing System for Performance-Based Navigation Arrivals,” paper presented at 32nd Digital Avionics Systems Conference (DASC), October 2013, available at http://www.aviationsystemsdivision.arc.nasa.gov/publications/atd1/tech-transfer/index.shtml.

system interactions prior to operational deployment is by HITLS. Full-fidelity HITLS can be expensive and time consuming. The FAA has done, or is planning to do, a few of these and cites cost as the main reason for not doing more. However, for purposes of learning, particularly for uncovering unexpected issues of multi-system, multi-person interactions, full fidelity is not necessary. One can gain a great deal from part-task simulations or even what are called “cognitive walk-throughs.” The latter are play-acting exercises where an experimenter confederate plays the part of another human (pilot, controller), and the “equipment” is a crude mock-up. These exercises would be much less costly than full-fidelity HITLS and could provide useful early-stage input into design and requirements.

One might like to use fast-time analytic models and simulations, but unfortunately there are few human-system analytic models that are very predictive, and they are also very context sensitive. There are very few analytic models that are up to being very helpful for NextGen, other than for modeling basic vision and hearing. One highly relevant model is that the time required for a human to receive an alert of some abnormality, understand the issue, make a decision, and take proper action exhibits a probability distribution with a very long “right-hand tail.”⁵ There is hard evidence for this. It means that when a human is a serial element in a system, even though the mean and median response times may be short, the wait time required to achieve 95 or 99 percent confidence will be very long.

Human factors has an important role to play in the FAA program management of system engineering and in acquisition. The research group in Atlantic City is used primarily for advanced concept exploration and testing and tuning of systems as they are deployed. The group is not currently used as part of system design, which is done by contractors, nor does it have direct input or sign-off authority in acquisition decisions. However, an FAA human factors group could represent the FAA workforce; provide continuity across time and contractor (so that, for instance, two contractors do not work at cross-purposes); represent training and concept-of-operations (CONOPS) issues that the FAA must face; and provide crucial input to key acquisition decisions. Although some of these issues can be specified by a program office in a contract, some of the human factors consequences are revealed only by experts, HITLS, and so on.

_______________

⁵ T.B. Sheridan, Human response is lognormal; plan on waiting if you want reliability, Ergonomics in Design 21(1):4-6, 2013.

Implications for Operations and Procedures

The medium-term plan will not fundamentally change the roles and activities of pilots and controllers. However, even with modest changes, experience shows that if the result is multiple (modest) changes to systems from what people are accustomed, misunderstandings and errors can result.

Looking ahead, the implications for operations and procedures of NextGen’s long-term goals and associated technical changes could be significant. Thus, procedure redesign and airspace design can become a large bottleneck to making progress. The 2013 Implementation Plan⁶ provides a nice summary of the FAA vision (at that point) of how things would eventually work gate-to-gate. That discussion makes clear that there are many new systems, all being developed under different programs and at different stages of implementation, being tested at different airports and coming online at different times. The timing of the implementation of the various elements complicates the human factors challenges. Within actual flight operations, there has been little opportunity as yet to observe interactions between the many new systems and procedures, since few airlines are equipped with required avionics beyond what exists a priori within the flight management systems. Further, a key assertion being made by proponents of “resilience engineering,” with which the committee concurs, is that errors and failures tend to occur when changes are introduced in systems, and the more simultaneous the changes, the greater the risks.

Some of the anticipated automation in NextGen will likely result in challenging and (at least initially) error-prone new tasks for controllers and pilots. Although automation is expected to provide benefits over the long term, major new tasks for pilots and controllers that merit close attention to procedure design, airspace design, and human factors include the following:

• Automation of navigation based on Global Positioning System and many new navigation aids, especially for arrivals but also including navigation around weather cells and surface movements. New computer-based decision aids are bound to make controllers more dependent on the computer advice given. How is that expected to affect policies for assigning authority and responsibility? New thinking is required beyond the naïve assumptions that the human operators must always be in charge. If computers can recognize invalid input, improper action and/or inattention, under what circumstances should the automation take over control

_______________

⁶ Federal Aviation Administration (FAA), NextGen Implementation Plan, Washington, D.C., June 2013.

• from the human controller? One could imagine a mechanism for rapid shifting of decision-making roles in off-nominal situations.

• A change from primarily voice to data for air-ground communication. This transition will inevitably create new opportunities for human error (for example, misreading of text input).
• New automation for anticipating and resolving aircraft conflicts, which is supposed to relieve controllers of having to stare at screens to vector aircraft moment to moment, enabling them to take on more planning functions, and enable flow controllers to anticipate traffic flow congestion around airports well upstream.
• A new need for both controllers and pilots to think in time differences as well as spatial geometry—which can get confusing.
• Greater demand than in the current air traffic control (ATC) system for real-time coordination between sector controllers, flow controllers, tower controllers, and pilots for some aspects of NextGen. For example, where controllers used to be responsible for only what was in their own sector, there is an effort is to get them to anticipate upstream and downstream activity more broadly by using new weather and traffic decision support tools. What means are being used to ensure that all parties to the cooperation are seeing the same picture?

Ultimately, realizing any intended benefits to the NAS from improved operations and procedures will require pursuing and completing operational integration. The committee learned that required navigation performance (RNP) and area navigation (RNAV) routes and procedures often go unexploited. Even if aircraft are equipped and pilots are trained, insufficient RNP/RNAV routes have been designed to make a significant difference in the overall performance of the NAS, and there seems little incentive for approach controllers to issue them to arriving aircraft that are properly equipped.⁷ This will require funding and commitment to see these things through. Funding development of new capabilities—many of them quite promising—and then curtailing deployment to economize on the backend is not prudent.

_______________

⁷ A detailed examination of the RNP approach at Midway airport with Southwest Airlines (SWA) equipage found that the system-wide fuel savings did not justify the equipment costs to SWA in and of itself. The benefits were more to other airlines operating out of O’Hare airport due to complex airport airspace interactions. See Akshay Belle, “A Methodology for Analysis of Metroplex Air Traffic Flows” Ph.D. dissertation, George Mason University, November 2013. Also see A. Belle, M. Wambsganss, and L. Sherry, “A Methodology for Airport Arrival Flow Analysis Using Track Data—A Case study for MDW Arrivals,” Integrated Communications, Navigation and Surveillance Conference (ICNS), 2013, 2013, http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6548547&tag=1.

A Living CONOPS

Early in the NextGen development, the FAA put out a CONOPS with some useful detail on the expected new roles of pilots and controllers.⁸ Since then, however, it has not been updated frequently, and what is available now does not seem to be aligned with the changes that have taken place in NextGen over time. The Implementation Plan has details about programs and includes flight operations in a very general sense, but has very little substance on operations from the pilot and controller perspectives. The 2007 NRC report Human-System Integration in the System Development Process⁹ emphasized the importance of a frequently updated visualization of the whole system design with off-ramp references to detailed working papers, and so on. This could be thought of as an updated working CONOPS, just as the system architecture should be thought of as a living architecture (and not static descriptions of systems after major decisions have been made). The CONOPS that was issued was perhaps useful to audiences outside of the FAA and NextGen, but not as helpful to system developers or others situated within stovepipes; it has likely resulted in inappropriate expectations about NextGen that persist today. A living CONOPS is related to the system architecture ideas discussed earlier. Both allow for clear, high-level exposition of NextGen and NAS properties that is kept up to date and that informs all stakeholders about the continuing evolution of the NAS. Although they are two distinct perspectives, both are useful.

_______________

⁸ Joint Planning and Development Office, Concept of Operations for the Next Generation Air Transportation System, Version 3.2, http://www.dtic.mil/dtic/tr/fulltext/u2/a535795.pdf.

⁹ NRC, Human-System Integration in the System Development Process: A New Look, The National Academies Press, Washington, D.C., 2007.

COSTS AND BENEFITS

The NAS is a national infrastructure to which significant resources are devoted. As such, it has numerous stakeholders, and there are few individuals or businesses in the country that do not have an interest in or expectations regarding its performance. Thus, the NAS, the FAA, and NextGen efforts are subject to significant scrutiny—not only from users (such as trade groups, airlines, airports, and affiliated labor groups), but from Congress, other federal agencies, and the flying public as well. And as a federal agency, the FAA must operate within the federal political environment and under whatever financial and performance constraints and expectations are produced within it.

The FAA lays out a business case for NextGen in The Business Case for the Next Generation Air Transportation System: FY 2013.¹⁰ The committee held several discussions with FAA staff as well to understand the analysis used to develop the costs and benefits of implementing NextGen. In summary, the FAA suggests that NextGen midterm improvements will generate $182 billion in benefits through 2020 and cost approximately $39 billion.

NextGen’s benefits are expected to accrue to stakeholders; however, many of those benefits cannot be fully realized without participation and (sometimes costly) adoption by the relevant stakeholders.¹¹ Nor is there a well-specified overview of what is and is not known about the value of various proposed levels of change (e.g., partial deployment of certain technologies or features). The architecture leadership community has a role to play with respect to managing system-level trade-offs and how they bear on cost and schedule, as discussed in Chapter 2.

Although many of the NextGen advances should benefit participants in the NAS writ large, the fact that some of the benefits may accrue to competitors could be a disincentive to participation by private entities. Thus, for some NextGen goals, the FAA is caught in a bind due to the distributed costs of deployment and the uncertainty of those costs if broad deployment does not occur. The expectation that economic benefits will sufficiently motivate airline equipment purchases may be misplaced, calling into question whether the anticipated voluntary uptake will occur.

_______________

¹⁰ FAA, The Business Case for the Next Generation Air Transportation System: FY 2013, Washington, D.C., 2013.

¹¹ For example, ADS-B must be installed not only in larger commercial aircraft but also in smaller general aviation aircraft to make full use of the system everywhere and there are costs (financial and also process) associated with deploying this technology.

NextGen plans require a substantial investment, both by the taxpayer via the FAA for infrastructure and by carriers and aircraft owners for equipage and training. At best, benefits—however quantified—to carriers will lag deployment costs, and benefits that accrue to the carriers will be less than the projected social benefits to the system as a whole. Whether something is cost beneficial to a specific entity (FAA, the airlines as a group, a specific airline, or general aviation) can vary greatly and poses a major challenge for NextGen equipage, implementation, training, and so on. Moreover, the required spending is in real dollars, while nearly two-thirds of the economic social benefit is quantified in the form of reduced delays to passengers, as is standard for Department of Transportation analyses of this sort.¹² The FAA does incorporate some savings in aircraft operations in its benefits total, but most of the anticipated benefits stem from estimated costs of passenger time and environmental and safety benefits;¹³ the passenger delay reduction accounts for $107 billion of the expected benefits.

The FAA does not fully control NextGen; many of the participants have to actively choose to acquire elements and participate. Voluntary systems are much more likely to succeed when the preferred configuration is locally preferred for each participant, and not just globally preferred on a cost-benefit basis. But, as outlined above, many of the benefits may be socially accrued, resulting in a cost-benefit equation may not actually be positive for many of the required decision makers. In addition, carriers typically expect a return on investment of less than 3 years, whereas the FAA’s development cycle can be much longer. These issues constitute a major risk. However, the solution to this problem is not necessarily technical. It may be that it would be best addressed by policy changes or other approaches, which are beyond the scope of this report.

Many of the benefits of NextGen cannot be meaningfully realized unless all, or nearly all, air carriers equip their fleets with the requisite technology.¹⁴ The carriers will also incur training costs, both for new equipage and for new procedures that use old equipage. For airlines to gain significant benefit, NextGen capabilities will need to be deployed at

_______________

¹² Polly Trottenberg, U.S. Department of Transportation, memorandum to Secretarial Officers and Modal Administrators, “Revised Departmental Guidance on Valuation of Travel Time in Economic Analysis,” September 28, 2011, http://www.dot.gov/sites/dot.dev/files/docs/vot_guidance_092811c.pdf.

¹³ The passenger value time is based on Department of Transportation guidance from 2011 and is $43.50 per hour, with a 1.6 percent growth per year. Quantifying environmental and safety impacts is also difficult and the methodology can found in the FAA’s report The Business Case for the Next Generation Air Transportation System (2013).

¹⁴ General aviation faces cost challenges as well, especially related to ADS-B Out avionics, which are still quite expensive compared to other operating and capital costs.

sufficient scale, and given the delay in implementing new procedures and technologies at major airports, airlines may not see benefits for some time. Some of the benefits from new technologies and associated procedures will not be realized until the system is operating at scale. However, some of the demand and capacity pressure present when NextGen was first envisaged have been reduced, which has reduced the urgency for achieving some of the anticipated benefits. In addition, the FAA’s current analysis does not take into account potentially negative costs related to possible security breaches absent improvements—a difficult thing to quantify, to be sure, but consideration of these sorts of aspects could be illuminating. Particular NextGen technologies and likely cost-benefit implications of each are discussed below.

• ADS-B Out. There is a mandate in place that will require ADS-B Out equipment by January 1, 2020, on all aircraft operating in nearly all NAS Controlled Airspace.¹⁵ The costs of the equipage must be borne by the operators, but the benefits mostly accrue to the FAA in the form of streamlined ATC procedures and information and (theoretically, at least) the retirement of some old and costly surveillance radar. This, of course, requires the mandate, because all aircraft have to be similarly equipped.
• ADS-B In. This capability, when available, could provide significant benefits to the aircraft, including much improved situational awareness, predictive traffic information, and delegated separation. Unfortunately, there is no technical specification for these details and no schedule for a mandated implementation to drive the development of one. Moreover, achieving some of these benefits will require changes in software and controller procedures. Without a credible implementation schedule, carriers will discount or dismiss the value of these capabilities.
• Performance-based navigation (PBN). Almost all Part 121 (Air Carrier) aircraft are equipped with some level of highly capable RNP, as are many general aviation aircraft. But use of these procedures at airports remains low.¹⁶ The problem with extracting associated benefits from the use of PBN is the absence of necessary procedure design by the FAA. In order to demonstrate progress some years ago, many existing procedures using ground-based navigational aids were simply converted using identical trajectories that simply overlay GPS waypoints. New procedures are required that truly utilize the precision of a modern flight management

_______________

¹⁵ FAA, “Automatic Dependent Surveillance-Broadcast Operations,” Advisory Circular, October 28, 2014, http://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_90-114A_FAA_Web_%282%29.pdf.

¹⁶ Matthew Hampton, Assistant IG for Aviation Audits, memorandum, “Audit Announcement—FAA Progress in Deploying Controller Automation Tools for Performance-Based Navigation Flight Procedures,” July 1, 2014.

• system in terminal procedures and in parallel-track en route navigation. The community has already invested heavily in this underutilized equipment, which makes them reluctant to rely on FAA schedules or commitments regarding other efforts.

• Other NextGen capabilities, like Data Communications (Data Comm) and Digital Voice, eventually will yield some efficiencies and reduce some human error but the emphasis has been on technology, not on quantified benefits. Although the FAA offers some modest incentives for Data Comm equipage, in general, there is little incentive for carriers to implement early and participate in a redesign of the airspace, which should be accompanying all this new technology. Instead, much of the airspace remains in the configurations established 50 years ago by the constraints of ground-based radio frequency navigational aids.

THE CHALLENGE OF BEING A SYSTEM INTEGRATOR

The FAA does not operate in a vacuum. When it comes to large government systems and software projects, it faces many of the same challenges as many other government agencies, among them are the following: a reliance on waterfall approaches to software development, difficulty in hiring and retaining skilled information technology (IT) pro-

fessionals, policies and requirements developed without sufficient technical or engineering input, and acquisition and budgeting processes not designed for the way modern software systems are built.¹⁷

The problems with air traffic control and airspace modernization, in the face of rapid technology change and challenging governmental budgeting requirements, have not been unique to the United States. Most modern industrialized countries have experienced the same problems. Most noticeable and relevant to the U.S. system are the Canadian, British, German, and Australian systems. Each has proceeded in its own way to separate the (mostly industrial) function of air traffic control from the essential governmental oversight function and responsibility that is demanded of sovereign nations under the Chicago Convention Treaty of 1944.¹⁸ The managerial and technical expertise required to constantly upgrade modern telecommunications systems is challenging for any government agency to attract and maintain. In addition, culturally, most government employees are risk averse and conservative in action due to the high level of public accountability and close oversight intrinsic to any government operation. This is not an environment in which most highly skilled engineers choose to be employed.¹⁹

Most NextGen software development is outsourced to contractors, thus obliging the FAA to act as the system integrator. However, the FAA’s NextGen team is ill-equipped—in terms of having sufficient broad and deep expertise and in terms of resources—to perform as a system integrator. Without sufficient system architecture competence and mature architectural approaches, the probability of success is compromised. Of particular note, the committee did not hear much from the FAA or its contractors about change management—especially in the case of important changes required by the agency rather than those managed entirely within the contractor’s own software and system development processes.

Second, the committee has a concern about the criteria being used to evaluate and assess software contractors. In briefings to the committee, there was significant emphasis placed on contractor software development maturity (e.g., capability maturity model integration, or CMMI) and less emphasis on contractor track record for value delivery. Federal procurements use a “best value” criterion above all others with strong

_______________

¹⁷ For more information about transforming large-scale government acquisition of software-intensive systems in a more agile manner, see NRC, Achieving Effective Acquisition of Information Technology in the Department of Defense, The National Academies Press, Washington, D.C., 2010.

¹⁸ The Convention on International Civil Aviation was signed on December 7, 1944, and is available at http://www.icao.int/publications/Documents/7300_orig.pdf.

¹⁹ Due to the nature of the challenges, threats, and incentives, the Department of Defense may be a possible exception to this observation.

emphasis on past performance. One element may be a CMMI rating, but that is one of many factors.²⁰ However, like the risk management process, the software and the acquisition approaches seem to be overly process driven, not outcome driven. The committee’s impression is that there was little apparent concern for increased efficiency that can be channeled into cost reduction, timeliness improvement, or quality improvement. While high ratings for capability maturity are desirable, they do not sufficiently reflect the quality of products delivered, or the effectiveness of the contractor in exploiting software to leverage existing capabilities, or exploit emerging technologies to best effect. At the same time, contractor efforts to work collaboratively with others on better integration are undoubtedly hampered by stovepiping and procurement regulations.

If the FAA is to succeed in both the medium and long term, it will require enhancements to its technical expertise. Architecture and systems engineering, which are needed for successful integration of capabilities and platforms into a coherent NAS system, have been undervalued. Program management and systems engineering process are important but are not a substitute for talent that can effectively guide the design and evolution of NextGen. This is especially important if the FAA continues to act as the systems integrator of NextGen programs. In that role, the FAA should maintain architectural leadership and not delegate architecture definition and control to contractors.

Today, the FAA relies greatly on its vendors and other external talent. Internally, the FAA depends on a very small number of individuals and lacks the critical mass that characterizes a vibrant and effective technical community. Digital communications will take on increasing importance as the NAS is modernized, so the FAA will need additional technical expertise in designing modern digital networks and protocols. Historically, air traffic control has relied heavily on analog voice communications, but with programs like Data Comm, digital communication will increasingly become primary. Cybersecurity is a challenge facing all who use modern computing and communications technology, and the potential threats and risks are magnified for critical infrastructure, like the systems that make

_______________

²⁰ Indeed, government contractors generally (beyond FAA contractors) have been required to complete CMMI assessments since the 1990s, but government software acquisition programs continue to suffer from delayed schedules and inadequate product performance. A 2014 GAO report on information technology reform initiatives noted that “federal IT projects too frequently fail and incur cost overruns and schedule slippages while contributing little to mission-related outcomes” (GAO, Information Technology: Reform Initiatives Can Help Improve Efficiency and Effectiveness, Statement of David A. Powner, Director, Information Technology Management Issues, Testimony Before the Subcommittee on Efficiency and Effectiveness of Federal Programs and the Federal Workforce, Committee on Homeland Security and Governmental Affairs, U.S. Senate, http://www.gao.gov/products/GAO-14-671T).

up the NAS. The FAA needs strong cybersecurity expertise in designing, implementing, integrating, and operating NextGen systems. Finally, while the FAA should emphasize organic talent, it will also be fruitful in today’s world for the FAA to regularly tap into outside communities of expertise. Even if the FAA were not acting as systems integrator, it would still need to be a “smart customer,”—meaning that it needs expertise that will enable it to effectively structure and manage its supplier relationships.

Developing and retaining this expertise will be a challenge. However, proceeding with inexperienced or less than the best personnel in key leadership positions is a significant risk.²¹ The FAA will likely need to tap into its “government-side” network of partners in federally funded research and development centers, systems engineering and technical assistance contractors, and similar organizations in order to gain access to sufficient expertise. It will also need to examine the incentive structure it creates for its primes (and into the supply chain as well) to better align incentives around the risk-managed, architecture-led processes above. These incentives can include earned-value models, rewards, and penalties. There also need to be processes for the various NextGen primes (and potentially bidders) to participate in the architecting process in a way that the architectural decisions that emerge will be respected and supported by those involved in program execution (rather than being used as an excuse to incur additional costs).

_______________

²¹ From NRC, Pre-Milestone A and Early-Phase Systems Engineering, 2008: “Perhaps the biggest risk of all in undertaking large development programs is to proceed with less than the best personnel, particularly in the key leadership positions in government and industry. High-quality program managers and system engineering leaders, in particular, are critical. High aptitude and extensive experience, combining to create high domain knowledge, are required for individuals to be fully effective in these positions” (p. 82).

SUPPORT FOR OPERATIONS AND MAINTENANCE

A common fallacy with software-intensive systems is that they can be built, deployed, and then operate with relatively little “maintenance” and modernization effort. The surprise, for those unfamiliar with such systems, is that operations and maintenance will very often include substantial modernization effort. This effort is needed both in response to new requirements and also in response to rapid growth and change in technological infrastructure. This is true for NextGen and the NAS, as the rest of this report has described, and this fact has implications for how the FAA should explain its needs to Congress and its overseers.

Congress plays an important and complex role in its relationship with the FAA and NextGen. One facet of its role is oversight, which it has done diligently ever since the formation of the FAA in 1958 and especially since the time of the controllers’ strike and the FAA’s response and later the Advanced Automation System (AAS) problems. However, in its role of authorization and appropriations, it has increasingly played an indirect role in the management of the FAA and has always acted as its banker. Since the AAS, Congress has passed legislation aimed at improving the FAA’s ability to modernize the ATC system.

In the 1996 Appropriations Bill, the FAA was given nearly unique authority to revise its acquisition regulations (FAA’s Acquisition Management System is broader and less prescriptive than the Federal Acquisition Regulation) and its personnel system. The aim was to allow the agency to attract the necessary technical personnel and to more rapidly and efficiently acquire the telecommunications equipment, software, and systems integration required of a modern air traffic control system. Some of these reforms were especially useful in the late 1990s and early 2000s. Despite these measures, today’s FAA, like many other government agencies, has trouble attracting and retaining sufficient technical talent to support its missions. And in 2004, the community was calling for even better performance and a more bold vision of the future. Congress then approved the formation of the Joint Planning and Development Office (JPDO), allowed for the new position of chief operating officer to be recruited from industry, and reorganized the FAA air traffic control organization, based on an airline industrial model. Congress further authorized the formation of an oversight committee composed of industry and government experts and managers. In spite of these proactive measures on the part of Congress, the FAA’s NextGen program is still moving at a pace slower than desired

and is now facing the new requirement of accommodating unmanned aircraft systems into civil airspace.

As for any large-scale government IT effort, a long-term commitment is important. Although Congress has been supportive of FAA efforts, as the above discussion makes clear, in the committee’s view, there is a specific need for support of ongoing maintenance and modernization (upgrades) and refreshing and modernizing of both hardware and software to provide reliable, cost-effective operation.²² Too often in government, funds are allocated for specific (new) programs or projects without sufficient allocation for the full life-cycle costs and for maintenance and refresh of existing (and still important) programs.²³

NextGen is and will be a continuously evolving system-of-systems and should not be thought of as one large monolith. NextGen needs to continually evaluate new technologies and approaches and then make decisions about what to incorporate and how. This means that there will be some reconceptualization going on while there is also a great deal of implementation happening, perhaps of components that are already being scheduled to be phased out. A system architecture, as discussed earlier in the report, is essential, always. But there must be flexibility for the system architecture to evolve, too, with consequent changes down the line leading ultimately to the implementation of a deployed system. In this respect, NextGen needs to be managed and evolved very much like other large systems (e.g., financial systems and retail systems) that are constantly evolving, requiring that management and financing recognize this need for evolvability. A large software-intensive system is not “bought” once and for all, but rather put into place and then continually maintained and evolved at continuing cost and effort.²⁴ Finally, as discussed in Chapter 1, Congress itself can partner with the FAA in acknowledging the changing context and adjusting expectations as appropriate.

_______________

²² Learning from the NavCanada experience would be useful. A financially stable, long-term business model is required to both organize and attract the necessary technical talent/expertise with a dependable income stream to provide project stability and predictability to design and deploy such a complex system.

²³ Lessons from the private sector suggest that there are many other equally significant opportunities for support of such systems, such as continuity and timeliness of funding, color of money, directed funding, addressing the lack of discretionary funds, and more readily moving funds where needed the most. These are common practices in running a corporation that are not typically available within the government structure.

²⁴ This is increasingly inconsistent with waterfall-style software development approaches that discourage revisiting early concepts and architectures and that can thwart innovation and evolution.