3

Recommendations

OVERVIEW

As the fraction of critical mission capability delivered by embedded weapons systems software grows, the U.S. Air Force (USAF) software sustainment capabilities will be increasingly challenged. With an eye toward the current issues faced by the USAF software sustainment enterprise, the committee consulted with technical experts in government, academia, and defense and nondefense industry to identify opportunities to improve future efficiency and effectiveness of USAF software sustainment.

The study committee’s recommendations cover six areas:

1. Accelerating adoption of DevSecOps where appropriate;
2. Building a robust talent pipeline for software;
3. Ensuring access to modern software tooling and infrastructure;
4. Tightening the loop between users and software developers;
5. Maintaining and managing USAF software inventories; and
6. Negotiating ownership of data rights and SILs.

ACCELERATING ADOPTION OF DEVSECOPS WHERE APPROPRIATE

Commercial software development has significantly evolved over the past two decades. A significant paradigm change was the shift to agile software development. Agile development practice rejected the top-down, requirements-driven approach

of the traditional waterfall development cycle for a series of shorter, dynamically planned development steps with scheduled activities. While there are many variants of agile development practices, most of these approaches emphasize the following:

• Smaller increments of work known generically as sprints;
• Small cross-functional teams with a strong focus on cross-team communications and coordination;
• Iterative and incremental refinement of capability;
• Direct access to stakeholder representatives, who can prioritize and refine functional requirements; and
• A focus on early development of a working prototype system that can be expanded and extended to achieve deployable functionality.

The committee found that practices that support the shift toward more agile software development methods included evolved software development automation tools, environments that supported rapid and frequent testing, and build cycles with many organizations adopting daily or continuous build and test processes to continuously maintain an operational system under development.

The move to agile development methodologies will require a review of some of the procedures used in program management to allow those procedures to have the latitude to accommodate the agile software development process. For example, MIL-STD-881D, April 9, 2018,¹ requires submittal of a work breakdown structure (WBS) early in the program planning using Appendix J of that document, while agile development requires the shorter, dynamically planned activity cycles allowing the program to build out its own WBS.

Over the past two decades, this paradigm shift in methodologies has been implemented at organizational scale in the form of Development, Security, and Operations (DevSecOps). DevSecOps recognizes the importance of integrating incremental development of software with frequent and incremental deployment in the operational environment. Rather than focusing on larger, high-risk block upgrades of software, DevSecOps focuses on the continuous delivery of smaller, more incremental features to the operational code baseline, leveraging automated tools for continuous testing, configuration management, and control of software, IT infrastructure, and hardware components. Advocates of DevSecOps note faster deployment cycles, reduction in failure rate, and shorter fix cycles for errors discovered post-deployment. The Development and Operations (DevOps) approach has been extended to DevSecOps to provide equivalent emphasis on integrating

___________________

¹ See Department of Defense (DoD), “Work Breakdown Structures for Defense Materiel Items,” last update June 19, 2018, https://quicksearch.dla.mil/qsDocDetails.aspx?ident_number=36026.

information security practices into the DevOps approach, including continuous automated security testing.

In these agile software methodologies, developers do not typically develop to an exacting specification. Instead, they work against a list of features that are prioritized jointly by the customer and the development team. This feature list is a living, not static, document—it evolves over time, reacting to changing customer priorities and lessons learned from incremental software deployments. These methodologies acknowledge that fielded software is incomplete and recognize that deployed software will continue to evolve. Software development teams and customers work together to prioritize and manage this queue of work—referred to as technical debt.

While USAF software sustainment organizations can and should benefit greatly from DevSecOps, there are unique aspects of some Department of Defense (DoD) software that will require more testing and analysis before deployment. Some USAF systems are designed to deliver lethal effects, and the USAF must use additional methods to ensure that these systems function as intended—such as formal methods, safety cases, rigorous operational testing, and stringent personal reliability programs.

Several characteristics are critical to implementation of DevSecOps and represent challenges to traditional USAF software methodologies. First, these methodologies require an empowered agent to act as the voice of the customer, with authority to add new feature requests and to prioritize among the remaining work to be done. Second, these methodologies require commitment to resource a standing team to provide software development services. Under these methodologies, one budgets for a standing software development capacity, rather than for a specific set of features with cost based on an estimated number of source lines of code (SLOC). Last, these methodologies require relief from burdensome formalized Authorization to Operate (ATO) and Operational Test (OT) processes that must be completed for each software delivery. To achieve the advantages of continuous delivery, ATO and OT processes must be integrated into the development process and operate at the speed of software delivery.

The Defense Innovation Board (DIB) report practices (see Box 3.1)² can be applied both to stand-alone software products and to embedded systems software. Hardware systems may require significant safety and performance testing prior to delivery to the fleet, but DevSecOps can still perform continuous delivery to a dedicated integration test asset, and developers can receive real-time feedback on their software from testers and operational surrogates.

___________________

² See J.M. McQuade, R.M. Murray, G. Louie, M. Medin, J. Pahlka, and T. Stephens, “Software Is Never Done: Refactoring the Acquisition Code for Competitive Advantage Defense Innovation Board,” last update March 13, 2019, https://media.defense.gov/2019/Mar/14/2002101480/-1/-1/0/DIB-SWAP_STUDY_REPORT[DRAFT],_LAST%20MODIFIED_13MAR2019.PDF.

Programs that have adopted these practices have been challenged to find software talent in the USAF organic developer workforce and in the defense industrial base with significant experience in DevSecOps. While agile programming methodologies have found increasing use in the defense industry and in USAF sustainment centers, few government organizations have transitioned to an integrated continuous delivery approach. Investment in tools, training, and expertise will be required to adopt current industry best practices.

DevSecOps is well suited to the needs of USAF sustainment. As noted earlier, software sustainment is fundamentally a process of continuous software evolution, and the practices of continuous development and deployment to operation incorporated in DevSecOps should permit software sustainers to deliver incremental software improvements to end users more frequently, prioritize defect removal and new feature requests, and rapidly receive feedback on deployed software to support iterative development. More responsive software practices should allow program managers to adjust timing of software update deployments to the fleet, permitting more rapid upgrade cycles, and increased responsiveness to user requests. However, this new capability must be accompanied by appropriate training on the use. There is a risk that in the rapid pace of capability delivery, the end user could fall behind simply because of the inability to utilize the feature due to insufficient training.

The structure of the USAF organic sustainment workforce also lends itself to the DevSecOps model of purchasing software services “by-the-team,” with standing dedicated teams supporting each software product with the output of the team sized by the complexity of the work at hand and the resources available to the program.

One issue the USAF software sustainment community has faced is the establishment of commercial-quality software factories. The term “software factory” sometimes brings up the wrong image of desk after desk of software engineers working on repetitive tasks. This is not what is meant by a software factory. To the contrary, a software factory is a development environment designed to make the software engineers as productive and innovative as possible by providing advanced automated software tools and computational resources. Commercial tools are evolving quickly and can be licensed. And cloud services can provide flexible, on-demand computational resources for testing and analysis of newly developed software to discover coding and architectural defects, performance issues, as well as vulnerabilities on a daily basis.

BUILDING A ROBUST TALENT PIPELINE FOR SOFTWARE

One of the key sources of variability in software productivity is the level of development talent available to software development teams. Computer scientists and programmers are not created equal and differences in capability can result in significant differences in performance. The USAF software sustainment centers have traditionally been co-located with maintenance depots, significantly separated from national centers of commercial software development. This approach has required the USAF to recruit, train, and retain a workforce that is generally separate from the large commercial software development communities located in coastal metropolitan areas. While the USAF reports that these local development communities have long program tenure and develop detailed narrow domain expertise on the systems, they are maintaining a very different approach from the short tenured staff associated with most commercial software development; this difference may also reflect a lack of ability to recruit and retain the most innovative, creative, and productive software talent.

Recruitment and retention of the USAF organic software maintenance workforce is constrained by personnel policies that do not differentiate the need to compete with private industry for software talent. Salary levels, rigid pay grades, and limited earnings growth potential dissuade the necessary software experts from pursuing government employment. Repeated studies have shown significant variability in efficiency among software developers,³ with some studies concluding that top-performing software developers can be up to 1,000 percent more efficient than their average peers. Given this variation, talent, skill, and performance are critical differentiators, and the USAF should consider process and policy changes to promote flexibility, attract top talent, recruit high-performing expert developers, and enable the retention of key staff.

Government software development work has a negative reputation among developers, and the USAF should pay critical attention to promoting and recognizing its top talent, improving its reputation as an employer of software talent, and enabling work environments to be attractive and compelling to staff.

___________________

³ See Construx, “Productivity Variations Among Software Developers and Teams: The Origin of 10x,” Blog, https://www.construx.com/blog/productivity-variations-among-software-developers-and-teams-the-origin-of-10x/, accessed June 2, 2020.

An alternative to and extension of the organic software workforce can be found in contracted technical talent, whether as fully outsourced efforts or as staff augmentation at USAF operating locations. Contractors may have fewer constraints than the USAF faces in competing for talent, but may present a more transient commitment to particular programs and projects, may face challenges with retention of technical expertise on particular projects from employee turnover, and may incur additional costs due to fee, profits, and administrative overheads and possible inefficiencies in direction and utilization. Sustainment organizations have felt compelled by regulation to manage organic and contractor software maintenance workforce mixes in a manner similar to that imposed on hardware sustainment efforts, but such regulations seem ill suited to the developmental nature of software sustainment. The USAF should carefully evaluate software sustainment contractor and organic workforce mixed teams with an eye toward accessing key talent, achieving operational and business efficiencies, and achieving maximum output per employee.

ENSURING ACCESS TO MODERN SOFTWARE TOOLING AND INFRASTRUCTURE

Over time, the requirements for any fielded software system evolve. This evolution can be the result of the obsolescence or upgrade of the software’s target host platform. This evolution can come from changes in users’ performance expectations, leading to the demand for new features. This evolution can arise from changes in adjacent systems or from fielding of new capabilities in networked systems. This evolution can arise from changes in policy, regulation, or process. This evolution can also arise from the evolution of the threat environment, including the emergence of new cybersecurity threats or awareness of new cybersecurity vulnerabilities. These evolved requirements and changes in targeted hardware platform specifications mean that a software system reset to its original configura-

tion state would no longer satisfy user expectations and might not be capable of operating at all.

The objective of software sustainment is fundamentally different from that of hardware sustainment. Instead, much of software sustainment is developmental work to bring new functionality to the system and to fix discovered defects and vulnerabilities. Rather than returning a system to its original state, the intent of software maintenance is the design, development, and delivery of a new product that offers incremental improvements over previous software versions.

Many of the issues associated with USAF software sustainment result from incorrectly seeing software sustainment as equivalent to hardware sustainment. This false equivalence results in practices, such as organic software development organizations collocated with hardware maintenance depots, the application of hardware-focused regulations and legislation to hardware development, and the use of USAF financial processes and personnel practices more suitable to hardware maintenance than to software development.

Recent studies by the Defense Science Board (DSB) and the DIB have reinforced the concern that the terms “software maintenance” and “software sustainment” are misnomers, and in reality, software is continuously in development.

The DSB Task Force on the Design and Acquisition of Software for Defense Systems⁴ noted: “The classic phases of acquisition include development, production, and sustainment. However, modern software is in continuous development. This creates a misalignment between the DoD’s processes and the reality of contemporary best practices.”

The DIB report “Software Is Never Done: Refactoring the Acquisition Code for Competitive Advantage”⁵ noted: “Thinking of software ‘procurement’ and ‘sustainment’ separately is also a problem: software is never ‘finished’ but must be constantly updated to maintain capability, address ongoing security issues and potentially add or increase performance.” The DIB report further noted:

___________________

⁴ See Department of Defense (DoD) Defense Science Board, 2018, “Design and Acquisition of Software for Defense Systems,” https://dsb.cto.mil/reports/2010s/DSB_SWA_Report_FINALdelivered2-21-2018.pdf, accessed June 2, 2020.

⁵ See J.M. McQuade, R.M. Murray, G. Louie, M. Medin, J. Pahlka, and T. Stephens, “Software Is Never Done: Refactoring the Acquisition Code for Competitive Advantage Defense Innovation Board,” last update March 13, 2019, https://media.defense.gov/2019/Mar/14/2002101480/-1/-1/0/DIB-SWAP_STUDY_REPORT[DRAFT],_LAST%20MODIFIED_13MAR2019.PDF.

Based on those observations and statements, as well as information received by the study committee, it becomes apparent that software sustainment is merely software development by another name, and should be treated as such. This perspective creates an opportunity to evolve USAF software sustainment practices by adopting industry best practices associated with the incremental and continuous delivery of software.

One unique challenge facing the USAF is the transition and handoff from contractor development to USAF organic staff for sustainment. In many programs, the original contractor maintains a significant presence even in the sustainment phase of the program, but the contractor workforce available for collocation at the USAF sustainment facility may differ significantly from the team involved earlier in the original software development, leading to challenges in knowledge transfer. In transition, the USAF may receive source code and formal design documentation, but there is the risk for significant loss of knowledge and experience in transition from the original prime development contractor. The DSB report quoted above noted some of the components critical to successful transition to sustainment, including: “all documentation, test files, coding, application programming interfaces (APIs), design documents, results of fault, performance tests conducted using the framework, and tools developed during the development.”⁷

The capacity of sustainment teams to continue to develop and evolve a software baseline, particularly for specialized embedded systems applications, is also limited by the availability of representative target host platform hardware to support development and testing. These target development platforms, or System Integration Laboratories (SILs), represent a significant investment for USAF programs and must be maintained in such a manner as to represent each of the operational hardware configurations of the system maintained in inventory. Where prime software system development contractors develop SILs, they should be deliverable to the USAF and relocated to a software sustainment facility as part of a program’s transition to sustainment. Access to an adequate number of SILs at USAF facilities is critical to scaling software sustainment activities.

___________________

⁶ J.M. McQuade, R.M. Murray, G. Louie, M. Medin, J. Pahlka, and T. Stephens, “Software Is Never Done: Refactoring the Acquisition Code for Competitive Advantage Defense Innovation Board,” last update March 13, 2019, https://media.defense.gov/2019/Mar/14/2002101480/-1/-1/0/DIB-SWAP_STUDY_REPORT[DRAFT],_LAST%20MODIFIED_13MAR2019.PDF.

⁷ See https://dsb.cto.mil/reports/2010s/DSB_SWA_Report_FINALdelivered2-21-2018.pdf.

USAF experience demonstrates the difficulty of estimating software development cost and schedule for initial development efforts as well as for sustainment efforts. Software estimation is plagued by significant differences in productivity from project to project, driven by complexity in task, target, and evolving requirements that may not be apparent until well into development. This is also the case when software development teams, and even members of the same teams, are not operating with the same processes and tools available. Most software estimation techniques rely on historic comparable performance to provide a basis of estimate, but because of the large variance in productivity between and across programs, these historic performance baselines often fail to reflect the experience encountered on any particular project. This task of estimating software development impacts resource-constrained sustainment activities, which compete for resources with hardware sustainment activities, and projects and programs can find themselves resourced at a level of effort that may not match user expectations for defect removal and feature delivery. This difficulty in estimating costs results in a misalignment of funding necessary to support software sustainment.

TIGHTENING THE LOOP BETWEEN USERS AND SOFTWARE DEVELOPERS

One of the key criteria for successful agile methodology development is the availability of empowered users, who have deep understanding and experience with the application domain, are empowered to approve sequencing of feature development and delivery, and are authorized to coordinate with other system stakeholders to negotiate and align competing priorities. This role is typically referred to as serving as “the voice of the customer.”

In typical USAF development practice, individuals and organizations authorizing and modifying requirements often are removed in time, space, and task orientation from actual system users. These individuals may provide budget and fiscal oversight to a development program, manage execution of the program from an acquisition perspective, or serve as requirements managers for a system. Agile methodologies development practice expects each of these actors to cede some

authority to empowered end users who engage with development teams in a tighter Observe, Orient, Decide, Act (OODA) loop.

In agile methodology, development occurs in short time windows referred to as sprints, and at the conclusion of each set of sprints, the remaining work to be done, including the features that were not addressed in previous sprints, new user requests received during the sprint, and new defects and rework discovered during the sprint, are reprioritized by users and developers in tight synchronization. This movement of remaining work permits users to accelerate mission-critical needs, reprioritize feature delivery to address fast-tempo changes in operational requirements, accept the need to live with certain software defects so that higher priority fixes can be implemented first, manage and accept risk, and gain confidence in the responsiveness of the software sustainment team. Developers gain critical real-time feedback to avoid wasted effort from misinterpretation of requirements, to negotiate schedule and resource commitments directly, to gain the flexibility to align and sequence tasks for increased throughput, and to enable development of alternative solutions through the receipt of creative feedback.

This interface between warfighter and software developers is most effective when teams are closely proximate or collocated with users. On programs with significant embedded hardware or significant security requirements, this collocation can create challenges owing to the need for secure access to hardware SILs.

THE IMPORTANCE OF MODULAR AND OPEN SOFTWARE ARCHITECTURE

While it is necessary to take steps to address the increasing software sustainment burden, it is also possible to reduce the burden through intelligent use of open software architectures that facilitate rapid and cost-effective future evolution. The lack of a well thought out architecture can result in software systems that are

difficult or impossible to maintain. Although individual open architectures should be designed to address program-specific life cycle objectives, the tenets of an open architecture can be similar across the USAF enterprise. By leveraging four tenets of modular open software architecture, the USAF can design and develop software systems that are inherently easier to maintain and sustain.

These recommended software architecture tenets are discussed further as follows:

Software Architecture Tenet 1: Develop modular software with well-defined interfaces among key components. There are both technical and organizational benefits surrounding this tenet. First, modular software is easier to sustain. It breaks larger software systems into manageable chunks, promotes effective unit testing, and minimizes the “ripple effect” of software changes. Next, having well-defined interfaces allows individual software components to be acquired and maintained by independent organizations, teams, and developers. Well-defined interfaces also help to mitigate some of the possible vendor lock owing to deep domain knowledge. Furthermore, it allows software components to be developed and sustained on different timelines and at different rates. Last, having truly open interfaces, rather than custom interfaces, between key software components helps prevents vendor lock. Historically, vendor lock has resulted in high costs and long timelines. Being able to recompete components within a system should vastly reduce this risk.

Software Architecture Tenet 2: Leverage commercial and open source middleware libraries and operating systems and minimize hardware-specific code. There are multiple benefits of leveraging commercial-off-the-shelf (COTS) and open source software. Their wide consumer base reduces cost to the government. Their broad adoption results in highly reliable solutions. Their investment in maintenance alleviates the government from having to maintain custom software. The benefits of COTS software must be balanced against the inherent risks. Commercial products come with the risk that they could be discontinued at any point. A COTS product may contain bugs that the vendor chose to not fix or the fix impacts the timeline of a weapon system operation. This risk is especially true for small solutions with

limited adoption. For this reason, it is best to leverage commercial products that are well established and have a broad user base.

Additionally, hardware-specific application software can be very difficult to maintain as hardware evolves. In many cases, low-level, complex code must be rewritten to work with new hardware features and configurations. Historically, this coding was done to achieve peak performance. While this needs to be examined on a case-by-case basis, there has been a trend in high-performance systems that shows that the majority of application code can be written in a hardware-agnostic way and still meet performance requirements. One strategy could be to implement the end-to-end application with hardware-agnostic code, such as device drivers, operating system services, and middleware functions, and then if necessary, optimize key operations with hardware-specific code to meet performance requirements. These operations are typically a small percentage of the total SLOC, and therefore reduce the burden during maintenance activities.

Software Architecture Tenet 3: Establish data rights that align with the life cycle strategy. Having access to the right level of data and source code should enable effective maintenance. Although it may seem obvious, establishing government purpose or unlimited rights on source code is, at a minimum, required to allow the government to maintain the code without being obligated to go back to the original equipment manufacturer (OEM). Alternatively, the restrictions experienced with proprietary software are minimized if a truly open architecture with appropriate standards is followed, thereby facilitating module replacement without the need for data rights. Rights on data should also be established and align with integration and test needs.

Software Architecture Tenet 4: Develop enterprise architectures. Developing open software architecture provides great sustainment and maintenance benefits to a single program. Establishing and deploying enterprise architecture can provide an order-of-magnitude greater benefit.⁸ However, it only makes sense to deploy an enterprise solution across programs with similar technical capabilities and life cycle strategies. For example, it would be beneficial to establish a single architectural approach for radar systems across the USAF. The first major benefit of enterprise architecture is that it allows for reuse across multiple programs. Some software components used on one radar system could easily be reused on another radar system. Next, because of reused software across multiple programs, there is left one (albeit larger) overall software architecture to maintain across that singular the enterprise. Last, the sustainment teams now need to learn only that one architecture, as opposed to multiple one-off solutions.

___________________

⁸ Committee discussion with SWEG personnel at Hill AFB, Ogden, Utah, on radar systems.

These recommended tenets are important but not sufficient alone. In addition, it is important to motivate program offices and contractors to build maintainable open software architectures. On the program office side, program managers are often motivated to go fast and cheap, and to focus on getting to an initial operating capability. This approach often is orthogonal to an open software architecture mindset. Program offices need to be given flexibility and incentives to allow for open software architectures, including both time and money allocations for the contractor. Likewise, contractors need to be motivated; otherwise, they may default to building a custom and unmaintainable solution that ensures that they must make any component upgrades. Establishing open software architecture requirements and verification methods at program onset is critical.

MAINTAINING AND MANAGING USAF SOFTWARE INVENTORIES

Today, the USAF supports a large number of legacy platforms and systems. These systems were suitable to the purpose at the time they were originally developed, but over time the ability to continuously support these systems has become increasingly expensive—requiring expertise in software languages, legacy code bases, tool chains, and hardware architectures that are no longer widely supported. Providing software sustainment for these legacy platforms will continue to be increasingly challenging as new requirements for security, requirements for rehosting to new hardware architectures and cloud-based environments, and new interface requirements proliferate.

The USAF should consider a more formal software life cycle portfolio inventory management process that would evaluate the cost of refactoring legacy systems, prioritize among the systems requiring support, acknowledge the obsolescence of many legacy code bases, institute a configuration freeze on those systems with key components that are no longer supported or supportable, and require investment in modernized replacements for critical software systems that can no longer be supported. Performing this triage on the sustainment inventory should permit resources to be concentrated on higher priority support programs and would permit concentration of investment for tools, SILs, and transition to DevSecOps for this priority inventory.

NEGOTIATING OWNERSHIP OF DATA RIGHTS AND SILS

Currently, the USAF appears to be lacking sufficient legal experts in IP within the area of software development (see the section “Data Rights Ownership and Usage Agreements,” in Chapter 2). This insufficiency is impacting the USAF’s ability to access needed code for various software development activities or at least the timely and efficient access to this code. For legacy systems, this issue is critical, and the USAF is unable to make cost-efficient and effective coding on these systems without reinstating the full relationship with the original contractor. These systems are often safety and mission related. The delays caused by the lack of contracting language to allow the USAF easier use or ownership rights to the code are unacceptable for such systems.

Government contracting teams should review contracting models for the incentives structures that can be dissuading contractors from allowing user access or ownership of the code by the USAF. Part of the contracting terms and conditions should account for the sustainment of the tool required for sustainment of software systems. These contracts should be reviewed on a regular basis to determine the necessity of sustaining such tool chains based on life cycles of the software. Contracting should be using models that enhance the team approach to such software development, including the movement of SILs from the contractor or its subcontractors to the USAF facilities.

Modern software tools are important to developing quality code in a timely and efficient manner. Commercial software companies that have adopted DevOps provide their developers with suites of automated tooling that help make the developers more efficient and allow for daily testing for functionality, defects, and cyber vulnerabilities. This idea of providing a software development environment with automated tooling is embodied in the concept of the software factory. There is a rich ecosystem of tool providers who are constantly improving their products. Due to security concerns, USAF’s software teams have often not had access to these commercial tools—or had them in environments that were distinct and separate from the cloud services used by commercial developers and maintained by the tool providers.

The USAF should use commercial tools to the maximum extent possible. Where necessary, the USAF could use secure implementations of these cloud services, but must recognize that the tool development may not transition to a secure cloud as quickly and easily as to commercially available clouds.

In addition, the organic USAF units that develop and sustain code are responsible for some systems that use legacy code for which current tooling may not be available. There is a burden, both financially and in workload, to maintain software tooling for these systems. The USAF must factor these costs (the costs of maintaining not only the mission software, but also the tooling required) into

decisions about retaining this legacy code or rewriting the code in more current and supportable languages.