Executive Summary
Modern software engineering practices, pioneered by the commercial software community, have begun transforming Department of Defense (DoD) software development, integration processes, and deployment cycle. DoD must further adopt and adapt these practices across the full defense software life cycle—and this adoption has implications for what is currently called “software maintenance” and “software sustainment” across the U.S. defense community.
In addition, studies by the Defense Innovation Board (DIB) and the Defense Science Board (DSB) have suggested that the terms software maintenance and software sustainment are misnomers and that, in reality, software is continuously in development. Software sustainment includes defect removal, vulnerability discovery, and patching, which coexist with feature addition and capability growth. It is counterproductive to spend too much effort trying to surmise the differences in these tasks. It is better to recognize the continuous development aspect of software, organize and support it with this understanding in mind, and provide a continuous flow of improved capabilities. Nevertheless, throughout this report, the term software sustainment will be used to represent the organic and contracted work that goes on during what is traditionally called the sustainment phase of U.S. Air Force (USAF) programs.
In conducting the present study, the committee found it clear that the USAF community responsible for software sustainment understands this concept and is in the process of transforming to more modern software engineering processes, practices, and tooling. Accelerating this transition should deliver the USAF and broader defense community benefits in cost, schedule, and capability—benefits
that are important in their own right, but even more important in today’s national security environment. The current software maintenance and sustainment community faces obstacles to this transition in culture, organization, funding, infrastructure, and staffing.
While the members of the study committee strongly support the transition to modern software engineering practices, such as agile development and Development, Security, and Operations (DevSecOps),1 the members recognize that not all software is alike. Web applications are not the same as embedded weapons systems or the nation’s nuclear command and control system. In addition, the USAF is maintaining, sustaining, and developing software that represents different generations, architectures, and coding languages, some of which are “orphans”: languages with few if any commercial uses, coding communities, or economic viability. For systems in development, there is a vibrant commercial infrastructure of software tools available to build the “software factories” that allow programming teams to be more productive. But for legacy systems built with archaic languages, and especially for those running on hardware no longer produced commercially, full tool suites may not exist. These legacy systems and the fact that not all software is alike suggests that “one size may not fit all” when it comes to software sustainment.
This situation also points out that when it comes to embedded weapons systems—the focus of this study—software, as important as it has become, does not exist in a vacuum. Software works in conjunction with hardware, other software modules, and users. The proper approach to continuous software development must take into consideration the viability and plans for the hardware, the training and proficiency of the users and operators, the environments in which the systems will be deployed, and the hazards and consequences of their use. Weapons systems with embedded software can benefit from the advances that the commercial software communities have pioneered, but also must recognize special requirements that arise from tight coupling, critical timing, integration with sensors and actuators, and safety, assurance, and lethality issues. Good software engineering requires good systems engineering from initiation of development to deployment.
With this in mind, the committee recommends the following:
Recommendation 1: The U.S. Air Force should continue its transition to the modern software engineering practices and procedures represented by agile development; Development, Security, and Operations (DevSecOps); software factories; and continuous development, continuous integration, and continuous deployment. Embedded weapons systems can benefit from
___________________
1 See GSA Tech Guides, “Understanding the Differences Between Agile and DevSecOps—From a Business Perspective,” https://tech.gsa.gov/guides/understanding_differences_agile_devsecops, accessed June 2, 2020.
these methodologies despite additional testing and safety requirements. In making this transition, the Air Force should evaluate each system for its unique characteristics, the availability of programmers, and tool suites for the software development environments required.
Recommendation 2: Software development requires a strong technical team, and the U.S. Air Force should address staffing concerns: recruitment, retention, development, and career paths. Modern software engineering requires well-trained, well-motivated personnel. The government at large and the Air Force in particular have been slow to adapt to a post-industrial world with the grade structure and salaries and productive work environment for knowledge workers such as software talent. The Air Force should manage its organic and contractor software personnel—in sustainment, development, and operations—in a more flexible way that acknowledges the need to acquire critical talent. Attention should be paid to alternative personnel structures as well as rapid and creative contracting processes.
Recommendation 3: The U.S. Air Force should provide access to modern software tooling and infrastructure to support its software teams and enable them to be more productive. This action will require investments in System Integration Laboratories (SILs), as well as changes in security standards and policies. Much of this tooling now requires access to commercial cloud services that are often inaccessible from government networks owing to Department of Defense information technology policy.
Recommendation 4: Modern software practices emphasize the importance of software teams working closely with the ultimate users of the software systems being created. For the U.S. Air Force, this relationship implies much tighter and more frequent interactions between the software engineers and the warfighters and other operational stakeholders. This relationship development is critical not only for the functionality desired but also for the user interface, a key enabler for successful deployment. In some cases, warfighter representatives may be embedded with development teams. In other cases, it may be appropriate for the development teams to be located closer to operational units. This tighter engagement of the actual software development on daily and weekly levels does not obviate the need for strong systems and software architecture to guide the overall development.
Recommendation 5: The U.S. Air Force should implement and widely adopt four tenets of modular open software architecture in programs across the enterprise: (1) develop modular software with well-defined interfaces
among key components; (2) leverage commercial and open source middleware libraries and operating systems and minimize hardware-specific code; (3) establish data rights that align with the life cycle strategy; and (4) develop enterprise architectures.
Recommendation 6: The U.S. Air Force should develop and maintain an inventory accounting for the software systems used by Air Force systems. This inventory should include documentation on the languages and environments needed for those systems, the hardware they run on, and the match to resources required. The Air Force should then use this inventory to realistically evaluate the cost of continued ownership of legacy systems, and where practical, either refactor systems with archaic software into modern languages and development environments or sunset the programs.
Recommendation 7: The U.S. Air Force should initiate the movement of System Integration Laboratories (SILs) to Air Force control, allowing the Air Force to have direct access for testing and evaluation using those SILs throughout the software development phase.
Recommendation 8: The U.S. Air Force should acquire data rights sufficient for software sustainment of its weapons systems, even when contractors hold agreements to sustain the system for the life cycle of the program. Embedded software development is strongly tied to the hardware on which it resides, the sensors and actuators with which the embedded software interacts, and the data rights of legacy tooling. This reemphasizes the need for System Integration Laboratories (SILs) where the hardware and software can be integrated and tested. Furthermore, embedded systems, especially legacy embedded systems, are often built on proprietary hardware and may use older operating systems and languages. These issues make it extremely important that the government have data rights sufficient for software sustainment, even in cases where contractor sustainment is planned for the life cycle of the program.
Recommendation 9: The U.S. Air Force should acquire the necessary legal expertise in software intellectual property law and regulations to develop contracting relationships that allow the Air Force to have user rights or ownership to both legacy code and code to be developed in the future. Air Force legal personnel should engage in discussions with contractors about user rights or easier access to code for systems that are under current development.
The committee believes that there will continue to be a need for both organic and contracted software engineering support for deployed systems. Several of the recommendations this committee has made are in the process of being implemented. Which organization performs these duties should depend on market conditions, as well as national security needs. This approach is not different from the current philosophy of contracted and organic sustainment writ large, but it must take into consideration the specific talent needed and available for software development.
