2

Current State of USAF Software Sustainment

OVERVIEW

This chapter is an assessment of the current state of software sustainment in the U.S. Air Force (USAF) and an examination of important trends both internal and external to the USAF that could severely impact its ability to upgrade and maintain software. USAF leadership interviewed by the study committee understood the critical importance of software to its mission. While the USAF is taking active measures to address the current sustainment workload, the ability to keep pace in the future will not be possible unless the USAF updates its software sustainment enterprise.

The USAF has an active and committed effort to sustain its software. Software sustainment is led by Software Engineering Groups (SWEGs) totaling approximately 5,000 employees at three locations:

1. Hill Air Force Base (AFB) in Ogden, Utah;
2. Tinker AFB in Oklahoma City, Oklahoma; and
3. Warner-Robins AFB in Warner-Robins, Georgia.

The SWEGs either perform or oversee almost 300 programs that support operations (e.g., B-1, F-16, C-17), mission (e.g., Jet Engine Test, Personnel Recovery Command and Control [PRC2], Global Positioning System [GPS]), and future programs that are or will be transitioning into sustainment (e.g., F-35). In

addition to the three SWEG locations, contractors provide substantial sustainment support.¹

The study committee visited two SWEGs and had many briefings and discussions with management and technical staff on those visits. The level of coordination and workload sharing among the three SWEGs was impressive. As an example of the level of coordination, leaders of the three organizations meet regularly and coordinate outreach to headquarters activities, balance workload across the three bases, and share best practices regarding hiring and technology.

However, the relationship between contractor organizations and the government SWEGs varied greatly. In some cases, the relationship was very good when the government and contractor teams shared workload and complementary skills. In other cases, the relationship was highly contentious when team responsibilities, data rights, source code access and control, and access to the full development environment were in question. While different SWEG teams had tensions with contractor groups depending on how different programs were run, contractors still play a critical role in providing expertise and support to USAF software sustainment processes.

From several presentations, USAF software engineers demonstrated a command of modern software engineering approaches and how to apply these in a judicious manner. Several USAF programs employed agile and Development, Security, and Operations (DevSecOps) software development tools and methods, and employed modern approaches to continuous integration for development and testing. Such projects typically involved development on modern information technology (IT) hardware platforms and the cloud.

Certain programs were harder to manage and improve when modern IT platforms and cloud services were unavailable. The USAF has a substantial base of legacy embedded systems and subsystems—some as old as 25 years. This state of systems aging drives SWEG staff to use dated technical approaches for development and testing, due to the lack of available tools for those programs. Staff at the SWEGs had a deep appreciation for software development across a wide range of systems, from software that is tightly coupled to hardware, such as flight safety systems, to software that leverages cloud technologies, such as those technologies used to manage command and control for rescue and recovery. USAF software sustainment teams would be able to utilize modern tools, if they had proper access to those tools.

Software development for radar systems, flight controls, and other crucial systems involves a tight coupling and interaction with hardware. The capacity of sustainment teams to continue to develop and evolve a software baseline, particularly for specialized embedded systems applications, is also limited by the availability of

___________________

¹ USAF staff was not able to quantify the level of contractor-performed sustainment. The level was estimated to be 50 percent.

representative target host platform hardware to support development and testing. These System Integration Laboratories (SILs) represent a significant investment for USAF programs and must be maintained in a manner to represent each of the operational hardware configurations of the system maintained in inventory. Where prime software system development contractors develop SILs, they must be deliverable to the USAF and relocated to a software sustainment facility, as part of a program’s transition to sustainment. Access to an adequate number of SILs is critical to scaling software sustainment activities and achieving the necessary schedule commitments.

From briefings and discussions with USAF and Department of Defense (DoD) leadership, it is apparent that the challenges facing sustainment are USAF-wide, not solely isolated at the SWEGs. The USAF believes that software sustainment can and must be more rapid and responsive to meet changing mission needs. The USAF needs to internalize the complexity of interrelated challenges, which are described in the following sections.

ORGANIZATIONAL STRUCTURE NOT ALIGNED FOR SOFTWARE SUSTAINMENT

From an organizational perspective, the SWEGs report up to the USAF Materiel Command (AFMC) through respective Air Logistics Complexes (ALCs). The ALCs are responsible for a wide range of activities from aircraft battle damage repair, maintenance of landing gear, overhaul of electronics, military construction, and software.² Activities and personnel that involve software engineering are managed under the same organizational and personnel structure as hardware repairs (Figure 2.1). The committee heard repeatedly that software is critical to the USAF, drives a majority of USAF platforms’ functionality, and represents a major risk for programs. That said, multiple individuals stated that ALC management’s breadth in scope limits the USAF’s ability to accommodate the specific challenges of managing software engineering personnel, facilities, and business practices. For example, software sustainment personnel are being treated like hardware and maintenance engineers. This is a source of organizational incompatibility that has a major impact on recruitment, retention, and organizational effectiveness.

Across AFMC, support for the software sustainment mission is mixed. The USAF Life Cycle Management Center (AFLCMC) was a key stakeholder and enthusiastic supporter in a number of efforts to modernize software practices. A key example of this support is the Personnel Recovery Command and Control (PRC2)

___________________

² See Hill Air Force Base, 2016, “Ogden Air Logistics Complex,” https://www.hill.af.mil/About-Us/Fact-Sheets/Display/Article/397018/ogden-air-logistics-complex/, accessed June 2, 2020.

program’s transformation from waterfall³ to a successful DevSecOps. Given the rapid changes in software engineering tools and technologies, this evolution would appear to be an area where science and technology (S&T) could help provide insight and facilitate a pathway for software research and development (R&D) to be targeted at sustainment activities. According to management and staff presenters, support from the Air Force Research Laboratory (AFRL) was not effective. Outside the USAF, Office of the Secretary of Defense (OSD) programs such as the Joint Federated Assurance Center (JFAC) that provide technical assistance in software assurance tools are useful, but they lack the critical mass to affect substantial change.

A key relationship in sustainment is between the SWEGs and the System Program Offices (SPOs). The SPOs provide funding support and direction for software sustainment. The quality of this technical and program direction varies across the SPOs. SPO representatives commented on the struggle to find qualified staff that can help manage software projects and programs. For one of the older aircraft programs (F-22), Hill AFB staff commented that there was no one at the SPO with whom to discuss software sustainment. In another program (F-35), the lack of SPO leadership experience in and understanding of software led to low

___________________

³ See M. Lotz, 2018, “Waterfall vs. Agile: Which Is the Right Development Methodology for Your Project?” Segue Technologies, https://www.seguetech.com/waterfall-vs-agile-methodology/, accessed June 2, 2020.

prioritizations of critical software investments over hardware upgrades. In some SPOs, key software management positions are unfilled, and when positions such as the system software architect are filled in other SPOs, the incumbents are often insufficiently trained and lack experience.

LACK OF AN ENTERPRISE STRATEGY

An enterprise strategy manages issues that involve and impact the entire USAF as an institution. While the SWEGs coordinate sharing program workload assignments and addressing problems such as recruiting, the committee observed no evidence of an USAF enterprise-wide approach to understanding and quantifying challenges in software sustainment and executing a coordinated investment strategy for system upgrades, technology upgrades, and so on. As a concrete example, the USAF had not surveyed the technical debt (i.e., obsolesce) for software deployed across its programs. Last, support for human technical capital is delegated to SWEGs without substantial support from USAF management.

The SWEGs participate in a broad range of software projects, from those using modern cloud technologies and DevSecOps programming to those developing code using obsolete computer programming languages, such as Jovial and Ada, to support 25-year-old microprocessors. In addition, the USAF continues to sustain important code for outdated operating systems, such as Microsoft Windows XP and Solaris. A key challenge for the USAF is retaining and training technical staff to develop software for obsolete operating systems, languages, and processors. First, there is a shortage of such expertise. Second, the culture expectations for many of new and recent hires are to continually upgrade skills to ensure job mobility. Developing expertise on obsolete software and hardware is not seen as career enhancing or adding to the preferred mobility within the culture of these staff members. In addition, modern tools used for continuous testing and software assurance are not available for these legacy software systems. Old software languages present

a national security risk because engineers cannot check old code for mistakes or quality assurance as effectively as new code.

USAF, SPOs, and SWEG leadership told the committee that there was a limited quantitative enterprise view or approach for software sustainment. There was no process in place to collect, categorize, and track how software is sustained across the breadth of the USAF programs. Information such as operating system, processor, purpose, size, and frequency of updates could be used to form the basis for quantifying obsolesce, justifying upgrades, and serving as a R&D driver at the highest USAF levels. While such a survey would be a significant undertaking, the need for this survey is demonstrated by the fact that the USAF lacks a clear picture on how enterprise-wide technical debt is accumulated and distributed across USAF programs. Such information could be used to formulate a USAF-wide strategy to understand when and how to upgrade software related to various service-wide risks, including those that are endogenous, such as staff and tools availability, and those that are exogenous, such as cybersecurity.

Over the past several years, the USAF has implemented a handful of pilot projects such as Kessel Run⁴ to drive modern software engineering practices into USAF acquisition and sustainment. While these projects generated promising results, there was a concern that the lessons learned, benefits, and shortcomings of such efforts would not be realized across the USAF. Without this enterprise approach, any “great ideas” implemented by the USAF cannot be put into the context of program life cycle sustainment needs or projections for future needs.

The USAF human capital strategy for recruiting, hiring, retention, and ascension is localized to the SWEGs. For example, Hill AFB preferred to recruit from universities that were local to the Utah and the Intermountain region, as opposed to nationally. In the section “Software Engineering Personnel Not Treated as a Strategic Asset,” later in this chapter, pay disparity for government software engineers is discussed. The impact of low pay for software engineers does not appear to be understood beyond the SWEGs. In addition, the impact of USAF-wide human capital strategy is not isolated to the SWEGs. Recruiting software engineering expertise into the SPO is a challenge, which is typically not seen as a viable career for software engineers. The lack of technical talent at the SPOs is a chronic and profound problem for both acquisition and sustainment, reducing the ability of the USAF to make appropriate trade-offs between software and non-software-related activities. From an enterprise perspective, the trends in workforce shortages, competition for talent from local technology sectors, and increasing software technology complexity across USAF platforms justifies an approach that requires a high-level view of USAF personnel management of software talent.

___________________

⁴ R.S. Cohen, 2019, The Air Force software revolution, Air Force Magazine, September 1, 2019, https://www.airforcemag.com/article/the-air-force-software-revolution/.

SYSTEMS ACQUISITION/SUSTAINMENT GAP

From an acquisition practices perspective, the USAF uses two distinct paradigms for the development and delivery of upgraded software capabilities. Where the upgrade is of significant cost, risk, and complexity—and particularly when it is associated with a major hardware systems upgrade—the USAF will create a separate acquisition program or project to manage and oversee the contracted delivery of upgraded capabilities. For most embedded software systems, where a continuous process of system improvements is expected over the system’s life cycle, the development and delivery of upgraded software capabilities is performed in an organic software sustainment activity. In both cases, the nature of the work to be performed is fundamentally identical—the delivery of an improved software configuration item capable of delivering improved functionality, while addressing evolved requirements and an evolved target-operating environment.

Although the fundamental technical nature of software development for sustainment versus acquisition is essentially the same, the USAF has historically faced challenges managing the transition from the presystems and system acquisition phases into the sustainment phase. These transition challenges are due to a number of factors, but the key theme is that the USAF has traditionally treated a program’s acquisitions and sustainment phases in a discontinuous and disjointed manner, with software being “thrown over the fence” from acquisition into sustainment. Historically, SPO system acquisitions rarely planned for software sustainment, until after Milestone C (MS C) (Figure 2.2). As such, SPOs are not sensitized to the need to understand the impact of shifting development work from contractors responsible for acquisition to a different group involving both government at the SWEGs and contractors. In addition, sustainment activities operate under different rules and regulations that require government involvement, like Core Logistics Capabilities,⁵ also referred to as “Core Requirements,”

___________________

⁵ See U.S. Code 10 USC 2464, “Core Logistics Capabilities,” June 1, 2020, https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title10-section2464&num=0&edition=prelim.

and the “50/50 Rule.”^6,7 All of these factors were compounded by SPOs focusing delivery and leaving sustainment challenges to be addressed after Milestone C.

The impact of this disjointed approach to acquisitions of software programs can be seen as one of the root causes for a number of chronic challenges, exemplified in the programmatic struggles in the F-35 program in developing a sustainment strategy. As per the briefings to the committee, the F-35 Joint Program Office (JPO) began planning for depot activation for software sustainment in March 2018 and completed a new vision in March 2019. The JPO has launched a number of positive new initiatives, such as workload sharing and siting government staff at contractor locations. Many of the new steps are vectored in the right direction, but this type of staff management needed to be established 5-10 years ago.⁸ In addition, F-22 sustainment faces challenges of lack of access to source code that inhibits USAF ability to perform system testing and debugging. This issue is linked directly with the data rights issue, which will be discussed in subsequent sections.

___________________

⁶ See U.S. Code 10 USC 2466, “Limitations on the Performance of Depot-Level Maintenance of Material,” June 1, 2020, https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title10section2466&num=0&edition=prelim.

⁷ See B. Kobren, 2018, Core and the 50-50 rule—a study in contrasts, Defense Acquisition University, https://www.dau.edu/training/career-development/logistics/blog/Core-and-the-50-50-Rule-%E2%80%93-A-Study-in-Contrasts, accessed June 2, 2020.

⁸ At Hill AFB in Ogden, Utah, there was extensive discussion as to the timing on engagement of the SPO and SWEGs on software sustainment.

During the course of SWEG visits, the USAF seemed to be internalizing these problems, developing lessons learned, and then using those lessons to pursue acquisition programs. Specifically, the Ground Based Strategic Deterrent (GBSD) program described efforts to change the ways sustainment is planned. In particular, they have started planning for sustainment during Milestone A (MS A). Other noteworthy efforts include populating the SPO with SWEG expertise, incorporating incentives for contractors to support transition, and securing necessary data rights for the technical baseline for software. This approach is a viable set of tactics to addressing the acquisition-to-sustainment transition.

Software is frequently upgraded and reengineered over the course of decades for a given program. In the committee’s discussions with Navy leadership, their view on software was substantially different from the USAF’s in that DoD software should not be treated in distinct phases of development and sustainment. Their view was that software is different from hardware and needs to be treated more as a product line that is continually evolved and adapted.

CONTRACTOR/GOVERNMENT TEAMS RANGING FROM GOOD TO NEAR-DYSFUNCTIONAL

In general, roles of the government versus the contractor appeared as an important unresolved and occasionally contentious issue. There were multiple USAF programs where government and contractor teams worked hand in hand, such as the MC130-J, complementing each other’s strengths to upgrade system software. In other cases, such as the F-22, government-contractor relations were extremely strained. The issues centered around two major issues: (1) the split between government and contractor work, and (2) ownership of the “technical baseline” for a program, such as the ability to understand and modify important system functions.

Over the course of the past three decades, the pendulum has swung from strong technical government management to weak technical oversight and back again to a more moderate positioning of government and contractor, according to SWEG personnel. Since the 1990s, DoD has been operating under the “50/50 Rule” and

the “Core Law,”^9,10 which direct that at least 50 percent of all depot work must be performed by the government. To a large extent, contractors and government groups appear to have reached equilibrium over the split of work. That said, the strategy regarding the split between government and contractor teams was an ongoing debate. For example, some in industry advocated that government take on more legacy programs, leaving the more advanced work to the contractor. While these rules and its implications for software sustainment do not ultimately inhibit work, the uncertainty on the role of the contractor deincentivizes a large and vital part of the U.S. defense industrial base from collaborating with the government.

The issue of ownership of the technical baseline is a much more contentious and critical issue. For software, the key issue is ownership or use of the data rights for source code, software design tools, technical documentation, interface descriptions, and so on. In some programs, contractors have prevented government access, occasionally citing intellectual property ownership. When negotiating with the contractors, the government is outclassed by industry experience and specific legal expertise. Programs such as the F-22 lack access to important source code and design information necessary for engineering improvement to the SILs, limiting government ability to make enhancements. This vendor lock-in and the lengthy time required for negotiating Engineering Change Proposals (ECPs) can be used during contract writing as justification for primarily using government sustainment capabilities over specific vendors.

For new programs such as the GBSD, the government staff has developed an acquisition strategy to provide incentives for contractors and pathways for government software data rights. If the USAF follows the pathway charted by programs such as GBSD, future programs should have a smoother journey associated with such issues. For legacy programs, the challenge of data rights for legacy software continues to be a challenge that requires deep technical and legal expertise to resolve. Unless SPOs are equipped with personnel who are able to identify and negotiate the technical, contractual, and legal issues, data rights issues will continue to plague programs and limit the government’s ability to rapidly upgrade and enhance software.

___________________

⁹ P.H. Porter, 2016, “Organic Versus Contractor Logistics Support for Depot-Level Repair: Factors That Drive Sub-Optimal Decisions,” Research Paper, Air War College, https://apps.dtic.mil/dtic/tr/fulltext/u2/1037198.pdf, accessed June 2, 2020; see also B. Kobren, 2018, Core and the 50-50 rule—a study in contrasts, Defense Acquisition University, https://www.dau.edu/training/career-development/logistics/blog/Core-and-the-50-50-Rule-%E2%80%93-A-Study-in-Contrasts, accessed June 2, 2020.

¹⁰ See J.A. Drezner, R. Raman, I. Blickstein, J. Ablard, M.A. Bradley, B. Eastwood, M. Falvo, et al., 2006, “Measuring the Statutory and Regulatory Constraints on DoD Acquisition: Research Design for an Empirical Study,” Technical Report, RAND Corporation, https://www.rand.org/content/dam/rand/pubs/technical_reports/2006/RAND_TR347.pdf, accessed June 2, 2020.

In the aggregate, the USAF estimated that the mix of software sustainment work performed by government and that performed by contractors was roughly split evenly. Within given programs, this split varies widely. Given the need to support rapid software development and overall shortages of software development staff, robust government-contractor relationships are key for rapid and efficient software sustainment for the USAF.

DATA RIGHTS OWNERSHIP AND USAGE AGREEMENTS

Software “data rights” refer to the government’s licensing rights to use and modify contractor software. Rights could provide permission to own and modify source code, technical documentation, interface descriptions, design diagrams, and so on. During the acquisition process, prime contractors and their subcontractors are charged with delivering a system for fielding. This lengthy and complex effort involves developing software that is frequently proprietary. Contractors typically view such software technologies as the crown jewels and foundational parts of their organizations and place restrictions on usage and sharing. After transition into sustainment, SWEGs must gain some level of access to software to perform sustainment activities. Data rights that were appropriate during acquisition are now in conflict with government priorities during sustainment. The committee repeatedly observed restrictions on how data rights hampered software sustainment efforts. (Note that this problem and has been described broadly for DoD in reports by the Defense Innovation Board [DIB].¹¹)

Data rights are critical for successful government sustainment, but these terms are often not clearly understood by the parties to the contract or defined in the initial contract for the system. The lack of access to source code can inhibit the government’s integration of new capabilities; identifying root causes for software failure; and, restrict government-led software assurance efforts. Regarding interface descriptions for software, these descriptions are important to identify how software or hardware components are integrated, which is critical to being able to maintain, test, and extend software. This challenge also extends to SILs. In several cases, inter-

___________________

¹¹ See Defense Innovation Board, “Software Acquisition and Practice (SWAP),” https://innovation.defense.gov/software/, accessed June 2, 2020.

face documents are not made available to the organic USAF organizations. This lack of access can include the design tools and program environments that have been developed by the contractors. In a few identified cases, the contractor has provided access to the source code but not the design tools, which severely limited the government’s ability to sustain the software.

There are several reasons why data rights are a recurring problem for programs. First, acquisition programs frequently wait to plan for sustainment until after contracts are awarded. To a large extent, the contractors have already negotiated data rights with the government and are not incentivized to modify their approach. Second, associated with this reason, the government has not clearly thought through or does not have contract clauses prepared to provide the incentives for contractors to share valuable intellectual property (IP). The government staff at the SWEGs frequently complained about the lack of IP legal expertise available to negotiate contract modifications and navigate other associated software licensing issues, such as permissive licensing. The limited IP staff with expertise in software IP is a key barrier to successfully navigating the morass of legal and technical issues. As a result, USAF software teams have minimum or no support in the systems’ formulation of contracts to allow the acquisition of the data rights or use of data rights and infrastructure accesses they need in order to adequately sustain the program. There is extremely little that can be done to improve the situation on legacy programs without agreement from the contractor community. Software engineers are not trained to navigate nuanced legal negotiations and contract acquisition matters, and they should not be the only ones responsible for trying to address this issue for their programs.

As discussed in the previous section, data rights are one critical element of government-contractor collaboration. That said, this is a complex technical-legal issue. For example, the Warner-Robins AFB SWEG group cited the C130-J as one of the most successful programs in terms of SWEG-SPO-contractor cooperation. Unfortunately, there are continuous smaller-scale issues with respect to proprietary marking of information and accessing original system configuration that continue to restrict government’s ability to sustain software.

CYBER SECURITY AND SOFTWARE ASSURANCE

The growing threat of cyber-attack on weapon systems is considered a significant game-changer regarding both systems design and software development. With respect to sustainment, this threat is manifesting in two ways. The first is the requirements for greater cyber security in the development and test environments. The second is the need for new tools to perform analysis of software for flaws and vulnerabilities. The SWEGs consider these areas a rapidly growing concern regarding staffing, facilities, and tools.

Related to cyber security, each SWEG has staff that supports various cybersecurity activities, such as information protection, access control, and accreditation of facilities. The more recent trend is the requirements to support higher levels of classification within Sensitive Compartmented Information Facility (SCIF) and Special Access Program (SAP)¹² bounds. One of the key challenges facing the SWEGs is that today cybersecurity strategies and tools are appropriate for business systems, but not designed for or suitable for weapons systems acquisitions and sustainment. For example, current best practice in cyber security is to retire and replace IT systems that are no longer supported by vendors. Due to the total cost of replacement and impact on mission, the USAF foresees a continuing need to sustain test stands and simulators that are supported by obsolete operating systems such as Windows XP and obsolete hardware platforms such as SGI and VAX computer systems. Last, government staff at the SWEGs lack the tools and experience to adapt cloud, agile, and DevSecOps environments to current USAF security requirements.

Software assurance (SA) represents a rapidly growing requirement for sustainment to provide “a level of confidence that software functions as intended and is free of vulnerabilities.”¹³ SA represents a deeply technical and broad field, which includes threat modeling, architecture analysis, static code analysis, dynamic analysis of software and systems, and detection of vulnerabilities in third-party software (e.g., real-time operating systems). For software development activities that employ modern language and programming environments, there is a rich and evolving set of tools that can be employed. Currently, the SWEGs are using a wide variety of commercial tools, such as

• Fortify Static Code Analysis;
• OWASP Zed Attack Proxy (ZAP);

___________________

¹² See Department of Defense (DoD), “Special Access Program (SAP) Policy,” Directive, last update February 4, 2020, https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodd/520507p.pdf?ver=2020-02-04-142942-827.

¹³ See National Institute of Standards and Technology Computer Security Resource Center, Glossary: “Software Assurance (SwA),” https://csrc.nist.gov/glossary/term/software_assurance, accessed June 2, 2020.

• Microsoft Code Analysis;
• Coverty unite test software;
• VectorCast; and
• Tenable Nessus vulnerability scanning software.

The challenge for modern environments is that while there are a large number of tools available, each has strengths and weakness that require evaluation for suitability for specific weapons platforms. For the many older weapon systems utilizing legacy versions of program languages, such as Ada, Jovial, and assembly, the challenge is much steeper. For such cases, tools such as static analyzers do not exist, requiring software engineers to visually inspect code. Ultimately, the challenge is that no set of tools can guarantee that software is free of exploitable vulnerabilities and that an engineering approach must be pursued to reduce risk.

INFRASTRUCTURE REPRESENTING CHALLENGES FOR USAF SOFTWARE SUSTAINMENT

One of the more surprising challenges facing the SWEGs were shortages and restrictions on facilities, networking, and IT support. The three SWEG facilities support a range of office space for software development and SILs. Given the steady growth in software sustainment staff, workload, and evolving requirements, the challenges are resulting in building or expanding existing office and lab facilities. Additionally, staff raised concerns during the site visit to Warner-Robins AFB regarding the poor quality of the networking and IT infrastructure.

To address office and lab space shortages, the SWEGs are pursuing multiple paths. Over the past several years, they have established dedicated buildings for software sustainment. In addition, they are pursuing expansions and new buildings on base. Due to the slow pace of military construction, Hill and Warner-Robins AFBs are also developing office sites external to their bases. The Hill AFB SWEG group is establishing office and laboratory space in nearby Ogden and Salt Lake City, Utah. The Warner-Robins AFB SWEG group is doing the same at Mercer University in nearby Macon, Georgia. In both cases, the external office and lab facilities are located closer to more populated areas and more collaborative and modern office space environments.

The second challenge, which appeared to be much more problematic, is the poor networking and IT support on the bases where the SWEGs currently operate. SWEG staff experienced delays as high as 45 minutes for sending a single e-mail. Furthermore, the USAF users were blocked from reaching websites required for

research and training, even when those websites were owned and operated by the U.S. government. Because of these issues, many cloud collaboration and development tools were either inaccessible or not usable. This lack of support was universally sited as a key source of frustration for both technical staff and management.

SOFTWARE ENGINEERING PERSONNEL NOT TREATED AS A STRATEGIC ASSET

USAF staffing levels supporting software sustainment has steadily grown over the past 10 years (Figure 2.3). Considering the anticipated growth of onboard and mission support software, the need for staffing is expected to continue to increase. The emerging need for skills in software engineering for enterprise and embedded systems coupled with the ever-increasing demands for the integration of artificial intelligence (AI) and machine learning (ML) will drive a need for multidisciplinary, high-end computer science talent. That said, there are several factors that threaten the USAF’s ability to maintain the required growth in staffing in numbers and qualifications to match future needs.

The most critical factor facing personnel are the pay grades available for technical staff, which impacts recruiting and retention. According to interviews with staff members, the highest grade available for technical, nonmanagement software engineers is a GS-13/10, which equates to about $100,000 to $120,000. For areas with emerging technology sectors like Salt Lake City, an entry-level computer science master’s degree or a bachelor of science degree with a few years of experience can easily achieve this grade of pay. In addition, these pay grades are not at the same level as other services, such as in the Navy, where nonmanagement software engineers can achieve GS-15 and S&T “super grades” paygrades.

As stated previously in this chapter, modern technology matters in the contest to recruit, retain, and inspire software engineering talent. Staffs seeking to enhance their career competitiveness have found little or no long-term value in working with languages such as Ada or Jovial, or environments without modern development and continuous integration tools. Also, talented CS staff members commented that they were lured away from working on legacy platforms, such as the F-16, to work on the more modern F-35. The panel saw many early-career professionals helping management drive the organizational adoption of agile program methods into the organizations, but the frustration with respect to using dated tools and environments was a significant factor to motivating staff to leave or consider leaving.