National Academies Press: OpenBook

Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief (2020)

Chapter: Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

Proceedings of a Workshop


March 2020

IN BRIEF

Software Sustainment and Maintenance of Weapons Systems for the United States Air Force
Proceedings of a Workshop—in Brief

On February 27-March 1, 2019, the National Academies of Sciences, Engineering, and Medicine’s Air Force Studies Board (AFSB) convened a workshop in Washington, D.C., on U.S. Air Force (USAF) software sustainment and maintenance of weapons systems. This workshop was conducted as part of a broader study conducted by the Committee on Air Force Software Sustainment and Maintenance of Weapon Systems that is focused on addressing challenges and identifying opportunities to effectively design and execute sustainment planning across USAF systems and platforms.

During the workshop, committee members heard from a wide range of subject-matter experts in software sustainment from government to academia to examine how the USAF can improve planning, implementation, and management of software sustainment and maintenance of weapons systems. The workshop is part of a larger study to

1. Assess how software that is embedded within weapons systems is currently sustained within the USAF.

2. Identify the unique requirements of software sustainment.

3. Develop and recommend a software sustainment work breakdown structure.

4. Identify the necessary personnel skill sets and core competencies for software sustainment.

The information summarized here reflects the knowledge and opinions of individual workshop participants and should not be viewed as a consensus of the workshop participants, the AFSB, or the National Academies. The purpose of the workshop was for the committee to gather information from field experts on the current state of USAF software sustainment and suggestions on how to improve the organization’s practices of sustaining software in weapons systems.

REVIEW OF PREVIOUS WORK

DEFENSE SCIENCE BOARD SOFTWARE REPORT

Paul Nielsen (Committee Chair) introduced and welcomed the workshop participants and led an opening discussion. Benjamin Smith (Office of the Secretary of the Air Force for Logistics and Product Support [SAF/AQD]) said that his department appreciates the support and outside independent look that this meeting offers. Nielsen began by giving a debrief of the February 2018 Defense Science Board’s


Image

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

(DSB’s) Design and Acquisition of Software for Defense Systems report,1 which examined the importance of software architectures and management approaches to software acquisition. Nielsen, who served as a member of the task force for the DSB report, explained the group’s findings and recommendations for the Department of Defense (DoD).

The DSB report emphasized that while software sustainment is sometimes perceived as routine maintenance, in reality, sustaining software code involves continuous development because every update is a change to the programming. Maintaining a software code’s relevance requires the adding new capabilities, analyzing data gathered on deployment, and combating cybersecurity vulnerabilities. Nielsen said that the USAF should budget a percentage of program funds to continuously improve its software over time.

After the DSB report brief, the committee discussed general perspectives regarding topics mentioned in the report. These topics included code quantity, quality, aging, and documentation. Nielsen argued that the quantity of source lines of ode (SLOC) is a very poor measure for assessing software quality, even though some contractors have obtained progress payments by the amount of SLOC they produce. Software, unlike hardware, does not degrade over time, so SLOC can live within an aircraft system for many years, even decades, without degradation. As such, documenting changes to software packages is essential to its long-term maintenance. Without detailed documentation from previous sustainers, Nielsen explained, we cannot know what changes have been made to the code or what vulnerabilities remain.

GOVERNMENT ACCOUNTABILITY OFFICE ON SOFTWARE SUSTAINMENT

Diana Maurer and Laura Czohara, (Government Accountability Office [GAO]) spoke with the committee about their 2019 report Weapons System Sustainment: DOD Needs to Better Capture and Report Software Sustainment Costs.2 The report focused on how investment in software sustainment is being tracked and reported.

Maurer said that differentiating software sustainment from other maintenance systems initiatives was challenging because software is an integral part of all weapons systems, and thus there is no single, coherent entity that can be isolated and given strict oversight, requirements, or guidelines. There is no sole software sustainment office within DoD, and there is no enterprise-wide software sustainment guidance. When discussing software sustainment, there is even difficulty in deciding what the term “software sustainment” means. The GAO team found that within DoD there are very different views on what software sustainment currently is and what people thought it should be.

The GAO found that the Navy has a different perspective than the Air Force and other services. The Navy views software sustainment as a continuous engineering process instead of a maintenance function. One of the workshop participants noted that the Navy has a devoted engineering ladder. When they train electrical engineers, there is no difference between maintenance and sustainment. They recognize that they are fixing problems whilst concurrently developing new things. Participants suggested that this could explain some of the sustainment workforce management differences between the Navy and other services.

Rebecca Winston (Consultant) warned that the DoD push for agile software development practices will mean that cost benchmarks will no longer be hard and fast and programs will not know the full cost of development and sustainment upfront due to agile practices’ need for flexibility.

____________

1 Defense Science Board, Design and Acquisition of Software for Defense Systems, February 2018, https://www.acq.osd.mil/dsb/reports/2010s/DSB_SWA_Report_FINALdelivered2-21-2018.pdf.

2 Government Accountability Office, Weapons System Sustainment: DOD Needs to Better Capture and Report Software Sustainment Costs, 2019, https://www.gao.gov/products/GAO-19-173.

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

AIR FORCE LEADERSHIP

DISCUSSION WITH STEVE WERT, PEO OF DIGITAL DIRECTORATE

Steve Wert (Program Executive Officer [PEO], Digital Directorate, Hanscom Air Force Base [AFB]) began his talk by mentioning that he wanted to focus on the National Defense Strategy with an agile software development priority. Wert’s group oversees 3,600 people in a broad collection of programs that includes homeland security, missile warning, weather systems, and mission planning for USAF fleets.

Wert described the “Integrated Defense Acquisition, Technology, and Logistics Life Cycle Management System” chart from the Defense Acquisitions University (DAU) as a woefully inadequate approach to software enterprise management. His goal is to convert programs to agile practices across his portfolio, which requires a completely different way of operating. Wert noted that Will Roper (Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics) strongly supported these efforts. Wert noted that in trying to change the system, his team has encountered many of the same workforce issues discussed at the workshop.

Wert stressed that while implementing institutional change is not his job, that level of involvement is essential to the long-term success of these processes. What his team can and has been doing is meeting with key end users with example product descriptions to get feedback on what capabilities the warfighter really needs. At this point, they have 12 applications in production across 27 product teams, with 4 product teams working on the F-35’s Automatic Logistics Information System (ALIS).

Ellen Chou (Director, AFSB) asked Wert how he is addressing the ALIS challenge. He tackled this challenge by enabling the Lockheed Martin teams to work with the Kessel Run teams. They are allowing other groups to modernize the ALIS system, because the USAF doesn’t want sole ownership of ALIS. Wert said that in the week following the workshop, his group will support the first two applications for ALIS, including lightweight utilities that work outside of ALIS to address things that ALIS does not do for end users. Now that his team has access to the source code for ALIS, they can work on improving the program.

Wert said that when a software sustainment team does not own the code they are working on, they are primarily charged with integrating code written by other teams. If the contractor misses a necessary element of the software package, new patches then need to be integrated, which causes program delays and missed milestones.

John Grosh (Lawrence Livermore National Laboratory) asked Wert how he managed the civilian software engineers working on classified programs. Wert responded that they have the civilian software engineers use stock photos to train a machine-learning software algorithm, and then classified data is analyzed by USAF personnel with the proper security clearances. When asked about job satisfaction within these teams, Wert said that they keep a more experienced civilian engineer as part of the team to provide context and mentorship. These teams also consist of product owners, engineers, user center designers, and coders. To mitigate mistakes made across shifts, knowledge sharing is prioritized across and within teams.

In his closing remarks, Wert stated that simply ordering a team 2,000 miles away to “be more agile” does not work. Transforming a team under new processes means restructuring and moving forward together as a team into the agile software development sphere.

____________

3 Defense Acquisitions University, “Integrated Defense Acquisition, Technology, and Logistics Life Cycle Management System,” https://timemilitary.files.wordpress.com/2013/02/acquisition_life_cycle_management.pdf.

4 Lockheed Martin, “Autonomic Logistics Information System (ALIS),” https://www.lockheedmartin.com/en-us/products/autonomic-logistics-information-system-alis.html.

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

To address the function that software sustainment is continuous development, Wert said they are renaming the “software maintenance groups” to “software engineering groups” (SWEGs) in order to better reflect the work they do.

DISCUSSION WITH SUSAN THORNTON, SAF/AQI

Susan Thornton (Director for Information Dominance Programs, Office of the Assistant Secretary of the Air Force for Acquisition [SAF/AQI]) began her presentation by explaining that SAF/AQI handles all of the USAF Command and Control acquisitions. This portfolio includes, but is not limited to, the following: Global Hawk, Distributed Common Ground System (DCGS), defensive cyber, business systems, financials, logistics, enterprise information technology (IT) from an acquisitions standpoint, as well as acting as an executive agent for common data connection for DoD. They are exploring whether software sustainment and maintenance should be dependent on an open architecture system rather than an integrated system to allow for the use of more flexible applications.

Thornton explained that when a program wants to update a weapons system with new software capabilities, there is a “color of money” issue, where mostly operations and maintenance (O&M) or research and development (R&D) money is used but there are no clear guidelines on how to use those categories for software development activities, nor is there a designated software sustainment category.

Thornton mentioned that a key challenge to improving software sustainment, weapons systems, and networks is authority to operate (ATO). If government software factories are given ATO, then they can use certain stacks and tools to certify development processes instead of having to certify the entirety of the underlying software. Thornton said that her office is looking at improving enterprise-wide IT infrastructure as well. In response to a question from Winston, she confirmed that the USAF is starting to think of IT as a foundation of capability rather than a capability in itself. Thornton cautioned that while there is recognition that IT is important to weapons systems development and software sustainment, funding needs to back up support in order to implement real change. SAF/AQI is trying to treat that decision at the enterprise level.

Thornton stressed that there are workforce expertise parameters that need to be addressed as programs are modernized and standardized. These concerns include what proportion of “organic” work (by government personnel without contractor support) versus contractor work a program wants—in other words, How much of the program does the USAF want to handle within the USAF software teams, and how much does the USAF want to purchase from third-party teams or acquire through AI algorithms? Thornton confirmed that the USAF is taking on more organic work than it has typically done in the past.

She also made the point that there is untapped potential in utilizing young airmen or airwomen who have grown up using modern technology. These young-in-field warfighters might not have science, technology, engineering, and mathematics (STEM) degrees, but they could have STEM capabilities through their hobbies and recreational activities, which the USAF is not tracking in a centralized database. Nielsen made the point that sometimes USAF software program teams have good people passionate about their work but who do not have access to the tools they would need in order to be more productive.

DISCUSSION WITH NICK GUERTIN, SOFTWARE ENGINEERING INSTITUTE

Nick Guertin (Senior Software Systems Engineer, Software Engineering Institute [SEI], Carnegie Mellon University) described his experiences across DoD—including but not limited to the USAF–that have provided a broad perspective to apply to the challenge of software sustainment. He is looking at Development, Security, and Operations (DevSecOps) for system processes of scale. The USAF is already analyzing hundreds of terabytes of data every week; Guertin wants the USAF to be able to continuously change to new challenges (see Figure 1).

Guertin admitted that the mechanisms by which we develop and sustain software are changing at a rapid rate and keeping pace is critical. Potential adversaries are also adaptive to new processes, and it is

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Image
FIGURE 1 Department of Defense software complexity and growth: explosive growth of source lines of code (SLOC) in avionics software. The SLOC for aircraft programs has increased substantially over time within the U.S. Air Force as newer and more complicated aircraft have been developed. SOURCE: Nick Guertin, Software Engineering Institute, “Software Sustainment and the Challenge of Modular Capability Innovation,” presentation to the workshop, February 28, 2019; adapted from Defense Science Board, Design and Acquisition of Software for Defense Systems, February 2018, www.acq.osd.mil/dsb/reports/2010s/DSB_SWA_Report_FINALdelivered2-21-2018.pdf.

nearly impossible to predict the next method or mechanism that will accelerate software development. Guertin said that a modular capability is the root element for flexibility.

The most cyber resistant software system, according to Guertin, is that which has been most recently developed, because it is new and not fully understood by potential adversaries. The software community can be a key partner in ensuring that we keep pace and ahead of potential malware. There are software groups around the country that are updating a secure, certified safe system and changing it weekly; but this isn’t being done within the USAF.

Guertin said that the USAF should do more user engagement to figure out what capabilities will benefit end users the most, and this means speaking with operators using the program in the field. A released software product is not static and continuously developing improved iterations of the product is a key part of sustaining that product. A sustainment team only stops improving something when the program is retired. Moving to a modular system could greatly improve software sustainment. If sustainers are changing a system to a collection of smaller packages, then they need to come in self-reliant pieces.

Software sustainment, according to Guertin, is synonymous with capability evolution. There is an infinite cycle of continuous capability improvements in the life cycle of a program, which is an essential element in agile development processes. This can improve performance, cost, and schedule models. When a hardware-heavy system has historically been put

What does it mean to sustain something that is continuously evolving?

— Nick Guertin

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Image
FIGURE 2 Capability evolution and delivery constantly in motion. SOURCE: Nick Guertin, Software Engineering Institute, “Software Sustainment and the Challenge of Modular Capability Innovation,” presentation to the workshop, February 28, 2019. The top graphic is adapted from Kharnagy’s “DevOps toolchain” image, licensed under CC BY-SA 4.0, https://commons.wikimedia.org/wiki/File:Devops-toolchain.svg. The bottom graphic is adapted from Defense Innovation Board, Software Is Never Done: Refactoring the Acquisition Code for Competitive Advantage, May 3, 2019, https://innovation.defense.gov/software.

into a sustainment phase of the life cycle, the focus has been “just keep the engine running.” This isn’t the case with software sustainment.

Guertin mentioned that the program offices need to make use of the digital environment created to support different operations based on different user needs. These digital environments could include modules, containerizations, and micro-servers. AI could help us determine what the battle space could look like. Each individual program has its own infrastructure from hardware to deployment. Individually they are costly, but as a service, the infrastructure can be streamlined, although this will require greater consensus on how programs should operate in these spheres. Guertin said the USAF should think of sustainment as a continuous engineering process and continue taking steps toward a DevSecOps framework (see Figure 2). The Air Force has traditionally had a very hardware-centric view of how to manage and maintain exquisite performance. Software sustainment and a software development mindset needs to be a part of the whole development community.

When discussing new forms of software development, Guertin said that he struggled at the beginning with too many prototype initiatives. If a prototype is ineffective, then you could get stuck with an ineffective prototype for a while. But good prototypes can be a strategic asset of infrastructure. Thomas Bruno (402nd SWEG, Robins AFB) argued that due to decreased aircraft availability for testing, more development tests are moving into system integration labs (SILs). Through SILs, software teams can test

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

a lot more components so they don’t have to take up aircraft time; it is like having the platform in a lab environment.

SOFTWARE SUSTAINMENT ON THE F-35 PROGRAM

Kate Gill (UK Software Sustainment Lead, F-35 Joint Program Office) discussed her work with software sustainment and the F-35 program. Gill began her discussion by stating that the F-35 program is focused on identifying core workforce competencies for who is tasked with software updates. The F-35 differs from other previous USAF aircraft due to its dramatically increased reliance on software and data processing to operate. Sustainment for the aircraft’s software will mean sustaining the data it gathers in addition to the aircraft’s internal software.

The F-35 program has to deal with international specifications. For instance, there is an environmental component for the cold weather Norway shipments that the Turkey shipments do not have to deal with. Different countries are using the system in different ways. Because the F-35 is a global solution, meetings also have to work around national holidays and government reporting benchmarks for different countries, so Gill said her calendar is complicated by trying to figure out when work can get done and when to host meetings. There are currently 42 configurations of the F-35 aircraft across 18 countries, so there is an issue created each time a new jet runs off the production line.

Despite its differences from other programs, the F-35 program and its sustainment must deal with the same challenges in sustaining and developing code as any other weapons system. Gill said that doesn’t work in the current framework of money discussions. Decisions were made for the F-35 program 27 years ago, and it has just gone from officially ending development and entering the sustainment-only phase of its life cycle. Lockheed Martin was very on-board with keeping engines running off the production line. But the real problem is with sustaining the software component of the program. Sustainers need to balance the circle of development, because the warfighter wants the maximum available capability he or she can get at every moment.

When Winston asked Gill how the program could apply agile software development metrics when it is married tightly to the current hardware construct, she said that the program could put boundaries out there and then manage the risk within those boundaries. A hardware wrapper can have a strong software development component. The difficulty emerges when a development team tries to modify critical components like the mission computer, which is interlinked with everything else working in the system. Gill said there could be a shift toward being more agile for the imbedded systems.

Gill said that the F-35 program is currently functioning normally from a funding standpoint, but there are groups that think that the USAF should dispose of the colors-of-money construct and allow greater agility when engaging Congress for funding. She said that while the USAF needs to be responsible and make the most out of the money that is given, being bound by funding categories is inhibiting the program on a daily basis. A single funding pool could help streamline development and inject greater flexibility for adapting to emerging challenges. Instead, program personnel are bogged down while attempting to move funds from different categories and justify how it fits, even if there is not a specific color of money for the software development activity being performed.

Winston made the point that for software sustainers, “all of the colors of money are set up for hardware, so no one is talking my language.” Heather Penney (Mitchell Institute for Aerospace Studies) stressed that the bureaucratic acquisitions processes need to fundamentally take into account that nothing in hardware exists nowadays without software. The USAF needs to change its outlook and treat everything as software, thereby making hardware the exception. The USAF should apply the agile software program management perspective to the hardware model.

Gill concluded by emphasizing that the USAF needs to incorporate the warfighter perspective in the program, or all the updates will be ineffective. Critical to the success of a program will be improving training and support, data links, cyber, GPS, and the whole system of how those aspects work in tandem with the aircraft.

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

INPUT FROM THE SOFTWARE ENGINEERING GROUPS

WORK BREAKDOWN STRUCTURE

Ryan Bishop (76th SWEG, Tinker AFB) led a discussion on the challenge of a work breakdown structure (WBS) for software sustainment within the USAF. Bishop stated that software sustainment work is typically wrapped up into thousands of hours of work. This is a core reporting perspective of where workload goes into the DoD. Bishop said that this is not a good metric within the larger software community, however. This construct still implies that software hours follow hardware hours, and that software is grafted on top of hardware.

When asked specifically about the WBS currently in use, Bishop said that his team uses the WBS for reporting and deciding where different workloads go. Before WBS, Bishop said that they already had a good idea of where work would go, but they were not good at reporting the core capabilities. They had some capabilities listed as a skill set, such as knowing the C++ programming language, but that might not be a capability in itself. The USAF does not have the ability to categorize software packages into buckets; a spreadsheet will not communicate everything that is going on with a program.

Bishop said that the two most relevant parts of the WBS are in core reporting and making depot recommendations. This helps to provide the minimum technical competence for a program. Bishop admitted that the current construct is not comprehensive, but they are trying to improve the reporting system. The current reporting tries to categorize things into buckets, but those categories do not cover everything. With radar programs, everything does fit into the categories, but that program is designed to tell you where everything goes. New radar programs might not have everything that fits into the current categories. Bishop said that the SWEGs are still good at deciding where workloads should go.

Winston asked if the only reason why we have the WBS is to report it up and if we could put it to a variety of uses. Michael Jennings (76th SWEG, Tinker AFB) confirmed that while in some ways the WBS is limited, it is one way of learning more about how much software is out there in the sustainment community. Jennings said that the WBS is focused on helping SAF/AQD know where the USAF can make better decisions, and there are a lot of decisions being made out there where the WBS has been useful, but the next step is to get a larger USAF view so that leadership can strategically plan where the USAF wants to apply organic systems and when to contract work outside the government. There is a balance between organic and industry work, as well as partnerships between the Navy and USAF working on other systems, so there are opportunities to share work across spectrums. There are a lot of platforms that do not have the most optimum capabilities. Bishop concluded his talk by saying his team is looking forward to making things better.

SOFTWARE ENGINEER PERSPECTIVES ON SOFTWARE SUSTAINMENT

A panel of SWEG personnel discussed workforce structure, management, and improvements that could be made. Brian Sun (76th SWEG, Tinker AFB) was the young-in-field engineer representative and introduced himself as working for almost 6 years on a high-performance computer. His experience has been informative, because school did not teach him how technology is used in the military sense. He said that a sustainer is able to fix a system if he or she knows what is going on, and it is gratifying to meet with end users to solve their problems.

Sun said that his position with the SWEG has been the best way to get his master’s degree as he pursues higher education, because of the tuition assistance the government provides. There are pathways to internships, and Ph.D. opportunities are encouraged. He continued that it is very rewarding to see a product to go from design to deployment, and that kind of work is more worthwhile than money. Sun was inspired by his first boss and the work he was involved with to stay within the government despite its challenges.

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

Derek Crane (517th SWEG, Hill AFB) said that he has been working at Hill AFB for almost 10 years, was an intern like Sun, and that he had an M.S. in computer engineering in imbedded systems. Crane said that for some of the modeling that you have to do for the higher-level domains, the electrical engineers would struggle because they do not have the parsing experience that you get as a software major. Nevertheless, electrical engineers “are brilliant so they can pick things up,” Crane concluded. This is one of the reasons why there is an uptick in electrical engineers being hired for the SWEG software sustainment work. Sun added that he was an electrical engineer that moved into the software community. He said that five people from his undergraduate university are now excelling at Tinker AFB, and as a result, they understand the open mission system engineering of modular systems and put those updates into the system as fast as possible.

Jennings discussed the efforts of the SWEGs to retain and recruit talent. From the time of inception, the SWEGs have grown 69 percent. They have about 29 percent of people in M.S. programs, and some are in Ph.D. programs. Sun said that when interviewing potential new hires, they not only look at if the person’s experience lines up with what the program needs, but also if the program needs line up with the person’s career goals. Sun told the story of a coworker who used to work at Boeing: the coworker specifically said in his interview that he wanted to do coding work, not management work, but on his first day on the job, they put him in a management position anyway. So he left and went to Tinker AFB so he could develop software. When discussing workforce pay grades, Devin Bright (402nd SWEG, Robins AFB) said that government software sustainers are not going to match what the commercial technology community can provide. Perhaps a technology manager will be able to meet salary levels more closely, but as Sun mentioned, there are a significant number of software engineers that want to stay on the development side. Nielsen mentioned that there are some software personnel at the second to highest paygrade band, but then there is nowhere to go from there. Jennings admitted that most employees leave for industry at the 2-year mark, and it is still difficult to retain people even beyond the 10-year mark.

In addition, Jennings said that even with anecdotal evidence of work-life balance being better at the SWEGs than in industry, the USAF needs more career-advancement opportunities for the technology career life path.

CAPITOL HILL PERSECTIVES ON SOFTWARE SUSTAINMENT

Gwenyth Woolwine (Majority Staff, Senate Armed Services Committee) and Jonathan Rayner (Military Legislative Assistant, Office of U.S. Representative Anthony Brown) spoke with the committee to provide congressional perspectives on software sustainment.

Woolwine said that one key issue that the Armed Services Committees have been tackling are based in acquisitions reform at large, but with her weapons systems experience, she said she saw software as driving everything. However, there has historically been more congressional oversight into hardware testing, assessment, and evaluation of cost than with software.

Nielsen said that maybe the USAF should take a software-focused approach, where hardware follows along behind. To this, Rayner responded that perhaps software sustainment changes shouldn’t be the focus, but maybe the USAF should be looking in how to change the hardware production process to be more like the software development model.

Woolwine said that the government needs to have enough expertise to recreate the industry base and to do contractor oversight. Oversight is already in place for hardware development, but finding expertise in oversight positions for software development teams might prove more difficult. Nevertheless, it is still essential to have the talent there and to maintain that backup capability. DoD needs to be able to make educated decisions on what is contracted out and what the government handles organically.

She continued by saying that Silicon Valley has a lot to offer. The government could be more involved by helping the companies the Services might want to contract someday. Rayner said that Pitch Day in New York, NY, run by Dr. Roper was a success in this sphere. Typically, it takes DoD months to

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

complete what Google can accomplish in 1 day, due to lengthy approval processes. Rayner said that it is difficult to conduct large changes within an enterprise in 1 day, but there seem to be steps in the right direction.

While Woolwine said she has visited successful organic capability groups in Texas and Boston, she admitted that there could be smaller-scale organic talent out there that her group hasn’t heard about. It is tricky to instigate change at scale within DoD, but there are changes happening for the better. Woolwine said that the USAF needs to get enough of the good stories in the production line and stand up those who are trying to fight the good fight.

In wrapping up the discussion, Woolwine said that there is a lot of room for software solutions to USAF’s management challenges. Software programs could help with hardware maintenance as well. This is an acquisitions, workforce, and culture problem, not just a technical software problem. A difference in the contract approach to software programs could mean a lot. There are partnerships in place, but Woolwine warned that some partnerships are at risk for falling apart.

DISCUSSION WITH JEFF BOLENG, SENIOR ADVISOR FOR SOFTWARE ACQUISITION

Jeff Boleng (Senior Advisor for Software Acquisition, Office of the Undersecretary of Defense for Acquisition and Sustainment) said that one of his initial goals is to stop people from using the term “software maintenance” in favor of “continuously delivering capability.” Nielsen added that the inaccuracy of the maintenance term is why the SWEGs changed their name to emphasize their engineering work. Being labeled as maintainers was influencing their workforce issues and gave “the impression that these are low educated wrench turners,” Boleng said. In reality, they are modernizers, offering continuous delivery of products that allow the program to maintain its competitiveness.

Boleng said that the USAF is very great at innovating, but does not scale that innovation very well. People are not being promoted and paid to the same scale that they want to be. Sometimes it is not a pay issue, but a responsibility issue. If people are not given more responsibility and given promotions that way, then they can leave out of frustration as well. Chou added that a recent article said that the inability to get security clearances hindered hiring people that programs needed. Boleng said that there was a 17-year-old without a high school diploma who won a hacking competition, but the USAF couldn’t give him a software engineering job. In addition, Winston said that the National Security Agency hired someone who was in the USAF pipeline for 9 months, so this is not a government-wide issue, but particularly a USAF one.

Boleng made the point that Kessel Run proved its success that quality software development and sustainment that is secure and deployable could be created without all of the regulations and hurdles other more traditional programs have to deal with. Nielsen said that Kessel Run has been highlighted in other reports, such as that by the DSB, so it maybe the AFSB committee’s consensus study can highlight what is being done at the USAF sustainment centers.

Nielsen added that classic obstacles for advancing the software sustainment enterprise include the color of money, leasing the right software tools, and getting access to the right Cloud-based services. The USAF could use Kessel Run as a training model for how to improve the larger system. Winston also made the point that Dr. Roper will not be in his current position forever, so when the money is spent and the ideas driver is gone, that is not a sustainable model.

Gill asked Boleng how the government could change congressional language to be more software-focused, and Boleng said that he thinks the appetite for changing in that direction is good. His team is encouraging DoD to change legislation. Options include dropping proposal language, creating a new acquisitions pathway for software, creating a new acquisitions title, and implementing a funding category specifically for software. All of the services within DoD approach software differently, so it is difficult to ask Congress to change their operations to benefit the USAF specifically. Congress doesn’t like the discrepancies of software reporting across the services, as discussed in the GAO presentation,

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

because constituents want to know what the money is being spent on.

Boleng said that instead of an entire new acquisitions pathway, a new set of payment pathways might be good enough. The USAF needs to be better about talking to the Hill and managing expectations for programs. Boleng said that the USAF needs to have access to the SLOC in order to manage the codes, and he suggested the USAF could instigate regulations and benchmarks to protect intellectual property (IP) rights of one company by preventing bringing that information over to another company’s project. Boleng said that the USAF needs to have organic software centers for weapons systems projects included on the program from day one. Bruno said that it has been done with a couple of programs and it has been great for those programs, but it is not consistent practice. The USAF should make this policy. The USAF needs to plan for tests and continuous developments earlier on in the life cycle of a program, and those tests will help the testing process overall by planning them in advance.

Penney mentioned that end user operators really value the capabilities that technology provides, and providing those capabilities to end users faster could make a difference. Bringing software sustainment into the life cycle sooner is a value proposition and could be added to the battle requirements for a program. Acquisitions personnel should understand software requirements, not just the valley of death in contracting and establishing lawyers in between proposals, prototypes, and programs. Software is the only way USAF warfighters get more capabilities.

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×

DISCLAIMER: This Proceedings of a Workshop—in Brief has been prepared by Catherine Puma as a factual summary of what occurred at the meeting. She was assisted by Steven Darbes and Ryan Murphy. The committee’s role was limited to planning the event. The statements made are those of the individual workshop participants and do not necessarily represent the views of all participants, the planning committee, or the National Academies. This Proceedings of a Workshop—in Brief was reviewer in draft form by Rebecca Winston (Winston Strategic Management Consulting), Brendan Godfrey (University of Maryland, College Park), and Michael Schneider (Lawrence Livermore National Laboratory), to ensure that it meets institutional standards for quality and objectivity. The review comments and draft manuscript remain confidential to protect the integrity of the process. All images are courtesy of workshop participants.

PLANNING COMMITTEE: Dr. Paul D. Nielsen (NAE), Chair, Carnegie Mellon Software Engineering Institute; Matthew Alexander, MIT Lincoln Laboratory; Lieutenant General (ret.) Ted Bowlds, USAF, IAI North America; John Grosh, Lawrence Livermore National Laboratory; Kevin Martin, HRL Laboratories, LLC; Heather Penney, Mitchell Institute for Aerospace Studies; Stephen Welby, Institute of Electrical and Electronics Engineers; and Rebecca Winston, Winston Strategic Management Consulting

RAPPORTEUR: Catherine Puma, Research Associate, Air Force Studies Board

STAFF: Ellen Chou, Director, Air Force Studies Board; George Coyle, Senior Program Officer; Ryan Murphy, Program Officer; Adrianna Hargrove, Finance Business Partner; Marguerite Schneider, Administrative Coordinator; Steven Darbes, Research Associate; Catherine Puma, Research Associate

SPONSORS: This workshop was supported by the U.S. Air Force.

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop—In Brief. Washington, D.C.: The National Academies Press. doi: https://doi.org/10.17226/25717.

Division on Engineering and Physical Sciences

Image

Copyright 2020 by the National Academy of Sciences. All rights reserved.

Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 1
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 2
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 3
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 4
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 5
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 6
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 7
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 8
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 9
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 10
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 11
Suggested Citation:"Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/25717.
×
Page 12
Software Sustainment and Maintenance of Weapons Systems for the United States Air Force: Proceedings of a Workshop–in Brief Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

On February 27-March 1, 2019, the National Academies of Sciences, Engineering, and Medicine's Air Force Studies Board convened a workshop in Washington, D.C., on U.S. Air Force (USAF) software sustainment and maintenance of weapons systems. This workshop was conducted as part of a broader study conducted by the Committee on Air Force Software Sustainment and Maintenance of Weapon Systems that is focused on addressing challenges and identifying opportunities to effectively design and execute sustainment planning across USAF systems and platforms. During the workshop, committee members heard from a wide range of subject-matter experts in software sustainment from government to academia to examine how the USAF can improve planning, implementation, and management of software sustainment and maintenance of weapons systems. This publication highlights the discussion and presentations of that event.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!