2

Examining How Sustainment Planning Should Be Evaluated Throughout the Development Process

Evaluating the effectiveness of sustainment planning during development must be an iterative and increasingly precise effort as a program’s design matures. It also must take into account any ongoing or planned changes in the overall sustainment enterprise that will be occurring during the life cycle of the new weapon system. Sustainment planning and execution for a U.S. Air Force (USAF) weapon system is intended to provide effective support outcomes such as availability, reliability, supportability, and operating and support costs, while making efficient use of public resources. To achieve these two outcomes, the system should be designed with both reliability and affordability as integral parts of the design process, since they cannot be economically “designed in” later. In addition, prudent decisions should be made at the appropriate point during the development phase concerning whether a weapons system should introduce new and innovative sustainment modernization approaches that may be adapted more broadly outside of the specific weapon system and/or whether sustainment processes, equipment, and business systems that are either in broader Air Force use or contemplated for use should be required for the new system.

As mentioned in Chapter 1, the most important decisions in sustainment planning should be made early and planning and evaluation of planning must begin in earnest during Material Solution Analysis and mature in specificity and knowledge at each ensuing stage of the acquisition process: technology development, engineering and manufacturing development, and initial operating capability. As the new system reaches full operational capability, the sustainment team evaluates execution

of the sustainment plan and makes adjustments to ensure the system is delivering the promised operational and cost outcomes and supporting overall sustainment enterprise goals, strategies, and plans.

The major tasks within a weapons system program to achieve these outcomes are the following:

1. Develop and implement a comprehensive product support strategy that focuses on and improves reliability and material availability, increases operational availability, reduces operating and support costs, and is fully consistent with and supports Air Force sustainment enterprise modernization plans.
2. Conduct cost analyses to support decisions about the product support strategy, including cost-benefit analyses.
3. Develop and implement appropriate product support arrangements (PSAs).
4. Adjust requirements and resource allocations to support implementation of the product support strategy.
5. Periodically review strategy and plan execution to ensure planned outcomes are being achieved.
6. Prior to a change in the product support strategy, or every 5 years, whichever occurs first, revalidate business case analyses performed for the strategy.¹

Effective evaluation consists of ensuring that these tasks are being performed with the needed professionalism, completeness, and timeliness necessary to support the weapons system and the Air Force’s overall sustainment enterprise. The program’s sustainment team, supervised at the enterprise level by the program manager, but led by the product support manager and with support from requirements, program management, engineering, and sustainment organizations, must from the outset, effectively address the following questions throughout the life cycle:

• When in the program’s life cycle should I take action?
• Based on where I am in the program’s life cycle, what action should I take?
• Once I decide on my course of action, how do I do it and how can I measure the effect of what I do?
• Is what I am doing having the expected outcome; if not, what should I do differently?

The focus of the following findings and recommendations is to help the USAF evaluate:

___________________

¹ Public Law 111-84, Section 108 of the 2010 National Defense Authorization Act, October 28, 2009.

• How is the program doing?
• How do we know?
• Should the program do something else?

Without being able to effectively answer these questions, the sustainment evaluation team will never know if they correctly answered the when, the what, and the how?

Finding 2-1: The Air Force does not systemically collect or evaluate information on the use of best practices or on the effects of life cycle sustainment decisions either for the enterprise as a whole or for individual programs.

The committee found that this is especially true for programs in the development phase, where practices and decisions have the greatest consequences on ensuring program success.² While the evaluation of sustainment planning and process at the program level could be improved, the committee determined that existing policies and guidance directing sustainment planning for programs were generally adequate. This finding is not evident for Air Force sustainment planning and execution at the enterprise level, where a lack of centralized planning and management renders evaluation difficult or immaterial.

U.S. Department of Defense (DoD) and USAF guidance, such as the Product Support Checklists, Product Support Guide, Supply Chain Reference Guides, Business Case Analysis Checklists, Guides for Achieving Reliability, Maintainability, and Availability, among others, are replete with both examples and requirements with various sustainment metrics.³Box 2.1 captures some of the key metrics for tracking sustainment activities.⁴

The committee recognizes the difficulty of capturing traditional sustainment metrics early in a weapons system’s life cycle. However, there are measures, surrogate measures using legacy systems, modeling, testing, and simulation techniques, engineering estimates, and analysis of early fielded system maintenance that should be used to inform Milestone A, B, and C decisions. For example, validating a contractually predicted A_o (uptime/uptime + downtime), which is now a required

___________________

² The study committee met with representatives from the Government Accountability Office to discuss an ongoing study on Department of Defense (DoD) weapon system development phase practices associated with designing for reliability. That study’s preliminary data support the conclusion that reliability is not adequately addressed during development.

³ See note 17 for the list of references and examples.

⁴ See “DoD Guide for Achieving Reliability, Availability, and Maintainability,” August 3, 2005, https://www.dsiac.org/sites/default/files/reference-documents/DoD%20RAM%20Guide%202005%20-%20Modified.pdf, and “DoD Supply Chain Metrics Guide,” March 3, 2016, https://www.acq.osd.mil/log/SCI/.policy_vault.html/Supply_Chain_Metrics_Guide_signed_3Mar2016.pdf.

key performance parameter (KPP), involves three variables. “Uptime” is the mean time between failures (how often the system breaks), “downtime” is the sum of the mean time to repair (how long the system takes to fix), and the average customer wait time (how long it takes to get parts). Consequently, if an A_o for a new aircraft is 85 percent, that implies that the system is either very reliable by design, quickly repairable or has a ready supply of expensive components.

This process requires asking the following series of questions:

• How do the OEM and program manager intend to achieve that A_o?
• Is that A_o being achieved in any legacy platform? If no, what is different about this platform from a design or support perspective that enables this A_o?

• What is the operational and financial risk of missing that A_o and what is the likelihood of not meeting the goal?
• What is the mitigation strategy for meeting those risks?

Simple answers to these questions will inform the sustainment course of action as early as Milestone A. If they are not asked, not answered, or not answered reliably early in the program, then intractable reliability, readiness, and cost issues will challenge the program throughout its life cycle.

Recommendation 2-1: The Air Force should task an appropriate Air Force sustainment and modernization organization to be responsible for ensuring that weapons systems programs have the standardized tools and resources they need to acquire the data necessary to effectively evaluate program sustainment information and to conduct necessary sustainment analysis and decision support during development. This organization should support an independent evaluation of the adequacy of sustainment planning and design efforts during development.

Recommendation 2-2: The Air Force should task an appropriate Air Force sustainment and modernization organization to train program executive officers, program managers, and product support managers to create, understand, and use meaningful sustainment metrics prior to Milestone A.

Specifically, sustainment metrics should

• Link to system-level sustainment metrics and objectives and to overall Air Force sustainment enterprise modernization plans;
• Be appropriate in scope and responsibility;
• Include specific and quantifiable units of measure;
• Include specific acceptable ranges or thresholds;
• Be able to motivate and inspire desired long-term behaviors;
• Be easily understood and accepted;
• Be easy to collect and verify;
• Be readily accessible; and
• Be analyzed to provide timely feedback.⁵

Finding 2-2: The Air Force has provided extensive guidance on the Life Cycle Sustainment Plan (LCSP) and other sustainment documents, but they reflect

___________________

⁵ DoD Product Support Manager Guidebook, Release: 2011, Update: December 2019, https://www.dau.edu/guidebooks/shared%20documents%20html/psm%20guidebook.aspx.

a general checklist mentality approach instead of emphasizing critical thinking and tailoring based on specific program needs.

The committee reviewed one guidance document: The “U. S. Air Force Product Support Managers Toolkit” contains 200 pages of checklists, 10 pages of acronyms, and 18 pages of references.⁶ The result is that Air Force LCSPs are largely template-driven, process-oriented checklists instead of meaningful outcome-focused strategic planning documents. The effect of this approach is that vital program outcomes such as reliability, availability, cost, and maintainability, all with significant operational and budgetary impacts, are frequently addressed with similar focus and priority as “Who is on the Integrated Product Support Team?” The case studies in Box 2.2 illustrate the lack of attention being considered to vital program sustainment metrics.

Recommendation 2-3: The Deputy Assistant Secretary of the Air Force for Logistics and Product Support (SAF/AQD) should revise Life Cycle Sustainment Plan planning guidance to ensure that processes and reporting demonstrate critical thinking, focus on expected outcomes, detail strategies for meeting sustainment key performance parameters and key system attributes, and include approaches for performing evaluations consistent with this guidance.

While all elements in the sustainment enterprise are important, a successful design for wartime capability, from a sustainment perspective, should be rooted in the following four principles:

1. Maximizing how much operational usage the platform delivers;
2. Minimizing how often the platform breaks;
3. Minimizing how long it takes for repairs and length of time awaiting material; and
4. Minimizing how much material it takes to support the system.

Getting these principles right in an affordable optimized balanced design throughout the life cycle should be the focus of the sustainment team and the centerpiece of the LCSP.

Additionally, the following question-focused guidelines may be helpful in shaping meaningful and detailed LCSPs. These guidelines may also form the basis for evaluating the program’s substantive progress at designing and testing for sustainment features and performance.

___________________

⁶ AFLCMC/LG, U.S. Air Force Product Support Tool Kit (PSTK), April 1, 2019.

• How will the original equipment manufacturer (OEM) achieve and maintain KPP and key system attributes (KSA) sustainment metrics over the course of the weapons system’s life cycle?
• Is the OEM’s plan to meet those KPPs and KSAs realistic given what is known about the performance of legacy platforms and the organic and commercial logistics systems that support those platforms?

• What are the operational and cost risks to the USAF if the program team does not meet those metrics? What is the probability of those risks and how should the Air Force mitigate those risks if they occur?
• How does the maintenance and support concepts result in meeting operational KPPs at the lowest cost to the USAF?
• What is the intellectual property strategy that enables the USAF to execute cost effective sustainment at the lowest cost possible?
• What is the program team’s approach toward addressing sustainment as an integral part of the acquisition strategy and the system’s design process?
• When sustainment KPPs and KSAs are at risk, what design tradeoffs can or should be made to defer or eliminate elements of the system design?

Finding 2-3: One of the potentially most important evaluation opportunities for a program’s sustainment plans is the evaluation of competing contractor proposals. However, the committee found that source selections currently do not provide either strong incentives to industry to innovate and design for sustainment or a basis to adequately evaluate sustainment design processes, approaches, and substantive content for their impacts on costs.

Historically sustainment costs have been considered inaccessible and too speculative to serve as a meaningful cost factor during source selection. The committee agrees that this is the case for cost evaluations and comparisons, but finds that design for sustainment comparisons between offerings can reasonably be included as a factor in technical (as opposed to cost) evaluations.

Finding 2-4: Individual programs represent large investments in modernization from an operational performance perspective, but they also serve as vehicles to introduce more innovative and modern sustainment approaches.

If proven successful at the program level, these innovative sustainment approaches may be applied across the entire Air Force as an enterprise-level solution. However, the committee found that current sustainment planning does not adequately take advantage of this opportunity. Based on the documents and presentations that the committee evaluated, the committee found nothing that precludes acquisition professionals from including a separate technical evaluation factor that considers total ownership costs (inclusive of manpower, inherent reliability, infrastructure requirements, fuel, repair concept) in the proposer’s design for sustainment for new platforms and major modernization efforts. This criterion would provide a powerful incentive to industry, especially in a competitive procurement, to design a system that not only delivers combat capability, but also would significantly improve sustainment performance and cost efficiencies. In some instances,

one could go even further and make effective evaluated price adjustments to cost proposals to reflect design features that provide high confidence of reduced sustainment costs relative to a government baseline. An example would be to give a discounted present value effective cost “credit” for reduced fuel consumption that would more than compensate for a more expensive engine in production.

Recommendation 2-4: The Air Force should include the proposer’s design for sustainment with technical justification based on total ownership cost analysis and trade-offs as a technical evaluation factor in all future solicitations for new platforms and major modernization efforts.

Air Force acquisition and contracting professionals may want to consider using evaluated price adjustments for high payoff/low risk sustainment cost reductions. This will incentivize industry to design a system that delivers superior combat capability, as well as significantly improve sustainment performance and cost efficiencies throughout the program’s life cycle.

Finding 2-5: Evaluation for sustainment planning and design during development is commonly performed using a program-defined and program unique approach.

There is no single Air Force enterprise sustainment advocate or architect to guide and implement an integrated and holistic strategy across the breadth of the Air Force’s life cycle management and sustainment activities. Some good examples found at the program level include: Predictive Maintenance and Condition Based Maintenance Plus (CBM+), additive manufacturing, and automation. Pursuing these initiatives is commendable. However, the absence of an empowered leader to coherently evaluate and implement innovation in common sustainment maintenance and modernization tasks hamper the Air Force’s ability to achieve effective enterprise-level solutions. The end result is that innovative sustainment modernization initiatives remain confined at the program level unless a willing program manager with flexible resources is able to promote and implement the solution across other programs. This approach confronts poor start-up economies of scale and the tight resource constraints that all individual program managers face.

The establishment of the Rapid Sustainment Office in 2018 marks a positive step toward addressing the absence of a standardized and centralized process to rapidly implement new technologies at scale across the Air Force. However, its 2-year test program model to demonstrate a positive return on investment may hinder its longer-term ability to effectively address sustainment of aging fleets while driving down costs and delivering faster solutions to the field.

Recommendations 2-5: The Air Force should develop a stronger Air Force sustainment enterprise function, including designating a sustainment and modernization functional leader.

The functional leaders should possess the following authorities:

1. Evaluate and designate common sustainment-maintenance tasks at the enterprise level;
2. Prioritize implementation of sustainment efforts like CBM+;
3. Evaluate the costs and benefits of alternative scalability options for organic or contractor support;
4. Advocate for necessary resources (budget); and
5. Engage as a functional lead and integrator across programs from conceptualization to modernization stages.

This leader should identify enterprise level evaluation metrics that empower horizontal analysis and direct input from logistics, sustainment, field, depot, and program offices aligned with the sustainment maturity matrix as captured in Figure 2.1.