Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary Recent years have seen a serious erosion in the ability of U.S. forces to field new weapons systems quickly in response to changing threats, as well as a large increase in the cost of these weapons systems. Today the military’s programs for developing weapons systems take two to three times longer to move from pro- gram initiation to system deployment than they did 30 years ago. This slowdown has occurred during a period in which threats have been changing more rapidly than ever and when technology advances and accumulated experience should have been accelerating rather than slowing the development process. Many causes for this trend have been suggested, including the increased complexity of the tasks and the systems involved from both technological and human/organizational perspectives; funding instability; loss of “mission urgency” after the end of the Cold War; bureaucracy, which increases cost and schedule but not value; and the need to satisfy the demands of an increasingly diverse user community. The difficulty of focusing on a specific, homogeneous, post-Cold War threat made problems even worse. Yet although the suggested causal fac- tors have merit, a common view is that better systems engineering (SE) could help shorten the time required for development, making it more like what it was 30 years ago. Simply stated, SE is the translation of a user’s needs into a definition of a system and its architecture through an iterative process that results in an effec- tive system design. SE applies over the entire program life cycle, from concept development to final disposal. Figure S-1 illustrates the Department of Defense (DOD) acquisition life cycle. The Committee on Pre-Milestone A Systems Engineering was tasked by the U.S. Air Force to examine the role that SE can play during the defense acquisi-  

 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING A B C Concept Technology System Development Production & Operations Refinement Development & Demonstration Deployment & Support System Life Cycle Acquisition Process Combat Developer Materiel Developer PM —Total Life Cycle System Manager Air Force Materiel Command Acquisition Framework High Less Ability to Ability to Influence LCC Influence (85% of Cost LCC Decisions Little Ability to (70-75% Made) Influence LCC (90-95% of Cost of Cost Decisions Minimum Ability to Influence LCC (95% of Cost Decisions (10%-15%) Made) Decisions Made) Made) (5%-10%) 28% Life Cycle Cost 72% Life Cycle Cost FIGURE S-1  DOD life cycle acquisition process. Points A, B, and C at the top of the figure represent Milestones A, B, and C. LCC, life cycle cost. SOURCE: Richard Andrews, 2003, An Overview of Acquisition Logistics. Fort Belvoir, Va.: Defense Acquisition Uni- versity. Available at http://www.afcea.org/events/pastevents/documents/Track4Session4- S-1 AMCEmphasisonCustomerFocusedITInitiatives.ppt#364,12,Slide 12. Last accessed on November 20, 2007. tion life cycle in addressing the root causes of program failure, especially during the pre-Milestone A and early phases of a program. Currently, few formal SE processes are applied to Air Force development programs before the Milestone A review. The committee has devoted much time and space in this report to trying to define a minimum set of systems engineering processes. Chapter 4, in particular, is devoted to this effort. The most important of these processes are summarized in the checklist in Box S-1 below in this summary (Box 4-1 in Chapter 4). A few of the things that need to be taken care of before Milestone A and just after it are the following: the consideration of alternative concepts (solutions) up front; the setting of clear, comprehensive key performance parameters (KPPs) and system requirements; and early attention to interfaces and interface complexity, to the concept of operations (CONOPS), and to the system verification approach. It is these early-stage processes that are covered in this report. The importance of   This is a result of the elimination in the 1990s of the development planning function that had existed in the Air Force Systems Command. 

Summary  stable requirements and funding between Milestone B and the achievement of initial operational capability (IOC) is stressed, as are processes including good configuration management and change control. The committee further stresses in the report what it regards as six of the most important process areas in its discus- sion of six “seeds of failure” in Chapter 4. SYSTEMS ENGINEERING AND THE DOD ACQUISITION life cycle The use of formal systems engineering practices throughout the life cycle of an acquisition program is critical to fielding the required system on time and within budget. Across the top of Figure S-1 are the points at which important management decisions are made: Milestones A, B, and C. Concept development and refinement occur before Milestone A, and further technology development, to reduce system design and development (SDD) risk, occurs before Milestone B. Only after Milestone B does a program become an enterprise with dedicated funding. Importantly, Figure S-1 shows that about three-quarters of total system life cycle costs are influenced by decisions made before the end of the concept refinement phase at Milestone A, while about three-quarters of life cycle funds are not actually spent until after Milestone C. This means that although high- q ­ uality SE is necessary during the entire acquisition cycle, the application of SE to decisions made in the pre-Milestone A period is critical to avoiding (or at least minimizing) cost and schedule overruns later in a program. Much of the value of early, high-quality SE will be manifested as success in fulfilling Milestone B requirements. mAIN findings and recommendations The committee’s main findings and recommendations are given below. Finding. Attention to a few critical systems engineering processes and functions particularly during preparation for Milestones A and B is essential to ensuring that Air Force acquisition programs deliver products on time and on budget. Today’s weapons systems provide unprecedented capabilities but also involve complex interfaces with external command, control, and communications systems and rely on a greater volume of software than ever before. Early decisions on the weapons system requirements and capabilities have a disproportionately large impact on program cost and schedule. The committee also recognizes that a lack of flexibility (a result of overly rigid processes or a lack of trust among program participants or stakeholders) can limit the ability of a program manager to change early decisions that warrant changing. The committee found many gaps and inconsistencies in the way that the Air Force manages pre-Milestone A activities. The committee heard from ­presenters 

 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING of some cases for which required documents were completed pro forma and filed away, never to be seen again, or for which required steps were skipped completely. The current practice of initiating programs at Milestone B denies the acquisition review authority the earlier opportunity (at Milestone A) to make judgments about the maturity of the technologies on which the program is based and to decide whether technologies need to be further developed prior to making a Milestone B commitment to system development and demonstration. Recommendation. The Air Force leadership should require that Milestones A and B be treated as critical milestones in every acquisition program and that a checklist such as the “Pre-Milestone A/B Checklist” suggested by the committee (see Box S-1 in this Summary) be used to judge successful completion. A rigorous, standard checklist of systems engineering issues should be addressed by each program through both the pre-Milestone A and pre- M ­ ilestone B phases. The committee’s recommended 20-item checklist is shown in Box S-1. While the committee considers that each item on the checklist is impor- tant, it calls attention to several items that warrant further discussion. Item 2 recognizes that the world changes too fast to be friendly to long development cycles. The committee believes that the Air Force should strive to structure major development programs so that initial deployment is achieved within, say, 3 to 7 years. Thirty years ago, this was a typical accomplishment—for example, nearly 40 years ago, the Apollo program put the first man on the Moon in fewer than 8 years. The development time issue is addressable by applying systems engineer- ing to Items 3, 4, and 13 through 15 before Milestones A and B. The definition of clear KPPs by Milestone A and clear requirements by Milestone B that can remain stable through IOC can be essential to an efficient development phase. It is also important that critical technologies be sufficiently mature prior to starting SDD. The committee observed that although today’s systems are not necessarily more complex internally than those of 30 years ago, their “external complexity” often is greater, because today’s systems are more likely to try to meet many diverse and sometimes contradictory requirements from multiple users. This kind of complexity can often lead to requirements being changed between Milestone B and IOC, and it can lead to relying on immature technology. Item 19 of the checklist stresses the importance of placing experienced, domain-knowledgeable managers in key program positions. The committee has observed that many of the truly extraordinary development programs of the past, such as Apollo, the Manhattan Project, the early imaging satellite programs, the U-2, the fleet ballistic missile system, and nuclear submarines, were managed by relatively small (and often immature) agencies with few established processes and controls. In that environment, dedicated managers driven by urgent missions accomplished feats that often seem incredible today. 

Summary  Box S-1 Pre-Milestone A/B Checklist Concept Development 1. Have at least two alternative concepts to meet the need been evaluated? The purpose of alternatives is to stimulate thinking to find the simplest, fast- est, and cheapest solution. 2. Can an initial capability be achieved within the time that the key program leaders are expected to remain engaged in their current jobs (normally less than 5 years or so after Milestone B)? If this is not possible for a complex major development program, can critical subsystems, or at least a key subset of them, be demonstrated within that time frame? Achieving capabilities or demonstrating critical subsystems while key program leaders remain engaged is important to get the capability into service quickly and cost-effectively and to begin the process of incremental improvements based on operational experience. 3. Will risky new technology have been matured before Milestone B? If not, is there an adequate risk mitigation plan? The development of risky new technology in parallel with a major development program can be costly in terms of both time and money. 4. Have external interface complexities (including dependencies on other programs) been identified and minimized? Is there a plan to mitigate their risks? Complex, ill-defined, external requirements and interfaces can be a major source of requirements instability during the development phase. This can be of particular importance when a system must operate in a system-of-systems environment. Key Performance Parameters and CONOPS 5. At Milestone A, have the KPPs been identified in clear, comprehensive, con- cise terms that are understandable to the users of the system? It is important that KPPs be expressed in terms understandable to all of the stakeholders. Failure to define the system’s KPPs simply and clearly at Mile- stone A is a first step to requirements instability and overruns later. continued 

 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Box S-1 Continued 6. At Milestone B, are the major system-level requirements (including all KPPs) defined sufficiently to provide a stable basis for the development through IOC? Beginning development without a complete list of stable requirements is one of the key “seeds of failure” described in Chapter 4 in this report. It is important to complete requirements trade-offs prior to the development phase. 7. Has a CONOPS been developed showing that the system can be operated to handle the expected throughput and meet response time requirements? It can be costly to discover too late that the system as designed cannot be operated to meet its requirements. Cost and Schedule Scoping 8. Are the major known cost and schedule drivers and risks explicitly identified, and is there a plan to track and reduce uncertainty? Identifying the major cost and schedule risk areas, with particular attention to this checklist and the six seeds of failure—inexperienced leadership, ex- ternal interface complexity, system complexity, incomplete requirements at Milestone B, immature technology, and high reliance on new software—can help focus management on these issues early. 9. Has the cost confidence level been accepted by the stakeholders for the p ­ rogram? It is important that stakeholders understand the degree of risk so that the stakeholders will not disrupt the program as inevitable development program surprises unfold later on. It will generally not be possible by Milestone A or Milestone B to identify all the risk areas that might surface later in a develop- ment program, but a frank, early disclosure of known potentials for risk can help sustain stakeholder support later on. Performance Assessment 10. Is there a sufficient collection of models and an appropriate simulation en- vironment to validate the selected concept and the CONOPS against the KPPs? continued 

Summary  Box S-1 Continued In large, complex programs, the development of models early on can be very important to later management of requirements changes and performance verification. 11. At Milestone B, do the requirements take into account likely future mission growth over the program life cycle? The committee advocates freezing new requirements and new technology insertion after Milestone B but also notes that making provisions in the initial requirements to facilitate later upgrades could have great long-term value. Architecture Development 12. Has the system been partitioned to define segments that can be indepen- dently developed and tested to the greatest degree possible? Effective partitioning of a complex system can greatly reduce its development cost. 13. By Milestone A, is there a plan to have information exchange protocols es- tablished for the whole system and its segments by Milestone B? Such a plan developed early on can greatly reduce interface problems later in the development phase when they would be more difficult and costly to fix. 14. At Milestone B, has the government structured the program plan to ensure that the contractor addresses the decomposition of requirements to hardware and software elements sufficiently early in the development program? The histories of programs with cost and schedule overruns are replete with e­xamples of large software developments that had to be redone because require­ments from the hardware side were assigned or determined late. Risk Assessment 15. Have the key risk drivers (not only the technology drivers) been identified? Identifying and managing risk early can pay large dividends; it is important to focus on the six “seeds of failure” (see item 8 above). continued 

 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Box S-1 Continued Program Implementation Strategy 16. Does the government have access over the life of the program to the talent required to manage the program? Does it have a strategy over the life of the program for using the best people available in the government, the FFRDCs, and the professional service industry? Seasoned management is critical; the government’s job is to find the best! 17. At Milestone A, is there a plan defining how the pre-Milestone B activity will be done, and by whom? Identifying the program and system managers early, identifying the FFRDC or SETA support needed, thinking through the use of competitive system concept contracts—all can have a decisive impact on the government’s ability to select the best concept, to define by Milestone B system requirements that can remain stable through IOC, and to select the best development contrac- tors. 18. Is there a top-level plan for how the total system will be integrated and t ­ested? A well-thought-out strategy for verifying system performance, including op- timum phasing of verification tests throughout the assembly process, and well-thought-out use of analytical models and external simulators can have a large positive impact on ultimate cost, schedule, and performance. 19. At Milestone B, have sufficiently talented and experienced program and sys- tems engineering managers been identified? Have they been empowered to tailor processes and to enforce requirements stability from Milestone B through IOC? Seasoned leaders in these areas are critical to maintaining focus and disci- pline through IOC. 20. Has the government attempted to align the duration of the program manager’s assignment with key deliverables and milestones in the program? A combination of assignment extension and time-certain milestones will help align incentives. NOTE: KPP, key performance parameter; CONOPS, concept of operations; IOC, initial opera- tional capability; FFRDC, federally funded research and development center; SETA, systems engineering and technical assistance. 

Summary  The committee believes that the accumulation of processes and controls over the years—well meant, of course—has stifled domain-based judgment that is necessary for timely success. Formal SE processes should be tailored to the application. But they cannot replace domain expertise. In connection with item 19, the committee recommends that the Air Force place great emphasis on put- ting seasoned, domain-knowledgeable personnel in key positions—particularly the program manager, the chief system engineer, and the person in charge of “requirements”—and then empower them to tailor standardized processes and procedures as they feel is necessary. One key pre-Milestone A task is the analysis of alternatives (AoA), which entails evaluating alternative concepts and comparing them in terms of capabili- ties, costs, risks, and so on. Checklist items 1 through 4, 12, and 13 should be completed before the AoA, while items 5 through 11 and 14 through 20 may be addressed after the AoA. Finding. The creation of a robust systems engineering process is critically d ­ ependent on having experienced systems engineers with adequate knowledge of the domain relevant to a contemplated program. While the systems engineering process is, broadly, reusable, it depends on having domain experts who are aware of what has gone wrong (and right) in the past recognize the potential to repeat the successes under new circumstances and avoid repeating the errors. Ideally, a person or persons with domain knowledge would have had experi- ence working on exactly the same problem, or at least a problem related to the one at hand. If that is not so (and it might not be if the problem has never been addressed before, as was the case for Apollo and nuclear submarines), the term could be taken to refer to academic training in the relevant field of engineering or science. It would also refer to the practice in critical thinking and problem solv- ing that comes with learning to be a systems engineer and then building on that foundation to gain the experiential knowledge and understanding of engineering in the context of an entire system. Systems engineering is enabled by tools that have been developed to assist in the management of systems engineering (not to be confused with the practice of systems engineering). Both industry and Air Force presenters told the committee that there are not enough domain-knowledgeable and experienced systems engineers to support all of the programs that need them. Recommendation. The Air Force should assess its needs for officers and civilians in the systems engineering field and evaluate whether either its internal training programs, which include assignments on Air Force programs that provide mentor- ing by experienced people and hands-on experience in the application of systems engineering principles, or external organizations are able to produce the required quality and quantity of systems engineers and systems engineering skills. Based on this assessment, the Air Force first should determine how and where students 

10 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING should be trained, in what numbers, and at what cost, and then implement a pro- gram that meets its needs. The Air Force needs to attract, develop, reward, and retain systems engineers across the full spectrum of relevant domains, engage them in the early (pre- M ­ ilestone A) phase of new programs (or modification programs), and sustain their participation throughout the life of the programs. One important step in this process would be to create an Air Force occupational code for systems engineer- ing so that engineers’ experience and education can be tracked and managed more effectively. The Air Force should support an internal systems engineering career track that rewards the mentoring of junior systems engineering personnel, pro- vides engineers with broad systems engineering experience, provides appropriate financial compensation to senior systems engineers, and enables an engineering career path into program management and operations. Finding. The government, federally funded research and development centers (FFRDCs), and industry all have important roles to play throughout the acquisi- tion life cycle of modern weapons systems. Since the need for a new or upgraded weapons system is most often first r ­ ecognized by the military user, it is appropriate for the military to codify its requirements and, with support from FFRDCs and independent systems engi- neering and technical assistance (SETA) contractors, to explore materiel and non­materiel solutions (such as doctrinal, organizational, or procedural changes) as well as to assess the potential for new technology to provide enhanced capa- bilities. While it is appropriate and usually desirable to engage development contractors in the pre-Milestone B process using competitive study contracts, the source selection for system development and demonstration should not be made until after the work associated with Milestones A and B is complete. Recommendation. Decisions made prior to Milestone A should be supported by a rigorous systems analysis and systems engineering process involving teams of users, acquirers, and industry representatives. Working together, government and industry can develop and explore solutions using systems engineering methodology to arrive at an optimal systems solution. Finding. The Air Force used to have a development planning organization that applied pre-Milestone A systems engineering processes to a number of successful programs, but that organization was allowed to lapse. The role of the Air Force development planning organization, which was within the Air Force Systems Command, was to provide standard evaluation tools and perform pre-Milestone A systems engineering functions across acquisition programs. The early 1990s saw an erosion of this front-end planning organization along with its funding as the Air Force Systems Command (now the Air Force Materiel Command) began to play a decreasing role in program execution. In 

Summary 11 the opinion of several speakers who met with the committee, one main reason for the erosion of funding was a lack of congressional support for the planning function. Recommendation. A development planning function should be established in the military departments to coordinate the concept development and refinement phase of all acquisition programs to ensure that the capabilities required by the country as a whole are considered and that unifying strategies such as network-centric operations and interoperability are addressed. The Air Force and the other military services should establish a development planning organization like that which existed in the early 1990s. The roles and functions of the various organizations involved in acquiring major weapons systems need to be clearly defined. The responsibility for execut- ing systems engineering and program management in the pre-Milestone A and B phases should be vested in the military departments that do the actual develop- ment planning functions. This should not be the responsibility of the Office of the Secretary of Defense (OSD) or of the Joint Staff. Instead, those offices need to enable the creation and functioning of military department development planning organizations with policy measures and, where appropriate, resources. The Joint Staff, under the auspices of the Joint Requirements Oversight Council (JROC), may help to define the requirements for major programs in the course of the development planning process, but it should not run the process itself. The existence of “joint” programs or a program such as Missile Defense, which has several related systems being developed by different military services, requires clear guidance from both OSD and the Joint Staff about who is in charge. These programs need to be harmonized and integrated by the responsible integrat- ing agency. However, development planning activities should still take place in the military departments where the expertise resides. Consequently, the develop- ment planning should be managed by that agency. While this committee cannot predict how Congress will view the revival of a good planning process to support pre-Milestone A program efforts, it is still important for the Air Force and DOD to make the case for the critical importance of this process before Congress and others. A development planning process is important not to start new programs, but rather to ensure that any new program (or a new start of any kind) is initiated with the foundation needed for success. Funding for this planning function needs to be determined by the military ser- vices, including both the acquisition communities and those (the warfighters) who generate the operational requirements. CONCLUDING THOUGHTS Many of the conclusions reached and recommendations made by the commit- tee are similar to those of previous reviews. Most of the past recommendations 

12 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING were never implemented, so one of this committee’s most critical thoughts relates to the importance of implementation. A sampling of key findings and recommen- dations from previous studies follows: • Government Accountability Office (GAO) , — Separate technology development from systems acquisition. Commit to a program only if the technology is sufficiently mature. Set the minimum Technology Readiness Level (TRL). — Stabilize the requirements early. — Employ systems engineering techniques before committing to product development. — Employ evolutionary approaches that pursue incremental increases in capability. — Address shortfalls in science, engineering, and program management staff. • National Defense Industrial Association (NDIA) — Increase SE awareness and recognize SE authority in the program formulation and decision process. — Incentivize career SE positions within the government. • Defense Science Board (DSB) — Overhaul the requirements process. — Stabilize acquisition tours. — Establish a robust SE capability. • Defense Acquisition Performance Assessment (DAPA) — Strategic technology exploitation is a key U.S. advantage. Opportuni- ties need to be identified early. — The U.S. economic and security environments have changed—for example, there are fewer prime contractors, smaller production runs, reduced plant capacity, fewer programs, and unpredictable threats. — The acquisition system must deal with instability of external funding.   Government Accountability Office (GAO), 2003, Defense Acquisitions: Improvements Needed in Space Systems Acquisition Manag­ement Policy, September. Available at http://www.gao.gov/new. items/d031073.pdf. Last accessed April 2, 2007.   GAO, 2005, Space Acquisitions: Stronger Development Practices and Investment Planning Needed to Address Continuing Problems, July. Available at http://www.gao.gov/new.items/d05891t.pdf. Last accessed April 2, 2007.   National Defense Industrial Association (NDIA) Systems Engineering Division, 2003, Task Re- port: Top Five Systems Engineering Issues in Defense Industry, January, Arlington, Va.: NDIA.   Defense Science Board/Air Force Scientific Advisory Board Joint Task Force, 2003, Acquisition of ­National Security Space Programs, May, Washington, D.C.: OUSD (AT&L).   Ronald Kadish, Gerald Abbott, Frank Cappuccio, Richard Hawley, Paul Kern, and Donald K ­ ozlowski, 2006, Defense Acquisition Performance Assessment. Available at http://www.acq.osd. mil/dapaproject/documents/DAPA-Report-web/DAPA-Report-web-feb21.pdf. Last accessed on April 2, 2007. 

Summary 13 — The DOD management model is based on a lack of trust. Quantity of review has replaced quality. There is no clear line of responsibility, authority, or accountability. — Oversight is preferred to accountability. — Oversight is complex, not process- or program-focused (as it should be). — The complexity of the acquisition process increases costs and draws out the schedule. — Incremental improvement applied solely to the “little a” acquisition process requires all processes to be stable—but they are not. The committee notes that successful implementation of these recommenda- tions requires the “zipper concept”—making connections at all levels, from the senior leadership of the Air Force and DOD down to the working levels within key program management offices and supervisory staffs.   The Acquisition—“Big A”—system is often believed to be a simple construct that efficiently integrates three independent processes: requirements, budgeting, and acquisition. “Little a,” on the other hand, refers to the acquisition process that focuses on “how to buy” in an effort to balance cost, schedule, and performance; it does not include requirements and budgeting.