National Academies Press: OpenBook
« Previous: 1 Introduction and Overview
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 26
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 27
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 28
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 29
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 30
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 31
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 32
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 33
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 34
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 35
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 36
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 37
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 38
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 39
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 40
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 41
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 42
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 43
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 44
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 45
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 46
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 47
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 48
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 49
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 50
Suggested Citation:"2 Relationship Between Systems Engineering and Program Outcome." National Research Council. 2008. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition. Washington, DC: The National Academies Press. doi: 10.17226/12065.
×
Page 51

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

2 Relationship Between Systems Engineering and Program Outcome introduction As discussed in Chapter 1, this committee places great importance on sys- tems engineering (SE) processes and their proper application by domain experts throughout the entire acquisition cycle, but particularly in the earliest stages of programs, that is, pre-Milestone A. This chapter discusses the development h ­ istory of a variety of past and ongoing programs, emphasizing the role of SE during pre-Milestone A and Milestone A-to-Milestone B time frames and deriv- ing key lessons from the program outcomes. The committee observed that programs which were successful in construct- ing a sound requirements baseline and financial/acquisition plan through the pre-Milestone A and Milestone A-to-Milestone B phase, using sound systems engineering processes, could succeed or fail. Those that failed were traced to poor post-Milestone B actions. However, those that had successful pre-­Milestone A and Milestone A-to-Milestone B phases were the only programs that succeeded. The committee observed that there were no successful programs that entered into Milestone B without the sound fundamentals provided by the rigor and discipline of the systems engineering and financial/programmatic planning afforded by the pre-Milestone A and Milestone A-to-Milestone B processes. In this context, pro- grams that succeeded were those that delivered their products within a reasonable margin of the original cost and schedule baseline. Programs that failed, in the committee’s view, may have delivered successful products but were well outside the reasonable expectations of the original program and were only successful in delivering products after the addition of substantial unplanned funding and a substantial extension of the original schedule. 26

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 27 While the committee attempted to address the Milestone A and B issues in the case studies, it found very little information because of poor formal documen- tation regarding what happened in the Milestone A and B phases. Much of what is reported comes from the collective knowledge of committee members who were familiar with the programs. The point is that there is generally insufficient docu- mented work done pre-Milestone A and B. The committee believed it important to include cases such as the C-5A to illustrate that good Milestone A and B work is not sufficient. It does not guarantee successful acquisition if poor decisions are made in source selection or during system design and development (SDD). While it is not possible in most instances to draw from the cases a clear causal relationship to the individual findings and recommendations of this report, the committee found the case studies to be of value and expects that they will also provide useful information for the reader. The important steps in the acquisition process are depicted as a continuous “thread” in Figure 2-1. Each segment of the thread is associated with a specific SE process. As Department of Defense (DOD) acquisition pulls the entire sys- tems engineering thread, the thread can break at many different points for many different reasons. Thread breakage from perturbations in the SE processes tends to result in cost and schedule overruns and performance degradation. The study schedule did not permit nor were the resources available to enable the committee to conduct in-depth, long-term studies of Air Force acquisition programs using a formal, rigorous, structured case study methodology. Instead, the committee had to use and to draw from published program data and formal Figure 2-1  Systems engineering thread from defense strategy to system operation. IOC, initial operational capability. SOURCE: Contributed by committee member John Griffin.

28 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING work already done by others (such as the Air Force Institute of Technology’s [AFIT’s] published systems engineering case studies), the presentations made to the committee on a sample of Air Force programs, and the knowledge and judg- ment of its members developed through long experience with such programs. The committee examined the formal systems engineering case studies that had been completed by AFIT’s Center for Systems Engineering. The AFIT case studies were based on the Friedman-Sage systems engineering case study meth- odology.  Each of the AFIT studies addressed the full program life cycle, includ- ing the early phases of a program (they were not specifically or strongly focused on pre-Milestone A, however). For other programs the committee examined, the committee could not con- duct analyses equivalent to the AFIT case studies. As the sponsor and committee anticipated, the committee did not find a wealth of data or proven models that could be used to strongly link a program’s pre-Milestone A systems engineering effort with later program outcomes. Still, these programs yielded lessons learned that applied to the study charge. The programs summarized in this chapter are these: • Space Based Infrared Systems (SBIRS), • Joint Direct Attack Munition (JDAM), • Future Combat Systems (FCS), • F-16 Fighting Falcon, • Turbine engine development: fighter jet engine, • Military Satellite Communications (MILSATCOM), • C-5A, and • B-2 Stealth Bomber. These examples were chosen to illustrate the diversity of programs to which SE is applied: for example, from relatively simple systems such as JDAM to complex systems of systems such as FCS and MILSATCOM. For brevity, program histories are summarized here; more detailed histories of the programs summarized by the committee can be found in the references provided in footnotes in this report. Other programs also considered in the committee’s deliberations are mentioned in this chapter and, while not sum- marized herein, can be accessed through the published documents referenced in this report.   Air Force Center for Systems Engineering Case Studies. Available at the website of the Air Force Institute of Technology at http://www.afit.edu/cse/cases.cfm. Last accessed on November 20, 2007.   See G. Friedman and A.P. Sage, 2005, “Case Studies of Systems Engineering and Management in Systems Acquisition,” Systems Engineering 7(1):84-97. The “Friedman-Sage” (F-S) SE case study methodology was used in the generation of the AFIT case studies; however, the AFIT case studies went well beyond the F-S methodology in the interview process and so on.

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 29 Program synopses This section highlights the application (or lack thereof) of SE to past and ongoing acquisition programs and their performance during various phases of their life cycles. Space Based Infrared Systems (SBIRS) Program SBIRS is designed to provide tracking and targeting capabilities for missile warning, missile defense, and technical intelligence. SBIRS is a successor to the highly successful Defense Support Program (DSP) satellite system, which performs the missions of early warning of strategic ballistic-missile launches, detection and reporting of tactical missile launches in theaters of interest, and, as secondary missions, detection of space launches and nuclear detonations. The histories of the SBIRS Program and DSP present a stark picture of how procurement time lines and costs have ballooned over the past 40 years. The contractor for the initial DSP development was selected in 1965, and a success- ful initial operational capability (IOC) was achieved just 5 years later. Since that time, DSP has been improved many times and remains a valuable part of U.S. space assets to this day. The successor to this program, now called SBIRS, was awarded for full-scale development in 1996. Since that time, numerous schedule slips and cost overruns have been announced, and the first launch is now sched- uled for 2009. During the SBIRS briefing to the committee, the presenter said that the program’s minimum goal is to produce a first flight article that is “at least somewhat better” than DSP. The SBIRS program has received much attention due to program execution issues. The program experienced Nunn-McCurdy breaches in 2002 and 2005, at which times it had to be recertified. The program had two additional Nunn- McCurdy breaches of 15 percent cost increase, but did not have to be recertified. The SBIRS program has completed Increment 1 and the delivery of its first two hosted sensors and is currently working to deliver its first geosynchronous Earth orbit (GEO) satellite; however, this milestone is projected to be reached extremely late relative to original program plans and will have considerably exceeded original cost projections. Systems Engineering Lessons The committee believes that the primary factors in this situation are as follows:   Col. Randall Weidenheimer, 2007, “Space Based Infrared Systems Wing,” Space and Missile Systems Center, USAF, presentation to the committee, January 31, 2007.   Nunn-McCurdy breach occurs when the program acquisition unit cost (PAUC) or average pro- A curement unit cost (APUC) exceeds the baseline value by 25 percent or more.

30 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING • The internal complexity of the SBIRS spacecraft, including complex focal plane arrays and the need for concomitant signal processing algorithms, is much more complex than that of the DSP spacecraft. • The external user interfaces for SBIRS are far more complex than those for the earlier DSP owing to requirements of the missile defense com- munity, tactical users, and the legacy early-warning role. • There was a high degree of requirements instability after Milestone B (or its equivalent), driven by shifting missile defense strategies and tactical requirements changes. • Full-scale development was undertaken while the sensor technology i ­ nitially planned was not mature. • There was a high reliance on new flight software. • There was inadequate oversight and accountability on the part of both the Air Force and the contractor. Each of these systems engineering issues was a significant contributor to the SBIRS Program’s failure to execute. However, since the issues of technology readiness, acquisition process, and requirements stability have been dealt with thoroughly in prior reviews of the SBIRS Program, the committee focuses here on the software issues, which have been overlooked elsewhere. The SBIRS Program represented a several-fold increase in the scope and scale of flight software relative to earlier Air Force programs and a manyfold increase over legacy systems in the SBIRS mission area. However, software systems engineering received insufficient emphasis and was not effective in early program formulation. Software and mission domain experience, in both the government and contractor teams, was inadequate to prevent or mitigate the impacts of inappropriate flowdown of requirements to the software, overly optimistic projections of software production rates, and inappropriate phasing of software development and testing with the system-level integration and test program. In fact, flight software still poses a risk to the execution of the remain- ing work in SBIRS. This risk is less related to how many lines of code can be written and tested in a given period of time than it is about whether or not the software system engineering was sufficient to produce the robust and resilient flight software system that will be needed over the current and succeeding phases of this complex program. Several other programs have recently experienced program execution issues late in the development phase as a result of failures to employ robust software systems engineering principles in the pre-Milestone A phase. These program execution issues are frequently not attributable to pure software causes; instead, they tend to occur (perhaps more insidiously) in programs with complex hardware/software interfaces or dependencies (such as with SBIRS). Another way to characterize the problem is that there was insuf- ficient coupling between the disciplines of systems engineering and software

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 31 engineering early in the program.  As a result of these problems, lessons are being learned, progress is being made, and improvements in practices and methodologies are being employed in some current pre-Milestone A programs (discussed below). Traditionally, there has been fairly loose contact between the disciplines of systems engineering and software engineering, both in academia and in practice. Typically, software engineering has contributed more to the formation of the discipline of systems engineering than the reverse, primarily because systems engineering as an academic discipline is a younger field. However, there is a notable trend lately of these two fields coming together, at least in academia. For example, the University of Southern California’s Systems Engineering Center recently merged into its Center for Software Engineering to form the Center for Systems and Software Engineering. These trends in academia are beginning to carry over into practice, leading to better up-front treatment of software systems engineering in the pre-Milestone A phase. A decade ago, when programs such as SBIRS were in their formative phases, program managers and chief engineers might have readily ignored the question of the feasibility of accomplishing stressing system requirements using software. Indeed, it had been common for software to be the fix-all for any design requirement that was deemed “too hard” for the hardware design—“We’ll just do it in software.” Today, the hardware/software trade space is more typically dealt with using the same level of rigor as is used in other system design trades. This increased contact between the disciplines of systems engineering and software engineering will allow early input into the requirements allocation process and trade space analysis by informed software specialists, thereby avoiding the later program execution pitfalls that result from having expected all of the hard prob- lems to be handled by the software. Specifically, the lessons learned from the SBIRS Program are being taken to heart in several more recent programs. For example, in the earliest phases of the Space Radar Program, the system program director chose to create a software division within the government program office, along with divisions focusing on the space vehicle, ground system, and systems engineering. With this organiza- tion, the program director sought to ensure that all pertinent systems engineering trade analyses and programmatic decisions would be vetted through all compet- ing needs of the program, and that the final decision options would be presented with a full characterization of the benefits and impacts associated with each discipline area, including software. Surely, the Space Radar Program will face its share of technical and programmatic challenges, but it will be informative to see to what degree potential software systems issues have been avoided or their impacts mitigated by this organizational approach to software systems engineer- ing in the formative phase of the program.   Robert N. Charette, 2005, “Why Software Fails,” IEEE Spectrum 42(9):42-49.

32 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING The committee notes here that one of the items on its systems engineering checklist (see Chapter 4) is to assess the methodology that the program has cho- sen to integrate systems engineering and software systems engineering. SBIRS and DOD Acquisition Reform The SBIRS acquisition was undertaken by the Air Force during a period in which “acquisition reform” was being strongly pursued. One of the premises of acquisition reform was to reduce life cycle cost through reduced government program office size and responsibility. The DOD began a program to reduce the size of the acquisition workforce over a period of several years. The Congress participated with direction in the Fiscal Year (FY) 1996, FY 1997, FY 1998, and FY 1999 Defense Authorization Acts to reduce the acquisition workforce.  A consequence of reducing the number of government acquisition personnel was the outsourcing of more acquisition work to industry. The SBIRS Program elected to delegate total system performance responsibility (TSPR) to the prime contractor. Assignment of TSPR to the prime contractor for systems of this scope and complexity is inappropriate if it leaves the government in a position of seek- ing insight into program execution and decisions, rather than assuring account- ability through active oversight. While there was nothing in the definition of TSPR that required no government involvement or that specified no government accountability, the implementation of TSPR on SBIRS was taken to an extreme.  A symptom of the flawed acquisition approach was the abrogation of government accountability for many of the pre-Milestone A systems engineering processes and products. Some acquisition reform initiatives (many in the Air Force directed under the heading of “Lightning Bolts”) were sent to the field for acquisition programs without a complete understanding of their consequences. While some seemed sensible in theory, many were applied inconsistently, resulting in unintended consequences. The committee notes that it would be a repeat of past failures to invent a series of SE reforms and mandate them in a “one size fits all” fashion without assessment and tailoring to the situation. Many of these same approaches worked fine in other programs (see the following case of JDAM). These points are discussed further in the systems engineering checklist presented in Chapter 4.   See the Chapter 3 section entitled “Congressional Actions to Cut DOD Acquisition Workforce” for additional discussion of these acts.   The issue here is the word “total.” The government program office cannot give up total responsibil- ity for cost and performance.   “Lightning Bolts” is the title given to a series of acquisition reform initiatives launched in 1994 by then Principal Deputy Assistant Secretary for Acquisition and Management for the Air Force Darleen A. Druyun. The initiatives, which included establishing a centralized acquisition support team to scrub all proposals over $10 million, developing a new System Program Office model, and reducing the number of military specifications and standards, were intended to make the acquisition and sustain- ment processes for the Air Force better, faster, and cheaper.

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 33 It would be a gross oversimplification to attribute all SBIRS program execu- tion issues solely to acquisition reform. Likewise, the program facts do not sup- port faulting systems engineering for all SBIRS execution issues. Nevertheless, poor SE did contribute to program troubles encountered thus far and might yet be a major contributor to execution issues with the remaining effort, particularly in the area of software systems engineering. The SBIRS case provides impetus to assess the level and quality of the integration of systems engineering and software systems engineering in ongoing programs. Joint Direct Attack Munition (JDAM) Program Description JDAM was initiated in late FY 1991 and had its roots in operation Desert Storm. It was during that conflict that military leaders realized the need for all- weather, extremely accurate bombs capable of being dropped from a number of aircraft platforms. The military arsenals were filled with hundreds of thousands of “dumb” gravity bombs. The military wanted to turn these unaided bombs into “smart” bombs using a strap-on kit. The kit would use Global Positioning System (GPS) satellite-guided signals and computer technology to deliver the bomb within 13 meters of its target, regardless of environmental conditions such as storms, darkness, and high winds. The JDAM Program operated in the same acquisition reform environment that SBIRS operated in, but the team did not select a TSPR approach, and the team’s perseverance paid off. The JDAM Program is a success by every measure: the JDAM team’s final proposal included an average unit production price (AUPP) between $14,000 and $15,000—down from an original cost target of $40,000 and an original cost estimate of $68,000. The JDAM team reduced its research and development costs from $380 million to $310 million and shortened the develop- ment program length from 46 months to 30 months. The total procurement cycle length was reduced from 15 years to 10 years. Military performance has continued to improve (e.g., product improvements have allowed the JDAM Program to con- tractually tighten performance accuracy from the original 13 meters to 5 meters). By February 2007, more than 150,000 JDAMs had been delivered to U.S. and international customers and had been integrated onto 10 U.S. and 5 international aircraft platforms. JDAM is a truly operationally effective system. There were many other innovative acquisition and systems engineering ini- tiatives employed by both the government (system program office [SPO] and user) and the industry teams.   For more details, see C. Ingols and L. Brem, 1998, Implementing Acquisition Reform: A Case Study on Joint Direct Attack Munitions (JDAM), Washington, D.C.: Defense Systems Management College. Available at http://www.acqnet.gov/comp/seven_steps/library/JDAMsuccess.pdf. Last ac- cessed on September 18, 2007.

34 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Systems Engineering Lessons The success of the JDAM Program was the result of many individual appli- cations of effective systems engineering and program management principles. Primary among these were that (1) the requirements were clearly defined prior to development and remained stable throughout the development phase, (2) the technology used was mature, and (3) the external interface complexity was well managed. Also of importance was the effective teaming between the government and contractors to form accountable Integrated Product Teams (IPTs). The IPT struc- ture within the JDAM Program instilled a motivation to all team members to engage in and contribute to the success of the program. Another key ingredient was the up-front focus on affordability. This was facilitated by contract provisions allowing the contractor’s profitability to increase as a result of cost reductions accrued through innovative technical and management approaches (including the selective use of commercial practices and specifications). Future Combat Systems (FCS) Description The Army’s Future Combat Systems (FCS) is, without question, the largest, most complex program in Army history. It is literally a “system of systems of systems.” Figure 2-2, from an Army briefing on FCS to the National Defense Industries Association, illustrates this complexity. The overarching objective of the program is to develop a lighter, more lethal force that could be deployed far more rapidly than the heavy forces that are a major proportion of the Army force structure today. To achieve the overwhelming lethality that is envisioned, the concept of network-centric warfare is the underlying theme. It is expected that, via an integrated, robust, self-adapting, self-healing network, pertinent situational data will be available to all leaders at all echelons of the Army ground force (i.e., they will have accurate situational awareness). This will support operations well within the enemy’s decision/awareness cycle and achieve increased lethality. These concepts were evolved by the Defense Advanced Research Projects Agency (DARPA) in cooperation with the Army and taken through early concept development. At that stage, the Army “constructed” the FCS Program. The deci- sion was made to take the concepts as defined at that time and, via a competitive bidding process, select a lead systems integrator (LSI) contractor to pull together the doctrinal concepts and codify them into a system of systems architecture. It was further decided to use Other Transactions Authority (OTA) as the program management vehicle, rather than the traditional milestone process. The idea was that early capabilities/requirements analysis leading to the systems of systems architecture and eventually to systems specifications could progress much faster without the layered milestone decision authority process. It was not envisioned

FIGURE 2-2  Conceptual illustration of the Future Combat Systems Program and its complexity as a system of systems. NOTE: SOSCOE, System of Systems Common Operating Environment. SOURCE: D. Emery, D. Bassett, and M. Schnaidt, “The Net-Centric Foxhole: Perspectives from Army Future Combat Systems,” presentation, October 25, 2006. San Diego, Calif.: National Defense Industrial Association 9th Annual Systems Engineering Conference. Available at http://www.dtic.mil/ndia/2006systems/Wednesday/emery.pdf. Last accessed on November 20, 2007. 35

36 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING that less rigor would or should be employed in the early capabilities/requirements determinations nor in the translation of capabilities/requirements to systems architecture and thence to systems specifications. Significant amounts of time, effort, and dollars were spent in platform and sensor technologies, in parallel with trying to define the integrating network. The LSI seemed to recognize this difficulty early on and set about developing an integrating computer hardware/software system entitled System of Systems Common Operating Environment (SOSCOE). Without some capability such as SOSCOE, there would have been little chance that the various platforms could ever be integrated. In parallel with SOSCOE, the network architecture began to be addressed. The network architecture early concept and definition work was con- strained, however, by a mandate from the government that its central core would be the Joint Tactical Radio Systems (JTRS) waveform architecture. While on the surface this did not appear to be a major obstacle, it soon became a problem in that the JTRS program itself was having technical and schedule problems and became a gating item in defining the network operating concepts, architecture, and topology. In summary, the activities that might be associated with normal pre-Milestone A work just did not coalesce. The program, under OTA, entered into full-scale development with many unresolved issues on systems capabilities/requirements and architecture. Costs were rising well above the original estimates, systems integration problems were growing, and the schedule was slipping well beyond the original estimates. Meanwhile, Congress was expressing real skepticism. As a result, the Secretary of the Army ordered that the program be restructured into the DOD-approved milestone process. Systems Engineering Lessons The FCS Program violates many of the precepts that the committee believes are important to success in the conventional system-development process. For example, the prime contract for FCS was let while requirements were very much still being traded off against desired capabilities. The Army has had difficulty stabilizing requirements changes between Milestone B and IOC, perhaps due to insufficient systems engineering in the early phases of program planning. A preliminary observation based on this case concerns the need for rigor in those activities associated with defining capabilities and requirements and overall systems architecture, independent of the formality of DOD milestone definitions, before entry to the SDD phase is authorized. It should be recognized that, as a potential program progresses through its early stages, better insight into technical feasibility and evolving needs will sometimes necessitate changing the requirements. If the changes become substantial, the program should be recycled through concept development before going ahead with full-scale SDD. A firm conclusion requires more analysis. The full suite of essential requirements should

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 37 be developed at Milestone A and be part of the reporting requirements at each of the program’s major reviews. Once SDD is authorized, a program should be under fairly rigorous configuration management, not only for the hardware and software but for the requirements, to assess both creep and stability. Tracking progress against original requirements should be a checklist item for reviews. This may be integrated into the key performance parameter (KPP) process. While the issues identified above would raise serious concerns in a conven- tional acquisition program, the committee notes that FCS is not a conventional development process. It is an LSI-managed procurement in which an LSI con- tractor (distinct from the development contractors) oversees the entire system engineering process. In fact, this process is set up deliberately to disregard some of the elements of good system development that the committee delineates in the SE checklist in Chapter 4. However, the ability to deal with such ­complexity and shifting design requirements is considered a strength of the LSI process. The committee does not know how this approach will turn out, as the program is still very much in the middle of the development process. If it is successful in delivering the desired transformational combat capabilities in a reasonable time and for a reasonable cost, it will serve as a model for future systems of systems procurements and will add new dimensions to the systems engineering art. F-16 Fighting Falcon Description The F-16 has a reputation for success spanning more than three decades. There are many accounts of how this legendary fighter was developed and of the issues that the program faced as it evolved over the years. One of these, an article by Richard P. Hallion entitled “A Troubling Past: Air Force Fighter Acquisition Since 1945,”10 is quoted in part below. Given how suitable the F-15 would ultimately prove to be for both the air superiority and air-to-ground roles, it is somewhat ironic that in 1968 (fearful that the Mach 2+ F-15 would turn out to be just another big, fast sled) Boyd, Spray, and the others began arguing for a highly agile, single-engine, and less- than-Mach 2 “austere” fighter, the so-called F-XX. They were unsuccessful in getting the Air Staff to redirect the F-15 program again—a wise decision on the part of the Air Force. Instead, the climate of thought that they proposed with the F-XX germinated at the end of the summer of 1971 in the so-called lightweight fighter program. The LWF program received a significant boost by a dramatic redirection of defense acquisition in June 1970, when then-president Richard M. Nixon’s “Blue Ribbon Defense Panel” recommended ending so-called total 10  Richard P. Hallion, 1990, “A Troubling Past: Air Force Fighter Acquisition Since 1945,” Airpower Journal 9(4):4-23.

38 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING package procurement and returning to competitive prototyping, something that had been abandoned since the late 1950s.* Ultimately this interest spawned a competitive fly-off between the General D ­ ynamics YF-16 and the Northrop YF-17, and out of this fly-off came both the F-16 and F-18 airplanes. Although ostensibly intended for technology demon­stration, there was little doubt that the “winning” aircraft would have an e ­ xcellent chance for full-scale production. In mid-January 1975, the Air Force declared the YF-16 the winner, awarding a contract for full-scale development. The first F-16A, which was a slightly larger and more refined aircraft than the YF-16 demonstrator, flew in December 1976. The Air Force activated its first F-16 squadron in January 1979, roughly a decade from the time the fighter mafia initially called for its development. Widespread foreign sales followed. (The YF-16/YF-17 competition was a win-win situation for both contestants, for the losing YF-17 was subsequently adopted, in greatly modified form, as the basis for the McDonnell Douglas F/A-18. Mirroring pilot opinion of the F-15 and F‑16, naval aviators generally were enthusiastic over its performance.) ** Unlike the F-15, the F-16 was a true fly-by-wire aircraft, using three com­puters constantly “voting” on each other’s performance to maintain control of what was basically an unstable airplane. The F-16 thus possessed superlative maneuver- ability, really making it a six-and-one-half-generation airplane, demonstrating performance only now being approached by foreign designs such as the Soviet MiG-29, the European fighter aircraft (EFA), Israeli Lavi, French Rafale, and Swedish Gripen. It is worth noting that going beyond the original air superiority intentions of its parents, the Air Force acquired the F-16 as a dual-role air-to- air and air-to-ground fighter-bomber. By acquiring it, the Air Force intended to *  Neufeld, Jacob. “The F-15 Eagle: Origins and Development, 1964–1972.” Air Power History 48, no. 1 (Spring 2001):4–21; M.B. Rothman. “Aerospace Weapon System Acqui- sition Milestones: A Data Base.” Rand Corp.: N-2599-ACQ. October 1987. ** Deborah L. Gable, “Acquisition of the F-16 Fighting Falcon (1972-1980),” Report 87-0900 ­(Maxwell AFB, Ala.: Air Command and Staff College, 1987), is a useful survey, copy in the history files at Headquarters Air Force Systems Command; Rothman, 62-65. Like the F-15, the F-16 proved infinitely more tractable than its century-series forebears. Further, I have benefited from conversations regarding the F-16’s flight control system with three noted test pilots who shepherded the plane from its YF-16 phase into full-scale development and on into operational service: Phillip Oestricher; Col Robert C. Ettinger, USAF, Retired; and Col David Milam, USAF. See also Spangenberg’s VFAX/ACF pro- gram memorandum. All aircraft tend to have quirks and faults of one sort or another. Un- fortunately, the very earliest production models of the F/A-18 have experienced significant problems with structural cracks, and this condition may limit the service lives of some 61 airplanes. See Barbara Amouyal and Robert Holzer, “Structural Flaws Halve Life of Early F/A-18 Hornets,” Defense News, 20 November 1989, 3. Somewhat balancing this is that as a combat airplane, the Hornet has been very impressive to its flight and ground crews from performance, reliability, and maintainability standpoints. The Navy is planning to procure a total of 1,157 Hornets.

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 39 complement the more expensive and capable F-15 carrying a mix of medium- and short-range air-to-air missiles with a cheaper swing-fighter carrying Side- winders that could assist in winning the air battle, and then fight air-land war. It is the F-16’s multimission capabilities that subsequently resulted in orders for 3,000 of this type aircraft, placing it among the most successful of postwar jet fighters. Systems Engineering Lessons The F-16 featured many innovations in the application of engineering and management concepts, but fundamentally the advantages that this aircraft p ­ ossesses reflect the shrewd application of available technology. Planners with extensive domain expertise were able to anticipate future warfighting environ- ments, understand the systems acquisition process, and comprehend the state of technology to meet the needs. A major lesson learned is that the F-16 Program applied sound systems engineering continuously through an evolutionary block upgrade of the weapon system. This has resulted in the evolution of the weapon system from a basic air- to-air fighter to one of the most sophisticated air-to-air and air-to-ground weapon systems in the world today. This example shows that SE is a life cycle effort and should be planned, programmed, and executed as such. The implications of such a life cycle approach are significant and hold key lessons and introduce difficult questions for application for future weapons sys- tems. The F-16s in the first block produced were very basic in their functional capability, providing for the integration of key technologies in the basic plat- form. Follow-on blocks would improve the combat capability in a predictable and stable engineering and management environment. The development time line, therefore, was shorter for the individual blocks and within a management and oversight time frame. This approach reduced the threat of instability in requirements and cost because the expectations were not only measurable but near term. The key enabler here was the ability and discipline to produce the first iteration of the platform with basic functionality—something not normally accepted in today’s process. SE processes must address a key question pre- and post-Milestone A: Can an initial capability be achieved within the time that the key program leaders are expected to remain engaged in their current jobs (normally less than 5 years or so after Milestone B)? If this is not possible for a complex major development program, can critical subsystems, or at least a key subset, be demonstrated within that time frame? The SE process should provide this alternative for decision ­ makers. The F-16 experience suggests that it can lead to very successful outcomes.

40 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Fighter Jet Engine Program Description Aircraft engine systems offer an illuminating case in point to demonstrate the value of systems engineering processes to incorporate sustainability from the earliest phases of a program.11 The engine systems referred to here are designated EG10-1, EG10-2, EG10-5, EG10-9. Collectively, these four engine systems are called the EG10 engine family. The original EG10-1 engine first qualified for use by the USAF in October 1973. Since that time, the contractor has improved on the design of the engine; its latest variant is the EG10-9 engine. In 1977, the Air Force organized the Propulsion System Program Office (SPO) in order to oversee the development of the EG10 fighter engine and its rival engine system that was being developed in parallel by another manufacturer. The Propulsion SPO was responsible for ensuring that the engine systems fulfilled the Air Force’s performance and sustainability needs. This organization of engineers and managers was also responsible for negotiating contracts between the Air Force and the engine manufacturers while assisting the manufacturers in solving problems that occurred during development, such as system integration issues and cost overruns. In many ways, the purpose of the SPO was to ensure that the engine’s development problems did not recur with the new engine system. The majority of SPO personnel were already veterans of previous engine system development projects and were already well acquainted with the many problems that an engine development program faces. This previous experience among team members was one of the most valuable resources in the success of the engine development program. The SPO not only interacted with the original equipment manufacturers (OEMs) but also with the warfighters who would eventually make use of the new engine’s capabilities. The SPO was responsible for surveying the warfight- ers in order to determine what they needed from the new engine in terms of maintainability and performance. Some SPO personnel had previously served with operational fighter squadrons as technicians and were able to bring firsthand knowledge of difficulties faced on the flight line. The SPO maintained a close relationship with the fighter squadrons to ensure that the warfighters’ interests were communicated to the OEMs. 11  This section is based on work done by committee member Wesley L. Harris, Massachusetts Institute of Technology (MIT), on case studies conducted on a series of Air Force fighter jet engine systems. Consistent with the bilateral agreement between the major American aerospace manufac- turer and MIT, the manufacturer is anonymous and the engine systems studied are given the aliases “EG10‑1,” “EG10-2,” and so on, collectively called the EG10 engine family. See “Sustainment Measures for Fighter Jet Engines” by Spencer L. Lewis and Wesley Harris, Society of Automotive Engineers, Inc., available at http://dspace.mit.edu/bitstream/1721.1/723/1/07_12_2001_Sustainment. pdf. Last accessed on June 29, 2007.

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 41 In the 1970s, Air Force Propulsion SPO managers instituted ground tests that more accurately simulated the operation of an engine throughout its lifetime. Designers used information on what throttle settings would be utilized throughout the life of the engine, given various expected mission types (e.g., air-to-ground missions, air-to-air missions, and escort missions). They then transformed a com- posite of these throttle settings into a simulation of the throttle settings at which the engine would be expected to perform. The EG10-5 eliminated many of the problems suffered by the EG10-1 and EG10-2 engines, such as in-flight stall stagnations and problems with the fuel control system. Additionally, because efforts were made to understand better how the engine would be employed in the field, the squadron owners of the engines were more satisfied with how they were able to maintain the system. The Air Force’s pro-sustainment policies did not cease upon completion of the EG10-5 project. The OEMs wished to continue producing engines for the F-15 and F-16 aircraft, and in order to do that, they had to continually improve the sustainability of their engines. Owing to the Air Force’s strong emphasis on sustainability, the two corporations installed a number of new features (such as electronic monitoring systems) making their systems more sustainable. It was in the midst of these innovations that the EG10-9 engine was designed. Systems Engineering Lessons Systems engineering contributed significantly to the improvements in reli- ability and maintainability of the evolving engines in the EG10 family of engines. These contributions may be classified as policy, technology, and process and tool development. Systems engineering thinking was used to develop an effective policy transition, managed by the Propulsion SPO, from a “nonsustainment” ideology to a “performance-with-sustainment” requirement. Technology contri- butions that were based on systems engineering included computer-aided design, modularity, electronic engine controls, and computer-aided logistics. In the area of processes and tools, systems engineering thinking enabled the development and use of IPTs and accelerated mission testing. These tools were captured in the transition from sequential engineering to concurrent engineering in the design and manufacture of jet engines. The history of the fighter engine development programs shows that designing early for sustainment and performance results in more affordable and agile life cycle options, and a greater flexibility for technical upgrades throughout the oper- ational lifetime of a system. Perhaps most importantly, however, it illustrates the value of having an experienced, domain-knowledgeable organization within the government (in this case, the Propulsion SPO) to manage acquisition programs.

42 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Military Satellite Communications (MILSATCOM) Description One of DOD’s most crucial goals is to create an interoperable system of systems aimed toward enhanced command, control, communications, computers, and intelligence. The mission of the MILSATCOM Program is to provide global, space-based communications capabilities supporting DOD and other government agency missions. MILSATCOM is itself a system of systems, providing narrowband, protected, wideband, and network communications capabilities to a wide range of military users. MILSATCOM must interface with a multitude of external stakeholders, some of which control key architectural specifications, or even performance requirements for the system (e.g., information assurance). The first mission area is the narrowband satellite communications (SATCOM) area that includes the UHF Follow-On (UFO) system and Mobile User Objective System (MUOS). Narrowband provides reliable service to mobile users. The wideband mission area includes the Defense Satellite Communica- tions System (DSCS), its follow-on Wideband Gapfiller System (WGS), and Global Broadcast Service (GBS). Wideband provides broadcast service, similar to DirecTV, and high-capacity data pipes. The protected mission area is supported by Milstar, advanced extremely high frequency (AEHF) system, and Polar MILSATCOM. Protected SATCOM provides highly secure and survivable communications. The Transformational Satellite Communications System (TSAT) is a next-generation MILSATCOM system, following WGS and AEHF. It will support both the protected and wide- band mission areas. In addition to the satellite systems, MILSATCOM includes a terminals seg- ment. The different SATCOM terminals communicate with one or more of the satellite systems. Examples include the Milstar Command Post Terminal, Secure Mobile Anti-Jam Reliable Tactical Terminal (SMART-T), Family of Advanced Beyond-Line-of-Sight Terminals (FAB-T), DSCS and GBS Terminals, Ground Multi-band Terminal (GMT), Airborne Integrated Terminal (AIT), Navy Multi- band Terminal (NMT), Warfighter Information Network-Tactical (WIN-T), High Capacity Communications Capability (HC3), and Multi-Band Multi-Mode Radio (MBMMR). Historically, the individual MILSATCOM systems were handled in a piece- meal fashion—each system focused only on what it needed to be individually successful, not considering how it might contribute to the broader MILSATCOM program. Each of the satellite systems, while being headquartered at the Air Force’s Space and Missile Systems Center, had its own user community, its own program director, program office, budget, schedule, and so on. In recent years, a combination of increasing integration of military opera- tions and a dramatic Air Force drawdown of technical and program management

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 43 talent have forced organizational change in the management of MILSATCOM programs. Today, the MILSATCOM Program is being handled as an enterprise under the MILSATCOM Joint Program Office (MJPO). Within the MJPO, each system is assessed not only on its own merits, but also on how it contributes to the various MILSATCOM mission areas (protected, wideband, narrowband SATCOM) and how it can contribute to meeting some or all of the needs of a particular user community (for example, supporting intelligence, surveillance, and reconnaissance data relay). The MJPO has taken several measures to reduce program risks and increase the probability of success in delivering MILSATCOM capabilities. These efforts have focused primarily on the integration impacts in four areas: across the breadth of individual programs, in the transition to operations, in the interactions among programs, and external influences on programs. As a result of interconnections between various MILSATCOM systems, the MJPO has also recognized the interdependencies among programs. The MJPO has started several initiatives to reduce the risks of any program’s adversely impacting others. These initiatives include the stand-up of the MILSATCOM chief engineer and the System of Systems Engineering, Architecture and Integration (SSEA&I) group; the MJPO Configuration Board (MCB); weekly senior-level functional reviews and discus- sions; and the MILSATCOM Integration Master Schedule (IMS). The MJPO cre- ated the MCB to track and manage the integration of interfaces, specifications and standards, and program efforts. This executive-level board, composed of MJPO program managers, is responsible for advising the MCB chair (MJPO director or deputy) on proposed contract change actions, and manages all MJPO configura- tion baselines throughout sustainment and any changes that have a cost impact. Despite the move toward unified program management represented by MJPO, in recent years the development of the MILSATCOM systems has been plagued with delays and cost growth that are a legacy of the earlier, fragmented manage- ment structure. Examples are the AEHF and WGS systems. Following the failure of Milstar Flight 3, AEHF was accelerated to complete worldwide protected SATCOM coverage. This decision forced significant adjust- ments to AEHF Flight 1, including a transition to an operations plan that is less than optimal. These changes resulted in the addition of a requirement that AEHF be backward-compatible with Milstar. Extra attention is being given to the AEHF system development effort to address issues on Milstar backward compatibil- ity, operations transition, cost overrun projections, and sustainment. The AEHF system procurement effort has had to provide a Nunn-McCurdy notification to Congress owing to late delivery of government-furnished products (especially for performing cryptographic functions) and replacement of critical electronic parts, causing a cost breach greater than 15 percent. AEHF had to replan the procure- ment effort, delaying delivery of the first satellite, and is now subject to increased scrutiny within the DOD and in Congress. The WGS system was contracted during the so-called Acquisition Reform era,

44 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING in which the government expected “commercial best practices” to be applied.12 The program viewed the WGS satellites as a mature product requiring little oversight. As it turned out, the large number of program interfaces and emergent program technical issues required both additional program insight and additional funding in critical areas. WGS, with a firm fixed-price (FFP) contract that limits the cost impacts to the government, had problems with fasteners that delayed the delivery of the satellites by 15 months. A combined government and con­tractor team applied systems engineering to identify and prioritize mission-­critical areas. This was an important step, especially in light of the fixed-price nature of the contract. The program encountered design, integration, and manufactur- ing problems due largely to the fact that the program was unable to benefit from continuing synergy on commercial SATCOM production lines. The Wideband Gapfiller System prime contractor had assumed a continued growth in the com- mercial ­SATCOM marketplace in its WGS FFP proposal. When the commercial S ­ ATCOM demand did not continue to grow, and in fact declined precipitously, the WGS prime contractor suffered substantial overruns on its FFP contract. System Engineering Lessons For the MILSATCOM Program to have been helped early by better systems engineering, each of the component elements would have needed to have been conceived in response to requirements from a consistent and coordinated set of users, advocated and funded by the same organization, and acquired, tested, and fielded in a coordinated way by an integrated program office. The cur- rent ­MILSATCOM Program was formed by combining preexisting satellite and t ­ erminal offices long after many important decisions had been made. In both AEHF and WGS, the control and coordination processes that the MJPO has now put in place would have resulted in much earlier identification and resolution of issues and coordinated, joint problem resolution. The AEHF program could have benefited greatly from earlier systems engi- neering that would have yielded a better understanding of the compatibility requirements with the legacy system and, in fact, understanding of the actual con- figuration of the sustained legacy Milstar system. A much more thorough analysis of the ability to accelerate AEHF in the face of immature technology would also have identified potential future problems and work-arounds. A more active man- 12  Several important points are worth noting about the differences between the DOD MILSATCOM system and commercial SATCOM. As previously noted, MILSATCOM is a system of systems. While commercial SATCOM may consist of a family of incrementally improved satellites, these satellites do not and need not respond to the range of stressing missions accomplished by MILSATCOM. Commercial users are by and large less disparate and naturally have a more stable and narrower requirements set. Also, while commercial SATCOM cannot be considered “low-tech,” it does not push the Technology Readiness Levels the way that MILSATCOM does. Finally, as a general rule, commercial SATCOM contracts are fixed-price production contracts, while DOD MILSATCOM contracts have been of the cost-plus type.

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 45 agement and engineering engagement with the National Security Agency (NSA) might have highlighted cryptographic issues sooner and allowed alternatives to be developed. A better understanding of program interfaces and of the critical dependence of a program on Technology Readiness Levels might have led to different con- tracting decisions on the Wideband Gapfiller System. Active government program management and earlier joint systems engineering would have resulted in earlier and cheaper problem resolution. While it is the mission of MJPO to deliver an integrated capability, many of the end users of the capability are not part of the Air Force Space Command (AFSPC), and their issues are not always well represented. The individual pro- gram managers do not and cannot control all aspects of key external requirements. While external users could always be better represented, the MILSATCOM SPO has made great strides in dealing with those users and understanding their issues. The MJPO has implemented a multipronged effort to provide a system- of-systems approach to the technical, business, and acquisition management of MILSATCOM products, and this approach is embodied in a wing-level systems engineering plan that addresses all issues at the system of systems level. Docu- mented processes, carried out at the enterprise (wing) level, ensure that standards exist and are enforced among program (product) elements and also ensure that artifacts and information are generated to support cross-program integration. By employing traditional systems engineering processes at the system of sys- tems level, MJPO is able to reduce surprises and limit unintended consequences of individual program decisions and at the same time to gain an integrated view of gaps and overlaps among the product lines. The implementation and successful execution of such a systems-of-systems approach require leadership, experienced personnel, communication, and the willingness to make difficult trade-off deci- sions that affect individual program optimization. C-5A Program Description The C-5A Program was characterized by an excellent pre-Milestone A and Milestone A-to-Milestone B process.13 The Development Planning organization at Aeronautical Systems Division, Wright-Patterson Air Force Base, conducted operational mission analyses and contracted with industry (Lockheed Martin Cor- poration, The Boeing Company, and McDonnell Douglas Corporation) to conduct supporting conceptual design analyses. The Development Planning organization also conducted parallel conceptual design analyses of potential aircraft designs 13  John M. Griffin, SES (Ret.), undated, C-5A Galaxy Systems Engineering Case, Wright-Patterson Air Force Base, Ohio: Center for ­Systems Engineering at the Air Force Institute of Technology. Avail- able at http://www.afit.edu/cse/cases.cfm. Last accessed on June 18, 2007.

46 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING that could meet the evolving functional baseline. The user was an integral part of the process, as were laboratory scientists and technologists. The result of the entire process was a weapons systems specification that represented the functional baseline and was technologically feasible within the existing state of the art. A SPO was established prior to the conduct of source selection; it was popu- lated with domain experts in all of the functional areas. The SPO cadre included a number of people from the Development Planning Directorate who participated in the early development of the requirements and were knowledgeable regarding the genesis of the program. In response to their perception of what their competitors would bid, ­Lockheed claimed an aggressively low value for its aircraft weight empty in its proposal. During the source selection, the Air Force convinced Lockheed to put the aircraft weight empty into its specification as a performance requirement equivalent to range and payload and other performance parameters. Then the Air Force put a financial penalty into the contract in the event that the contractor’s aircraft weight exceeded the specification value, and Lockheed signed this contract. To compli- cate matters further, the contract type was a firm, fixed-price contract for both development and initial production under the new contract strategy that was being pursued by the Air Force, called Total Package Procurement. As a final constraint to the contract, a complementary financial penalty was included for each month that Lockheed missed the contract first flight date. As the source selection came to a close, the three contractors were debriefed as to their strengths and weaknesses. The Air Force advised Lockheed that its assessment of the proposed aircraft showed the wing area to be deficient by some 400 square feet. Lockheed hastily redesigned the aircraft wing in just 4 days and resubmitted its proposal. The time available to redesign the wing was inadequate to conduct a systems engineering assessment of the new design; most of the technical parameters were updated as a ratio of the wing areas. This included the weight empty, which had been estimated with optimism for the original design and was now even more optimistic. In retrospect, this change contributed to a further disconnect of the cost estimate from the technical baseline. The net effect of all this activity during source selection was to excessively constrain costs, schedule, and performance to such tight and rigorous limits that there was no hope of being able to perform to the requirements of the contract. As the design progressed to preliminary design review (PDR), it was clear to Lockheed that it could not meet the weight requirement. Lockheed proposed to the SPO that the weight requirement be removed from the contract, that the contractor would meet all of the performance requirements remaining in the specification, and that the engine contract would be increased by the $5 million needed to redesign the engine for an increase in engine thrust. The engine con- tractor was more than willing to redesign the engine for increased thrust, since its competitors had already seen the need to increase thrust to compensate for weight growth that was occurring in Boeing’s commercial 747. The Air Force

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 47 System Program Office disapproved the request and, in fact, sent the contractor a cure notice.14 Lockheed agreed to withdraw its offer and then directed its in- house engineering staff of designers to remove more weight from the structure of the aircraft. A competent system engineering assessment would have revealed that the resulting aircraft would be operationally unsuitable, having a limited load factor and reduced fatigue life. This was eventually recognized and accepted, and production was halted after 81 aircraft were completed. Major reviews of the program and the design of the structure were con- ducted under the leadership of the Air Force. The combined government and industry team conducted an extensive systems engineering study of multiple options before selecting a new program approach. The new wing was rede- signed to meet strength and life requirements. This was accomplished using basically the same design approach and adding 14,000 pounds to correct the problems. This new wing was retrofitted to the first 81 aircraft, and it was also included in the next buy of 59 C-5B aircraft. The Air Force spent over $2 bil- lion more than the original budget, and it took 10 years longer to finish the production with fewer aircraft than originally planned. Lockheed petitioned the United States government through the Congress for a $250 million loan to avoid bankruptcy. Systems Engineering Lessons The C-5A case study makes the point that the pre-Milestone A and M ­ ilestone-A-to-Milestone B requirements process was efficient and effective. The C-5A Program should have succeeded based on the strength of the pre- Milestone A and Milestone A-to-Milestone B process. The point of failure on the C-5A Program occurred during source selection and was the result of poor appli- cation of the systems engineering process by both Lockheed and the Air Force during source selection after Milestone B. Because of the actions taken by the program participants, the program was doomed to failure from the first day of the contract. The lesson is that the systems engineering process in the pre-Milestone A and from the Milestone A-to-Milestone B time frame is an absolutely neces- sary function, but it is not sufficient. The process must be executed accurately and completely throughout the continuum of program acquisition. Figure 2-3 shows the systems engineering graphic for the C-5A. It shows the systems engineering process in green for the requirements development phase and in red from contract go-ahead and on, and again turning to green for the retrofit and the C-5B, albeit at the expense of time and money. 14  Before terminating a contract for default, the contracting officer may issue a written notice called a “cure notice.” The notice allows the contractor 10 days to “cure” any defects. If the failure to perform is not cured within 10 days, the contracting officer may issue a notice of termination for default.

48 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING • Solid early studies of trade-offs FA • Mock-ups for roll-on, roll-off A • Stable requirements FNA C-5 FSA • Requirement management Sustainment CD (Contract award & beyond) AOA Prioritization of requirements M/S B OT • Significant structural PDR consequences (re-wing) CDR DT Figure 2-3  The C-5A systems engineering graphic showing an assessment of the applica­ tion of the process. SOURCE: Contributed by committee member John Griffin. B-2 Stealth Bomber Program 2-3 Description As was the case with the C-5A discussed above, the B-2 case demonstrates a sound requirements process before Milestone A and successful systems engi- neering through Milestone B. The B-2 encountered problems just prior to PDR 2 when the design team completed an integrated analysis of the structure, flight controls, and aerodynamics. The design review showed that the aircraft had insufficient control power and structural rigidity to damp structural loads while providing the required maneuvering margins in the presence of turbulence. The integrated design team, led by Northrop Grumman and including the major subcontractors, Boeing and Vought, and the Air Force customer, completed the redesign in 4 months and received approval for the new changes through the company and the Air Force management structure in 3 months. Notwithstand- ing its rapid systems engineering and design response to correct the problem, the subsystem design could not recover from the changes and caused a 6 month slip to the effective completion of critical design review (CDR). In this case, the maturity of the design and confluence of three technical disciplines in a new con- cept led to the realization that a redesign was necessary. The design team quickly developed a new approach, and the management team responded with new plans. However, the subsystems designers and the subcontractors did not have sufficient

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 49 time to recover their previous work and were late to CDR. This was not a failure in requirements, but rather a recovery of the design and a lag in recovery by one segment of the team. Systems Engineering Lessons The B-2 case study illustrates that even when the early systems engineering process is done well, the acquisition process is fraught with peril because of the unknowns and complications that arise in any program. However, from the case studies and from listening to the testimony of the briefers, it is clear that program offices and industry teams staffed with domain experts equipped to handle technical and programmatic difficulties are best suited to respond quickly and effectively to the problems when they arise. Managers who are inexperienced in handling problems such as occurred in this case may be unable to prevent the full collapse of a program. The acquisition process is by nature a complicated and delicate one that can easily be driven to instability. This point is underscored and somewhat amplified by comparing the C-5A partici- pants’ actions and those taken by the B-2 team. The C-5A Program certainly had domain experts, as did the B-2. However, the C-5A Program was complicated by the extreme controversy of the program, the fixed-price Total Package Procure- ment contract approach, the contractor’s aggressive claims, and the Air Force’s reluctance to re-open the contract after it was signed, all of which combined in a complicated dynamic and contributed to the less-than-domain-expert decisions. In the B-2 case, the program proceeded with the agreement of all parties, not- withstanding that the Northrop Grumman program management stated concern that the schedule of CDR was at risk. This risk was eventually realized, but it was not a requirements issue. Rather, it was a technical issue that surfaced as a result of the need for the aircraft redesign to meet the requirements on contract from the start of Milestone B. Figure 2-4 presents the systems engineering graphic for the B-2, showing that it turns red at PDR and then recovers to green at CDR, albeit a year late and nearly $1 billion over the estimate at completion. SHARED FINDINGS AND LESSONS LEARNED AMONG CASES Although each of the case studies summarized above revealed some unique findings, several key findings and lessons learned are shared among the programs. These are summarized here: • There is a need for an appropriate level of SE talent and leadership early in the program, with clear lines of accountability and authority. Senior SE personnel should be experienced in the product(s) domain, with strong skills in architecture development, requirement management, analysis,

50 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING • Solid early studies of trade-offs FA with combined SPO, user, contractor teams FNA FSA • Strong integration of B-2 CD requirements and design AOA Sustainment M/S B • System redesign OT PDR 1 PDR 2 CDR DT FIGURE 2-4  The B-2 systems engineering graphic showing an assessment of the applica- tion of the process. SOURCE: Contributed by committee member John Griffin. modeling and simulation, affordability analysis, and specialty engineering disciplines (e.g., reliability, maintainability, survivability, system security, and technology maturity management). 2-4 • There is a need to establish and nurture a collaborative user/acquirer/ industry team pre-Milestone A to perform system trade-offs and manage overall system complexity. Today, there are often significant disconnects in the hand-offs between users, acquirers, requirements developers, indus- try, and others. Some of the “best practices” include structured collabora- tion among these members. • One must clearly establish a complete and stable set of system-level requirements and products at Milestone A. While requirements creep is a real problem that must be addressed, some degree of requirements flexibility is also necessary as lessons involving feasibility and practical- ity are learned and insights are gained as technology is matured and the development subsequently proceeds. Certainly control is necessary, but not an absolute freeze. Also, planning ahead for most likely change pos- sibilities through architectural choices should be encouraged, but deliber- ately managed, a concept encouraged herein. A typical program execution team has a program manager (PM)-level SE integration team (SEIT), with

RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME 51 responsibility, authority, and accountability to perform the SE functions (including analysis, modeling and simulation, architecture development, requirements management, and so on). Some of the “program discipline” needs to be in pre-Milestone A management. • It is necessary to manage the maturity of technologies prior to Milestone B and to avoid reliance on immature technologies. Technology maturity and risk mitigation plans should be carefully managed as an integral part of program plans. The above represent lessons learned as a result of problems that arose or successes that were achieved on past or current programs. They are applicable in ­general. It is crucial for programs currently being formulated or beginning the acquisition process, TSAT and Space Radar being cases in point, that these ­lessons be applied early. It is incumbent on senior operational and acquisition leadership to enforce the discipline implied by these findings and shared lessons.

Next: 3 Systems Engineering Workforce »
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition Get This Book
×
Buy Paperback | $49.00 Buy Ebook | $39.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The ability of U.S. military forces to field new weapons systems quickly and to contain their cost growth has declined significantly over the past few decades. There are many causes including increased complexity, funding instability, bureaucracy, and more diverse user demands, but a view that is gaining more acceptance is that better systems engineering (SE) could help shorten development time. To investigate this assertion in more detail, the US Air Force asked the NRC to examine the role that SE can play during the acquisition life cycle to address root causes of program failure especially during pre-milestone A and early program phases. This book presents an assessment of the relationship between SE and program outcome; an examination of the SE workforce; and an analysis of SE functions and guidelines. The latter includes a definition of the minimum set of SE processes that need to be accounted for during project development.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!