2
Relationship Between Systems Engineering and Program Outcome

INTRODUCTION

As discussed in Chapter 1, this committee places great importance on systems engineering (SE) processes and their proper application by domain experts throughout the entire acquisition cycle, but particularly in the earliest stages of programs, that is, pre-Milestone A. This chapter discusses the development history of a variety of past and ongoing programs, emphasizing the role of SE during pre-Milestone A and Milestone A-to-Milestone B time frames and deriving key lessons from the program outcomes.

The committee observed that programs which were successful in constructing a sound requirements baseline and financial/acquisition plan through the pre-Milestone A and Milestone A-to-Milestone B phase, using sound systems engineering processes, could succeed or fail. Those that failed were traced to poor post-Milestone B actions. However, those that had successful pre-Milestone A and Milestone A-to-Milestone B phases were the only programs that succeeded. The committee observed that there were no successful programs that entered into Milestone B without the sound fundamentals provided by the rigor and discipline of the systems engineering and financial/programmatic planning afforded by the pre-Milestone A and Milestone A-to-Milestone B processes. In this context, programs that succeeded were those that delivered their products within a reasonable margin of the original cost and schedule baseline. Programs that failed, in the committee’s view, may have delivered successful products but were well outside the reasonable expectations of the original program and were only successful in delivering products after the addition of substantial unplanned funding and a substantial extension of the original schedule.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 26
2 Relationship Between Systems Engineering and Program Outcome INTRODuCTION As discussed in Chapter 1, this committee places great importance on sys- tems engineering (SE) processes and their proper application by domain experts throughout the entire acquisition cycle, but particularly in the earliest stages of programs, that is, pre-Milestone A. This chapter discusses the development history of a variety of past and ongoing programs, emphasizing the role of SE during pre-Milestone A and Milestone A-to-Milestone B time frames and deriv- ing key lessons from the program outcomes. The committee observed that programs which were successful in construct- ing a sound requirements baseline and financial/acquisition plan through the pre-Milestone A and Milestone A-to-Milestone B phase, using sound systems engineering processes, could succeed or fail. Those that failed were traced to poor post-Milestone B actions. However, those that had successful pre-Milestone A and Milestone A-to-Milestone B phases were the only programs that succeeded. The committee observed that there were no successful programs that entered into Milestone B without the sound fundamentals provided by the rigor and discipline of the systems engineering and financial/programmatic planning afforded by the pre-Milestone A and Milestone A-to-Milestone B processes. In this context, pro- grams that succeeded were those that delivered their products within a reasonable margin of the original cost and schedule baseline. Programs that failed, in the committee’s view, may have delivered successful products but were well outside the reasonable expectations of the original program and were only successful in delivering products after the addition of substantial unplanned funding and a substantial extension of the original schedule. 

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe While the committee attempted to address the Milestone A and B issues in the case studies, it found very little information because of poor formal documen- tation regarding what happened in the Milestone A and B phases. Much of what is reported comes from the collective knowledge of committee members who were familiar with the programs. The point is that there is generally insufficient docu- mented work done pre-Milestone A and B. The committee believed it important to include cases such as the C-5A to illustrate that good Milestone A and B work is not sufficient. It does not guarantee successful acquisition if poor decisions are made in source selection or during system design and development (SDD). While it is not possible in most instances to draw from the cases a clear causal relationship to the individual findings and recommendations of this report, the committee found the case studies to be of value and expects that they will also provide useful information for the reader. The important steps in the acquisition process are depicted as a continuous “thread” in Figure 2-1. Each segment of the thread is associated with a specific SE process. As Department of Defense (DOD) acquisition pulls the entire sys- tems engineering thread, the thread can break at many different points for many different reasons. Thread breakage from perturbations in the SE processes tends to result in cost and schedule overruns and performance degradation. The study schedule did not permit nor were the resources available to enable the committee to conduct in-depth, long-term studies of Air Force acquisition programs using a formal, rigorous, structured case study methodology. Instead, the committee had to use and to draw from published program data and formal FIGURE 2-1 Systems engineering thread from defense strategy to system operation. IOC, initial operational capability. SOURCE: Contributed by committee member John Griffin.

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING work already done by others (such as the Air Force Institute of Technology’s [AFIT’s] published systems engineering case studies1), the presentations made to the committee on a sample of Air Force programs, and the knowledge and judg- ment of its members developed through long experience with such programs. The committee examined the formal systems engineering case studies that had been completed by AFIT’s Center for Systems Engineering. The AFIT case studies were based on the Friedman-Sage systems engineering case study meth- odology. 2 Each of the AFIT studies addressed the full program life cycle, includ- ing the early phases of a program (they were not specifically or strongly focused on pre-Milestone A, however). For other programs the committee examined, the committee could not con- duct analyses equivalent to the AFIT case studies. As the sponsor and committee anticipated, the committee did not find a wealth of data or proven models that could be used to strongly link a program’s pre-Milestone A systems engineering effort with later program outcomes. Still, these programs yielded lessons learned that applied to the study charge. The programs summarized in this chapter are these: • Space Based Infrared Systems (SBIRS), • Joint Direct Attack Munition (JDAM), • Future Combat Systems (FCS), • F-16 Fighting Falcon, • Turbine engine development: fighter jet engine, • Military Satellite Communications (MILSATCOM), • C-5A, and • B-2 Stealth Bomber. These examples were chosen to illustrate the diversity of programs to which SE is applied: for example, from relatively simple systems such as JDAM to complex systems of systems such as FCS and MILSATCOM. For brevity, program histories are summarized here; more detailed histories of the programs summarized by the committee can be found in the references provided in footnotes in this report. Other programs also considered in the committee’s deliberations are mentioned in this chapter and, while not sum- marized herein, can be accessed through the published documents referenced in this report. 1Air Force Center for Systems Engineering Case Studies. Available at the website of the Air Force Institute of Technology at http://www.afit.edu/cse/cases.cfm. Last accessed on November 20, 2007. 2 See G. Friedman and A.P. Sage, 2005, “Case Studies of Systems Engineering and Management in Systems Acquisition,” Systems engineering 7(1):84-97. The “Friedman-Sage” (F-S) SE case study methodology was used in the generation of the AFIT case studies; however, the AFIT case studies went well beyond the F-S methodology in the interview process and so on.

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe PROgRAM SyNOPSES This section highlights the application (or lack thereof) of SE to past and ongoing acquisition programs and their performance during various phases of their life cycles. Space Based Infrared Systems (SBIRS) Program SBIRS is designed to provide tracking and targeting capabilities for missile warning, missile defense, and technical intelligence. SBIRS is a successor to the highly successful Defense Support Program (DSP) satellite system, which performs the missions of early warning of strategic ballistic-missile launches, detection and reporting of tactical missile launches in theaters of interest, and, as secondary missions, detection of space launches and nuclear detonations. The histories of the SBIRS Program and DSP present a stark picture of how procurement time lines and costs have ballooned over the past 40 years. The contractor for the initial DSP development was selected in 1965, and a success- ful initial operational capability (IOC) was achieved just 5 years later. Since that time, DSP has been improved many times and remains a valuable part of U.S. space assets to this day. The successor to this program, now called SBIRS, was awarded for full-scale development in 1996. Since that time, numerous schedule slips and cost overruns have been announced, and the first launch is now sched- uled for 2009. During the SBIRS briefing to the committee, the presenter said that the program’s minimum goal is to produce a first flight article that is “at least somewhat better” than DSP.3 The SBIRS program has received much attention due to program execution issues. The program experienced Nunn-McCurdy breaches4 in 2002 and 2005, at which times it had to be recertified. The program had two additional Nunn- McCurdy breaches of 15 percent cost increase, but did not have to be recertified. The SBIRS program has completed Increment 1 and the delivery of its first two hosted sensors and is currently working to deliver its first geosynchronous Earth orbit (GEO) satellite; however, this milestone is projected to be reached extremely late relative to original program plans and will have considerably exceeded original cost projections. Systems engineering Lessons The committee believes that the primary factors in this situation are as follows: 3 Col. Randall Weidenheimer, 2007, “Space Based Infrared Systems Wing,” Space and Missile Systems Center, USAF, presentation to the committee, January 31, 2007. 4A Nunn-McCurdy breach occurs when the program acquisition unit cost (PAUC) or average pro- curement unit cost (APUC) exceeds the baseline value by 25 percent or more.

OCR for page 26
0 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING • The internal complexity of the SBIRS spacecraft, including complex focal plane arrays and the need for concomitant signal processing algorithms, is much more complex than that of the DSP spacecraft. • The external user interfaces for SBIRS are far more complex than those for the earlier DSP owing to requirements of the missile defense com- munity, tactical users, and the legacy early-warning role. • There was a high degree of requirements instability after Milestone B (or its equivalent), driven by shifting missile defense strategies and tactical requirements changes. • Full-scale development was undertaken while the sensor technology initially planned was not mature. • There was a high reliance on new flight software. • There was inadequate oversight and accountability on the part of both the Air Force and the contractor. Each of these systems engineering issues was a significant contributor to the SBIRS Program’s failure to execute. However, since the issues of technology readiness, acquisition process, and requirements stability have been dealt with thoroughly in prior reviews of the SBIRS Program, the committee focuses here on the software issues, which have been overlooked elsewhere. The SBIRS Program represented a several-fold increase in the scope and scale of flight software relative to earlier Air Force programs and a manyfold increase over legacy systems in the SBIRS mission area. However, software systems engineering received insufficient emphasis and was not effective in early program formulation. Software and mission domain experience, in both the government and contractor teams, was inadequate to prevent or mitigate the impacts of inappropriate flowdown of requirements to the software, overly optimistic projections of software production rates, and inappropriate phasing of software development and testing with the system-level integration and test program. In fact, flight software still poses a risk to the execution of the remain- ing work in SBIRS. This risk is less related to how many lines of code can be written and tested in a given period of time than it is about whether or not the software system engineering was sufficient to produce the robust and resilient flight software system that will be needed over the current and succeeding phases of this complex program. Several other programs have recently experienced program execution issues late in the development phase as a result of failures to employ robust software systems engineering principles in the pre-Milestone A phase. These program execution issues are frequently not attributable to pure software causes; instead, they tend to occur (perhaps more insidiously) in programs with complex hardware/software interfaces or dependencies (such as with SBIRS). Another way to characterize the problem is that there was insuf- ficient coupling between the disciplines of systems engineering and software

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe engineering early in the program. 5 As a result of these problems, lessons are being learned, progress is being made, and improvements in practices and methodologies are being employed in some current pre-Milestone A programs (discussed below). Traditionally, there has been fairly loose contact between the disciplines of systems engineering and software engineering, both in academia and in practice. Typically, software engineering has contributed more to the formation of the discipline of systems engineering than the reverse, primarily because systems engineering as an academic discipline is a younger field. However, there is a notable trend lately of these two fields coming together, at least in academia. For example, the University of Southern California’s Systems Engineering Center recently merged into its Center for Software Engineering to form the Center for Systems and Software Engineering. These trends in academia are beginning to carry over into practice, leading to better up-front treatment of software systems engineering in the pre-Milestone A phase. A decade ago, when programs such as SBIRS were in their formative phases, program managers and chief engineers might have readily ignored the question of the feasibility of accomplishing stressing system requirements using software. Indeed, it had been common for software to be the fix-all for any design requirement that was deemed “too hard” for the hardware design—“We’ll just do it in software.” Today, the hardware/software trade space is more typically dealt with using the same level of rigor as is used in other system design trades. This increased contact between the disciplines of systems engineering and software engineering will allow early input into the requirements allocation process and trade space analysis by informed software specialists, thereby avoiding the later program execution pitfalls that result from having expected all of the hard prob- lems to be handled by the software. Specifically, the lessons learned from the SBIRS Program are being taken to heart in several more recent programs. For example, in the earliest phases of the Space Radar Program, the system program director chose to create a software division within the government program office, along with divisions focusing on the space vehicle, ground system, and systems engineering. With this organiza- tion, the program director sought to ensure that all pertinent systems engineering trade analyses and programmatic decisions would be vetted through all compet- ing needs of the program, and that the final decision options would be presented with a full characterization of the benefits and impacts associated with each discipline area, including software. Surely, the Space Radar Program will face its share of technical and programmatic challenges, but it will be informative to see to what degree potential software systems issues have been avoided or their impacts mitigated by this organizational approach to software systems engineer- ing in the formative phase of the program. 5 Robert N. Charette, 2005, “Why Software Fails,” Ieee Spectrum 42(9):42-49.

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING The committee notes here that one of the items on its systems engineering checklist (see Chapter 4) is to assess the methodology that the program has cho- sen to integrate systems engineering and software systems engineering. SBIrS and DOD acquisition reform The SBIRS acquisition was undertaken by the Air Force during a period in which “acquisition reform” was being strongly pursued. One of the premises of acquisition reform was to reduce life cycle cost through reduced government program office size and responsibility. The DOD began a program to reduce the size of the acquisition workforce over a period of several years. The Congress participated with direction in the Fiscal Year (FY) 1996, FY 1997, FY 1998, and FY 1999 Defense Authorization Acts to reduce the acquisition workforce. 6 A consequence of reducing the number of government acquisition personnel was the outsourcing of more acquisition work to industry. The SBIRS Program elected to delegate total system performance responsibility (TSPR) to the prime contractor. Assignment of TSPR to the prime contractor for systems of this scope and complexity is inappropriate if it leaves the government in a position of seek- ing insight into program execution and decisions, rather than assuring account- ability through active oversight. While there was nothing in the definition of TSPR that required no government involvement or that specified no government accountability, the implementation of TSPR on SBIRS was taken to an extreme. 7 A symptom of the flawed acquisition approach was the abrogation of government accountability for many of the pre-Milestone A systems engineering processes and products. Some acquisition reform initiatives (many in the Air Force directed under the heading of “Lightning Bolts”8) were sent to the field for acquisition programs without a complete understanding of their consequences. While some seemed sensible in theory, many were applied inconsistently, resulting in unintended consequences. The committee notes that it would be a repeat of past failures to invent a series of SE reforms and mandate them in a “one size fits all” fashion without assessment and tailoring to the situation. Many of these same approaches worked fine in other programs (see the following case of JDAM). These points are discussed further in the systems engineering checklist presented in Chapter 4. 6 See the Chapter 3 section entitled “Congressional Actions to Cut DOD Acquisition Workforce” for additional discussion of these acts. 7The issue here is the word “total.” The government program office cannot give up total responsibil- ity for cost and performance. 8 “Lightning Bolts” is the title given to a series of acquisition reform initiatives launched in 1994 by then Principal Deputy Assistant Secretary for Acquisition and Management for the Air Force Darleen A. Druyun. The initiatives, which included establishing a centralized acquisition support team to scrub all proposals over $10 million, developing a new System Program Office model, and reducing the number of military specifications and standards, were intended to make the acquisition and sustain- ment processes for the Air Force better, faster, and cheaper.

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe It would be a gross oversimplification to attribute all SBIRS program execu- tion issues solely to acquisition reform. Likewise, the program facts do not sup- port faulting systems engineering for all SBIRS execution issues. Nevertheless, poor SE did contribute to program troubles encountered thus far and might yet be a major contributor to execution issues with the remaining effort, particularly in the area of software systems engineering. The SBIRS case provides impetus to assess the level and quality of the integration of systems engineering and software systems engineering in ongoing programs. Joint Direct Attack Munition (JDAM) Program Description JDAM was initiated in late FY 1991 and had its roots in operation Desert Storm. It was during that conflict that military leaders realized the need for all- weather, extremely accurate bombs capable of being dropped from a number of aircraft platforms. The military arsenals were filled with hundreds of thousands of “dumb” gravity bombs. The military wanted to turn these unaided bombs into “smart” bombs using a strap-on kit. The kit would use Global Positioning System (GPS) satellite-guided signals and computer technology to deliver the bomb within 13 meters of its target, regardless of environmental conditions such as storms, darkness, and high winds. The JDAM Program operated in the same acquisition reform environment that SBIRS operated in, but the team did not select a TSPR approach, and the team’s perseverance paid off. The JDAM Program is a success by every measure: the JDAM team’s final proposal included an average unit production price (AUPP) between $14,000 and $15,000—down from an original cost target of $40,000 and an original cost estimate of $68,000. The JDAM team reduced its research and development costs from $380 million to $310 million and shortened the develop- ment program length from 46 months to 30 months. The total procurement cycle length was reduced from 15 years to 10 years. Military performance has continued to improve (e.g., product improvements have allowed the JDAM Program to con- tractually tighten performance accuracy from the original 13 meters to 5 meters). By February 2007, more than 150,000 JDAMs had been delivered to U.S. and international customers and had been integrated onto 10 U.S. and 5 international aircraft platforms. JDAM is a truly operationally effective system. There were many other innovative acquisition and systems engineering ini- tiatives employed by both the government (system program office [SPO] and user) and the industry teams.9 9 For more details, see C. Ingols and L. Brem, 1998, Implementing acquisition reform: a Case Study on Joint Direct attack Munitions (JDaM), Washington, D.C.: Defense Systems Management College. Available at http://www.acqnet.gov/comp/seven_steps/library/JDAMsuccess.pdf. Last ac- cessed on September 18, 2007.

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING Systems engineering Lessons The success of the JDAM Program was the result of many individual appli- cations of effective systems engineering and program management principles. Primary among these were that (1) the requirements were clearly defined prior to development and remained stable throughout the development phase, (2) the technology used was mature, and (3) the external interface complexity was well managed. Also of importance was the effective teaming between the government and contractors to form accountable Integrated Product Teams (IPTs). The IPT struc- ture within the JDAM Program instilled a motivation to all team members to engage in and contribute to the success of the program. Another key ingredient was the up-front focus on affordability. This was facilitated by contract provisions allowing the contractor’s profitability to increase as a result of cost reductions accrued through innovative technical and management approaches (including the selective use of commercial practices and specifications). Future Combat Systems (FCS) Description The Army’s Future Combat Systems (FCS) is, without question, the largest, most complex program in Army history. It is literally a “system of systems of systems.” Figure 2-2, from an Army briefing on FCS to the National Defense Industries Association, illustrates this complexity. The overarching objective of the program is to develop a lighter, more lethal force that could be deployed far more rapidly than the heavy forces that are a major proportion of the Army force structure today. To achieve the overwhelming lethality that is envisioned, the concept of network-centric warfare is the underlying theme. It is expected that, via an integrated, robust, self-adapting, self-healing network, pertinent situational data will be available to all leaders at all echelons of the Army ground force (i.e., they will have accurate situational awareness). This will support operations well within the enemy’s decision/awareness cycle and achieve increased lethality. These concepts were evolved by the Defense Advanced Research Projects Agency (DARPA) in cooperation with the Army and taken through early concept development. At that stage, the Army “constructed” the FCS Program. The deci- sion was made to take the concepts as defined at that time and, via a competitive bidding process, select a lead systems integrator (LSI) contractor to pull together the doctrinal concepts and codify them into a system of systems architecture. It was further decided to use Other Transactions Authority (OTA) as the program management vehicle, rather than the traditional milestone process. The idea was that early capabilities/requirements analysis leading to the systems of systems architecture and eventually to systems specifications could progress much faster without the layered milestone decision authority process. It was not envisioned

OCR for page 26
FIGURE 2-2 Conceptual illustration of the Future Combat Systems Program and its complexity as a system of systems. NOTE: SOSCOE, System of Systems Common Operating Environment. SOURCE: D. Emery, D. Bassett, and M. Schnaidt, “The Net-Centric Foxhole: Perspectives from Army Future Combat Systems,” presentation, October 25, 2006. San Diego, Calif.: National Defense Industrial Association 9th Annual Systems Engineering Conference. Available at http://www.dtic.mil/ndia/2006systems/Wednesday/emery.pdf. Last accessed on November 20, 2007. 

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING that less rigor would or should be employed in the early capabilities/requirements determinations nor in the translation of capabilities/requirements to systems architecture and thence to systems specifications. Significant amounts of time, effort, and dollars were spent in platform and sensor technologies, in parallel with trying to define the integrating network. The LSI seemed to recognize this difficulty early on and set about developing an integrating computer hardware/software system entitled System of Systems Common Operating Environment (SOSCOE). Without some capability such as SOSCOE, there would have been little chance that the various platforms could ever be integrated. In parallel with SOSCOE, the network architecture began to be addressed. The network architecture early concept and definition work was con- strained, however, by a mandate from the government that its central core would be the Joint Tactical Radio Systems (JTRS) waveform architecture. While on the surface this did not appear to be a major obstacle, it soon became a problem in that the JTRS program itself was having technical and schedule problems and became a gating item in defining the network operating concepts, architecture, and topology. In summary, the activities that might be associated with normal pre-Milestone A work just did not coalesce. The program, under OTA, entered into full-scale development with many unresolved issues on systems capabilities/requirements and architecture. Costs were rising well above the original estimates, systems integration problems were growing, and the schedule was slipping well beyond the original estimates. Meanwhile, Congress was expressing real skepticism. As a result, the Secretary of the Army ordered that the program be restructured into the DOD-approved milestone process. Systems engineering Lessons The FCS Program violates many of the precepts that the committee believes are important to success in the conventional system-development process. For example, the prime contract for FCS was let while requirements were very much still being traded off against desired capabilities. The Army has had difficulty stabilizing requirements changes between Milestone B and IOC, perhaps due to insufficient systems engineering in the early phases of program planning. A preliminary observation based on this case concerns the need for rigor in those activities associated with defining capabilities and requirements and overall systems architecture, independent of the formality of DOD milestone definitions, before entry to the SDD phase is authorized. It should be recognized that, as a potential program progresses through its early stages, better insight into technical feasibility and evolving needs will sometimes necessitate changing the requirements. If the changes become substantial, the program should be recycled through concept development before going ahead with full-scale SDD. A firm conclusion requires more analysis. The full suite of essential requirements should

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe In the 1970s, Air Force Propulsion SPO managers instituted ground tests that more accurately simulated the operation of an engine throughout its lifetime. Designers used information on what throttle settings would be utilized throughout the life of the engine, given various expected mission types (e.g., air-to-ground missions, air-to-air missions, and escort missions). They then transformed a com- posite of these throttle settings into a simulation of the throttle settings at which the engine would be expected to perform. The EG10-5 eliminated many of the problems suffered by the EG10-1 and EG10-2 engines, such as in-flight stall stagnations and problems with the fuel control system. Additionally, because efforts were made to understand better how the engine would be employed in the field, the squadron owners of the engines were more satisfied with how they were able to maintain the system. The Air Force’s pro-sustainment policies did not cease upon completion of the EG10-5 project. The OEMs wished to continue producing engines for the F-15 and F-16 aircraft, and in order to do that, they had to continually improve the sustainability of their engines. Owing to the Air Force’s strong emphasis on sustainability, the two corporations installed a number of new features (such as electronic monitoring systems) making their systems more sustainable. It was in the midst of these innovations that the EG10-9 engine was designed. Systems engineering Lessons Systems engineering contributed significantly to the improvements in reli- ability and maintainability of the evolving engines in the EG10 family of engines. These contributions may be classified as policy, technology, and process and tool development. Systems engineering thinking was used to develop an effective policy transition, managed by the Propulsion SPO, from a “nonsustainment” ideology to a “performance-with-sustainment” requirement. Technology contri- butions that were based on systems engineering included computer-aided design, modularity, electronic engine controls, and computer-aided logistics. In the area of processes and tools, systems engineering thinking enabled the development and use of IPTs and accelerated mission testing. These tools were captured in the transition from sequential engineering to concurrent engineering in the design and manufacture of jet engines. The history of the fighter engine development programs shows that designing early for sustainment and performance results in more affordable and agile life cycle options, and a greater flexibility for technical upgrades throughout the oper- ational lifetime of a system. Perhaps most importantly, however, it illustrates the value of having an experienced, domain-knowledgeable organization within the government (in this case, the Propulsion SPO) to manage acquisition programs.

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING Military Satellite Communications (MILSATCOM) Description One of DOD’s most crucial goals is to create an interoperable system of systems aimed toward enhanced command, control, communications, computers, and intelligence. The mission of the MILSATCOM Program is to provide global, space-based communications capabilities supporting DOD and other government agency missions. MILSATCOM is itself a system of systems, providing narrowband, protected, wideband, and network communications capabilities to a wide range of military users. MILSATCOM must interface with a multitude of external stakeholders, some of which control key architectural specifications, or even performance requirements for the system (e.g., information assurance). The first mission area is the narrowband satellite communications (SATCOM) area that includes the UHF Follow-On (UFO) system and Mobile User Objective System (MUOS). Narrowband provides reliable service to mobile users. The wideband mission area includes the Defense Satellite Communica- tions System (DSCS), its follow-on Wideband Gapfiller System (WGS), and Global Broadcast Service (GBS). Wideband provides broadcast service, similar to DirecTV, and high-capacity data pipes. The protected mission area is supported by Milstar, advanced extremely high frequency (AEHF) system, and Polar MILSATCOM. Protected SATCOM provides highly secure and survivable communications. The Transformational Satellite Communications System (TSAT) is a next-generation MILSATCOM system, following WGS and AEHF. It will support both the protected and wide- band mission areas. In addition to the satellite systems, MILSATCOM includes a terminals seg- ment. The different SATCOM terminals communicate with one or more of the satellite systems. Examples include the Milstar Command Post Terminal, Secure Mobile Anti-Jam Reliable Tactical Terminal (SMART-T), Family of Advanced Beyond-Line-of-Sight Terminals (FAB-T), DSCS and GBS Terminals, Ground Multi-band Terminal (GMT), Airborne Integrated Terminal (AIT), Navy Multi- band Terminal (NMT), Warfighter Information Network-Tactical (WIN-T), High Capacity Communications Capability (HC3), and Multi-Band Multi-Mode Radio (MBMMR). Historically, the individual MILSATCOM systems were handled in a piece- meal fashion—each system focused only on what it needed to be individually successful, not considering how it might contribute to the broader MILSATCOM program. Each of the satellite systems, while being headquartered at the Air Force’s Space and Missile Systems Center, had its own user community, its own program director, program office, budget, schedule, and so on. In recent years, a combination of increasing integration of military opera- tions and a dramatic Air Force drawdown of technical and program management

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe talent have forced organizational change in the management of MILSATCOM programs. Today, the MILSATCOM Program is being handled as an enterprise under the MILSATCOM Joint Program Office (MJPO). Within the MJPO, each system is assessed not only on its own merits, but also on how it contributes to the various MILSATCOM mission areas (protected, wideband, narrowband SATCOM) and how it can contribute to meeting some or all of the needs of a particular user community (for example, supporting intelligence, surveillance, and reconnaissance data relay). The MJPO has taken several measures to reduce program risks and increase the probability of success in delivering MILSATCOM capabilities. These efforts have focused primarily on the integration impacts in four areas: across the breadth of individual programs, in the transition to operations, in the interactions among programs, and external influences on programs. As a result of interconnections between various MILSATCOM systems, the MJPO has also recognized the interdependencies among programs. The MJPO has started several initiatives to reduce the risks of any program’s adversely impacting others. These initiatives include the stand-up of the MILSATCOM chief engineer and the System of Systems Engineering, Architecture and Integration (SSEA&I) group; the MJPO Configuration Board (MCB); weekly senior-level functional reviews and discus- sions; and the MILSATCOM Integration Master Schedule (IMS). The MJPO cre- ated the MCB to track and manage the integration of interfaces, specifications and standards, and program efforts. This executive-level board, composed of MJPO program managers, is responsible for advising the MCB chair (MJPO director or deputy) on proposed contract change actions, and manages all MJPO configura- tion baselines throughout sustainment and any changes that have a cost impact. Despite the move toward unified program management represented by MJPO, in recent years the development of the MILSATCOM systems has been plagued with delays and cost growth that are a legacy of the earlier, fragmented manage- ment structure. Examples are the AEHF and WGS systems. Following the failure of Milstar Flight 3, AEHF was accelerated to complete worldwide protected SATCOM coverage. This decision forced significant adjust- ments to AEHF Flight 1, including a transition to an operations plan that is less than optimal. These changes resulted in the addition of a requirement that AEHF be backward-compatible with Milstar. Extra attention is being given to the AEHF system development effort to address issues on Milstar backward compatibil- ity, operations transition, cost overrun projections, and sustainment. The AEHF system procurement effort has had to provide a Nunn-McCurdy notification to Congress owing to late delivery of government-furnished products (especially for performing cryptographic functions) and replacement of critical electronic parts, causing a cost breach greater than 15 percent. AEHF had to replan the procure- ment effort, delaying delivery of the first satellite, and is now subject to increased scrutiny within the DOD and in Congress. The WGS system was contracted during the so-called Acquisition Reform era,

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING in which the government expected “commercial best practices” to be applied.12 The program viewed the WGS satellites as a mature product requiring little oversight. As it turned out, the large number of program interfaces and emergent program technical issues required both additional program insight and additional funding in critical areas. WGS, with a firm fixed-price (FFP) contract that limits the cost impacts to the government, had problems with fasteners that delayed the delivery of the satellites by 15 months. A combined government and contractor team applied systems engineering to identify and prioritize mission-critical areas. This was an important step, especially in light of the fixed-price nature of the contract. The program encountered design, integration, and manufactur- ing problems due largely to the fact that the program was unable to benefit from continuing synergy on commercial SATCOM production lines. The Wideband Gapfiller System prime contractor had assumed a continued growth in the com- mercial SATCOM marketplace in its WGS FFP proposal. When the commercial SATCOM demand did not continue to grow, and in fact declined precipitously, the WGS prime contractor suffered substantial overruns on its FFP contract. System engineering Lessons For the MILSATCOM Program to have been helped early by better systems engineering, each of the component elements would have needed to have been conceived in response to requirements from a consistent and coordinated set of users, advocated and funded by the same organization, and acquired, tested, and fielded in a coordinated way by an integrated program office. The cur- rent MILSATCOM Program was formed by combining preexisting satellite and terminal offices long after many important decisions had been made. In both AEHF and WGS, the control and coordination processes that the MJPO has now put in place would have resulted in much earlier identification and resolution of issues and coordinated, joint problem resolution. The AEHF program could have benefited greatly from earlier systems engi- neering that would have yielded a better understanding of the compatibility requirements with the legacy system and, in fact, understanding of the actual con- figuration of the sustained legacy Milstar system. A much more thorough analysis of the ability to accelerate AEHF in the face of immature technology would also have identified potential future problems and work-arounds. A more active man- 12Several important points are worth noting about the differences between the DOD MILSATCOM system and commercial SATCOM. As previously noted, MILSATCOM is a system of systems. While commercial SATCOM may consist of a family of incrementally improved satellites, these satellites do not and need not respond to the range of stressing missions accomplished by MILSATCOM. Commercial users are by and large less disparate and naturally have a more stable and narrower requirements set. Also, while commercial SATCOM cannot be considered “low-tech,” it does not push the Technology Readiness Levels the way that MILSATCOM does. Finally, as a general rule, commercial SATCOM contracts are fixed-price production contracts, while DOD MILSATCOM contracts have been of the cost-plus type.

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe agement and engineering engagement with the National Security Agency (NSA) might have highlighted cryptographic issues sooner and allowed alternatives to be developed. A better understanding of program interfaces and of the critical dependence of a program on Technology Readiness Levels might have led to different con- tracting decisions on the Wideband Gapfiller System. Active government program management and earlier joint systems engineering would have resulted in earlier and cheaper problem resolution. While it is the mission of MJPO to deliver an integrated capability, many of the end users of the capability are not part of the Air Force Space Command (AFSPC), and their issues are not always well represented. The individual pro- gram managers do not and cannot control all aspects of key external requirements. While external users could always be better represented, the MILSATCOM SPO has made great strides in dealing with those users and understanding their issues. The MJPO has implemented a multipronged effort to provide a system- of-systems approach to the technical, business, and acquisition management of MILSATCOM products, and this approach is embodied in a wing-level systems engineering plan that addresses all issues at the system of systems level. Docu- mented processes, carried out at the enterprise (wing) level, ensure that standards exist and are enforced among program (product) elements and also ensure that artifacts and information are generated to support cross-program integration. By employing traditional systems engineering processes at the system of sys- tems level, MJPO is able to reduce surprises and limit unintended consequences of individual program decisions and at the same time to gain an integrated view of gaps and overlaps among the product lines. The implementation and successful execution of such a systems-of-systems approach require leadership, experienced personnel, communication, and the willingness to make difficult trade-off deci- sions that affect individual program optimization. C-5A Program Description The C-5A Program was characterized by an excellent pre-Milestone A and Milestone A-to-Milestone B process.13 The Development Planning organization at Aeronautical Systems Division, Wright-Patterson Air Force Base, conducted operational mission analyses and contracted with industry (Lockheed Martin Cor- poration, The Boeing Company, and McDonnell Douglas Corporation) to conduct supporting conceptual design analyses. The Development Planning organization also conducted parallel conceptual design analyses of potential aircraft designs 13 John M. Griffin, SES (Ret.), undated, C-a Galaxy Systems engineering Case, Wright-Patterson Air Force Base, Ohio: Center for Systems Engineering at the Air Force Institute of Technology. Avail- able at http://www.afit.edu/cse/cases.cfm. Last accessed on June 18, 2007.

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING that could meet the evolving functional baseline. The user was an integral part of the process, as were laboratory scientists and technologists. The result of the entire process was a weapons systems specification that represented the functional baseline and was technologically feasible within the existing state of the art. A SPO was established prior to the conduct of source selection; it was popu- lated with domain experts in all of the functional areas. The SPO cadre included a number of people from the Development Planning Directorate who participated in the early development of the requirements and were knowledgeable regarding the genesis of the program. In response to their perception of what their competitors would bid, Lockheed claimed an aggressively low value for its aircraft weight empty in its proposal. During the source selection, the Air Force convinced Lockheed to put the aircraft weight empty into its specification as a performance requirement equivalent to range and payload and other performance parameters. Then the Air Force put a financial penalty into the contract in the event that the contractor’s aircraft weight exceeded the specification value, and Lockheed signed this contract. To compli- cate matters further, the contract type was a firm, fixed-price contract for both development and initial production under the new contract strategy that was being pursued by the Air Force, called Total Package Procurement. As a final constraint to the contract, a complementary financial penalty was included for each month that Lockheed missed the contract first flight date. As the source selection came to a close, the three contractors were debriefed as to their strengths and weaknesses. The Air Force advised Lockheed that its assessment of the proposed aircraft showed the wing area to be deficient by some 400 square feet. Lockheed hastily redesigned the aircraft wing in just 4 days and resubmitted its proposal. The time available to redesign the wing was inadequate to conduct a systems engineering assessment of the new design; most of the technical parameters were updated as a ratio of the wing areas. This included the weight empty, which had been estimated with optimism for the original design and was now even more optimistic. In retrospect, this change contributed to a further disconnect of the cost estimate from the technical baseline. The net effect of all this activity during source selection was to excessively constrain costs, schedule, and performance to such tight and rigorous limits that there was no hope of being able to perform to the requirements of the contract. As the design progressed to preliminary design review (PDR), it was clear to Lockheed that it could not meet the weight requirement. Lockheed proposed to the SPO that the weight requirement be removed from the contract, that the contractor would meet all of the performance requirements remaining in the specification, and that the engine contract would be increased by the $5 million needed to redesign the engine for an increase in engine thrust. The engine con- tractor was more than willing to redesign the engine for increased thrust, since its competitors had already seen the need to increase thrust to compensate for weight growth that was occurring in Boeing’s commercial 747. The Air Force

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe System Program Office disapproved the request and, in fact, sent the contractor a cure notice.14 Lockheed agreed to withdraw its offer and then directed its in- house engineering staff of designers to remove more weight from the structure of the aircraft. A competent system engineering assessment would have revealed that the resulting aircraft would be operationally unsuitable, having a limited load factor and reduced fatigue life. This was eventually recognized and accepted, and production was halted after 81 aircraft were completed. Major reviews of the program and the design of the structure were con- ducted under the leadership of the Air Force. The combined government and industry team conducted an extensive systems engineering study of multiple options before selecting a new program approach. The new wing was rede- signed to meet strength and life requirements. This was accomplished using basically the same design approach and adding 14,000 pounds to correct the problems. This new wing was retrofitted to the first 81 aircraft, and it was also included in the next buy of 59 C-5B aircraft. The Air Force spent over $2 bil- lion more than the original budget, and it took 10 years longer to finish the production with fewer aircraft than originally planned. Lockheed petitioned the United States government through the Congress for a $250 million loan to avoid bankruptcy. Systems engineering Lessons The C-5A case study makes the point that the pre-Milestone A and Milestone-A-to-Milestone B requirements process was efficient and effective. The C-5A Program should have succeeded based on the strength of the pre- Milestone A and Milestone A-to-Milestone B process. The point of failure on the C-5A Program occurred during source selection and was the result of poor appli- cation of the systems engineering process by both Lockheed and the Air Force during source selection after Milestone B. Because of the actions taken by the program participants, the program was doomed to failure from the first day of the contract. The lesson is that the systems engineering process in the pre-Milestone A and from the Milestone A-to-Milestone B time frame is an absolutely neces- sary function, but it is not sufficient. The process must be executed accurately and completely throughout the continuum of program acquisition. Figure 2-3 shows the systems engineering graphic for the C-5A. It shows the systems engineering process in green for the requirements development phase and in red from contract go-ahead and on, and again turning to green for the retrofit and the C-5B, albeit at the expense of time and money. 14 Before terminating a contract for default, the contracting officer may issue a written notice called a “cure notice.” The notice allows the contractor 10 days to “cure” any defects. If the failure to perform is not cured within 10 days, the contracting officer may issue a notice of termination for default.

OCR for page 26
 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING • Solid early studies of trade-offs FA • Mock-ups for roll-on, roll-off A • Stable requirements FNA C-5 FSA Sustainment • Requirement management CD (Contract award & beyond) AOA Prioritization of requirements M/S B OT • Significant structural PDR consequences (re-wing) DT CDR FIGURE 2-3 The C-5A systems engineering graphic showing an assessment of the applica- tion of the process. SOURCE: Contributed by committee member John Griffin. B-2 Stealth Bomber Program 2-3 Description As was the case with the C-5A discussed above, the B-2 case demonstrates a sound requirements process before Milestone A and successful systems engi- neering through Milestone B. The B-2 encountered problems just prior to PDR 2 when the design team completed an integrated analysis of the structure, flight controls, and aerodynamics. The design review showed that the aircraft had insufficient control power and structural rigidity to damp structural loads while providing the required maneuvering margins in the presence of turbulence. The integrated design team, led by Northrop Grumman and including the major subcontractors, Boeing and Vought, and the Air Force customer, completed the redesign in 4 months and received approval for the new changes through the company and the Air Force management structure in 3 months. Notwithstand- ing its rapid systems engineering and design response to correct the problem, the subsystem design could not recover from the changes and caused a 6 month slip to the effective completion of critical design review (CDR). In this case, the maturity of the design and confluence of three technical disciplines in a new con- cept led to the realization that a redesign was necessary. The design team quickly developed a new approach, and the management team responded with new plans. However, the subsystems designers and the subcontractors did not have sufficient

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe time to recover their previous work and were late to CDR. This was not a failure in requirements, but rather a recovery of the design and a lag in recovery by one segment of the team. Systems engineering Lessons The B-2 case study illustrates that even when the early systems engineering process is done well, the acquisition process is fraught with peril because of the unknowns and complications that arise in any program. However, from the case studies and from listening to the testimony of the briefers, it is clear that program offices and industry teams staffed with domain experts equipped to handle technical and programmatic difficulties are best suited to respond quickly and effectively to the problems when they arise. Managers who are inexperienced in handling problems such as occurred in this case may be unable to prevent the full collapse of a program. The acquisition process is by nature a complicated and delicate one that can easily be driven to instability. This point is underscored and somewhat amplified by comparing the C-5A partici- pants’ actions and those taken by the B-2 team. The C-5A Program certainly had domain experts, as did the B-2. However, the C-5A Program was complicated by the extreme controversy of the program, the fixed-price Total Package Procure- ment contract approach, the contractor’s aggressive claims, and the Air Force’s reluctance to re-open the contract after it was signed, all of which combined in a complicated dynamic and contributed to the less-than-domain-expert decisions. In the B-2 case, the program proceeded with the agreement of all parties, not- withstanding that the Northrop Grumman program management stated concern that the schedule of CDR was at risk. This risk was eventually realized, but it was not a requirements issue. Rather, it was a technical issue that surfaced as a result of the need for the aircraft redesign to meet the requirements on contract from the start of Milestone B. Figure 2-4 presents the systems engineering graphic for the B-2, showing that it turns red at PDR and then recovers to green at CDR, albeit a year late and nearly $1 billion over the estimate at completion. SHARED FINDINgS AND LESSONS LEARNED AMONg CASES Although each of the case studies summarized above revealed some unique findings, several key findings and lessons learned are shared among the programs. These are summarized here: • There is a need for an appropriate level of SE talent and leadership early in the program, with clear lines of accountability and authority. Senior SE personnel should be experienced in the product(s) domain, with strong skills in architecture development, requirement management, analysis,

OCR for page 26
0 pre-MILeSTONe a aND earLY-pHaSe SYSTeMS eNGINeerING • Solid early studies of trade-offs with combined SPO, user, contractor FA teams FNA FSA B-2 • Strong integration of requirements and CD design Sustainment AOA M/S B • System redesign OT PDR 1 PDR 2 DT CDR FIGURE 2-4 The B-2 systems engineering graphic showing an assessment of the applica- tion of the process. SOURCE: Contributed by committee member John Griffin. modeling and simulation, affordability analysis, and specialty engineering disciplines (e.g., reliability, maintainability, survivability, system security, and technology maturity management). 2-4 • There is a need to establish and nurture a collaborative user/acquirer/ industry team pre-Milestone A to perform system trade-offs and manage overall system complexity. Today, there are often significant disconnects in the hand-offs between users, acquirers, requirements developers, indus- try, and others. Some of the “best practices” include structured collabora- tion among these members. • One must clearly establish a complete and stable set of system-level requirements and products at Milestone A. While requirements creep is a real problem that must be addressed, some degree of requirements flexibility is also necessary as lessons involving feasibility and practical- ity are learned and insights are gained as technology is matured and the development subsequently proceeds. Certainly control is necessary, but not an absolute freeze. Also, planning ahead for most likely change pos- sibilities through architectural choices should be encouraged, but deliber- ately managed, a concept encouraged herein. A typical program execution team has a program manager (PM)-level SE integration team (SEIT), with

OCR for page 26
 reLaTIONSHIp BeTWeeN SYSTeMS eNGINeerING aND prOGraM OuTCOMe responsibility, authority, and accountability to perform the SE functions (including analysis, modeling and simulation, architecture development, requirements management, and so on). Some of the “program discipline” needs to be in pre-Milestone A management. • It is necessary to manage the maturity of technologies prior to Milestone B and to avoid reliance on immature technologies. Technology maturity and risk mitigation plans should be carefully managed as an integral part of program plans. The above represent lessons learned as a result of problems that arose or successes that were achieved on past or current programs. They are applicable in general. It is crucial for programs currently being formulated or beginning the acquisition process, TSAT and Space Radar being cases in point, that these lessons be applied early. It is incumbent on senior operational and acquisition leadership to enforce the discipline implied by these findings and shared lessons.