Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 Systems Engineering Functions and Guidelines INTRODUCTION In this chapter the committee defines the most critical systems engineering (SE) activities in the pre-Milestone A/B phase. Six key functions are discussed, along with four key SE outputs for the Milestone A/B activities. The commit- tee does not attempt to capture every function that may be involved, but instead focuses on the broad functions that have the greatest impact on ultimate system performance. The intent is that the chapter highlight the key functions from the viewpoint of the program manager, not define a âhow toâ manual for systems engineers. The committee discusses guidelines for managers and systems engineering practitioners in the pre-Milestone A/B phase of a program. In preparing these guidelines, the committee identifies those items that address the causes of the large decline in development program productivity and the attendant cost and schedule growth over the past 30 years, as discussed at the beginning of Chap- terÂ 1. These guidelines cover the following topics: â¢ The âsix seeds of failureâ: a discussion of six areas of risk that the com- mittee believes are worthy of particular attention in the pre-Milestone B phases and that are the basis for the recommendations that the committee makes later; â¢ Other potential causes of development productivity decline: a discussion of several often-mentioned causes of development productivity decline, including talent diminishment in government and industry, and excessive oversight; 75
76 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING â¢ Obsolete and nonrelevant systems engineering processes: a discussion warning against the use of legacy processes and guidelines that do not add value to the tasks at hand; â¢ General processes and practices for systems engineering: a discussion of some of the processes that are critical after Milestone B, including modeling and simulation, control of lower-level requirements, and change control; and â¢ A checklist: some of the committeeâs most important findings, provided to guide systems engineering activities and output before Milestone B. Pre-Milestone A SystemS Engineering Functions The prerequisite for starting the systems engineering process is a user- defined need (or outcome). The purpose of systems engineering in this phase is to define an optimal system concept and concept of operations (CONOPS) that satis- fies the need and also fits within budgetary constraints. The systems engineering outputs in this phase include a concise statement of key performance parameters (KPPs), a CONOPS description, a program implementation strategy, and models to support performance assessment, cost estimation, and risk management in subsequent phases. These functions are aggregated in the following broad areas, shown in Fig- ureÂ 4-1: concept creation, preliminary CONOPS, performance assessment, archi- tecture development, risk assessment, and cost scoping. These functions are reiterative and thus occur in parallel, to produce the key outputs from the pre- Milestone A systems engineering process: KPPs, CONOPS, supporting models, and program implementation strategy. â¢ Concept Creation â¢ Preliminary CONOPS â¢ Performance Assessment â¢ Key Performance Parameters â¢ Architecture Development â¢ Program Strategy â¢ Risk Assessment â¢ CONOPS â¢ Cost Scoping â¢ Supporting Models FIGURE 4-1â Critical pre-Milestone A systems engineering functions and outputs. NOTE: CONOPS, concept of operations. 4-1
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 77 Functions The functional elements are discussed briefly below. 1. System concept creation. This is a creative process for selecting a basic approach for a system to achieve the desired outcomes. The following guidelines are important in this phase: â¢ Consider alternatives. It is important that at least two alternative con- cepts be considered before locking in a solution. â¢ Consider the "time value of capability.â Seek concepts that can deliver initial capability within about 5 years (measured from Milestone B). Each year that a needed capability is delayed has a cost to those who need it, and delays the availability of operational data and experi- ence to guide subsequent improvements. Development programs that are successful and timely at initial operational capability (IOC) often become platforms on which many other capabilities are subsequently added. The world changes too fast to be friendly to long development cycles. Further, extended delivery times run the risk of the system becoming obsolete before deployment and can be an indication that the concept is excessively complex or excessively dependent on immature technology in its first delivery. Indeed, one insidious effect of long development cycles prior to IOC is that they create the temptation to add new emerging technologies and to further tune the requirements, which have the effect of further increasing the pre-IOC development cycle. Historical precedent shows that many complex major systems can achieve IOC in less than about 5 years. The Air Force has a detailed analysis of alternatives (AoA) process that can be invoked in this phase. In Air Force Instruction (AFI) 10-601, the role of the AoA process is described as follows: The AoA provides information that helps the decision makers select the most cost effective alternative(s), in order to satisfy a mission need or eliminate an operational gap/shortfall in capability. It compares alternative solutions on the basis of operational effectiveness, and cost. It documents the analytical and operational rationale for choosing the preferred alternative(s). It also helps to justify the need for starting, stopping or continuing an acquisition program. In its grandest form, this process is extensive in defining methodology and documentation requirements and should be tailored to meet the needs of each program. â Office of Aerospace Studies Analysis of Alternatives (AoA), 2003, Guidance in Support of AFI 10- 60 (revised September 22, 2003), Kirtland Air Force Base, N.Mex.: Office of Aerospace Studies.
78 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING 2. Develop a CONOPS. At this stage, the CONOPS is a top-level description of how a system and its operators and users will interact to produce the required capability, including operation in a system of systems. At ÂMilestone A, the CONOPS needs to be sufficient to ensure that the Âchosen concept is capable of operating to produce the desired outcomes and time lines. A key precursor of a CONOPS is often a description of the other systems with which the system of interest will be interacting and an understanding of what the user expects the system to do in its operating environment. 3. System performance assessment. This is the analytical process of pre- dicting the performance of the system concepts evaluated. It needs to be sufficient to predict the ultimate performance of the system in operation. Often system performance models developed in this phase become the basis for supporting the governmentâs management of the development phase. The following guidelines are important to consider: â¢ The models must typically also provide the basis for setting segment performance requirements and assessing the impact of deviations from expected performance after Milestone A. â¢ Modeling and simulation often constitute a powerful tool, starting with the evaluation of concepts and moving on through the entire develop- ment phase and even into the operational phase. While this discipline may not always begin in the pre-Milestone A phase, models begun in this phase can add rigor to the assessment of concepts as well as being powerful management tools in subsequent phases. 4. System architecture development. A well-known approach to addressing a complex problem is to break it into parts that can be addressed separately. Architecture here refers to the partitioning of the system into separately definable and procurable parts, the structuring of interfaces between the system and the outside world, and the structuring of interfaces (physical, functional, and data) among the segments. Through careful partitioning, architecture can minimize complexity and thereby development risk. The following guidelines are important to remember in this phase: â¢ Seek independence of the segments. Good architecture seeks to achieve segments that can be developed and tested separately to reduce the complexity of the development phase. â¢ Simplify interfaces. In selecting the architecture, seek to minimize the entropy of the implementation by making the interfaces among the segments as simple as possible. â¢ Plan for testability. It is important that the system designer consider the testability of the parts alone, of the system as a whole, and of any system of systems implications as early as possible to minimize the impact of undiscovered issues later. â¢ Plan for integration. The system designer must have a vision of how the parts can be integrated and tested to verify end-to-end performance
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 79 in the operating environment. Modeling and simulation can often play a major role in the overall test and verification plan. 5. Risk assessment. A pre-Milestone A goal is to identify the key risk Âdrivers to set the stage for effective risk management in the procurement and development phases. The six key risk drivers identified in the next section are areas to focus on in risk assessment. High-risk areas may be found not only in technical areas, but also in the availability of critical human resources, stable funding, industrial base capacity, and so on. 6. Cost scoping. The systems engineering process in the pre-Milestone B phases normally includes very rough estimates of a systemâs development and operating costs sufficient to determine if the concept and CONOPS are consistent with expected budgetary constraints. The system designers should try to avoid the low bias in cost estimating often introduced by the desire of the government and potential contractors to sell the pro- gram. This is usually best accomplished, in this early phase, by top-level comparisons on a segment-by-segment basis with actual cost experi- ence on recent programs. For revolutionary concepts, less-well-grounded techniques may be unavoidable. Models developed in this phase may be matured later to support âshould costsâ in evaluating contractor bids. Outputs The output categories are described briefly as follows. 1. Definition of key performance parameters. This is the primary output of systems engineering in the pre-Milestone A period. These KPPs will drive the subsequent procurement and development stages of the program. It is important that they are defined concisely and specifically. They should be few in number, but specific in describing the primary capabilities desired. They should be sufficient to provide the basis from which all lower-level requirements are derived. Ideally, they are defined in terms that are under- standable to the users of the system rather than in highly complex techni- cal language. In the committeeâs experience, the failure of the designers to define the systemâs KPPs simply and clearly at Milestone A and the top-level requirements at Milestone B is the first step to requirements instability and overruns later. The committee believes that the top-level KPPs and system require- ments can be the basis for all lower-level requirements. Committee mem- bers have had experiences with complex, multisegmented systems where a half-dozen top-level requirements drove all lower-level requirements. Often, system availability, maintainability, and/or reliability requirements provide the basis for important derived requirements that are not covered by other capability requirements. The imposition of this discipline can
80 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING TABLE 4-1â Summary of Pre-Milestone A Systems Engineering Function and Output Guidelines Function/Output Summary Guideline System concept â¢ Complexity should be minimized both within the system and with development regard to the systemâs external interfaces. â¢ The use of high-risk, immature technologies should be avoided. â¢ Favor concepts that can achieve initial operational capability (IOC) in fewer than about 5 years. â¢ At least two alternative concepts should be evaluated before selecting a final concept. System â¢ Partition to achieve segments that can be procured and tested separately. architecture â¢ Minimize the complexity of the interfaces among the segments. â¢ Establish data, structure, and architecture standards where appropriate. â¢ Maintain the independence of the functional requirements. Concept of CONOPS needs to be developed sufficiently to ensure that the concept can: operations â¢ Perform in the environment in which it will operate, (CONOPS) â¢ Handle the expected throughput, and â¢ Meet response time requirements. Performance Performance analysis should be sufficient to: analysis â¢ Predict performance against mission needs, â¢ Assess the impact of segment-level performance on end-to-end performance, and â¢ Include the development of system performance models to support performance assessment in the acquisition phase. Top-level Requirements at this stage should: requirements â¢ Be broad and few in number, generation â¢ Drive the derivation of all lower-level requirements on the program, and â¢ Remain largely unchanged through the first development cycle and the achievement of IOC. Risk Risk identification should: identification â¢ Identify the top-level risk factors that are inherent in the concept, architecture, and CONOPS. Cost estimates Cost estimates should: â¢ Provide a high level of confidence that the concept and CONOPS are consistent with budgetary constraints, and â¢ If necessary, facilitate the development of a cost model that can be extended later to support âshould costsâ for contractor-proposed solutions. Key KPPs at this stage should be: performance â¢ Broad and few in number; parameters â¢ Comprehensive and clearly and simply defined covering required (KPPs) capabilities, availability, and reliability; â¢ Sufficiently complete to be the source of all lower-level requirements; and â¢ Sufficiently mature that little change is needed after Milestone B, and prior to IOC.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 81 have a powerful effect in protecting against requirements creep driven by well-meaning specialists who want to insert their favorite processes and technologies into the program. The Department of Defense (DOD) Guide for Integrating Systems Engineering into DOD Acquisition Contracts, Version 1.0 has this to say on the importance of getting the requirements right: Sound system requirements (including performance) are the backbone of a good technical strategy and resultant plan (as documented in the SEP and related plans). The performance requirements, as a minimum, must be commensurate with satisfying the threshold for the critical operational (including sustainment and support) requirements (e.g., Key Performance Parameters (KPPs)) and balanced with program cost, schedule, and risk constraints. If these elements are not balanced at the start of the SDD phase, the program has a high probability of incurring cost increases, suffering schedule delays, and/or deficient performance of the end product. 2. CONOPS. A draft CONOPS is needed to show that the system can be operated to meet the usersâ objectives. 3. Supporting models. Models to validate system performance and cost projections in phases A and B are important to support the governmentâs management of subsequent stages. 4. Program implementation strategy. The final output from a typical M Â ilestone A activity should be a strategy for implementing the program going forward. This typically would include a plan and time line for achieving Milestone B, an approach to mitigating the most critical iden- tified risks, and an approach to establishing cost projection credibility before Milestone B. Table 4-1 summarizes some of the committeeâs observations on the key functions and outputs. Six Drivers of Cost, Development Time, and Performance Risk that are Addressable by Systems Engineering Processes The committee heard wide-ranging views from its guest speakers, members, and others about the twofold-or-greater growth in program cost and completion time over the past 30 years. The committee believes that understanding the real drivers of this large increase in development times and costs can help identify â Department of Defense, 2006, Guide for Integrating Systems Engineering into DOD Acquisition Contracts, Version 1.0, December 11. Available at http://www.acq.osd.mil/se/publications.htm. Last accessed on June 20, 2007.
82 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING 1. Inexperienced Leadership 2. External 3. System 4. Incomplete or Interface Complexity Unstable Complexity Requirements at 5. Reliance on Immature Milestone B Technology 6. Reliance on Large Amounts of New Software FIGURE 4-2â The âsix seeds of failureâ to avoid or manage during the pre-Milestone A and B phase. the role that early applications of systems engineering processes could play in reversing this trend. The committeeâs approach was 4-2to harness its collective experience along with the lessons derived from the case studies in Chapter 2. It also tested hypothetical drivers of the problem by comparing current practices with those that existed in the past when some of the most remarkable program successes were achieved in fewer than 5 years from Milestone B. While this was at best an anecdotal analysis, the committee examined the myriad factors that may have contributed to serious development issues and identified six factors that are pervasive sources of poor performance and that are addressable through sound systems engineer- ing processes. The committee calls these factors, illustrated in Figure 4-2, the six seeds of failure. They are inexperienced leadership; external interface complexity; system complexity; incomplete or unstable requirements at Milestone B; reliance on immature technology; and reliance on large amounts of new software. A brief description of the rationale behind each of these factors follows. Inexperienced Leadership Perhaps the biggest risk of all in undertaking large development programs is to proceed with less than the best personnel, particularly in the key leadership positions in government and industry. High-quality program managers and system engineering leaders, in particular, are critical. High aptitude and extensive experi- ence, combining to create high domain knowledge, are required for individuals to be fully effective in these positions. In evaluating program management and system engineering experience, one should consider both the length of experience and the number of programs that the candidates have participated in. Experience
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 83 in at least two major development environments can be very valuable. Since not every program can have the best, it is important for government executives to assign the best to the most difficult problems. Programs that exhibit the highest levels of the risk factors described here, particularly high complexity, are good candidates for the best leadership team. Pre-Milestone A Mitigation For programs with high levels of external interface complexity and internal technical complexity, the government should seek the highest-aptitude, most- experienced people it can muster for the program manager and system engineer- ing manager roles going forward into the Pre-Milestone B phase, and it should keep these people in place through IOC whenever possible. Proven strength and the ability to minimize complexity and control changes after Milestone B through IOC in the face of unrelenting pressure both from inside the program team and from external users and sponsors, are important. External Interface Complexity One characteristic of very complex system developments during World War II and the early Cold War years was the simplicity and urgency of the needs and missions. Beating the Germans to the atomic bomb, the Russians to the Moon, penetrating the Iron Curtain, and so on, provided clear, urgent goals that galva- nized the sponsors of the complex systems and focused and empowered the gov- ernment and contractor teams. Such clear, driving missions, and the simple user interfaces that they required, allowed the program team to develop its concepts quickly and to keep the top-level requirements stable until IOC. In the post-Cold War era, the immediacy of the threats often seems less apparent, and programs often try to serve many missions and users with a single system, or system of systems. The interaction of multiple systems that were not designed together (e.g., military satellite communications [MILSATCOM], see Chapter 2), often termed âsystems of systems,â also can greatly increase the dif- ficulty of creating a stable requirements base for a new system. The concept of network-centric operations, in particular, can introduce external complexity. The complex processes necessary to coordinate these communities of interest seem to have hidden costs that can add many years to the development cycle and lead to substantial budget overruns. In addition, systems dependent on highly complex external interfaces can be far more difficult to operate after deployment. Pre-Milestone A Mitigation Systems engineering should treat the minimization of external interface complexity as a key driver in selecting concepts and architecture. Simplifying and
84 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING standardizing the ways that external users access the system and seeking to mini- mize the degree to which the systemâs capabilities must be tailored specifically for individual users can help. Program managers may also be able to minimize the pressure for pre-IOC changes from the external user community by striving to create a sense of time urgency to aid in stabilizing requirements. System Complexity The flexibility and capability enabled by advances in electronics technol- ogy and software provide systems designers with far more options than their p Â redecessors enjoyed 30 years ago. The downside is that these new capabili- ties can tempt designers into unnecessarily complex concepts and designs that impose a âcost of internal complexityâ similar to the external complexity costs described above. In particular, there can be a tendency to assign poorly-thought- out functions and options to the software. Often, late-arriving requirements are implemented in software because the perceived schedule risk of redesigning hardware is unacceptable. Pre-Milestone A Mitigation Systems engineering should treat complexity minimization as a key driver in selecting concepts and architecture. Architecture selection can have a powerful effect on controlling complexity. A proven approach to solving large, complex problems is to partition the problem into smaller pieces that can be addressed separately. Partitioning the system to create parts that can be separately developed and tested and that minimize the complexity of internal and external interfaces can be very important. Incomplete or Unstable Requirements at Milestone B It is not unusual for programs to proceed beyond Milestone A with unsettled KPPs and beyond Milestone B with unsettled top-level requirements. This can be a major schedule and cost driver, particularly after Milestone B. As the missions and user communities have become more complex, the government is finding it much more difficult to resolve competing views of system concepts and performance requirements. As a result, it is often tempting to go through the process of selecting the development contractor team before resolving these issues. The well-meaning goal is to get the contractor teamâs help in resolving these issues. The result, how- ever, is often that large development teams start without sufficient direction, and the contractor teams with their high âburn ratesâ waste time and money participat- ing in âwhat-ifâ requirements debates that do not converge quickly. Another driver of unstable requirements is funding instability. This phenom- enon, familiar to all, is a result of political processes, not systems engineering.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 85 But its effects on requirements stability, cost, and schedule are significant and insidious. A committee member told of his experience with the Defense Satellite Communications System (DSCS) Program and how the rate of design and pro- duction accelerated dramatically, and how the costs dropped significantly below expectations when a fixed-price, multiyear procurement contract was authorized by the Congress and, in effect, froze funding and changes to the requirements. Pre-Milestone A Mitigation Federally funded research and development centers (FFRDCs), systems engi- neering and technical assistance (SETA) contractors, and small study contracts to potential development contractors can all be used to resolve concept and require- ments issues prior to Milestone A. It is important not to proceed with the develop- ment phase until the top-level concept and requirements are finalized. Also, tight partnership and collaboration between the users/sponsors and the developer are critical to stabilize requirements in the formative phase ofÂ a program. Reliance on Immature Technology The use of unproven technology in large system developments can intro- duce a high risk of schedule and cost growth. The committee has seen examples of large procurements in which high-risk technology was seen as a part of the rationale for justifying a program. The committee believes that the use of mature technology should be a prime goal in concept and architecture selection. When the concept and architecture selections identify technologies that are risky, alter- native concepts should be found, or the program strategy must accept that the system must wait for the technology to be demonstrated successfully before entering system design and development. Pre-Milestone A Mitigation The system designer should seek to avoid the use of new, unproven technol- ogy in large development programs with important missions. The committee believes that technology development should be conducted separately from large operational program developments if possible. As an alternative to pursuing such technology in parallel with post-Milestone B development, the systems engineers might consider making provisions in the architecture and systems design at Milestone B to allow subsequent technology â Lt Gen Ronald Kadish (USAF, ret.), Booz Allen Hamilton, âDefense Acquisition Performance Assessment: Report Summary Briefing,â presentation to the committee, February 28, 2007. â National Research Council, 1993, An Examination of the Air Forceâs Pre-Milestone One ÂPlanning/ Decision Process, Washington, D.C.: National Academy Press.
86 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING insertions of expected new technology, if such flexibility can be accomplished without adding substantial design complexity or requirements instability. Reliance on Large Amounts of New Software Large, complex software elements have been the source of high costs and long development times on many programs. The committee believes that most often this is not due to the difficulty of the software design, but rather reflects an inadequate definition of the software requirements prior to initiating the software development effort. Poorly-thought-out and ambiguous requirements seem to be an even greater addiction in software than in systems as a whole. A common view is that âafter all, we can always change it later.â The tendency to accept far too much complexity in software requirements is also a common problem. Pre-Milestone A Mitigation Perhaps the most important actions that the systems engineers can take to minimize the risk of software disasters are to get the software functional alloca- tions clearly defined at Milestone B and to constrain the software such that the software element can be delivered within about 18 to 24 months or less. In 2000, the Defense Science Board stressed the following guidelines for software pro- gram structure that the committee believes are worthy of attention: â¢ Aggressively limit development time to no more than 18 months; â¢ Minimize complexity; â¢ Highly incentivize development; â¢ Allow program management to trade functionality for time and stability; â¢ Have good processes, but value past performance over process; â¢ Use an iterative, not a waterfall, development process; and â¢ Develop an executable architecture first. Contrary to intuition, the integration of commercial off-the-shelf (COTS) products often results in high technical risk in DOD acquisitions. COTS integra- tion needs to be planned and assessed in the overall system context. Performance issues in the context of the overall system architecture need to be taken into account. Any modification to COTS software usually implies life cycle mainte- nance of the entire COTS product. Integration issues are often more challeng- ing than in commercial applications. With reference to the âPre-Milestone A/B Checklistâ (Box 4-1, which appears below in the section entitled âPre-Milestone A/B Checklistâ), early consideration should be given to COTS risks. Sources of â Defense Science Board (DSB), 2000, Task Force on Defense Software, November, Washington, D.C.: OUSD (AT&L). Available at http://www.acq.osd.mil/dsb/reports/defensesoftware.pdf. Last accessed on June 27, 2007.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 87 useful information tailored to the acquisition community, including known risks, publications, case studies, courses, and a risk assessment tool called the COTS Utilization Risk Evaluation, are maintained by the Software Engineering Institute (see www.sei.cmu.edu/cbs). OTHER Possible PERFORMANCE DRIVERS In addition to the six seeds of failure discussed above, committee members and others discussed a number of other factors as potential sources of the decline in development productivity over the years. Four of these were most frequently mentioned: diminished talent in the government, diminished talent in industry, excessive focus on cost and schedule to the detriment of performance, and excessive oversight. While it may be correct to identify each of these factors (in particular the diminished talent and domain experience of key personnel), the committee believes that the others may be less significant than many believe and that they are certainly not excuses for poor performance or reasons not to look for solutions in good systems engineering and program management. Table 4-2 briefly summarizes committee observations on these often-cited issues. TABLE 4-2â Other Cited Sources of Development Productivity Decline Source of Decline Committee Observation Diminished talent in It is not clear that the talent in the government is less capable government to manage than it was 30 years ago. The âgood old daysâ may not have acquisitions been as good as we think they were. In fact, in some ways the depth of talent today is greater owing to the entrance of women and minorities into the available talent pool, and the much larger and more experienced government-industry talent pool to draw from. But there are issues involving the number and the domain experience of available personnel in government. Diminished talent in Same observation as above. industry Focus on cost and The committee believes that programs benefit from strong schedule to the detriment commitment to cost and schedule, as well as to performance of performance and supportability. Excessive oversight This may be the most important of these issues. The committee believes that the most insidious effect of excessive oversight may be its contribution to requirements and funding instability between Milestone B and initial operational capability. Another negative effect is the tendency of high-level external âoversightâ to defocus the management team from the development job and direct its energy instead to providing air cover for the program.
88 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Obsolete and NonRelevant Systems Engineering Processes At least one major prime contractor known to the committee has decided to eliminate the term âsystems engineeringâ altogether after finding that many of the accumulated documented processes in government, academia, and industry are useless. Processes can be a valuable tool to capture the lessons of the past. However, in the committeeâs experience, the accumulation of such processes over many years, largely implemented to address one specific development problem or another, can lead to programs being driven by rules that are irrelevant, obsolete, or excessive for the real-world job. Process requirements generated by well-meaning people who do their work without the benefit of real program experience can also lead to non-value-added work. The adverse effects of obsolete and nonrelevant process requirements can be minimized by allowing systems engineering and program management the leeway to tailor compliance with required processes to suit the needs of each specific program. General PolicIes and Best Practices for Systems Engineering IN ALL PHASES The previous sections of this chapter concern guidelines and practices for program-specific systems engineering efforts during pre-Milestone A and B phases. This section addresses some of the most important general guidelines and practices for personnel and organizations responsible for systems engineering policy, methods, and tools that apply to the entire program life cycle. The prime contractor assumes much of the responsibility for systems engi- neering after Milestone B, with the government involved in oversight, review, and approval. Some of the pre-Milestone A functions are complete (such as concept development). Others, such as architecture and CONOPS development, move to a more detailed level. New functions come into play, such as change control, configuration control, interface definition and management, and detailed modeling and simulation of operational scenarios. Avoiding requirements creep as low-level requirements are developed by the contractors is an important gov- ernment systems engineering role after Milestone A. The following paragraphs discuss the committeeâs views on several key systems engineering functions after Milestone B. Modeling and Simulation Modeling and simulation are major components of the systems engineering process throughout the life cycle of a system. As a system moves through the acquisition cycle, modeling and simulation transition from more aggregated and general forms to higher-resolution and higher-fidelity forms. Early in the cycle, constructive models tend to be prevalent, while later the activity shifts toward
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 89 human-in-the-loop and mixed-mode simulation to support operational testing and evaluation. As a system moves into operational use, live rehearsal may become the most prevalent form in use. It is now possible to represent varying portions of a system with simula- tions throughout the development, integration, and testing processes even before real hardware and software become available, making it unnecessary to wait for the completion of development before any substantial integration testingÂ can be done. During the early stages of planning for a new program, management should recognize the need to acquire or develop appropriate data necessary to enable the timely use of modeling and simulation prior to Milestone A. Another challenge that must be addressed is that of maintaining consistency across the different levels of modeling and simulation aggregation and fidelity. Moving from the low- to medium- to high-fidelity models, the outputs of one stage will generally serve to set performance thresholds for the next stage, helping to define what constitutes a successful concept or design. The potential bidders for the systems to be acquired will also play a significant role in defining the simulation environment and by building contractor-Âspecific models expressing both the behavior and performance of their concepts. At each level of aggregation and for each iteration of the models, there will be systems engineering tasks to validate both the models and the environment, ensuring that the laws of physics are taken into account and that behaviors are both consistent and explicable. Modeling, simulation, and analysis will play a major role in identifying key design parametersâthe âlong poles in the tentââwhich are associated with potential risk, sensitivity, or uncertainty and on which the ability of the design to meet important requirements will depend. Whether the parameter is processing throughput or response time or the acquisition cost per removal-free operating hour, early identification provides guidance to the designers and focus to risk- mitigation planning activities such as early benchmarking or development of alternatives. Many program managers fail to recognize the return-on-investment of plan- ning for and using modeling and simulation up front in the systems engineering process, and therefore they reduce or eliminate the budget for modeling and simulation from their overall budgets. A relatively minor up-front investment in modeling and simulation can actually reduce significantly the cost of the develop- ment program by identifying problems early and reducing both the time and the cost of full-scale testing. Modeling and simulation have become ever more central to the development of modern systems. Unprecedented advances in digital processing have made high-fidelity representation of systems and subsystems in computer models pos- sible from the simplest of systems to the most complex. This has made it possible to examine the projected performance of systems over wide excursions of design
90 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING and environmental assumptions very early in the development process, even prior to Milestone A. Todayâs modeling and simulation tools make it possible to per- form extensive system of systems simulations and evaluate alternate architectures at affordable cost and early enough to make a difference. Systems of Systems The committee had numerous discussions on âsystems of systemsâ (SoS) and found the term ill-defined and overused. However, there are some important issues that arise when a system designed to provide a primary capability is used in conjunction with other systems to provide a different capability. The definition of SoS that the committee adopted is as follows: systems of systems are groups of systems, each of which individually provides its own mis- sion capability, that can be operated collectively to achieve an independent, and usually larger, common mission capability. Often systems of systems are not initially developed together but rather are formed by operating systems, initially developed separately into a SoS to achieve a larger objective. The Armyâs Future Combat Systems (FCS) Program is an example of a SoS in which the constituent systems are being procured together. This SoS consists of platforms, weapons, surveillance systems, and so on that can operate separately or together in combat. While the design of systems of systems may impose somewhat different engi- neering processes, the committee believes that, in general, most of the elements of good systems engineering apply to systems of systems as well, and the committee has not attempted to differentiate between them in this report. However, these types of system aggregations increase the complexity of external interfaces that are difficult for a development program office to define and control. As pointed out earlier, high levels of external interface complexity constitute one of the six seeds of failure and deserve specific attention. Thus the use of memorandums of understanding and other cross-organization means of communication and control are particularly important in these types of programs. System Dynamic Modeling System dynamics seeks to understand, through qualitative and quantitative models, the time-dependent behavior of managed systems and how information feedback governs their behavior. System dynamics also seeks to design robust information feedback structures and control policies through simulation and optimization. System dynamic modeling has the potential to allow development program performance to be predicted with far greater accuracy by modeling interactions â K.G. Cooper, 1994, âThe $2,000 Hour: How Managers Influence Project Performance Through the Rework Cycle,â Project Management Journal 15(1):11-24.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 91 among internal and external social and environmental factors. There is evidence that it is possible to model a program based on its organization, management poli- cies, staffing, experience level by skill, productivity, overtime, attrition, morale, hiring, training, requirements changes, quality, required rework, and so on, and to calibrate the organizationâs processes with historical data. The resulting system dynamics model could be used to help establish appropriate policies and guide program management to insightful decisions that affect program success. While the committee did not hear any accounts of the successful use of system Âdynamics modeling on government development programs, the committee believes that this discipline deserves further evaluation as a systems engineering and program management support tool. While system dynamics modeling is relevant to both the system being developed and the development process itself, there are many equally good (some would argue substantially better) modeling methods that can deal with the p Â roduct being acquired. Discrete time simulation lends itself to modeling systems governed by physics, and discrete event simulations tend to be used for systems governed by statistical processes. System dynamics seems uniquely relevant to pre-Milestone A activities in modeling the process of acquisition. The challenge during the pre-Milestone A period is to capture the potential interactions between the organizational, political, economic, and policy components of the acquisition system as much as it is to capture the interactions between the components of the system to be acquired. System dynamics models might provide great value, and their power to stimulate insight might lead to significant advances in the practice of systems acquisition. Testing and Evaluation The discussion of modeling and simulation above can be expanded to include the total testing and evaluation (T&E) function. Testing and evaluation could and should be used to validate key outputs of pre-Milestone A systems engineering. This requires coordination between the planning and systems engineering com- munity and the T&E community, to ensure harmony and linkage between what is tested and validated and what is defined in the front end of programs. Indeed, one of the best forcing functions to get people to agree on what is expected is to get detailed agreement on the tests that, if passed, demonstrate a successful completion. Taking the position that no requirement exists until the test that it must meet has been defined is a valuable tool in managing successful projects, in the committeeâs experience. Also, the smart linkage between the planning and testing communities, including the wise application of modeling and simulation, can actually reduce the need for some costly, time-consuming, full-scale testing. â Professor Jay W. Forrester of the Massachusetts Institute of Technology has been a prominent researcher in this area. Further discussion of the topic can be found at http://sysdyn.clexchange. org/sd-intro/home.html. Last accessed on November 20, 2007.
92 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING The committee believes that a review of how T&E is incorporated in the develop- ment planning process, how modeling and simulation can be used effectively in T&E, and how T&E supports program validation can be of immense value. Cost and Schedule Performance Estimating Large cost and schedule overruns on development programs have become the norm, and they continue to seriously undermine confidence in the acquisition process, stress the nationâs resources, and diminish the capability of the nation to respond quickly to new threats. Cost and schedule results are heavily impacted by SE decisions in the pre-Milestone B phases, and subsequent performance cost and schedule efficiency can be significantly affected by the application of good SE discipline during the development phase. The two, cost and schedule performance, are almost always inextricably linked. That is, the causes of cost deviation affect schedule, and vice versa. These causes can be discussed in three groups: (1) those that are the result of program execution performance, (2) those that result from technical estimating uncer- tainty, and (3) those that result from competition-driven biases in the initial cost and schedule estimates. By âexecution performance,â the committee refers to how closely ultimate cost and schedule performance mirror the cost and schedule performance obtain- able by an optimally managed program. Since optimum management on a devel- opment program is not possible, the variances here are always negative (overruns) that must be offset by recognizing this fact in the initial estimates. This report has attempted to identify guidelines for systems engineering on development programs that can enable the programs to be executed as closely as possible to this theoretical optimum. These guidelines are summarized in this chapter and in the committeeâs checklist (see the section below titled âPre- M Â ilestone A/B Checklistâ). Elsewhere in the report, the committee also addresses Air Force-wide issuesâsuch as the training of system engineers and certain Air Force organizational issues covered in the committeeâs recommendationsâthat can better enable these guidelines to be executed on all programs. In comparing todayâs program performance to that of similar programs of years ago, the com- mittee examined the actual Milestone B-to-IOC times rather than performance- to-budget, so that it would be comparing the effects of program execution, not the estimating errors discussed below. By âtechnical estimating uncertainty,â the committee means the uncertainty about exactly what must be done, how long a development will take, and how much it will cost, given the fact that the job has not been done before. One might assume that this uncertainty would have a zero mean, that is, that error would sometimes be too high and sometimes too low. But in real life, the committeeâs experience is that this uncertainty almost always leads to a low estimate of the cost and schedule required. This may be because it is unconsciously assumed that
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 93 the program execution will be optimum and that all of the tasks to be done have been identified. In real life, execution can never be optimum, and at the outset there is always a failure to identify a significant fraction of the work needed. Good program managers know this and build contingencies into their cost and schedule estimates. By âcompetition-driven biases,â the committee means the desire of govern- ment and industry supporters of a program in the pre-Milestone A and B phases to sell the program to the executive and legislative branch budget officials and of the competing contractorsâ desire to win. The competing contractors, for what are usually cost-plus contracts, are highly incentivized to bid the âlowest cred- ible costâ to winâas one said, âItâs better to be on contract and underbid than be on the street.â A now-retired senior aerospace industry executive is reported to have told one of his losing teams, âThere is no excuse for losing a cost-plus procurement on cost.â The result of governmentâs and industryâs desire to sell their programs and contractorsâ desire to win creates a âconspiracy of hopeââan expression given some prominence by the Defense Acquisition Performance Assessment (DAPA) panel. It should be noted that the estimating errors created by the second and third issues above cause a program to be planned with cost and schedule objectives that are not achievable, and that cause significant energy to be expended by gov- ernment and contractor management teams as the variances are recognized and plans painfully redone. The systems engineering process can play a role in mitigating these factors by developing cost projection models during the pre-Milestone A and B phases. These models seem to work best when they are rooted in real-life cost experience on similar developments, with adjustment made for differences in complexity and novelty in the programs being compared. Success-oriented schedules, with insufficient time and budget margins for the unknown development problems that are sure to come, are a classic attribute of underestimated programs. The committee believes that systems engineering and program management on every large development program should develop a cost model and discipline late in the pre-Milestone A phase and refine the model through Milestone B. Requirements Creep and Requirements Traceability Matrices After Milestone A and to a greater degree after Milestone B, the contractor, with help from government overseers, will be developing detailed requirements at the segment, subsystem, and component levels. Systems engineering in govern- â Ronald Kadish, Gerald Abbott, Frank Cappuccio, Richard Hawley, Paul Kern, and Donald K Â ozlowski, 2006, Defense Acquisition Performance Assessment. Available at http://www.acq.osd. mil/dapaproject/documents/DAPA-Report-web/DAPA-Report-web-feb21.pdf. Last accessed on AprilÂ 2, 2007.
94 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING ment and industry must control this process so that each lower-level requirement is directly responsive to top-level system requirements or KPPs. There is a strong tendency for subsystem and component engineers to expand the cost and development difficulty by writing specifications based on the âbest the supplier can do,â or on personal preference, rather than on what the system needs. This ratcheting up of requirements can have unexpected cost and schedule impacts. Requirements traceability matrices can be useful tools for making sure that each lower-level requirement can be justified by a system-level require- ment. This helps enforce the discipline that all lower-level requirements must be t Âracable to one of a few well-thought-out KPPs and top-level system requirements discussed earlier in this chapter. Change Control and Configuration Management Change control becomes a very important activity after Milestone B. It plays a critical role in guarding against the top-down and bottom-up requirements creep and requirement instability, discussed above with respect to the six seeds of fail- ure. During the pre-Milestone A period, the KPPs of the system (or system of sys- tems) should be established and agreed to by the user/sponsor, the developer, and senior management authority. These KPPs should be placed under tight change control at Milestone A. Thereafter, as the program progresses through Milestone B and beyond, more detailed requirements and specifications will be developed, all linked and traceable to the KPPs.Â The government or its designated integrator needs to maintain control of the top-level requirements and exercise a heavy bias in the direction of minimizing changes before IOC, even if it means deferring great new ideas that can be put in the initial deployment; this needs to be done because the cost of such changes can be hidden and unappreciated even by the contractors who are asked to provide estimates for them. In its oversight role, the government should also ensure that the contractors are controlling lower-level, derived requirements, so that only requirements and requirements changes needed to meet the top-level end objectives are considered prior to IOC. Intersystem and Intersegment Interface Management Another key role for the systems engineers after Milestone B is to ensure that the interfaces among the segments and among the system and its users are clearly defined early in the design phase. A consequence of good architecture that parti- tions a system into separately procurable segments is that the interfaces among them need to be defined early and precisely, as they become key design drivers. There will always be a tendency for the segment designers themselves to want to define these interfaces late in the design phase when their designs are complete and the needed interface support is obvious. The problem with this approach is that it will invariably require the designers of interfacing segments to redesign
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 95 with adverse schedule and cost impacts. The development of standards for data passed across interfaces, including data protocols, units of measure, coordinate systems, and so on, can be valuable in managing interfaces. The establishment of communities of interest across the interface boundaries for specific issues, such as data semantics, can also be useful. Sharing Best Practices with Other Agencies This study was requested by the Air Force; however, the committeeâs conclu- sions and recommendations apply to any acquisition program within the entire Department of Defense, regardless of the military department or service affilia- tion. Likewise, there are major acquisition and systems development programs in other government agencies that could also benefit from the recommendations of this study. For instance, the National Aeronautics and Space Administration (NASA) has embarked on the development of numerous new programs to accom- plish the goals of the new U.S. space exploration policy to return to the Moon and travel to Mars and beyond. This new policy requires new space-launch systems, a replacement for the space shuttle, robotic technologies, and so on. Many of the required developments will be systems whose complexity will rival, if not exceed, the complexities and challenges of even the most critical and largest national security defense programs. NASAâs companion to the DOD acquisition process has elements similar to those described in Chapter 1. A good systems engineer- ing process that includes up-front development planning is just as beneficial to NASA as it is to DOD. The Federal Aviation Administration (FAA) also faces the development of major new systems in the future to meet the goals of the Next Generation Air Transportation System. This program requires the improvement of every node of the air transportation system (from the curbside origination of a traveler to the curbside destination of that traveler) to meet the demands of a future air system where superlarge airliners and very small air-taxis may mix with unmanned aerial vehicles. Again, the complexities of these developments beg for a good systems engineering process with good development planning. In the cases of both NASA and the FAA, these developments will not occur in isolation within the specific agency. Instead, many of these efforts will be made with the cooperation of the DOD. Thus, cooperation in sharing good processes will not be a luxury. It will be a necessity. â Federal Aviation Administration, 2007, Online Fact Sheet: Next Generation Air Transportation System 2006 Progress Report, March 14. Available at http://www.faa.gov/news/fact_sheets/news_ story.cfm?newsId=8336. Last accessed on May 17, 2007.
96 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Pre-Milestone A/B Checklist A summary of some of the most important outputs of the systems engineer- ing process is presented here in the form of a checklist. Box 4-1 lists the outputs that a program or systems engineering manager might use in evaluating the completeness of the effort in each of the systems engineering functional areas described earlier. This checklist applies to Milestones A and B. Often a checklist item needed in preliminary form at Milestone A must be completed to a higher level of maturity at Milestone B. The committee notes in Chapter 2 that it believes the Air Force should include Milestone A in more of its development programs. PREVIOUS RELEVANT STUDIES, FINDINGS, AND RECOMMENDATIONS Several previous studies have commented on various aspects of the appli- cation of systems engineering to government acquisition programs. Their key findings and recommendations, which are in good agreement with those of this committee, are summarized below. National Research Council Report In 1993, the National Research Councilâs (NRCâs) Air Force Studies Board (AFSB) completed a study entitled An Examination of the Air Forceâs Pre- Milestone One Planning/Decision Process.10 In essence, that study reached conclusions similar to those of the General Accounting Office (now the Gov- ernment Accountability Office; GAO) study discussed below: namely, that the critical leverage point in any program is matching the customerâs needs with the developerâs resources. The AFSB study concluded: âWhere a strong partnership exists (e.g., as between the Air Force Space Command [AFSPACECOM] and the AFMC) the subsequent transition from user to developer is not a problemâ (p. 2). This statement implies that there must be tight collaboration between user and developer in all pre-Milestone A activities, especially in all systems engineering activities. Defense Science Board and Air Force Scientific Advisory Board Report (Young Panel Report) In August 2002, the Under Secretary of Defense for Acquisition, Technology and Logistics; the Secretary of the Air Force; and the Under Secretary of the Air Force/Director of the National Reconnaissance Office chartered the Defense Sci- ence Board/Air Force Scientific Advisory Board Joint Task Force on Acquisition 10â National Research Council, 1993, An Examination of the Air Forceâs Pre-Milestone One Planning/ Decision Process, Washington, D.C.: National Academy Press.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 97 Box 4-1 Pre-Milestone A/B Checklist Concept Development 1. Have at least two alternative concepts to meet the need been evaluated? The purpose of alternatives is to stimulate thinking to find the simplest, fast- est, and cheapest solution. 2. Can an initial capability be achieved within the time that the key program leaders are expected to remain engaged in their current jobs (normally less than 5 years or so after Milestone B)? If this is not possible for a complex major development program, can critical subsystems, or at least a key subset of them, be demonstrated within that time frame? Achieving capabilities or demonstrating critical subsystems while key program leaders remain engaged is important to get the capability into service quickly and cost-effectively and to begin the process of incremental improvements based on operational experience. 3. Will risky new technology have been matured before Milestone B? If not, is there an adequate risk mitigation plan? The development of risky new technology in parallel with a major development program can be costly in terms of both time and money. 4. Have external interface complexities (including dependencies on other programs) been identified and minimized? Is there a plan to mitigate their risks? Complex, ill-defined, external requirements and interfaces can be a major source of requirements instability during the development phase. This can be of particular importance when a system must operate in a system-of-systems environment. Key Performance Parameters and CONOPS 5. At Milestone A, have the KPPs been identified in clear, comprehensive, con- cise terms that are understandable to the users of the system? It is important that KPPs be expressed in terms understandable to all of the stakeholders. Failure to define the systemâs KPPs simply and clearly at Mile- stone A is a first step to requirements instability and overruns later. continued
98 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Box 4-1 Continued 6. At Milestone B, are the major system-level requirements (including all KPPs) defined sufficiently to provide a stable basis for the development through IOC? Beginning development without a complete list of stable requirements is one of the key âseeds of failureâ described in Chapter 4 in this report. It is important to complete requirements trade-offs prior to the development phase. 7. Has a CONOPS been developed showing that the system can be operated to handle the expected throughput and meet response time requirements? It can be costly to discover too late that the system as designed cannot be operated to meet its requirements. Cost and Schedule Scoping 8. Are the major known cost and schedule drivers and risks explicitly identified, and is there a plan to track and reduce uncertainty? Identifying the major cost and schedule risk areas, with particular attention to this checklist and the six seeds of failureâinexperienced leadership, ex- ternal interface complexity, system complexity, incomplete requirements at Milestone B, immature technology, and high reliance on new softwareâcan help focus management on these issues early. 9. Has the cost confidence level been accepted by the stakeholders for the p Â rogram? It is important that stakeholders understand the degree of risk so that the stakeholders will not disrupt the program as inevitable development program surprises unfold later on. It will generally not be possible by Milestone A or Milestone B to identify all the risk areas that might surface later in a develop- ment program, but a frank, early disclosure of known potentials for risk can help sustain stakeholder support later on. Performance Assessment 10. Is there a sufficient collection of models and an appropriate simulation en- vironment to validate the selected concept and the CONOPS against the KPPs? continued
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 99 Box 4-1 Continued In large, complex programs, the development of models early on can be very important to later management of requirements changes and performance verification. 11. At Milestone B, do the requirements take into account likely future mission growth over the program life cycle? The committee advocates freezing new requirements and new technology insertion after Milestone B but also notes that making provisions in the initial requirements to facilitate later upgrades could have great long-term value. Architecture Development 12. Has the system been partitioned to define segments that can be indepen- dently developed and tested to the greatest degree possible? Effective partitioning of a complex system can greatly reduce its development cost. 13. By Milestone A, is there a plan to have information exchange protocols es- tablished for the whole system and its segments by Milestone B? Such a plan developed early on can greatly reduce interface problems later in the development phase when they would be more difficult and costly to fix. 14. At Milestone B, has the government structured the program plan to ensure that the contractor addresses the decomposition of requirements to hardware and software elements sufficiently early in the development program? The histories of programs with cost and schedule overruns are replete with eÂxamples of large software developments that had to be redone because requireÂments from the hardware side were assigned or determined late. Risk Assessment 15. Have the key risk drivers (not only the technology drivers) been identified? Identifying and managing risk early can pay large dividends; it is important to focus on the six âseeds of failureâ (see item 8 above). continued
100 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Box 4-1 Continued Program Implementation Strategy 16. Does the government have access over the life of the program to the talent required to manage the program? Does it have a strategy over the life of the program for using the best people available in the government, the FFRDCs, and the professional service industry? Seasoned management is critical; the governmentâs job is to find the best! 17. At Milestone A, is there a plan defining how the pre-Milestone B activity will be done, and by whom? Identifying the program and system managers early, identifying the FFRDC or SETA support needed, thinking through the use of competitive system concept contractsâall can have a decisive impact on the governmentâs ability to select the best concept, to define by Milestone B system requirements that can remain stable through IOC, and to select the best development contrac- tors. 18. Is there a top-level plan for how the total system will be integrated and t Âested? A well-thought-out strategy for verifying system performance, including op- timum phasing of verification tests throughout the assembly process, and well-thought-out use of analytical models and external simulators can have a large positive impact on ultimate cost, schedule, and performance. 19. At Milestone B, have sufficiently talented and experienced program and sys- tems engineering managers been identified? Have they been empowered to tailor processes and to enforce requirements stability from Milestone B through IOC? Seasoned leaders in these areas are critical to maintaining focus and disci- pline through IOC. 20. Has the government attempted to align the duration of the program managerâs assignment with key deliverables and milestones in the program? A combination of assignment extension and time-certain milestones will help align incentives. NOTE: KPP, key performance parameter; CONOPS, concept of operations; IOC, initial opera- tional capability; FFRDC, federally funded research and development center; SETA, systems engineering and technical assistance.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 101 of National Security Space Programs to review the acquisition of national security space programs, identify and characterize systemic problems, and recommend improvements. Four key points were made in the Young panel report (the task forceâs final report, frequently referred to as the âYoung panel reportâ because the joint task force was chaired by A. Thomas Young):11 â¢ Cost has replaced mission success as the primary driver in managing acquisi- tion processes, resulting in excessive technical and schedule risk. We must reverse this trend and reestablish mission success as the overarching principle for program acquisition. â¢ The space acquisition system is strongly biased to produce unrealistically low cost estimates throughout the acquisition process. These estimates lead to unrealistic budgets and unexecutable programs. â¢ Government capabilities to lead and manage the acquisition process have seriously eroded. On this count, we strongly recommend that the government address acquisition staffing, reporting integrity, systems engineering capabili- ties, and program manager authority. â¢ While the space industrial base is adequate to support current programs, long-term concerns exist. The chairs of the Defense Science Board (DSB) and Air Force Scientific Advisory Board (AFSAB) emphasized an overall government underappreciation for the importance of appropriately staffed and trained systems engineers for man- aging the technologically demanding and unique aspects of space programs. In the Young panel report, the top five issues related to systems engineering were deemed to be the following: â¢ Lack of awareness of the importance, value, timing, accountability, and organizational structure of systems engineering for programs; â¢ General lack of availability within government and industry of ade- quate, qualified systems engineering resources for allocation to major programs; â¢ Insufficient systems engineering tools and environments to effectively execute systems engineering on programs; â¢ Lack of consistent and effective application of requirements definition, development, and management; and â¢ Poor initial program formulation. 11â Defense Science Board/Air Force Scientific Advisory Board Joint Task Force, 2003, Acquisition of National ÂSecurity Space Programs, Washington, D.C.: OUSD (AT&L). Available at http://www. acq.osd.mil/dsb/reports/space.pdf. Last accessed on April 2, 2007.
102 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING National Defense Industrial Association Report The summary recommendations of the recent report by the National Defense Industrial Association (NDIA) were as follows:12 â¢ Ensure institutionalization of effective systems engineering practices into program planning and execution. â¢ Integrate engineering planning within the acquisition life cycle to ensure adequate time and effort for SE early in the program life cycle. â¢ Emphasize the application of systems engineering practices and resources to the capability definition process to address warfighter needs and translation into executable programs. â¢ Grow systems engineering expertise through training, career incentives, and broadening âsystems thinkingâ into other disciplines. â¢ Strengthen and clarify policy and guidance regarding the use of collaborative environments, models, simulations, and other automated tools. Government Accountability Office Reports Pertinent to this committeeâs discussion of pre-Milestone A SE, the GAO, in its 2003 report Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy,13 stated that the most leveraged decision point in a program is matching the customerâs needs with the developerâs resources. This initial decision sets the stage for the eventual outcome. In successful pro- grams, negotiations and trade-offs occur before product development is started. As an example, GAO pointed out that at the time the decision was made to accelerate the Advanced Extremely High Frequency (AEHF) Program (see the M Â ILSATCOM case summary in Chapter 2), DOD had neither the funding nor the manpower to accomplish the task. The GAO also reported that a primary problem affecting Air Force space programs is that often the users refuse to relax rigid requirements to more closely match technical capabilities that are achievable. For instance, for one element of the Space Based Infrared Systems (SBIRS) Program (see the SBIRS case study in Chapter 2), it became apparent that a lack of knowledge of program challenges led to overly optimistic schedules and budgets. Attempts to stay on schedule by 12â National Defense Industrial Association, 2006, Top Five Systems Engineering Issues Within DepartÂ ment of Defense and Defense Industry. Available at http://www.ndia.org/Content/ÂContentGroups/Â Divisions1/Systems_Engineering/Top_5_Systems_Engineering_Issues.pdf. Last accessed on AprilÂ 2, 2007. 13â Government Accountability Office, 2003, Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy, September. Available at http://www.gao.gov/new.items/ d031073.pdf. Last accessed April 2, 2007.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 103 approving critical milestones without meeting critical criteria led to higher costs and even further schedule erosion.14 The 2005 GAO study Space Acquisitions: Stronger Development Practices and Investment Planning Needed to Address Continuing Problems15 was done at the request of Congress for input on problems relating to DODâs space systems acquisition. To meet this request, the GAO drew on its previous reports related to the causes of acquisition problems, underlying incentives and pressures, and potential solutions. The testimony resulting from the study partially repeated conclusions from the 2003 GAO report listed above. It concluded that programs typically do not achieve a match between requirements and resources at program start. Specifically: â¢ Either requirements are not adequately defined early, or they are changed dramatically once the program is underway. â¢ Technologies are typically not mature enough to be included in product development. â¢ There are deficiencies in the space acquisition workforce, contractor capabilities, and funding available for testing of space technologies. â¢ There is a tendency for programs to take on technology development that should occur in the science and technology (S&T) community. â¢ DOD starts more programs than it can afford in the long run, forcing programs to underestimate cost and overpromise capability. â¢ The most pertinent of GAOâs recommendations was to employ the tech- niques of SE to close gaps between available technologies and customer needs before committing to new product development. Also, GAO recom- mended that DOD develop plans for addressing the shortage of staff with science and engineering backgrounds.16 Defense Acquisition Performance Assessment The major findings of the 2006 Defense Acquisition Performance Assessment (DAPA) study17 were these: â¢ Strategic technology exploitation is a key U.S. advantage. Opportunities need to be identified early. 14â Ibid. 15â Government Accountability Office, 2005, Space Acquisitions: Stronger Development Practices and Investment Planning Needed to Address Continuing Problems, July 12. Available at http://www. gao.gov/new.items/d05891t.pdf. Last accessed April 2, 2007. 16â Ibid. 17â Ronald Kadish, Gerald Abbott, Frank Cappuccio, Richard Hawley, Paul Kern, and Donald K Â ozlowski, 2006, Defense Acquisition Performance Assessment. Available at http://www.acq.osd.mil/ dapaproject/documents/DAPA-Report-web/DAPA-Report-web-feb21.pdf. Last accessed on AprilÂ 2, 2007.
104 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING â¢ The U.S. economic and security environments have changed; for example, there are fewer prime contractors, smaller production runs, reduced plant capacity, fewer programs, unpredictable threats. â¢ The acquisition system must deal with instability of external funding. â¢ The DOD management model is based on a lack of trust. Quantity of review has replaced quality. There is no clear line of responsibility, a Â uthority, or accountability. â¢ Oversight is preferred to accountability. â¢ Oversight is complex, not process- or program-focused (as it should be). â¢ The complexity of the acquisition process increases costs and draws out the schedule. â¢ Incremental improvement applied solely to the âlittle aâ acquisition pro- cess18 requires all processes to be stableâbut they are not. The DAPA report dealt with larger issues of acquisition culture and policy. A member of this committee, who was the chair of the DAPA study, emphasized that a successful response to the instabilities caused by the current process or proper program initiation as envisioned requires early and detailed SE practices. FINDINGS AND RECOMMENDATIONS The considerations discussed in this chapter led the committee to develop the following findings and recommendations: Finding 4-1. Attention to a few critical systems engineering processes and func- tions particularly during preparation for Milestones A and B is essential to ensur- ing that Air Force acquisition programs deliver products on time and on budget. Todayâs weapons systems provide unprecedented capabilities but also involve complex interfaces with external command, control, and communications systems and rely on a greater volume of software than ever before. Early decisions on the weapons system requirements and capabilities have a disproportionately large impact on program cost and schedule. The committee also recognizes that a lack of flexibility (a result of overly rigid processes or a lack of trust among program participants or stakeholders) can limit the ability of a program manager to change early decisions that warrant changing. The committee found many gaps and inconsistencies in the way that the Air Force manages pre-Milestone A activities. The committee heard from Âpresenters of some cases in which required documents were completed pro forma and filed away, never to be seen again, or for which required steps were skipped 18â The AcquisitionââBig Aââsystem is often believed to be a simple construct that efficiently integrates three independent processes: requirements, budgeting, and acquisition. âLittle a,â on the other hand, refers to the acquisition process that focuses on âhow to buyâ in an effort to balance cost, schedule, and performance; it does not include requirements and budgeting.
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 105 completely. The current practice of initiating programs at Milestone B denies the acquisition review authority the earlier opportunity (at Milestone A) to make judgments about the maturity of the technologies on which the program is based and to decide whether technologies need to be further developed prior to making a Milestone B commitment to system development and demonstration. Recommendation 4-1. The Air Force leadership should require that Milestones A and B be treated as critical milestones in every acquisition program and that a checklist such as the âPre-Milestone A/B Checklistâ suggested by the committee (see Box 4-1 in this chapter) be used to judge successful completion. A rigorous, standard checklist of systems engineering issues should be addressed by each program through both the pre-Milestone A and pre-Milestone B phases. The committeeâs recommended 20-item checklist is shown in Box 4â1. While the committee considers that each item on the checklist is important, it calls attention to several items that warrant further discussion: â¢ Checklist item 2 recognizes that the world changes too fast to be friendly to long development cycles. The committee believes that the Air Force should strive to structure major development programs so that initial deployment is achieved within, say, 3 to 7 years. Thirty years ago, this was a typical accomplishmentâfor example, nearly 40 years ago, the Apollo program put the first man on the Moon in fewer than 8 years. â¢ The development time issue is addressable by applying systems engi- neering to items 3, 4, and 13 through 15 before Milestones A and B. The definition of clear KPPs by Milestone A and clear requirements by Milestone B that can remain stable through IOC can be essential to an efficient development phase. It is also important that critical technologies be sufficiently mature prior to starting SDD. The committee observed that although todayâs systems are not necessarily more complex internally than those of 30 years ago, their external complexity often is greater, because todayâs systems are more likely to try to meet many diverse and sometimes contradictory requirements from multiple users. This kind of Âcomplexity can often lead to requirements being changed between ÂMilestone B and IOC, and it can lead to relying on immature technology. â¢ Item 19 of the checklist stresses the importance of placing experienced, domain-knowledgeable managers in key program positions. The com- mittee has observed that many of the truly extraordinary development programs of the past, such as Apollo, the Manhattan Project, the early imaging satellite programs, the U-2, the fleet ballistic missile system, and nuclear submarines, were managed by relatively small (and often immature) agencies with few established processes and controls. In that environment, dedicated managers driven by urgent missions accomplished feats that often seem incredible today. The committee believes that the
106 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING accumulation of processes and controls over the yearsâwell meant, of courseâhas stifled domain-based judgment that is necessary for timely success. Formal SE processes should be tailored to the application. But they cannot replace domain expertise. In connection with Item 19, the committee recommends that the Air Force place great emphasis on putting seasoned, domain-knowledgeable personnel in key positionsâparticu- larly the program manager, the chief system engineer, and the person in charge of ârequirementsââand then empower them to tailor standardized processes and procedures as they feel is necessary. One key pre-Milestone A task is the analysis of alternatives (AoA), which entails evaluating alternative concepts and comparing them in terms of capabili- ties, costs, risks, and so on. Checklist items 1 through 4, 12, and 13 should be completed before the AoA, while items 5 through 11 and 14 through 20 may be addressed after the AoA. Finding 4-2. The Air Force used to have a development planning organization that applied pre-Milestone A systems engineering processes to a number of successful programs, but that organization was allowed to lapse. The role of the Air Force development planning organization, which was within the Air Force Systems Command, was to provide standard evaluation tools and per- form pre-Milestone A systems engineering functions across acquisition programs. The early 1990s saw an erosion of this front-end planning organization along with its funding as the Air Force Systems Command (now the Air Force Materiel Com- mand [AFMC]) began to play a decreasing role in program execution. In the opinion of several speakers who met with the committee, one main reason for the erosion of funding was a lack of congressional support for the planning function. The specific budget âprogram elementâ (PE) for the Air Force planning function was PE 65808. This PE funded a robust planning process in the Air Force for many years. Again, several programs noted in Chapter 2 had their roots in the pre-Milestone A analysis, concept development, and Âprototyping funded by PE 65808 in the past. The funding for PE 65808 began to decline in the early 1990s as Congress started to reduce the funds appropriated. The rationale for this decline is not documented. However, one committee member recalls specific meetings with key congressional staffers who were responsible for approving the funding of PE 65808. At one of these meetings a staffer stated that this PE ultimately leads to new programs that Congress will have to fund in the future, and so if Congress does not fund PE 65808, then the Air Force will not develop new unaffordable systems. Recommendation 4-2. A development planning function should be established in the military departments to coordinate the concept development and refinement phase of all acquisition programs to ensure that the capabilities required by the
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 107 country as a whole are considered and that unifying strategies such as network- centric operations and interoperability are addressed. The Air Force and the other military services should establish a development planning organization like that which existed in the early 1990s. The roles and functions of the various organizations involved in acquiring major weapons systems need to be clearly defined. The responsibility for execut- ing systems engineering and program management in the pre-Milestone A and B phases should be vested in the military departments that do the actual develop- ment planning functions. This should not be the responsibility of the Office of the Secretary of Defense (OSD) or of the Joint Staff. Instead, those offices need to enable the creation and functioning of military department development planning organizations with policy measures, and, where appropriate, resources. The Joint Staff, under the auspices of the Joint Requirements Oversight Council (JROC), may help to define the requirements for major programs in the course of the development planning process, but it should not run the process itself. The existence of âjointâ programs or a program such as Missile Defense, which has several related systems being developed by different military services, requires clear guidance from both OSD and the Joint Staff about who is in charge. These programs need to be harmonized and integrated by the responsible integrat- ing agency. However, development planning activities should still take place in the military departments where the expertise resides. Consequently, the develop- ment planning should be managed by that agency. While this committee cannot predict how Congress will view the revival of a good planning process to support pre-Milestone A program efforts, it is still important for the Air Force and DOD to make the case for the critical importance of this process before Congress and others. A development planning process is important not to start new programs, but rather to ensure that any new program (or a new start of any kind) is initiated with the foundation needed for success. Funding for this planning function needs to be determined by the military ser- vices, including both the acquisition communities and those (the warfighters) who generate the operational requirements. CONCLUDING THOUGHTS Many of the conclusions reached and recommendations made by the commit- tee are similar to those of previous reviews. Most of the past recommendations were never implemented, so one of this committeeâs most critical thoughts relates to the importance of implementation. Successful implementation of these recom- mendations requires the âzipper conceptââmaking connections at all levels, from the senior leadership of the Air Force and DOD down to the working levels within key program management offices and supervisory staffs.