Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 75
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition 4 Systems Engineering Functions and Guidelines INTRODUCTION In this chapter the committee defines the most critical systems engineering (SE) activities in the pre-Milestone A/B phase. Six key functions are discussed, along with four key SE outputs for the Milestone A/B activities. The committee does not attempt to capture every function that may be involved, but instead focuses on the broad functions that have the greatest impact on ultimate system performance. The intent is that the chapter highlight the key functions from the viewpoint of the program manager, not define a “how to” manual for systems engineers. The committee discusses guidelines for managers and systems engineering practitioners in the pre-Milestone A/B phase of a program. In preparing these guidelines, the committee identifies those items that address the causes of the large decline in development program productivity and the attendant cost and schedule growth over the past 30 years, as discussed at the beginning of Chapter 1. These guidelines cover the following topics: The “six seeds of failure”: a discussion of six areas of risk that the committee believes are worthy of particular attention in the pre-Milestone B phases and that are the basis for the recommendations that the committee makes later; Other potential causes of development productivity decline: a discussion of several often-mentioned causes of development productivity decline, including talent diminishment in government and industry, and excessive oversight;
OCR for page 76
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition Obsolete and nonrelevant systems engineering processes: a discussion warning against the use of legacy processes and guidelines that do not add value to the tasks at hand; General processes and practices for systems engineering: a discussion of some of the processes that are critical after Milestone B, including modeling and simulation, control of lower-level requirements, and change control; and A checklist: some of the committee’s most important findings, provided to guide systems engineering activities and output before Milestone B. PRE-MILESTONE A SYSTEMS ENGINEERING FUNCTIONS The prerequisite for starting the systems engineering process is a user-defined need (or outcome). The purpose of systems engineering in this phase is to define an optimal system concept and concept of operations (CONOPS) that satisfies the need and also fits within budgetary constraints. The systems engineering outputs in this phase include a concise statement of key performance parameters (KPPs), a CONOPS description, a program implementation strategy, and models to support performance assessment, cost estimation, and risk management in subsequent phases. These functions are aggregated in the following broad areas, shown in Figure 4-1: concept creation, preliminary CONOPS, performance assessment, architecture development, risk assessment, and cost scoping. These functions are reiterative and thus occur in parallel, to produce the key outputs from the pre-Milestone A systems engineering process: KPPs, CONOPS, supporting models, and program implementation strategy. FIGURE 4-1 Critical pre-Milestone A systems engineering functions and outputs. NOTE: CONOPS, concept of operations.
OCR for page 77
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition Functions The functional elements are discussed briefly below. System concept creation. This is a creative process for selecting a basic approach for a system to achieve the desired outcomes. The following guidelines are important in this phase: Consider alternatives. It is important that at least two alternative concepts be considered before locking in a solution. Consider the "time value of capability.” Seek concepts that can deliver initial capability within about 5 years (measured from Milestone B). Each year that a needed capability is delayed has a cost to those who need it, and delays the availability of operational data and experience to guide subsequent improvements. Development programs that are successful and timely at initial operational capability (IOC) often become platforms on which many other capabilities are subsequently added. The world changes too fast to be friendly to long development cycles. Further, extended delivery times run the risk of the system becoming obsolete before deployment and can be an indication that the concept is excessively complex or excessively dependent on immature technology in its first delivery. Indeed, one insidious effect of long development cycles prior to IOC is that they create the temptation to add new emerging technologies and to further tune the requirements, which have the effect of further increasing the pre-IOC development cycle. Historical precedent shows that many complex major systems can achieve IOC in less than about 5 years. The Air Force has a detailed analysis of alternatives (AoA) process that can be invoked in this phase. In Air Force Instruction (AFI) 10-601,1 the role of the AoA process is described as follows: The AoA provides information that helps the decision makers select the most cost effective alternative(s), in order to satisfy a mission need or eliminate an operational gap/shortfall in capability. It compares alternative solutions on the basis of operational effectiveness, and cost. It documents the analytical and operational rationale for choosing the preferred alternative(s). It also helps to justify the need for starting, stopping or continuing an acquisition program. In its grandest form, this process is extensive in defining methodology and documentation requirements and should be tailored to meet the needs of each program. 1 Office of Aerospace Studies Analysis of Alternatives (AoA), 2003, Guidance in Support of AFI 10-60 (revised September 22, 2003), Kirtland Air Force Base, N.Mex.: Office of Aerospace Studies.
OCR for page 78
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition Develop a CONOPS. At this stage, the CONOPS is a top-level description of how a system and its operators and users will interact to produce the required capability, including operation in a system of systems. At Milestone A, the CONOPS needs to be sufficient to ensure that the chosen concept is capable of operating to produce the desired outcomes and time lines. A key precursor of a CONOPS is often a description of the other systems with which the system of interest will be interacting and an understanding of what the user expects the system to do in its operating environment. System performance assessment. This is the analytical process of predicting the performance of the system concepts evaluated. It needs to be sufficient to predict the ultimate performance of the system in operation. Often system performance models developed in this phase become the basis for supporting the government’s management of the development phase. The following guidelines are important to consider: The models must typically also provide the basis for setting segment performance requirements and assessing the impact of deviations from expected performance after Milestone A. Modeling and simulation often constitute a powerful tool, starting with the evaluation of concepts and moving on through the entire development phase and even into the operational phase. While this discipline may not always begin in the pre-Milestone A phase, models begun in this phase can add rigor to the assessment of concepts as well as being powerful management tools in subsequent phases. System architecture development. A well-known approach to addressing a complex problem is to break it into parts that can be addressed separately. Architecture here refers to the partitioning of the system into separately definable and procurable parts, the structuring of interfaces between the system and the outside world, and the structuring of interfaces (physical, functional, and data) among the segments. Through careful partitioning, architecture can minimize complexity and thereby development risk. The following guidelines are important to remember in this phase: Seek independence of the segments. Good architecture seeks to achieve segments that can be developed and tested separately to reduce the complexity of the development phase. Simplify interfaces. In selecting the architecture, seek to minimize the entropy of the implementation by making the interfaces among the segments as simple as possible. Plan for testability. It is important that the system designer consider the testability of the parts alone, of the system as a whole, and of any system of systems implications as early as possible to minimize the impact of undiscovered issues later. Plan for integration. The system designer must have a vision of how the parts can be integrated and tested to verify end-to-end performance
OCR for page 79
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition in the operating environment. Modeling and simulation can often play a major role in the overall test and verification plan. Risk assessment. A pre-Milestone A goal is to identify the key risk drivers to set the stage for effective risk management in the procurement and development phases. The six key risk drivers identified in the next section are areas to focus on in risk assessment. High-risk areas may be found not only in technical areas, but also in the availability of critical human resources, stable funding, industrial base capacity, and so on. Cost scoping. The systems engineering process in the pre-Milestone B phases normally includes very rough estimates of a system’s development and operating costs sufficient to determine if the concept and CONOPS are consistent with expected budgetary constraints. The system designers should try to avoid the low bias in cost estimating often introduced by the desire of the government and potential contractors to sell the program. This is usually best accomplished, in this early phase, by top-level comparisons on a segment-by-segment basis with actual cost experience on recent programs. For revolutionary concepts, less-well-grounded techniques may be unavoidable. Models developed in this phase may be matured later to support “should costs” in evaluating contractor bids. Outputs The output categories are described briefly as follows. Definition of key performance parameters. This is the primary output of systems engineering in the pre-Milestone A period. These KPPs will drive the subsequent procurement and development stages of the program. It is important that they are defined concisely and specifically. They should be few in number, but specific in describing the primary capabilities desired. They should be sufficient to provide the basis from which all lower-level requirements are derived. Ideally, they are defined in terms that are understandable to the users of the system rather than in highly complex technical language. In the committee’s experience, the failure of the designers to define the system’s KPPs simply and clearly at Milestone A and the top-level requirements at Milestone B is the first step to requirements instability and overruns later. The committee believes that the top-level KPPs and system requirements can be the basis for all lower-level requirements. Committee members have had experiences with complex, multisegmented systems where a half-dozen top-level requirements drove all lower-level requirements. Often, system availability, maintainability, and/or reliability requirements provide the basis for important derived requirements that are not covered by other capability requirements. The imposition of this discipline can
OCR for page 80
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition TABLE 4-1 Summary of Pre-Milestone A Systems Engineering Function and Output Guidelines Function/Output Summary Guideline System concept development Complexity should be minimized both within the system and with regard to the system’s external interfaces. The use of high-risk, immature technologies should be avoided. Favor concepts that can achieve initial operational capability (IOC) in fewer than about 5 years. At least two alternative concepts should be evaluated before selecting a final concept. System architecture Partition to achieve segments that can be procured and tested separately. Minimize the complexity of the interfaces among the segments. Establish data, structure, and architecture standards where appropriate. Maintain the independence of the functional requirements. Concept of operations (CONOPS) CONOPS needs to be developed sufficiently to ensure that the concept can: Perform in the environment in which it will operate, Handle the expected throughput, and Meet response time requirements. Performance analysis Performance analysis should be sufficient to: Predict performance against mission needs, Assess the impact of segment-level performance on end-to-end performance, and Include the development of system performance models to support performance assessment in the acquisition phase. Top-level requirements generation Requirements at this stage should: Be broad and few in number, Drive the derivation of all lower-level requirements on the program, and Remain largely unchanged through the first development cycle and the achievement of IOC. Risk identification Risk identification should: Identify the top-level risk factors that are inherent in the concept, architecture, and CONOPS. Cost estimates Cost estimates should: Provide a high level of confidence that the concept and CONOPS are consistent with budgetary constraints, and If necessary, facilitate the development of a cost model that can be extended later to support “should costs” for contractor-proposed solutions. Key performance parameters (KPPs) KPPs at this stage should be: Broad and few in number; Comprehensive and clearly and simply defined covering required capabilities, availability, and reliability; Sufficiently complete to be the source of all lower-level requirements; and Sufficiently mature that little change is needed after Milestone B, and prior to IOC.
OCR for page 81
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition have a powerful effect in protecting against requirements creep driven by well-meaning specialists who want to insert their favorite processes and technologies into the program. The Department of Defense (DOD) Guide for Integrating Systems Engineering into DOD Acquisition Contracts, Version 1.02 has this to say on the importance of getting the requirements right: Sound system requirements (including performance) are the backbone of a good technical strategy and resultant plan (as documented in the SEP and related plans). The performance requirements, as a minimum, must be commensurate with satisfying the threshold for the critical operational (including sustainment and support) requirements (e.g., Key Performance Parameters (KPPs)) and balanced with program cost, schedule, and risk constraints. If these elements are not balanced at the start of the SDD phase, the program has a high probability of incurring cost increases, suffering schedule delays, and/or deficient performance of the end product. CONOPS. A draft CONOPS is needed to show that the system can be operated to meet the users’ objectives. Supporting models. Models to validate system performance and cost projections in phases A and B are important to support the government’s management of subsequent stages. Program implementation strategy. The final output from a typical Milestone A activity should be a strategy for implementing the program going forward. This typically would include a plan and time line for achieving Milestone B, an approach to mitigating the most critical identified risks, and an approach to establishing cost projection credibility before Milestone B. Table 4-1 summarizes some of the committee’s observations on the key functions and outputs. SIX DRIVERS OF COST, DEVELOPMENT TIME, AND PERFORMANCE RISK THAT ARE ADDRESSABLE BY SYSTEMS ENGINEERING PROCESSES The committee heard wide-ranging views from its guest speakers, members, and others about the twofold-or-greater growth in program cost and completion time over the past 30 years. The committee believes that understanding the real drivers of this large increase in development times and costs can help identify 2 Department of Defense, 2006, Guide for Integrating Systems Engineering into DOD Acquisition Contracts, Version 1.0, December 11. Available at http://www.acq.osd.mil/se/publications.htm. Last accessed on June 20, 2007.
OCR for page 82
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition FIGURE 4-2 The “six seeds of failure” to avoid or manage during the pre-Milestone A and B phase. the role that early applications of systems engineering processes could play in reversing this trend. The committee’s approach was to harness its collective experience along with the lessons derived from the case studies in Chapter 2. It also tested hypothetical drivers of the problem by comparing current practices with those that existed in the past when some of the most remarkable program successes were achieved in fewer than 5 years from Milestone B. While this was at best an anecdotal analysis, the committee examined the myriad factors that may have contributed to serious development issues and identified six factors that are pervasive sources of poor performance and that are addressable through sound systems engineering processes. The committee calls these factors, illustrated in Figure 4-2, the six seeds of failure. They are inexperienced leadership; external interface complexity; system complexity; incomplete or unstable requirements at Milestone B; reliance on immature technology; and reliance on large amounts of new software. A brief description of the rationale behind each of these factors follows. Inexperienced Leadership Perhaps the biggest risk of all in undertaking large development programs is to proceed with less than the best personnel, particularly in the key leadership positions in government and industry. High-quality program managers and system engineering leaders, in particular, are critical. High aptitude and extensive experience, combining to create high domain knowledge, are required for individuals to be fully effective in these positions. In evaluating program management and system engineering experience, one should consider both the length of experience and the number of programs that the candidates have participated in. Experience
OCR for page 83
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition in at least two major development environments can be very valuable. Since not every program can have the best, it is important for government executives to assign the best to the most difficult problems. Programs that exhibit the highest levels of the risk factors described here, particularly high complexity, are good candidates for the best leadership team. Pre-Milestone A Mitigation For programs with high levels of external interface complexity and internal technical complexity, the government should seek the highest-aptitude, most-experienced people it can muster for the program manager and system engineering manager roles going forward into the Pre-Milestone B phase, and it should keep these people in place through IOC whenever possible. Proven strength and the ability to minimize complexity and control changes after Milestone B through IOC in the face of unrelenting pressure both from inside the program team and from external users and sponsors, are important. External Interface Complexity One characteristic of very complex system developments during World War II and the early Cold War years was the simplicity and urgency of the needs and missions. Beating the Germans to the atomic bomb, the Russians to the Moon, penetrating the Iron Curtain, and so on, provided clear, urgent goals that galvanized the sponsors of the complex systems and focused and empowered the government and contractor teams. Such clear, driving missions, and the simple user interfaces that they required, allowed the program team to develop its concepts quickly and to keep the top-level requirements stable until IOC. In the post-Cold War era, the immediacy of the threats often seems less apparent, and programs often try to serve many missions and users with a single system, or system of systems. The interaction of multiple systems that were not designed together (e.g., military satellite communications [MILSATCOM], see Chapter 2), often termed “systems of systems,” also can greatly increase the difficulty of creating a stable requirements base for a new system. The concept of network-centric operations, in particular, can introduce external complexity. The complex processes necessary to coordinate these communities of interest seem to have hidden costs that can add many years to the development cycle and lead to substantial budget overruns. In addition, systems dependent on highly complex external interfaces can be far more difficult to operate after deployment. Pre-Milestone A Mitigation Systems engineering should treat the minimization of external interface complexity as a key driver in selecting concepts and architecture. Simplifying and
OCR for page 84
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition standardizing the ways that external users access the system and seeking to minimize the degree to which the system’s capabilities must be tailored specifically for individual users can help. Program managers may also be able to minimize the pressure for pre-IOC changes from the external user community by striving to create a sense of time urgency to aid in stabilizing requirements. System Complexity The flexibility and capability enabled by advances in electronics technology and software provide systems designers with far more options than their predecessors enjoyed 30 years ago. The downside is that these new capabilities can tempt designers into unnecessarily complex concepts and designs that impose a “cost of internal complexity” similar to the external complexity costs described above. In particular, there can be a tendency to assign poorly-thought-out functions and options to the software. Often, late-arriving requirements are implemented in software because the perceived schedule risk of redesigning hardware is unacceptable. Pre-Milestone A Mitigation Systems engineering should treat complexity minimization as a key driver in selecting concepts and architecture. Architecture selection can have a powerful effect on controlling complexity. A proven approach to solving large, complex problems is to partition the problem into smaller pieces that can be addressed separately. Partitioning the system to create parts that can be separately developed and tested and that minimize the complexity of internal and external interfaces can be very important. Incomplete or Unstable Requirements at Milestone B It is not unusual for programs to proceed beyond Milestone A with unsettled KPPs and beyond Milestone B with unsettled top-level requirements. This can be a major schedule and cost driver, particularly after Milestone B. As the missions and user communities have become more complex, the government is finding it much more difficult to resolve competing views of system concepts and performance requirements. As a result, it is often tempting to go through the process of selecting the development contractor team before resolving these issues. The well-meaning goal is to get the contractor team’s help in resolving these issues. The result, however, is often that large development teams start without sufficient direction, and the contractor teams with their high “burn rates” waste time and money participating in “what-if” requirements debates that do not converge quickly. Another driver of unstable requirements is funding instability. This phenomenon, familiar to all, is a result of political processes, not systems engineering.
OCR for page 85
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition But its effects on requirements stability, cost, and schedule are significant and insidious. A committee member told of his experience with the Defense Satellite Communications System (DSCS) Program and how the rate of design and production accelerated dramatically, and how the costs dropped significantly below expectations when a fixed-price, multiyear procurement contract was authorized by the Congress and, in effect, froze funding and changes to the requirements.3 Pre-Milestone A Mitigation Federally funded research and development centers (FFRDCs), systems engineering and technical assistance (SETA) contractors, and small study contracts to potential development contractors can all be used to resolve concept and requirements issues prior to Milestone A. It is important not to proceed with the development phase until the top-level concept and requirements are finalized. Also, tight partnership and collaboration between the users/sponsors and the developer are critical to stabilize requirements in the formative phase of a program.4 Reliance on Immature Technology The use of unproven technology in large system developments can introduce a high risk of schedule and cost growth. The committee has seen examples of large procurements in which high-risk technology was seen as a part of the rationale for justifying a program. The committee believes that the use of mature technology should be a prime goal in concept and architecture selection. When the concept and architecture selections identify technologies that are risky, alternative concepts should be found, or the program strategy must accept that the system must wait for the technology to be demonstrated successfully before entering system design and development. Pre-Milestone A Mitigation The system designer should seek to avoid the use of new, unproven technology in large development programs with important missions. The committee believes that technology development should be conducted separately from large operational program developments if possible. As an alternative to pursuing such technology in parallel with post-Milestone B development, the systems engineers might consider making provisions in the architecture and systems design at Milestone B to allow subsequent technology 3 Lt Gen Ronald Kadish (USAF, ret.), Booz Allen Hamilton, “Defense Acquisition Performance Assessment: Report Summary Briefing,” presentation to the committee, February 28, 2007. 4 National Research Council, 1993, An Examination of the Air Force’s Pre-Milestone One Planning/ Decision Process, Washington, D.C.: National Academy Press.
OCR for page 98
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition At Milestone B, are the major system-level requirements (including all KPPs) defined sufficiently to provide a stable basis for the development through IOC? Beginning development without a complete list of stable requirements is one of the key “seeds of failure” described in Chapter 4 in this report. It is important to complete requirements trade-offs prior to the development phase. Has a CONOPS been developed showing that the system can be operated to handle the expected throughput and meet response time requirements? It can be costly to discover too late that the system as designed cannot be operated to meet its requirements. Cost and Schedule Scoping Are the major known cost and schedule drivers and risks explicitly identified, and is there a plan to track and reduce uncertainty? Identifying the major cost and schedule risk areas, with particular attention to this checklist and the six seeds of failure—inexperienced leadership, external interface complexity, system complexity, incomplete requirements at Milestone B, immature technology, and high reliance on new software—can help focus management on these issues early. Has the cost confidence level been accepted by the stakeholders for the program? It is important that stakeholders understand the degree of risk so that the stakeholders will not disrupt the program as inevitable development program surprises unfold later on. It will generally not be possible by Milestone A or Milestone B to identify all the risk areas that might surface later in a development program, but a frank, early disclosure of known potentials for risk can help sustain stakeholder support later on. Performance Assessment Is there a sufficient collection of models and an appropriate simulation environment to validate the selected concept and the CONOPS against the KPPs?
OCR for page 99
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition In large, complex programs, the development of models early on can be very important to later management of requirements changes and performance verification. At Milestone B, do the requirements take into account likely future mission growth over the program life cycle? The committee advocates freezing new requirements and new technology insertion after Milestone B but also notes that making provisions in the initial requirements to facilitate later upgrades could have great long-term value. Architecture Development Has the system been partitioned to define segments that can be independently developed and tested to the greatest degree possible? Effective partitioning of a complex system can greatly reduce its development cost. By Milestone A, is there a plan to have information exchange protocols established for the whole system and its segments by Milestone B? Such a plan developed early on can greatly reduce interface problems later in the development phase when they would be more difficult and costly to fix. At Milestone B, has the government structured the program plan to ensure that the contractor addresses the decomposition of requirements to hardware and software elements sufficiently early in the development program? The histories of programs with cost and schedule overruns are replete with examples of large software developments that had to be redone because requirements from the hardware side were assigned or determined late. Risk Assessment Have the key risk drivers (not only the technology drivers) been identified? Identifying and managing risk early can pay large dividends; it is important to focus on the six “seeds of failure” (see item 8 above).
OCR for page 100
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition Program Implementation Strategy Does the government have access over the life of the program to the talent required to manage the program? Does it have a strategy over the life of the program for using the best people available in the government, the FFRDCs, and the professional service industry? Seasoned management is critical; the government’s job is to find the best! At Milestone A, is there a plan defining how the pre-Milestone B activity will be done, and by whom? Identifying the program and system managers early, identifying the FFRDC or SETA support needed, thinking through the use of competitive system concept contracts—all can have a decisive impact on the government’s ability to select the best concept, to define by Milestone B system requirements that can remain stable through IOC, and to select the best development contractors. Is there a top-level plan for how the total system will be integrated and tested? A well-thought-out strategy for verifying system performance, including optimum phasing of verification tests throughout the assembly process, and well-thought-out use of analytical models and external simulators can have a large positive impact on ultimate cost, schedule, and performance. At Milestone B, have sufficiently talented and experienced program and systems engineering managers been identified? Have they been empowered to tailor processes and to enforce requirements stability from Milestone B through IOC? Seasoned leaders in these areas are critical to maintaining focus and discipline through IOC. Has the government attempted to align the duration of the program manager’s assignment with key deliverables and milestones in the program? A combination of assignment extension and time-certain milestones will help align incentives. NOTE: KPP, key performance parameter; CONOPS, concept of operations; IOC, initial operational capability; FFRDC, federally funded research and development center; SETA, systems engineering and technical assistance.
OCR for page 101
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition of National Security Space Programs to review the acquisition of national security space programs, identify and characterize systemic problems, and recommend improvements. Four key points were made in the Young panel report (the task force’s final report, frequently referred to as the “Young panel report” because the joint task force was chaired by A. Thomas Young):11 Cost has replaced mission success as the primary driver in managing acquisition processes, resulting in excessive technical and schedule risk. We must reverse this trend and reestablish mission success as the overarching principle for program acquisition. The space acquisition system is strongly biased to produce unrealistically low cost estimates throughout the acquisition process. These estimates lead to unrealistic budgets and unexecutable programs. Government capabilities to lead and manage the acquisition process have seriously eroded. On this count, we strongly recommend that the government address acquisition staffing, reporting integrity, systems engineering capabilities, and program manager authority. While the space industrial base is adequate to support current programs, long-term concerns exist. The chairs of the Defense Science Board (DSB) and Air Force Scientific Advisory Board (AFSAB) emphasized an overall government underappreciation for the importance of appropriately staffed and trained systems engineers for managing the technologically demanding and unique aspects of space programs. In the Young panel report, the top five issues related to systems engineering were deemed to be the following: Lack of awareness of the importance, value, timing, accountability, and organizational structure of systems engineering for programs; General lack of availability within government and industry of adequate, qualified systems engineering resources for allocation to major programs; Insufficient systems engineering tools and environments to effectively execute systems engineering on programs; Lack of consistent and effective application of requirements definition, development, and management; and Poor initial program formulation. 11 Defense Science Board/Air Force Scientific Advisory Board Joint Task Force, 2003, Acquisition of National Security Space Programs, Washington, D.C.: OUSD (AT&L). Available at http://www.acq.osd.mil/dsb/reports/space.pdf. Last accessed on April 2, 2007.
OCR for page 102
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition National Defense Industrial Association Report The summary recommendations of the recent report by the National Defense Industrial Association (NDIA) were as follows:12 Ensure institutionalization of effective systems engineering practices into program planning and execution. Integrate engineering planning within the acquisition life cycle to ensure adequate time and effort for SE early in the program life cycle. Emphasize the application of systems engineering practices and resources to the capability definition process to address warfighter needs and translation into executable programs. Grow systems engineering expertise through training, career incentives, and broadening “systems thinking” into other disciplines. Strengthen and clarify policy and guidance regarding the use of collaborative environments, models, simulations, and other automated tools. Government Accountability Office Reports Pertinent to this committee’s discussion of pre-Milestone A SE, the GAO, in its 2003 report Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy,13 stated that the most leveraged decision point in a program is matching the customer’s needs with the developer’s resources. This initial decision sets the stage for the eventual outcome. In successful programs, negotiations and trade-offs occur before product development is started. As an example, GAO pointed out that at the time the decision was made to accelerate the Advanced Extremely High Frequency (AEHF) Program (see the MILSATCOM case summary in Chapter 2), DOD had neither the funding nor the manpower to accomplish the task. The GAO also reported that a primary problem affecting Air Force space programs is that often the users refuse to relax rigid requirements to more closely match technical capabilities that are achievable. For instance, for one element of the Space Based Infrared Systems (SBIRS) Program (see the SBIRS case study in Chapter 2), it became apparent that a lack of knowledge of program challenges led to overly optimistic schedules and budgets. Attempts to stay on schedule by 12 National Defense Industrial Association, 2006, Top Five Systems Engineering Issues Within Department of Defense and Defense Industry. Available at http://www.ndia.org/Content/ContentGroups/Divisions1/Systems_Engineering/Top_5_Systems_Engineering_Issues.pdf. Last accessed on April 2, 2007. 13 Government Accountability Office, 2003, Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy, September. Available at http://www.gao.gov/new.items/d031073.pdf. Last accessed April 2, 2007.
OCR for page 103
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition approving critical milestones without meeting critical criteria led to higher costs and even further schedule erosion.14 The 2005 GAO study Space Acquisitions: Stronger Development Practices and Investment Planning Needed to Address Continuing Problems15 was done at the request of Congress for input on problems relating to DOD’s space systems acquisition. To meet this request, the GAO drew on its previous reports related to the causes of acquisition problems, underlying incentives and pressures, and potential solutions. The testimony resulting from the study partially repeated conclusions from the 2003 GAO report listed above. It concluded that programs typically do not achieve a match between requirements and resources at program start. Specifically: Either requirements are not adequately defined early, or they are changed dramatically once the program is underway. Technologies are typically not mature enough to be included in product development. There are deficiencies in the space acquisition workforce, contractor capabilities, and funding available for testing of space technologies. There is a tendency for programs to take on technology development that should occur in the science and technology (S&T) community. DOD starts more programs than it can afford in the long run, forcing programs to underestimate cost and overpromise capability. The most pertinent of GAO’s recommendations was to employ the techniques of SE to close gaps between available technologies and customer needs before committing to new product development. Also, GAO recommended that DOD develop plans for addressing the shortage of staff with science and engineering backgrounds.16 Defense Acquisition Performance Assessment The major findings of the 2006 Defense Acquisition Performance Assessment (DAPA) study17 were these: Strategic technology exploitation is a key U.S. advantage. Opportunities need to be identified early. 14 Ibid. 15 Government Accountability Office, 2005, Space Acquisitions: Stronger Development Practices and Investment Planning Needed to Address Continuing Problems, July 12. Available at http://www.gao.gov/new.items/d05891t.pdf. Last accessed April 2, 2007. 16 Ibid. 17 Ronald Kadish, Gerald Abbott, Frank Cappuccio, Richard Hawley, Paul Kern, and Donald Kozlowski, 2006, Defense Acquisition Performance Assessment. Available at http://www.acq.osd.mil/dapaproject/documents/DAPA-Report-web/DAPA-Report-web-feb21.pdf. Last accessed on April 2, 2007.
OCR for page 104
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition The U.S. economic and security environments have changed; for example, there are fewer prime contractors, smaller production runs, reduced plant capacity, fewer programs, unpredictable threats. The acquisition system must deal with instability of external funding. The DOD management model is based on a lack of trust. Quantity of review has replaced quality. There is no clear line of responsibility, authority, or accountability. Oversight is preferred to accountability. Oversight is complex, not process- or program-focused (as it should be). The complexity of the acquisition process increases costs and draws out the schedule. Incremental improvement applied solely to the “little a” acquisition process18 requires all processes to be stable—but they are not. The DAPA report dealt with larger issues of acquisition culture and policy. A member of this committee, who was the chair of the DAPA study, emphasized that a successful response to the instabilities caused by the current process or proper program initiation as envisioned requires early and detailed SE practices. FINDINGS AND RECOMMENDATIONS The considerations discussed in this chapter led the committee to develop the following findings and recommendations: Finding 4-1. Attention to a few critical systems engineering processes and functions particularly during preparation for Milestones A and B is essential to ensuring that Air Force acquisition programs deliver products on time and on budget. Today’s weapons systems provide unprecedented capabilities but also involve complex interfaces with external command, control, and communications systems and rely on a greater volume of software than ever before. Early decisions on the weapons system requirements and capabilities have a disproportionately large impact on program cost and schedule. The committee also recognizes that a lack of flexibility (a result of overly rigid processes or a lack of trust among program participants or stakeholders) can limit the ability of a program manager to change early decisions that warrant changing. The committee found many gaps and inconsistencies in the way that the Air Force manages pre-Milestone A activities. The committee heard from presenters of some cases in which required documents were completed pro forma and filed away, never to be seen again, or for which required steps were skipped 18 The Acquisition—“Big A”—system is often believed to be a simple construct that efficiently integrates three independent processes: requirements, budgeting, and acquisition. “Little a,” on the other hand, refers to the acquisition process that focuses on “how to buy” in an effort to balance cost, schedule, and performance; it does not include requirements and budgeting.
OCR for page 105
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition completely. The current practice of initiating programs at Milestone B denies the acquisition review authority the earlier opportunity (at Milestone A) to make judgments about the maturity of the technologies on which the program is based and to decide whether technologies need to be further developed prior to making a Milestone B commitment to system development and demonstration. Recommendation 4-1. The Air Force leadership should require that Milestones A and B be treated as critical milestones in every acquisition program and that a checklist such as the “Pre-Milestone A/B Checklist” suggested by the committee (see Box 4-1 in this chapter) be used to judge successful completion. A rigorous, standard checklist of systems engineering issues should be addressed by each program through both the pre-Milestone A and pre-Milestone B phases. The committee’s recommended 20-item checklist is shown in Box 4-1. While the committee considers that each item on the checklist is important, it calls attention to several items that warrant further discussion: Checklist item 2 recognizes that the world changes too fast to be friendly to long development cycles. The committee believes that the Air Force should strive to structure major development programs so that initial deployment is achieved within, say, 3 to 7 years. Thirty years ago, this was a typical accomplishment—for example, nearly 40 years ago, the Apollo program put the first man on the Moon in fewer than 8 years. The development time issue is addressable by applying systems engineering to items 3, 4, and 13 through 15 before Milestones A and B. The definition of clear KPPs by Milestone A and clear requirements by Milestone B that can remain stable through IOC can be essential to an efficient development phase. It is also important that critical technologies be sufficiently mature prior to starting SDD. The committee observed that although today’s systems are not necessarily more complex internally than those of 30 years ago, their external complexity often is greater, because today’s systems are more likely to try to meet many diverse and sometimes contradictory requirements from multiple users. This kind of complexity can often lead to requirements being changed between Milestone B and IOC, and it can lead to relying on immature technology. Item 19 of the checklist stresses the importance of placing experienced, domain-knowledgeable managers in key program positions. The committee has observed that many of the truly extraordinary development programs of the past, such as Apollo, the Manhattan Project, the early imaging satellite programs, the U-2, the fleet ballistic missile system, and nuclear submarines, were managed by relatively small (and often immature) agencies with few established processes and controls. In that environment, dedicated managers driven by urgent missions accomplished feats that often seem incredible today. The committee believes that the
OCR for page 106
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition accumulation of processes and controls over the years—well meant, of course—has stifled domain-based judgment that is necessary for timely success. Formal SE processes should be tailored to the application. But they cannot replace domain expertise. In connection with Item 19, the committee recommends that the Air Force place great emphasis on putting seasoned, domain-knowledgeable personnel in key positions—particularly the program manager, the chief system engineer, and the person in charge of “requirements”—and then empower them to tailor standardized processes and procedures as they feel is necessary. One key pre-Milestone A task is the analysis of alternatives (AoA), which entails evaluating alternative concepts and comparing them in terms of capabilities, costs, risks, and so on. Checklist items 1 through 4, 12, and 13 should be completed before the AoA, while items 5 through 11 and 14 through 20 may be addressed after the AoA. Finding 4-2. The Air Force used to have a development planning organization that applied pre-Milestone A systems engineering processes to a number of successful programs, but that organization was allowed to lapse. The role of the Air Force development planning organization, which was within the Air Force Systems Command, was to provide standard evaluation tools and perform pre-Milestone A systems engineering functions across acquisition programs. The early 1990s saw an erosion of this front-end planning organization along with its funding as the Air Force Systems Command (now the Air Force Materiel Command [AFMC]) began to play a decreasing role in program execution. In the opinion of several speakers who met with the committee, one main reason for the erosion of funding was a lack of congressional support for the planning function. The specific budget “program element” (PE) for the Air Force planning function was PE 65808. This PE funded a robust planning process in the Air Force for many years. Again, several programs noted in Chapter 2 had their roots in the pre-Milestone A analysis, concept development, and prototyping funded by PE 65808 in the past. The funding for PE 65808 began to decline in the early 1990s as Congress started to reduce the funds appropriated. The rationale for this decline is not documented. However, one committee member recalls specific meetings with key congressional staffers who were responsible for approving the funding of PE 65808. At one of these meetings a staffer stated that this PE ultimately leads to new programs that Congress will have to fund in the future, and so if Congress does not fund PE 65808, then the Air Force will not develop new unaffordable systems. Recommendation 4-2. A development planning function should be established in the military departments to coordinate the concept development and refinement phase of all acquisition programs to ensure that the capabilities required by the
OCR for page 107
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition country as a whole are considered and that unifying strategies such as network-centric operations and interoperability are addressed. The Air Force and the other military services should establish a development planning organization like that which existed in the early 1990s. The roles and functions of the various organizations involved in acquiring major weapons systems need to be clearly defined. The responsibility for executing systems engineering and program management in the pre-Milestone A and B phases should be vested in the military departments that do the actual development planning functions. This should not be the responsibility of the Office of the Secretary of Defense (OSD) or of the Joint Staff. Instead, those offices need to enable the creation and functioning of military department development planning organizations with policy measures, and, where appropriate, resources. The Joint Staff, under the auspices of the Joint Requirements Oversight Council (JROC), may help to define the requirements for major programs in the course of the development planning process, but it should not run the process itself. The existence of “joint” programs or a program such as Missile Defense, which has several related systems being developed by different military services, requires clear guidance from both OSD and the Joint Staff about who is in charge. These programs need to be harmonized and integrated by the responsible integrating agency. However, development planning activities should still take place in the military departments where the expertise resides. Consequently, the development planning should be managed by that agency. While this committee cannot predict how Congress will view the revival of a good planning process to support pre-Milestone A program efforts, it is still important for the Air Force and DOD to make the case for the critical importance of this process before Congress and others. A development planning process is important not to start new programs, but rather to ensure that any new program (or a new start of any kind) is initiated with the foundation needed for success. Funding for this planning function needs to be determined by the military services, including both the acquisition communities and those (the warfighters) who generate the operational requirements. CONCLUDING THOUGHTS Many of the conclusions reached and recommendations made by the committee are similar to those of previous reviews. Most of the past recommendations were never implemented, so one of this committee’s most critical thoughts relates to the importance of implementation. Successful implementation of these recommendations requires the “zipper concept”—making connections at all levels, from the senior leadership of the Air Force and DOD down to the working levels within key program management offices and supervisory staffs.
OCR for page 108
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition This page intentionally left blank.