Pre-Milestone A and Early-Phase Systems Engineering

A Retrospective Review and Benefits for Future Air Force Systems Acquisition

Committee on Pre-Milestone A Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition

Air Force Studies Board

Division on Engineering and Physical Sciences

NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS

Washington, D.C.
www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Pre-Milestone A and Early-Phase Systems Engineering A Retrospective Review and Benefits for Future Air Force Systems Acquisition Committee on Pre-Milestone A Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition Air Force Studies Board Division on Engineering and Physical Sciences

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This is a report of work supported by Grant FA9550-06-1-0549 between the U.S. Air Force and the National Academy of Sciences. Any opinions, findings, conclusions, or recom- mendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for the project. International Standard Book Number-13: 978-0-309-11475-2 International Standard Book Number-10: 0-309-11475-6 Limited copies of this report are Additional copies are available from: available from: Air Force Studies Board The National Academies Press National Research Council 500 Fifth Street, N.W. 500 Fifth Street, N.W. Lockbox 285 Washington, DC 20001 Washington, DC 20055 (202) 334-3111 (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area) Internet, http://www.nap.edu Copyright 2008 by the National Academy of Sciences. All rights reserved. Printed in the United States of America

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examina- tion of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering com- munities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1

OCR for page R1
COMMITTEE ON PRE-MILESTONE A SySTEMS ENgINEERINg: A RETROSPECTIvE REvIEW AND BENEFITS FOR FuTuRE AIR FORCE SySTEMS ACquISITION PAUL G. KAMINSKI, Technovation, Inc., Chair LESTER L. LYLES, U.S. Air Force (retired), Vice Chair DEV A. BANERJEE, The Boeing Company THOMAS W. BLAKELY, Lockheed Martin Aeronautics Company NATALIE W. CRAWFORD, RAND Corporation STEPHEN E. CROSS, Georgia Institute of Technology, Georgia Tech Research Institute GILBERT F. DECKER, Independent Consultant LLEWELLYN S. DOUGHERTY, Raytheon Company JOHN V. FARR, Stevens Institute of Technology JAMES H. FREY, Frey Associates ROBERT A. FUHRMAN, Lockheed Martin Corporation (retired) DAVID J. GORNEY, The Aerospace Corporation JOHN M. GRIFFIN, Griffin Consulting WESLEY L. HARRIS, Massachusetts Institute of Technology RONALD T. KADISH, Booz Allen Hamilton ROBERT H. LATIFF, Science Applications International Corporation ALDEN V. MUNSON, JR., Office of the Director of National Intelligence MARK K. WILSON, Mark Wilson Consulting Staff MICHAEL A. CLARKE, Board Director JAMES C. GARCIA, Study Director GREGORY EYRING, Senior Program Officer CARTER W. FORD, Associate Program Officer WILLIAM E. CAMPBELL, Senior Program Associate LANITA R. JONES, Program Associate (until July 2007) DETRA BODRICK-SHORTER, Administrative Coordinator 

OCR for page R1
AIR FORCE STuDIES BOARD LAWRENCE J. DELANEY, Independent Consultant, Chair TAYLOR W. LAWRENCE, Raytheon Company, Vice Chair FRANK J. CAPPUCCIO, Lockheed Martin Corporation THOMAS DARCY, EADS North America Defense Company STEVEN D. DORFMAN, Hughes Electronics (retired) PAMELA A. DREW, Boeing Integrated Defense Systems KENNETH E. EICKMANN, U.S. Air Force (retired) JOHN V. FARR, Stevens Institute of Technology RAND H. FISHER, Lockheed Martin Corporation JACQUELINE GISH, Northrop Grumman Corporation KENNETH C. HALL, Duke University WESLEY L. HARRIS, Massachusetts Institute of Technology LESLIE KENNE, LK Associates DONALD J. KUTYNA, U.S. Air Force (retired) GREGORY S. MARTIN, GS Martin Consulting DEBASIS MITRA, Bell Laboratories CHANDRA KUMAR N. PATEL, Pranalytica, Inc. ROBERT F. RAGGIO, Dayton Aerospace, Inc. GENE W. RAY, GMT Ventures LOURDES SALAMANCA-RIBA, University of Maryland MARVIN R. SAMBUR, Headquarters, U.S. Air Force (retired) LYLE H. SCHWARTZ, Air Force Office of Scientific Research (retired) EUGENE L. TATTINI, Jet Propulsion Laboratory Staff MICHAEL A. CLARKE, Director GREGORY EYRING, Senior Program Officer JAMES C. GARCIA, Senior Program Officer DANIEL E.J. TALMAGE, JR., Program Officer CARTER W. FORD, Associate Program Officer MARTA VORNBROCK, Associate Program Officer DETRA BODRICK-SHORTER, Administrative Coordinator CHRIS JONES, Financial Associate WILLIAM E. CAMPBELL, Senior Program Associate LANITA R. JONES, Program Associate (until July 2007) URRIKKA B. WOODS, Program Associate ENITA A. WILLIAMS, Research Associate i

OCR for page R1
Preface and Acknowledgments This study was requested by the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering. The main goal was to examine the role that systems engineering can play during the defense acquisition life cycle in addressing the root causes of program failure, especially during the pre-Milestone A and early phases of a program. As chair and vice chair of the study committee, we extend special thanks to the committee members for their commitment and diligence, which enabled us to complete the task successfully. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evi- dence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Elliot I. Axelband, The RAND Corporation, Dennis M. Buede, Innovative Decisions, Inc., Frank B. “Ted” Campbell, USAF (retired), Lockheed Martin Corporation, Donald L. Cromer, USAF (retired), Hughes Space and Communications Company (retired), Don R. Kozlowski, NAE, The Boeing Company (formerly McDonnell Douglas Corp.) (retired), Annette J. Krygiel, Independent Consultant, ii

OCR for page R1
iii prefaCe aND aCKNOWLeDGMeNTS Louis S. Metzger, The MITRE Corporation, William B. Rouse, NAE, Georgia Institute of Technology, Andrew P. Sage, NAE, George Mason University, John P. Stenbit, NAE, TRW, Inc. (retired), A. Thomas Young, NAE, Lockheed Martin Corporation (retired), and Gary Ziegler, Independent Consultant. Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommenda- tions, nor did they see the final draft of the report before its release. The review of this report was overseen by Thom J. Hodgson, NAE, North Carolina State University. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were care- fully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution. The committee acknowledges and appreciates the contribution of the mem- bers of the Air Force Studies Board (AFSB) who developed the study statement of task in concert with the Air Force sponsor. The AFSB was established in 1996 as a unit of the National Research Council at the request of the United States Air Force. The AFSB brings to bear broad military, industrial, and academic scien- tific, engineering, and management expertise on Air Force technical challenges and other issues of importance to senior Air Force leaders. The board discusses potential studies of interest, develops and frames study tasks, ensures proper project planning, suggests potential committee members and reviewers for reports produced by fully independent ad hoc study committees, and convenes meetings to examine strategic issues. The board members, listed on page vi, were not asked to endorse the committee’s conclusions or recommendations, nor did they review the final draft of this report before its release, although board members with appropriate expertise may be nominated to serve as formal members of study committees or as report reviewers. The committee is very grateful to the Air Force sponsor for its dedicated support throughout the study and for the efforts of the National Research Council staff. Paul G. Kaminski, Chair Lester L. Lyles, Vice Chair Committee on Pre-Milestone A Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition

OCR for page R1
Contents SUMMARY 1 1 INTRODUCTION AND OVERVIEW 14 Systems Engineering, 17 Department of Defense Acquisition Process, 20 History of Air Force Development Planning, 21 Statement of Task and Committee Approach, 22 2 RELATIONSHIP BETWEEN SYSTEMS ENGINEERING 26 AND PROGRAM OUTCOME Introduction, 26 Program Synopses, 29 Space Based Infrared Systems (SBIRS) Program, 29 Joint Direct Attack Munition (JDAM) Program, 33 Future Combat Systems (FCS), 34 F-16 Fighting Falcon, 37 Fighter Jet Engine Program, 40 Military Satellite Communications (MILSATCOM), 42 C-5A Program, 45 B-2 Stealth Bomber Program, 48 Shared Findings and Lessons Learned Among Cases, 49 ix

OCR for page R1
x CONTeNTS 3 SYSTEMS ENGINEERING WORKFORCE 52 Introduction, 52 Production of Systems Engineers by U.S. Universities, 53 Production of Systems Engineers by U.S. Industry, 54 The Role of Federally Funded Research and Development Centers, 56 Systems Engineering FFRDCs, 56 Studies and Analysis and “Technology Transition” FFRDCs, 57 Systems Engineering Training and Education Within the Air Force, 57 The Genesis of the Air Force Center for Systems Engineering, 57 U.S. Air Force Academy Training in Systems Engineering and Systems Engineering Management, 59 Current Inventory of Air Force Officers Assigned and Trained in the Scientist, Engineer, and Acquisition Manager Career Fields, 61 Air Force Civilian Systems Engineering Positions, 62 Effects of Program Budget Decision 720, 66 Congressional Actions to Cut DOD Acquisition Workforce, 68 The Future Engineering Force, 69 Revitalizing the Acquisition Corps, 70 Concluding Thoughts, 71 Questions That Need to Be Addressed by DOD, 71 Contractor and Government Considerations, 71 Findings and Recommendations, 72 4 SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES 75 Introduction, 75 Pre-Milestone A Systems Engineering Functions, 76 Functions, 77 Outputs, 79 Six Drivers of Cost, Development Time, and Performance Risk That Are Addressable by Systems Engineering Processes, 81 Inexperienced Leadership, 82 External Interface Complexity, 83 System Complexity, 84 Incomplete or Unstable Requirements at Milestone B, 84 Reliance on Immature Technology, 85 Reliance on Large Amounts of New Software, 86 Other Possible Performance Drivers, 87 Obsolete and Nonrelevant Systems Engineering Processes, 88 General Policies and Best Practices for Systems Engineering in All Phases, 88 Modeling and Simulation, 88 Systems of Systems, 90

OCR for page R1
xi CONTeNTS System Dynamic Modeling, 90 Testing and Evaluation, 91 Cost and Schedule Performance Estimating, 92 Requirements Creep and Requirements Traceability Matrices, 93 Change Control and Configuration Management, 94 Intersystem and Intersegment Interface Management, 94 Sharing Best Practices with Other Agencies, 95 Pre-Milestone A/B Checklist, 96 Previous Relevant Studies, Findings, and Recommendations, 96 National Research Council Report, 96 Defense Science Board and Air Force Scientific Advisory Board Report (Young Panel Report), 96 National Defense Industrial Association Report, 102 Government Accountability Office Reports, 102 Defense Acquisition Performance Assessment, 103 Findings and Recommendations, 104 Concluding Thoughts, 107 APPENDIXES A Biographical Sketches of Committee Members 111 B Meetings and Speakers 120 C What Is Systems Engineering? 124

OCR for page R1

OCR for page R1
Acronyms AEHF advanced extremely high frequency AETC Air Education and Training Command AFIT Air Force Institute of Technology AFMC Air Force Materiel Command AFMC/EN AFMC Engineering Directorate AFMCI AFMC Instruction AFSAB Air Force Scientific Advisory Board AFSC Air Force Systems Command AFSPC Air Force Space Command AIAA American Institute of Aeronautics and Astronautics AIT Airborne Integrated Terminal AMT accelerated mission test AoA analysis of alternatives APUC average procurement unit cost AUPP average unit production price C4ISR command, control, communications, computers, intelligence, surveillance, and reconnaissance CAD computer-aided design CAM computer-aided manufacturing CASE computer-aided software engineering CD concept development CDD capability development document CDR critical design review CIP Component Improvement Program xiii

OCR for page R1
xi aCrONYMS CONOPS concept of operations COTS commercial off-the-shelf CPD capabilities production document CR concept refinement CRRA capability review and risk assessment CSE Center for Systems Engineering CSEP certified systems engineering professional DAB Defense Acquisition Board DAPA Defense Acquisition Performance Assessment DARPA Defense Advanced Research Projects Agency DAU Defense Acquisition University DCS defense communications system DE domain experts DNRO Director of the National Reconnaissance Office DOD Department of Defense DDAF Department of Defense Architecture Framework DODI DOD Instruction DOTMLPF doctrine, organization, training, materiel, leadership, personnel, and facilities DSB Defense Science Board DSCS Defense Satellite Communications System DSP Defense Support Program DT development test ENSIP engine structural integrity program ESC Electronic Systems Center EVM earned value management FAA Federal Aviation Administration FAB-T Family of Advanced Beyond-Line-of-Sight Terminals FCS Future Combat Systems FFP firm fixed price FFRDC federally funded research and development center FNA functional needs analysis FRPDR full rate production decision review FSA functional solutions analysis GAO Government Accountability Office GBS Global Broadcast Service GEO geosynchronous Earth orbit GMT Ground Multiband Terminal

OCR for page R1
x aCrONYMS GOTS government off-the-shelf GPS Global Positioning System HC3 High Capacity Communications Capability ICD initial capabilities document IDA Institute for Defense Analyses IDE intermediate development education IMS Integration Master Schedule INCOSE International Council on Systems Engineering IOC initial operational capability IPT integrated product team IRM integrated risk management plan JASSM Joint Air-to-Surface Standoff Missile JCIDS Joint Capability Integration and Development System JDAM Joint Direct Attack Munition JROC Joint Requirements Oversight Council JTRS joint tactical radio system KPP key performance parameter LCC life cycle cost LSI lead systems integrator M&S modeling and simulation MBMMR Multi-Band Multi-Mode Radio MCB MJPO Configuration Board MIG MILSATCOM integration group MILSATCOM military satellite communications MJPO MILSATCOM Joint Program Office MS milestone MUOS mobile user objective system NAE National Academy of Engineering NDIA National Defense Industrial Association NMS national military strategy NMT Navy Multiband Terminal NRO National Reconnaissance Office NSA National Security Agency NSS national security strategy NSSO National Security Space Office

OCR for page R1
xi aCrONYMS OEM original equipment manufacturer OJT on-the-job training OSD Office of the Secretary of Defense OSS&E operational safety, suitability, and effectiveness OT operational test OTA Other Transactions Authority OUSD (AT&L) Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics PAUC program acquisition unit cost PBD program budget decision PDR preliminary design review PE program element PM program manager R&D research and development ROI return on investment S&E scientist and engineer S&T science and technology SATCOM satellite communications SBIRS Space Based Infrared Systems SDD system design and development SDOE system design and operational effectiveness SE systems engineering SEAG systems engineering advisory group SEI Software Engineering Institute SEIT systems engineering integration team SEM systems engineering management SEP systems engineering plan SES Senior Executive Service SETA systems engineering and technical assistance SMART-T Secure Mobile Anti-Jam Reliable Tactical Terminal SMC Space and Missile Systems Center SoS system of systems SOSCOE System of Systems Common Operating Environment SPO system program office SR space radar SRD system requirements document SSEA&I System of Systems Engineering, Architecture and Integration STE staff technical equivalent STSS Space Tracking and Surveillance System SWAMP Software Acquisition Management Plan

OCR for page R1
xii aCrONYMS T&E test and evaluation TD technology development TDS technology development strategy TMD theater missile defense TPM technical performance measure TRL Technology Readiness Level TSAT Transformational Satellite Communications System TSPR total system performance responsibility UFO Ultra-High Frequency Follow-On USAF United States Air Force USAFA U.S. Air Force Academy USD AT&L Under Secretary of Defense for Acquisition, Technology and Logistics WGS Wideband Gapfiller Satellite WIN-T Warfighter Information Network-Tactical

OCR for page R1