Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
Pre-Milestone A and Early-Phase Systems Engineering
A Retrospective Review and Benefits for Future Air Force Systems Acquisition
Committee on Pre-Milestone A Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
Air Force Studies Board
Division on Engineering and Physical Sciences
NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu
OCR for page R2
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This is a report of work supported by Grant FA9550-06-1-0549 between the U.S. Air Force and the National Academy of Sciences. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-11475-2
International Standard Book Number-10: 0-309-11475-6
Limited copies of this report are available from:
Air Force Studies Board
National Research Council
500 Fifth Street, N.W.
Washington, DC 20001
(202) 334-3111
Additional copies are available from:
The National Academies Press
500 Fifth Street, N.W. Lockbox 285 Washington, DC 20055 (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area) Internet, http://www.nap.edu
Copyright 2008 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
OCR for page R3
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
www.national-academies.org
OCR for page R4
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
This page intentionally left blank.
OCR for page R5
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
COMMITTEE ON PRE-MILESTONE A SYSTEMS ENGINEERING: A RETROSPECTIVE REVIEW AND BENEFITS FOR FUTURE AIR FORCE SYSTEMS ACQUISITION
PAUL G. KAMINSKI,
Technovation, Inc.,
Chair
LESTER L. LYLES,
U.S. Air Force (retired),
Vice Chair
DEV A. BANERJEE,
The Boeing Company
THOMAS W. BLAKELY,
Lockheed Martin Aeronautics Company
NATALIE W. CRAWFORD,
RAND Corporation
STEPHEN E. CROSS,
Georgia Institute of Technology, Georgia Tech Research Institute
GILBERT F. DECKER, Independent Consultant
LLEWELLYN S. DOUGHERTY,
Raytheon Company
JOHN V. FARR,
Stevens Institute of Technology
JAMES H. FREY,
Frey Associates
ROBERT A. FUHRMAN,
Lockheed Martin Corporation (retired)
DAVID J. GORNEY,
The Aerospace Corporation
JOHN M. GRIFFIN,
Griffin Consulting
WESLEY L. HARRIS,
Massachusetts Institute of Technology
RONALD T. KADISH,
Booz Allen Hamilton
ROBERT H. LATIFF,
Science Applications International Corporation
ALDEN V. MUNSON, JR.,
Office of the Director of National Intelligence
MARK K. WILSON,
Mark Wilson Consulting
Staff
MICHAEL A. CLARKE, Board Director
JAMES C. GARCIA, Study Director
GREGORY EYRING, Senior Program Officer
CARTER W. FORD, Associate Program Officer
WILLIAM E. CAMPBELL, Senior Program Associate
LANITA R. JONES, Program Associate (until July 2007)
DETRA BODRICK-SHORTER, Administrative Coordinator
OCR for page R6
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
AIR FORCE STUDIES BOARD
LAWRENCE J. DELANEY, Independent Consultant, Chair
TAYLOR W. LAWRENCE,
Raytheon Company,
Vice Chair
FRANK J. CAPPUCCIO,
Lockheed Martin Corporation
THOMAS DARCY,
EADS North America Defense Company
STEVEN D. DORFMAN,
Hughes Electronics (retired)
PAMELA A. DREW,
Boeing Integrated Defense Systems
KENNETH E. EICKMANN,
U.S. Air Force (retired)
JOHN V. FARR,
Stevens Institute of Technology
RAND H. FISHER,
Lockheed Martin Corporation
JACQUELINE GISH,
Northrop Grumman Corporation
KENNETH C. HALL,
Duke University
WESLEY L. HARRIS,
Massachusetts Institute of Technology
LESLIE KENNE,
LK Associates
DONALD J. KUTYNA,
U.S. Air Force (retired)
GREGORY S. MARTIN,
GS Martin Consulting
DEBASIS MITRA,
Bell Laboratories
CHANDRA KUMAR N. PATEL,
Pranalytica, Inc.
ROBERT F. RAGGIO,
Dayton Aerospace, Inc.
GENE W. RAY,
GMT Ventures
LOURDES SALAMANCA-RIBA,
University of Maryland
MARVIN R. SAMBUR,
Headquarters, U.S. Air Force (retired)
LYLE H. SCHWARTZ,
Air Force Office of Scientific Research (retired)
EUGENE L. TATTINI,
Jet Propulsion Laboratory
Staff
MICHAEL A. CLARKE, Director
GREGORY EYRING, Senior Program Officer
JAMES C. GARCIA, Senior Program Officer
DANIEL E.J. TALMAGE, JR., Program Officer
CARTER W. FORD, Associate Program Officer
MARTA VORNBROCK, Associate Program Officer
DETRA BODRICK-SHORTER, Administrative Coordinator
CHRIS JONES, Financial Associate
WILLIAM E. CAMPBELL, Senior Program Associate
LANITA R. JONES, Program Associate (until July 2007)
URRIKKA B. WOODS, Program Associate
ENITA A. WILLIAMS, Research Associate
OCR for page R7
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
Preface and Acknowledgments
This study was requested by the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering. The main goal was to examine the role that systems engineering can play during the defense acquisition life cycle in addressing the root causes of program failure, especially during the pre-Milestone A and early phases of a program. As chair and vice chair of the study committee, we extend special thanks to the committee members for their commitment and diligence, which enabled us to complete the task successfully.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
Elliot I. Axelband, The RAND Corporation,
Dennis M. Buede, Innovative Decisions, Inc.,
Frank B. “Ted” Campbell, USAF (retired), Lockheed Martin Corporation,
Donald L. Cromer, USAF (retired), Hughes Space and Communications Company (retired),
Don R. Kozlowski, NAE, The Boeing Company (formerly McDonnell Douglas Corp.) (retired),
Annette J. Krygiel, Independent Consultant,
OCR for page R8
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
Louis S. Metzger, The MITRE Corporation,
William B. Rouse, NAE, Georgia Institute of Technology,
Andrew P. Sage, NAE, George Mason University,
John P. Stenbit, NAE, TRW, Inc. (retired),
A. Thomas Young, NAE, Lockheed Martin Corporation (retired), and
Gary Ziegler, Independent Consultant.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Thom J. Hodgson, NAE, North Carolina State University. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
The committee acknowledges and appreciates the contribution of the members of the Air Force Studies Board (AFSB) who developed the study statement of task in concert with the Air Force sponsor. The AFSB was established in 1996 as a unit of the National Research Council at the request of the United States Air Force. The AFSB brings to bear broad military, industrial, and academic scientific, engineering, and management expertise on Air Force technical challenges and other issues of importance to senior Air Force leaders. The board discusses potential studies of interest, develops and frames study tasks, ensures proper project planning, suggests potential committee members and reviewers for reports produced by fully independent ad hoc study committees, and convenes meetings to examine strategic issues. The board members, listed on page vi, were not asked to endorse the committee’s conclusions or recommendations, nor did they review the final draft of this report before its release, although board members with appropriate expertise may be nominated to serve as formal members of study committees or as report reviewers.
The committee is very grateful to the Air Force sponsor for its dedicated support throughout the study and for the efforts of the National Research Council staff.
Paul G. Kaminski, Chair
Lester L. Lyles, Vice Chair
Committee on Pre-Milestone A Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
OCR for page R9
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
Contents
SUMMARY
1
1
INTRODUCTION AND OVERVIEW
14
Systems Engineering,
17
Department of Defense Acquisition Process,
20
History of Air Force Development Planning,
21
Statement of Task and Committee Approach,
22
2
RELATIONSHIP BETWEEN SYSTEMS ENGINEERING AND PROGRAM OUTCOME
26
Introduction,
26
Program Synopses,
29
Space Based Infrared Systems (SBIRS) Program,
29
Joint Direct Attack Munition (JDAM) Program,
33
Future Combat Systems (FCS),
34
F-16 Fighting Falcon,
37
Fighter Jet Engine Program,
40
Military Satellite Communications (MILSATCOM),
42
C-5A Program,
45
B-2 Stealth Bomber Program,
48
Shared Findings and Lessons Learned Among Cases,
49
OCR for page R10
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
3
SYSTEMS ENGINEERING WORKFORCE
52
Introduction,
52
Production of Systems Engineers by U.S. Universities,
53
Production of Systems Engineers by U.S. Industry,
54
The Role of Federally Funded Research and Development Centers,
56
Systems Engineering FFRDCs,
56
Studies and Analysis and “Technology Transition” FFRDCs,
57
Systems Engineering Training and Education Within the Air Force,
57
The Genesis of the Air Force Center for Systems Engineering,
57
U.S. Air Force Academy Training in Systems Engineering and Systems Engineering Management,
59
Current Inventory of Air Force Officers Assigned and Trained in the Scientist, Engineer, and Acquisition Manager Career Fields,
61
Air Force Civilian Systems Engineering Positions,
62
Effects of Program Budget Decision 720,
66
Congressional Actions to Cut DOD Acquisition Workforce,
68
The Future Engineering Force,
69
Revitalizing the Acquisition Corps,
70
Concluding Thoughts,
71
Questions That Need to Be Addressed by DOD,
71
Contractor and Government Considerations,
71
Findings and Recommendations,
72
4
SYSTEMS ENGINEERING FUNCTIONS AND GUIDELINES
75
Introduction,
75
Pre-Milestone A Systems Engineering Functions,
76
Functions,
77
Outputs,
79
Six Drivers of Cost, Development Time, and Performance Risk That Are Addressable by Systems Engineering Processes,
81
Inexperienced Leadership,
82
External Interface Complexity,
83
System Complexity,
84
Incomplete or Unstable Requirements at Milestone B,
84
Reliance on Immature Technology,
85
Reliance on Large Amounts of New Software,
86
Other Possible Performance Drivers,
87
Obsolete and Nonrelevant Systems Engineering Processes,
88
General Policies and Best Practices for Systems Engineering in All Phases,
88
Modeling and Simulation,
88
Systems of Systems,
90
OCR for page R11
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
System Dynamic Modeling,
90
Testing and Evaluation,
91
Cost and Schedule Performance Estimating,
92
Requirements Creep and Requirements Traceability Matrices,
93
Change Control and Configuration Management,
94
Intersystem and Intersegment Interface Management,
94
Sharing Best Practices with Other Agencies,
95
Pre-Milestone A/B Checklist,
96
Previous Relevant Studies, Findings, and Recommendations,
96
National Research Council Report,
96
Defense Science Board and Air Force Scientific Advisory Board Report (Young Panel Report),
96
National Defense Industrial Association Report,
102
Government Accountability Office Reports,
102
Defense Acquisition Performance Assessment,
103
Findings and Recommendations,
104
Concluding Thoughts,
107
APPENDIXES
A Biographical Sketches of Committee Members
111
B Meetings and Speakers
120
C What Is Systems Engineering?
124
OCR for page R12
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
This page intentionally left blank.
OCR for page R13
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
Acronyms
AEHF advanced extremely high frequency
AETC Air Education and Training Command
AFIT Air Force Institute of Technology
AFMC Air Force Materiel Command
AFMC/EN AFMC Engineering Directorate
AFMCI AFMC Instruction
AFSAB Air Force Scientific Advisory Board
AFSC Air Force Systems Command
AFSPC Air Force Space Command
AIAA American Institute of Aeronautics and Astronautics
AIT Airborne Integrated Terminal
AMT accelerated mission test
AoA analysis of alternatives
APUC average procurement unit cost
AUPP average unit production price
C4ISR command, control, communications, computers, intelligence, surveillance, and reconnaissance
CAD computer-aided design
CAM computer-aided manufacturing
CASE computer-aided software engineering
CD concept development
CDD capability development document
CDR critical design review
CIP Component Improvement Program
OCR for page R14
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
CONOPS concept of operations
COTS commercial off-the-shelf
CPD capabilities production document
CR concept refinement
CRRA capability review and risk assessment
CSE Center for Systems Engineering
CSEP certified systems engineering professional
DAB Defense Acquisition Board
DAPA Defense Acquisition Performance Assessment
DARPA Defense Advanced Research Projects Agency
DAU Defense Acquisition University
DCS defense communications system
DE domain experts
DNRO Director of the National Reconnaissance Office
DOD Department of Defense
DDAF Department of Defense Architecture Framework
DODI DOD Instruction
DOTMLPF doctrine, organization, training, materiel, leadership, personnel, and facilities
DSB Defense Science Board
DSCS Defense Satellite Communications System
DSP Defense Support Program
DT development test
ENSIP engine structural integrity program
ESC Electronic Systems Center
EVM earned value management
FAA Federal Aviation Administration
FAB-T Family of Advanced Beyond-Line-of-Sight Terminals
FCS Future Combat Systems
FFP firm fixed price
FFRDC federally funded research and development center
FNA functional needs analysis
FRPDR full rate production decision review
FSA functional solutions analysis
GAO Government Accountability Office
GBS Global Broadcast Service
GEO geosynchronous Earth orbit
GMT Ground Multiband Terminal
OCR for page R15
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
GOTS government off-the-shelf
GPS Global Positioning System
HC3 High Capacity Communications Capability
ICD initial capabilities document
IDA Institute for Defense Analyses
IDE intermediate development education
IMS Integration Master Schedule
INCOSE International Council on Systems Engineering
IOC initial operational capability
IPT integrated product team
IRM integrated risk management plan
JASSM Joint Air-to-Surface Standoff Missile
JCIDS Joint Capability Integration and Development System
JDAM Joint Direct Attack Munition
JROC Joint Requirements Oversight Council
JTRS joint tactical radio system
KPP key performance parameter
LCC life cycle cost
LSI lead systems integrator
M&S modeling and simulation
MBMMR Multi-Band Multi-Mode Radio
MCB MJPO Configuration Board
MIG MILSATCOM integration group
MILSATCOM military satellite communications
MJPO MILSATCOM Joint Program Office
MS milestone
MUOS mobile user objective system
NAE National Academy of Engineering
NDIA National Defense Industrial Association
NMS national military strategy
NMT Navy Multiband Terminal
NRO National Reconnaissance Office
NSA National Security Agency
NSS national security strategy
NSSO National Security Space Office
OCR for page R16
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
OEM original equipment manufacturer
OJT on-the-job training
OSD Office of the Secretary of Defense
OSS&E operational safety, suitability, and effectiveness
OT operational test
OTA Other Transactions Authority
OUSD (AT&L) Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics
PAUC program acquisition unit cost
PBD program budget decision
PDR preliminary design review
PE program element
PM program manager
R&D research and development
ROI return on investment
S&E scientist and engineer
S&T science and technology
SATCOM satellite communications
SBIRS Space Based Infrared Systems
SDD system design and development
SDOE system design and operational effectiveness
SE systems engineering
SEAG systems engineering advisory group
SEI Software Engineering Institute
SEIT systems engineering integration team
SEM systems engineering management
SEP systems engineering plan
SES Senior Executive Service
SETA systems engineering and technical assistance
SMART-T Secure Mobile Anti-Jam Reliable Tactical Terminal
SMC Space and Missile Systems Center
SoS system of systems
SOSCOE System of Systems Common Operating Environment
SPO system program office
SR space radar
SRD system requirements document
SSEA&I System of Systems Engineering, Architecture and Integration
STE staff technical equivalent
STSS Space Tracking and Surveillance System
SWAMP Software Acquisition Management Plan
OCR for page R17
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
T&E test and evaluation
TD technology development
TDS technology development strategy
TMD theater missile defense
TPM technical performance measure
TRL Technology Readiness Level
TSAT Transformational Satellite Communications System
TSPR total system performance responsibility
UFO Ultra-High Frequency Follow-On
USAF United States Air Force
USAFA U.S. Air Force Academy
USD AT&L Under Secretary of Defense for Acquisition, Technology and Logistics
WGS Wideband Gapfiller Satellite
WIN-T Warfighter Information Network-Tactical
OCR for page R18
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition
This page intentionally left blank.
