The Role of
Experimentation
Campaigns
in the Air Force
Innovation Life Cycle
Committee on the Role of Experimentation Campaigns in the Air Force
Innovation Life Cycle: A Study
Air Force Studies Board
Division on Engineering and Physical Sciences
A Report of
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by Grant FA9550-14-1-0127 from the U.S. Air Force. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the view of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-45112-3
International Standard Book Number-10: 0-309-45112-4
Digital Object Identifier: 10.17226/23676
Copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2016 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle. Washington, DC: The National Academies Press. doi: 10.17226/23676.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.
Reports document the evidence-based consensus of an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and committee deliberations. Reports are peer reviewed and are approved by the National Academies of Sciences, Engineering, and Medicine.
Proceedings chronicle the presentations and discussions at a workshop, symposium, or other convening event. The statements and opinions contained in proceedings are those of the participants and have not been endorsed by other participants, the planning committee, or the National Academies of Sciences, Engineering, and Medicine.
For information about other products and activities of the Academies, please visit nationalacademies.org/whatwedo.
COMMITTEE ON THE ROLE OF EXPERIMENTATION CAMPAIGNS IN THE AIR FORCE INNOVATION LIFE CYCLE: A STUDY
LESTER L. LYLES (Gen., USAF, Ret.), NAE,1 Independent Consultant, Co-Chair
ALEX MILLER, University of Tennessee, Co-Chair
TED F. BOWLDS (Lt. Gen., USAF, Ret.), The Spectrum Group
CHARLES R. “CR” DAVIS (Lt. Gen., USAF, Ret.), Seabury Aerospace & Defense
BLAISE J. DURANTE, U.S. Air Force (retired)
ANTONIO L. ELIAS, NAE, Orbital Sciences Corporation
IVY ESTABROOKE, Utah Science, Technology and Research Agency
DAVID E. HAMILTON, JR., Eagle Aerie, Inc.
BERNADETTE JOHNSON, Massachusetts Institute of Technology, Lincoln Laboratory
WILLIAM JOHNSON, WMJ Associates LLC
JOSEPH LAWRENCE, National Defense University
ROBERT ANDREW KIRK MITCHELL, NAE, Independent Consultant
BENJAMIN RILEY, Georgia Insitute of Technology, Research Institute
JOEL SERCEL, ICS Associates, Inc.
DANIEL WARD, Dan Ward Consulting, LLC
Staff
JOAN FULLER, Director, Air Force Studies Board
GEORGE J. COYLE, Study Director
STEVEN G. DARBES, Research Assistant
DIONNA C. ALI, Research Assistant
ADRIANNA HARGROVE, Senior Program Assistant/Financial Assistant
CHRIS JONES, Financial Manager
MARGUERITE E. SCHNEIDER, Administrative Coordinator
NANDA RAMANUJAM, Consultant
___________________
1 National Academy of Engineering.
AIR FORCE STUDIES BOARD
DOUGLAS M. FRASER, Doug Fraser, LLC, Chair
DONALD C. FRASER, NAE, Charles Stark Draper Laboratory (retired), Vice Chair
BRIAN A. ARNOLD, Peachtree City, Georgia
ALLISON ASTORINO-COURTOIS, National Security Innovations, Inc.
TED F. BOWLDS, The Spectrum Group
STEVEN R.J. BRUECK, University of New Mexico
FRANK J. CAPPUCCIO, Cappuccio and Associates, LLC
BLAISE J. DURANTE, U.S. Air Force (retired)
BRENDAN B. GODFREY, University of Maryland, College Park
MICHAEL A. HAMEL, Lockheed Martin Space Systems Company
DANIEL E. HASTINGS, Massachusetts Institute of Technology
RAYMOND E. JOHNS, JR., Flight Safety International
ROBERT H. LATIFF, R. Latiff Associates
NANCY G. LEVESON, NAE, Massachusetts Institute of Technology
MARK J. LEWIS, Science and Technology Policy Institute
ALEX MILLER, University of Tennessee
OZDEN OCHOA, Texas A&M University
RICHARD V. REYNOLDS, Van Fleet Group, LLC
STARNES E. WALKER, University of Delaware
DEBORAH WESTPHAL, Toffler Associates
DAVID A. WHELAN, NAE, Boeing Defense, Space, and Security
REBECCA WINSTON, Winston Strategic Management Consulting
MICHAEL I. YARYMOVYCH, NAE, Sarasota Space Associates
Staff
JOAN FULLER, Director
ALAN H. SHAW, Deputy Director
DIONNA C. ALI, Research Assistant
GEORGE J. COYLE, Senior Program Officer
STEVEN G. DARBES, Research Assistant
CARTER W. FORD, Program Officer
ADRIANNA HARGROVE, Senior Program Assistant/Financial Assistant
CHRIS JONES, Financial Manager
ANDREW J. KREEGER, Program Officer
MARGUERITE E. SCHNEIDER, Administrative Coordinator
Preface
BACKGROUND
The Air Force (USAF) has continuously sought to improve the speed with which it develops new capabilities to accomplish its various missions in air, space, and cyberspace. Arguably, the pinnacle of these efforts was reached during the time of the Air Force Systems Command (AFSC), which operated a mature and powerful development planning process called VANGUARD.1 Over the years, however, VANGUARD eroded and eventually disappeared, primarily because its funding declined, and AFSC then became the technology-focused part of the acquisition organization of the combined Air Force Materiel Command (AFMC). Funding for development planning was at one time a robust program element (PE) 65808. As Congress reduced this PE and as the new AFMC focused more on the sustainment of weapon systems, development planning withered to become a largely lost art and science.
In an attempt to address this recognized deficiency and recapture the way that it used to do development planning, the Air Force commissioned a number of
___________________
1 For example, Air Force propulsion research in the late 1950s and early 1960s created the technological base that took turbojet propulsion into the era of the so-called high-bypass-ratio fan jet, making possible large-capacity airlifters (of which the C-5A was first) and airliners (typified by the Boeing 747) and transforming both military and civil aeronautics. Over the same period, Air Force research on electronic flight controls generated an equivalent technological base supporting the emergence of the first inherently unstable fly-by-wire aircraft, the YF-16, Have Blue, and Tacit Blue, progenitors of the immensely successful F-16, F-117, and B-2.
studies over the past several years by the National Research Council’s (NRC’s)2 Air Force Studies Board. Those studies included the following:3
- Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Acquisition (2008),
- Evaluation of U.S. Air Force Preacquisition Technology Development (2011),
- Assessment to Enhance Air Force and Department of Defense Prototyping for the New Defense Strategy: A Workshop Summary (2013), and
- Development Planning: A Strategic Approach to Future Air Force Capabilities (2014).
The Development Planning (DP) study, in 2014, was the impetus for Air Force leadership to embrace the need to address DP completely and to initiate several policy and organizational changes in a revamped Air Force capability development process. The Executive Summary of the Charter for Air Force Capability Development, presented in its entirety in Appendix D, reads as follows:
To deliver timely and effective Air Force capabilities for the Joint Warfighter, AF leadership needs a mechanism to recognize the highest priority operational challenges and opportunities and align them with strategy, planning, programming, requirements and acquisition activities across the enterprise. . . .” This new Capability Development process is the 21st Century embodiment of the VANGUARD and Development Planning processes of the past. It encourages innovation, experimentation, and, prototyping. It also encourages doing these things rapidly to answer questions about the art of the possible for future planning as well as funding.
The need for rapid innovation, driven by a combination of experimentation and prototyping, gave rise to the present study, in which the committee convened by the National Academies of Sciences, Engineering, and Medicine considers how properly designed and executed experimentation campaigns can help restore the level of innovation the U.S. Air Force once enjoyed.
STATEMENT OF TASK AND STUDY APPROACH
The Office of the Assistant Secretary of the Air Force for Science, Technology and Engineering requested both the convening of a workshop, Role of Experimentation Campaigns in the Air Force Innovation Life Cycle, and the current study
___________________
2 Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council are used in an historical context identifying programs prior to July 1.
3 These reports of the National Research Council were published by National Academies Press, Washington, D.C.
on this same topic. Together, the workshop and the study would be an integral component of the thrust to reinvigorate development planning.
In response, the National Academies set forth a statement of task to guide the ad hoc Committee on the Role of Experimentation Planning in the Air Force Innovation Life Cycle as it wrote the current study. The committee, consisting of industry, academia, and retired members of government was formed and tasked with the following:
- Define the current state of practice of experimentation within the Air Force (AF).
- Assess the role of experimentation in the innovation life cycle and address how it can support the Air Force’s future technology requirements.
- Evaluate the role of risk management and experimentation in the innovation life cycle.
- Evaluate current legislative, organizational, or other barriers that limit the use of experimentation within the AF.
- Review and recommend best practices for incentivizing experimentation based on an assessment of comparable S&T organizations.
- Recommend metrics that can be implemented across the AF to monitor and assess the use and value of experimentation.
- In addition, the committee will address any other factors deemed to be relevant, such as organizational structure or concepts of operation that could enhance the likelihood of successfully implementing a robust experimentation program in the Air Force acquisition community.
To accomplish these tasks, the committee met seven times, including in a kickoff meeting, a workshop, and in five meetings to collect information and initiate the report writing process (see Appendix C for meeting agendas). The committee met with active and retired senior leaders and officers and civilians from the Air Force, Army, Navy, and Marines, all of whom have been actively involved with experimentation and development planning. The committee also gained valuable perspective and insights from the senior-level (C-suite4) management of innovative companies in both defense and nondefense businesses, the venture capital industry, and academia.
While very conscious of the title of this study, the committee recognized the essential role of experimentation in innovation. It focused on how to improve innovation within the Air Force development organization and in its culture and
___________________
4 The C-suite includes all those senior company executives who report directly to the chief executive officer of a company (e.g., the chief financial officer, the chief operating officer, or the chief technology officer.
planning. Experimentation itself is a major tool in the innovation cycle and is addressed as such.
ROLE OF THE AIR FORCE STUDIES BOARD
The Air Force Studies Board provides advice to senior USAF leaders based on independent research. An ad hoc committee was formed under the auspices of the Air Force Studies Board to carry out the research needed to offer such advice, which would restore the earlier level of technological innovation by increasing experimentation and experimentation campaigns. This report describes the committee’s research into highly innovative organizations, compares the best practices it found there with the practices it observes in today’s Air Force, and proposes actions to close the gaps.
Gen (USAF, RET.) Lester L. Lyles
Alex Miller
William B. Stokely Chair of Management University of Tennessee College of Business Administration
Co-Chairs, Committee on the Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: A Study
Acknowledgment of Reviewers
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
Robert Barish, NAM,1 University of Illinois, Chicago,
Meyer J. Benzakein, NAE,2 Ohio State University,
Paul Bevilaqua, NAE, Lockheed Martin Aeronautics Company (retired),
Thomas J. Burns, Defense Advanced Research Projects Agency,
Stephen E. Cross, Georgia Institute of Technology,
Brendan B. Godfrey, University of Maryland, College Park,
Paul G. Kaminski, NAE, Technovation, Inc.,
George W. Sutton, NAE, Analysis and Applications, Inc., and
Michael Yarymovych, NAE, Sarasota Space Associates.
___________________
1 National Academy of Medicine.
2 National Academy of Engineering.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Robert E. Schafrik, NAE, General Electric Aviation (retired), and Wesley L. Harris, Massachusetts Institute of Technology, NAE, who were responsible for making certain that independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Contents
Best Practices in Highly Innovative Organizations
Current Practices in Today’s Air Force
2 EXPERIMENTATION FOR INNOVATION: BEST PRACTICES IN HIGHLY INNOVATIVE ORGANIZATIONS
A Brief Review of Relevant Research
What It Takes to Build an Innovative Organization
Cradle-to-Grave Aspects of Experimentation and Innovation
The Valley of Death—The Universal Problem
Smart Experimentation Campaigns
USAF’s Long History of Experimentation Campaigns
Experimentation Processes and Tools
Experimentation Campaign Tools
3 EXPERIMENTATION FOR INNOVATION: CURRENT AIR FORCE PRACTICE
Smart Experimentation Campaigns
Experimentation Leadership and Organization
Experimentation Processes and Tools
4 FINDINGS AND RECOMMENDATIONS FOR CLOSING GAPS
B Committee Member Biographies
D Charter for Air Force Capability Development
(SecAF Signed 2 June 2016)
Acronyms
$AVE |
Surfing Aircraft Vortices for Energy |
3D |
three-dimensional |
ABL | Airborne Laser |
ACC |
Air Combat Command |
AF |
Air Force |
AFB |
Air Force Base |
AF CDC |
Air Force Capability Development Council |
AFMC |
Air Force Materiel Command |
AFRL |
Air Force Research Laboratory |
AFSB |
Air Force Studies Board |
AFSC |
Air Force Systems Command |
AFSOC |
Air Force Special Operations Command |
AMC |
Air Mobility Command |
AOC |
Air Operations Center |
BENS | Business Executives for National Security |
BMDO |
Ballistic Missile Development Organization |
CAOC-X | Combined Air Operations Center-Experimental |
CDC |
Capability Development Council |
CDWG |
Capabilities Development Working Group |
CEO |
chief operating officer |
CHOP |
Countermeasures Hands-On Program |
CNC |
computer numerically controlled |
CoCOM |
Combatant Command |
CRP |
Commercialization Readiness Program |
CSAF |
Chief of Staff of the [United States] Air Force |
CTO |
chief technology officer |
DARPA | Defense Advanced Research Projects Agency |
DIUx |
Defense Innovation Unit Experimental |
DOD |
Department of Defense |
DP |
Development Planning |
ECCT | Enterprise Capability Collaboration Team |
FY | fiscal year |
GAO | Government Accountability Office |
GNP |
gross national product |
HQAF | Air Force Headquarters |
MAJCOM | Major Command |
MDAP |
major development acquisition programs |
MWR |
morale, welfare, and recreation |
N-Star | NASA Solar Electric Propulsion Technology Application Readiness |
NACA |
National Advisory Committee for Aeronautics |
NASA |
National Aeronautics and Space Administration |
NRC |
National Research Council |
OODA | observe, orient, decide, act |
PE | program element |
PI |
principal investigator |
R&D | research and development |
RCO |
Rapid Capabilities Office |
RCS |
radar cross-section |
RDT&E |
research, development, technology, and evaluation |
S&T | science and technology |
SAB |
Scientific Advisory Board |
SAF/AQ |
Assistant Secretary of the Air Force (Acquisition) |
SBIR |
Small Business Innovation Research |
SDPE |
Strategic Development Planning and Experimentation |
SecAF |
Secretary of the Air Force |
SIMAF |
Simulation and Analysis Facility |
SOCOM |
Special Operations Command |
SOF AT&L |
U.S. Special Operations Forces Acquisition, Technology, and Logistics |
STTR | Small Business Technology Transfer |
TRL | technology readiness level |
UAV | unmanned air vehicle |
USAF |
U.S. Air Force |
USAFE |
U.S. Air Forces in Europe |
USD AT&L |
Office of the Under Secretary of Defense Acquisition, Technology and Logistics |
VC | venture capital |
This page intentionally left blank.