The Role of Experimentation
Campaigns in the Air Force
Innovation Life Cycle
Proceedings of a Workshop
Committee on the Role of Experimentation Campaigns
in the Air Force Innovation Life Cycle
Air Force Studies Board
Division on Engineering and Physical Sciences
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This is a report of work supported by Grant FA9550-14-1-0127 with the U.S. Air Force. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the view of any organization or agency that provided support for the project.
Digital Object Identifier: 10.17226/23602
Limited copies of this report are available from:
Air Force Studies Board
National Academies of Sciences, Engineering, and Medicine
500 Fifth Street, NW
Washington, DC 20001
Copyright 2016 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.
Reports document the evidence-based consensus of an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and committee deliberations. Reports are peer reviewed and are approved by the National Academies of Sciences, Engineering, and Medicine.
Proceedings chronicle the presentations and discussions at a workshop, symposium, or other convening event. The statements and opinions contained in proceedings are those of the participants and have not been endorsed by other participants, the planning committee, or the National Academies of Sciences, Engineering, and Medicine.
For information about other products and activities of the Academies, please visit nationalacademies.org/whatwedo.
COMMITTEE ON THE ROLE OF EXPERIMENTATION CAMPAIGNS IN THE AIR FORCE INNOVATION LIFE CYCLE
LESTER L. LYLES (Gen., USAF, Ret.), NAE,1 Independent Consultant, Co-Chair
ALEX MILLER, University of Tennessee, Co-Chair
TED F. BOWLDS (Lt. Gen., USAF, Ret.), The Spectrum Group
CHARLES R. “CR” DAVIS (Lt. Gen., USAF, Ret.), Seabury Aerospace & Defense
BLAISE J. DURANTE, U.S. Air Force (Ret.)
ANTONIO L. ELIAS, NAE, Orbital Sciences Corporation
IVY ESTABROOKE, Utah Science Technology and Research Agency
DAVID E. HAMILTON, JR., Eagle Aerie Inc.
BERNADETTE JOHNSON, Massachusetts Institute of Technology Lincoln Laboratory
WILLIAM JOHNSON, WMJ Associates LLC
JOSEPH LAWRENCE, National Defense University
ROBERT ANDREW KIRK MITCHELL, NAE, Independent Consultant
BENJAMIN P. RILEY, Georgia Tech Research Institute
JOEL SERCEL, ICS Associates Inc.
DANIEL WARD, Dan Ward Consulting, LLC
Staff
JOAN FULLER, Director, Air Force Studies Board
GEORGE J. COYLE, Senior Program Officer
MARGUERITE E. SCHNEIDER, Administrative Coordinator
STEVEN G. DARBES, Research Assistant
DIONNA C. ALI, Research Assistant
___________________
1 NAE, National Academy of Engineering.
AIR FORCE STUDIES BOARD
DOUGLAS M. FRASER, Doug Fraser, LLC, Chair
DONALD C. FRASER, NAE,1 Charles Stark Draper Laboratory (retired), Vice Chair
BRIAN A. ARNOLD, Peachtree City, Georgia
ALLISON ASTORINO-COURTOIS, National Security Innovations, Inc.
TED F. BOWLDS, The Spectrum Group
STEVEN R.J. BRUECK, University of New Mexico
FRANK J. CAPPUCCIO, Cappuccio and Associates, LLC
BLAISE J. DURANTE, U.S. Air Force (Ret.)
BRENDAN B. GODFREY, University of Maryland, College Park
MICHAEL A. HAMEL, Lockheed Martin Space Systems Company
DANIEL E. HASTINGS, Massachusetts Institute of Technology
RAYMOND E. JOHNS, JR., Flight Safety International
ROBERT H. LATIFF, R. Latiff Associates
NANCY G. LEVESON, NAE, Massachusetts Institute of Technology
MARK J. LEWIS, Institute for Defense Analyses Science and Technology Policy Institute
ALEX MILLER, University of Tennessee
OZDEN OCHOA, Texas A&M University
RICHARD V. REYNOLDS, The VanFleet Group, LLC
STARNES E. WALKER, University of Delaware
DEBORAH WESTPHAL, Toffler Associates
DAVID A. WHELAN, NAE, Boeing Defense, Space, and Security
REBECCA WINSTON, Winston Strategic Management Consulting
MICHAEL I. YARYMOVYCH, NAE, Sarasota Space Associates
Staff
JOAN FULLER, Director
ALAN H. SHAW, Deputy Director
GEORGE J. COYLE, Senior Program Officer
CARTER W. FORD, Program Officer
ANDREW J. KREEGER, Program Officer
DIONNA C. ALI, Research Assistant
STEVEN G. DARBES, Research Assistant
CHRIS JONES, Financial Manager
MARGUERITE E. SCHNEIDER, Administrative Coordinator
___________________
1 NAE, National Academy of Engineering.
Preface
The U.S. Air Force Office of the Assistant Secretary for Science, Technology and Engineering asked the Air Force Studies Board of the National Academies of Sciences, Engineering, and Medicine to define and assess the current use of experimentation campaigns within the Air Force, evaluate barriers to their use, and make recommendations to increase their use.
To accomplish this work, the Committee on the Role of Experimentation Campaigns in the Air Force Innovation Life Cycle was established to (1) organize a workshop and (2) complete a consensus study with recommendations on how to best institutionalize experimentation within the Air Force. (Appendix A contains the committee’s statement of task, and Appendix B contains biographical sketches of the committee members.) The goal of the workshop was to bring together practitioners of experimentation methodologies from a broad cross-section of the community with the intention of using these inputs to guide the subsequent consensus study activity.
The Workshop on the Role of Experimentation Campaigns in the Air Force Innovation Cycle was held in Washington, D.C., on January 27-29, 2016. Participants at the workshop presented a broad range of issues, experiences, and insights related to experimentation, experimentation campaigns, and innovation. These proceedings provide a summary of the workshop presentations; however, no attempt was made to develop consensus findings and recommendations. The agenda for the workshop appears in Appendix C, and Appendix D lists the workshop participants. A list of acronyms and abbreviations is provided in the front matter.
The Air Force Studies Board and the committee recognize the contributions of the workshop participants and appreciate this opportunity to provide input on a subject of importance to the Air Force.
A NOTE ABOUT THE ABSTRACTS
The abstracts in these proceedings are those provided by or approved by the speakers; presentations shown on the agenda but without a corresponding transcript are those for which the speaker did not provide an abstract or declined the offer to include an abstract. The abstracts have been lightly edited for clarity.
This page intentionally left blank.
Contents
2 THE PROBLEM AS SEEN BY SENIOR AIR FORCE LEADERS: THE NEED FOR EXPERIMENTATION
3 AIR FORCE PRACTICES VERSUS BEST PRACTICES
Experimentation Inside the Air Force
What the Rapid Capabilities Office Does That More of the Air Force Should Consider Doing
Global Hawk: Advanced Concept Technology Demonstration Case Study on Successes and Obstacles
Quick Reaction Effort to Provide an Air-Enabled Cyber Capability
Experimentation Outside the Air Force
Motiv Space Systems—A “New Space” Company
Disruptive Innovation: What It Is and Why It Is Important
Taxonomy and Language of Innovation
Innovation Models from the Venture Capital World
Lessons from Venture Capital Industry: How Innovation and Risk Are Managed
4 BETTER EXPERIMENTATION IN THE AIR FORCE: BARRIERS AND LEVERS
Making Experimentation Part of the Culture
Life Cycle Management Center Perspectives on Barriers
Perspective on Acquisition Barriers to Innovation and Experimentation
Acronyms
ACTD | Advanced Concept Technology Demonstration |
AF | Air Force |
AFLCMC | Air Force Life Cycle Management Center |
AFMC | Air Force Materiel Command |
AMRAAM | Advanced Medium-range Air to Air Missile |
ATP | authorization to proceed |
C4ISR |
Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance |
COG | Cyber Operations Group |
COS | Cyber Operations (or Ops) Squadron |
COTS | commercial off-the-shelf (most common); Commercial Orbital Transportation Services (specialized space usage) |
CW | Cyber Wing |
DARPA |
Defense Advanced Research Projects Agency |
DIUx | Defense Innovation Unit-Experimental |
DoD | Department of Defense |
DP | Development Planning |
DT&E | Developmental Test and Engineering |
FAR |
Federal Acquisition Regulation |
HMS |
His (Her) Majesty’s Ship |
IA |
information assurance |
ID/IQ | Indefinite Delivery/Indefinite Quantity |
JPL |
Jet Propulsion Laboratory |
MVP | minimum viable product |
NASA |
National Aeronautics and Space Administration |
NATO | North Atlantic Treaty Organization |
NDAA | National Defense Authorization Act |
OSD |
Office of the Secretary of Defense |
OT | other transactions |
OTA | Other Transactions Authority |
PEM |
program element monitor |
PEO | Program Executive Office |
PM | program manager |
POR | program of record |
PPP | public-private partnership |
RCO |
Air Force Rapid Capabilities Office |
RPO | Rendezvous Proximity Operations |
SaaS |
software as a service |
SAF/AQ | USAF Office of the Assistant Secretary (Acquisitions) |
SAF/AQR | USAF Office of the Assistant Secretary for Science, Technology and Engineering |
SBIR | Small Business Innovation Research |
SIMAF | Simulation and Analysis Facility |
TRL |
technology readiness level (e.g., TRL 3) |
UAV |
unmanned air vehicle |
UFP | unit flyaway price |
USAF | U.S. Air Force |
USC | U.S. Code |
VC |
venture capital |