National Academies Press: OpenBook
« Previous: 2 The Problem as Seen by Senior Air Force Leaders: The Need for Experimentation
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

3

Air Force Practices Versus Best Practices

EXPERIMENTATION INSIDE THE AIR FORCE

WHAT THE RAPID CAPABILITIES OFFICE DOES THAT MORE OF THE AIR FORCE SHOULD CONSIDER DOING

David Hamilton CEO at Eagle Aerie, Inc., and Former Director of the Rapid Capabilities Office

The Rapid Capabilities Office Overview

The Rapid Capabilities Office (RCO) originated with former Air Force Chief of Staff General John Jumper and former Secretary of the Air Force James Roche to expedite the testing, evaluation, and fielding of select combat support and weapons systems for warfighters. At the time, Jumper and Roche were frustrated with the inability of the existing acquisition system to keep up with the demands of the warfighter and felt that the Air Force had lost something of its ability to develop and deploy new systems. The RCO was designed around a “return to the future” concept that would attempt to apply older methods from previous programs, such as the F-117 Nighthawk, to the modern acquisition needs of the warfighter. The RCO reports directly to senior leadership and maintains the largest classified fund and, at the same time, with the smallest staff of any Air Force Program Executive Office (PEO).

The RCO faced resistance from its inception. Several existing organizations in the Research and Development and Acquisition system opposed the organization because of the different procedures and processes it used. Bureaucratic “road blocks” were a regular occurrence. As a result, the RCO needed to be strategic with regard to how it dealt with other acquisition organizations; thus, every RCO project faced business as well as technical challenges.

Despite the perception that the current rules and regulations are too restrictive, the RCO found that it had quite a bit of leeway, as long as it had senior leadership support. The RCO leadership met regularly with Air Force senior leadership and monthly with the Secretary of Defense. The connection with top leadership was the key to the RCO’s success. The RCO avoided bureaucratic “roadblocks” by changing the terminology used by the RCO; involving a variety of different specialty personnel, including lawyers, contracting officers, financial personnel,

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

and so on; and ensuring effective communication with other acquisition groups. When the RCO encountered challenges, it could call upon top-level support to achieve its goals.

The RCO’s staff, in addition to being diverse in professional background and expertise, also consisted of handpicked personnel who demonstrated innovative thinking and risk taking. The RCO provided them with an environment conducive to innovation and risk taking. Additionally, the environment set up by the RCO and its connection to the warfighter changed the perception of system and weapons development processes. The connection to the warfighter and the necessity to meet the warfighter’s constantly changing needs greatly enhanced morale and motivation in the RCO and drove decision making and planning; the needs of the warfighter set the “milestone” schedule and pace of each project.

Washington, D.C., Air Defense System Experimentation Case Study

The RCO was tasked with setting up an enhanced air defense system for Washington, D.C., to defend against a possible terrorist attack. The RCO settled on a plan to deploy a ground-based AMRAAM (Advanced Medium-range Air to Air Missile) system. Air Force leadership agreed and wanted the system in place before the next inauguration. The RCO developed a plan to purchase, integrate, and deploy a Swedish ground launch system, using a Spanish communications configuration, employing existing AMRAAMs, and built to be air launched. The ground launch system would also need to integrate with existing radars and other defense systems. The RCO’s proposal was met with skepticism by other service branches and acquisition groups. Nevertheless, the tests of the system at Eglin Air Force Base were successful. The system was rapidly integrated with existing defense infrastructure and deployed around the capital. The combination of a critical need, senior leadership support, and a campaign of experimentation led to the rapid and successful deployment of an effective new air defense system in and around the capital.

GLOBAL HAWK: ADVANCED CONCEPT TECHNOLOGY DEMONSTRATION CASE STUDY ON SUCCESSES AND OBSTACLES

Robert Mitchell Consultant, Former Vice President for Northrop Grumman Aerospace Systems

Background

Global Hawk, initially called Tier II+, is a high-altitude, long-endurance, unmanned system comprising an air vehicle with an array of intelligence, surveillance, and reconnaissance sensors, a ground-control system, and communications architecture capable of command and control and sensor data dissemination world-wide. Global Hawk is now a well-established U.S. Air Force (USAF) program of record. Other variants include the U.S. Navy Triton, and there are German and North Atlantic Treaty Organization (NATO) variants as well as several systems under consideration by other nations.

Advanced Concept Technology Demonstration

What is truly unique about Tier II+ is the origin of the program as a Defense Advanced Research Projects Agency (DARPA) Advanced Concept Technology Demonstration (ACTD), developed in accordance with Section 845, “Other Transactions Authority.” The language included in the 1994 Defense Authorization Bill stated the following:

“Other Transactions” for prototype projects are acquisition instruments that generally are not subject to the federal laws and regulations governing procurement contracts. As such, they are not required to comply with the Federal Acquisition Regulation (FAR), its supplements, or laws that are limited in applicability to procurement contracts.

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

Aside from the technology challenge associated with such a system, the program was also an experiment in how to change the way we do business programmatically and technically; hence, the results are relevant for consideration by this panel.

Program Structure
Contracting Approach

DARPA leadership embraced the other transactions language, and the Tier II+ contract, awarded to Teledyne Ryan Aeronautical in mid-1995, was described as a “new way of doing business.” The contracting approach was unique; the initial one-page agreement asked for the best capability within a $10 million ($1994) average unit flyaway price (UFP) for vehicles 11 through 20. The agreement, signed by Larry Lynn, head of DARPA, and Robert A.K. Mitchell, president and CEO of Teledyne Ryan Aeronautical, also allowed that either party could “walk” at any time during development. The final contract included a performance specification, but all of the numbers were expressed as goals that could be traded, as necessary, to stay within the hard $10 million UFP requirement.

Program Management

Mr. Lynn established a policy of insight versus oversight, and Mr. Mitchell restructured Ryan Aeronautical to create a new commercial Strategic Business Unit. Leadership and staffing on both sides were highly selective, focused on engineering capability, and willing to work in a close knit government/contractor environment with a joint commitment to succeed. The program was run on an extremely aggressive schedule with tight control of requirements, disciplined configuration control, and an accurate understanding of program cost and schedule status. Critical decisions were made quickly and issues were resolved expeditiously, if necessary, by the creation of task teams and action centers. The entire organization was collaborative with little distinction between contractor and government. The program managers and every integrated product team worked together in the same physical locations. Neither party ever wanted to exercise its right to “walk.”

Technical Performance

There are many benefits for an ISR system that can operate at high altitude for long endurance. At 65,000 feet, the Tier II+ aircraft would fly in relatively benign atmospheric conditions and well above most convective weather. Flights above 60,000 feet are also above the U.S. Class A-controlled airspace and above all other traffic except for similar high-altitude aircraft such as the U-2, and from 65,000 feet, the sensors can cover a large geographical area. However, the design of such an aircraft is extremely challenging. To be successful, the Tier II+ unmanned air vehicle (UAV) had to be autonomous from take-off through landing, the sensors were required to provide coverage over large geographical areas to high resolution, and the communications architecture had to enable reliable command and control as well as wide-band sensor data transmission and downlink from halfway around the globe. And, in many respects, the operational environment for a high-altitude, long-endurance UAV is comparable to the spacecraft environment—power is limited, temperatures are extreme, and thermal management requires a complex balance of heat absorption and rejection through radiation. Fault tolerance, failure identification, very high system reliability, and the associated redundancy management system are absolutely critical to success. Aerodynamically, the aircraft had to operate in a narrow speed and altitude range bounded by stall limit and mach limit (often called the coffin corner). To achieve these capabilities, a four-part strategy of (1) experience, (2) off the shelf, (3) early test, and (4) margin for growth was adopted. All of the government representatives were experts. Ryan had decades of UAV experience, as had the other team members. All critical subsystems were selected from existing programs, and every aspect of the system was tested to failure, including the vehicle management system. A software stress-testing approach was developed at the program authorization to proceed (ATP) and continued up to first flight. By that time, every imaginable failure and the system response had been simulated or tested; there was absolutely no

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

tolerance for a failure in flight. The inevitable growth in size, weight, power, and software was planned for and was accommodated within the margins, so there was no need for any major weight reduction or other recovery initiatives. Moreover, the $10 million UFP was firm, so everything had to “buy its way” onto the aircraft; there were no capabilities that did not contribute to military utility.

Results

The results of the Tier II+ development program are worth noting. From a program management standpoint, all of the critical decisions were already made by mutual agreement, so program reviews were short and only a means of formally documenting the design progress. Every element of the system was tested to the extent there were no surprises. The schedule was 7/24 with no breaks (including holidays). Roll-out occurred at month 21 and the first vehicle was shipped to Edwards Air Force Base and was ready to fly in 24 months, effectively from a clean sheet start. At that time, however, a parallel program (Tier III Minus/Dark Star) had suffered a catastrophic loss during take-off, so the Tier II+ program went into a holding mode while all of the systems and software were re-checked and re-tested, delaying the first flight by 9 months. There were no significant changes made to the system, and the first flight actually occurred in February 1998, 33 months from ATP. Cost performance was also notable, and with the exception of some growth in the sensor area, the entire development was conducted within budget. Finally, the $10 million UFP would have been achieved except for congressionally mandated funding and procurement constraints.

Summary and Conclusions

The Tier II+ ACTD was a major success, due to the 1994 Defense Authorization Bill Section 845, “Other Transactions Authority,” and the ensuing commitment by both government and industry to work in a different and highly innovative way. The contracting approach was innovative, the program organization was innovative, and the design itself was innovative. The entire program could be described as a highly successful and innovative technology and organizational experiment.

QUICK REACTION EFFORT TO PROVIDE AN AIR-ENABLED CYBER CAPABILITY

Chet Wall 90th COS Technical Advisor, 24th AF, 688th CW, 318th COG

On January 27, 2016, Mr. Chet Wall presented his remarks to the workshop on the 24th Air Force’s “Quick Reaction Effort to Provide an Air-enabled Cyber Capability.”

EXPERIMENTATION OUTSIDE THE AIR FORCE

MOTIV SPACE SYSTEMS—A “NEW SPACE” COMPANY

Brett Lindenfeld Vice President of Operations, Motiv Space Systems

Innovation Lessons from a Startup

Motiv Space Systems was founded by three colleagues in 2014. We had previously worked together at another small business for several years, and before that had first started work at the Jet Propulsion Laboratory (JPL). Our previous small company had been acquired by a much larger aerospace and defense company. Ultimately, it became clear that there were still significant opportunities for the unique qualities a small business brings to the market

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

place. Thus, despite the risks involved in launching a startup, we felt compelled to start anew and Motiv was born. While there was risk, we recognized the potential for value creation for ourselves and our potential customers. There was also the opportunity to control our own destiny and to work with people that we enjoyed working with.

How We Managed Risk

At the outset, our approach to managing risk in a startup was to develop dual-use technology and products that would be developed through government contracts but could eventually find uses in commercial market segments. This approach would help to buffer the ups and downs in the government procurement cycle. We also adopted a broad systems engineering view and pursued opportunities in the motion control arena (broadly, “anything that moves”) so that we would not pigeon-hole ourselves when we did not know what would be successful. We were opportunistic and experimented with many ideas—for example, acquiring a license from Caltech technology that had been developed by one of our first employees when they were at NASA JPL. As a small company, we were nimble enough to decide and execute the license in a matter of weeks, when it could easily have taken more than a year at a larger, more bureaucratic organization.

Following these key decisions, we continued to develop a platform based on our skill sets and strove to identify how we could apply what we knew while opportunistically solving customer challenges. With each win, we gained recognition for our work to the point we competitively won a significant robotic element on the Mars 2020 Rover mission. Our work has focused on developing platform solutions—in robotics, in sensor systems, in robotic manipulators (robotic arms), and components and subsystems for the same—but we also began to experiment and integrate our solutions across these platforms to create novel versatile solutions such as an advanced Rendezvous Proximity Operations (RPO) sensor suite for a technology development program. Our view was that while some of these may be niche solutions, they could expand to promising high-value opportunities that we would not know unless we experimented with new ideas and put them in front of customers.

Key Factors for Motiv’s Success

Our ongoing success story could be summarized in terms of a few critical lessons.

  • Pursue incremental and iterative development. We actively experimented with different ideas and iterated until we got strong positive feedback in terms of contract wins.
  • Be opportunistic. We did not limit ourselves to building only products that we believed the customer would want, but rather based our product and service offerings on what the customer was telling us. If we did not have a particular capability, we would license it or build smart partnerships with other companies to deliver a solution that had clear value.
  • Stay agile. As a small business, we knew that we could not afford to have too many people onboard and not be able to support them as contract dollars fluctuated. Instead, we have maintained and continue to modestly grow a small staff of about 15 people, but we have access to flexible staffing through our team of subcontractors and consultants so that we have, effectively, a team of about 23 full-time employees. However, our relationships with our consultants, crucially, is based not only on economics but also on trust and mutual value. The consultants are treated just like full-time employees and fully integrated into projects but also allowed conveniences such as working remotely.

Many of these lessons have been drawn from the practices at the original Skunkworks, where a blindingly fast iterative development process, undistracted by bureaucracy, was the norm. Through such practices, Motiv has worked to value all of its employees as trusted partners and to respect them as talented, productive individuals. The result has been year-over-year revenue growth of 100 percent.

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
How Government Could Engage Small Business Better (for Mutual Benefit)

Our company has worked with different parts of the U.S. government and has observed that more could be done to efficiently harness the power, talent, and creativity within small businesses that want to do business with the government. For example, contracting processes could be simplified and shortened so that the focus is on unleashing the creativity of these companies rather than in zealously avoiding contracting mistakes. In our opinion, the government would be better off failing (making contracting mistakes) in some cases than expending the money and time to prevent failure with 100 percent certainty. By focusing on the contract alone and not trusting the power of the individuals and teams, a lot of value is wasted. Instead, the government should expand programs like the SBIRs (Small Business Innovation Research), which are good models of contracting approaches that are efficient and simplified and focus on trusting, empowering, and supporting the contractors, just as we do our employee-partners.

DISRUPTIVE INNOVATION: WHAT IT IS AND WHY IT IS IMPORTANT

TAXONOMY AND LANGUAGE OF INNOVATION

Joel Sercel Founder and Principal Engineer, Trans Astra

To innovate successfully, an organization must first distinguish between the different types of innovation, commensurate with its goals and limitations, before it can decide how to foster such innovation. For example, the innovation that sustains the current order and status quo (sustaining innovation) is often of two varieties—evolutionary innovation improves a product in an existing market in ways that customers are expecting (e.g., fuel injection system versus carburetor), while revolutionary innovation (also called discontinuous or radical innovation) is innovation that is unexpected but nevertheless does not destroy existing markets (e.g., automobile versus horse-drawn carriages), both providing similar solutions for human transportation.1

On the other hand, a disruptive innovation is an innovation that creates a new market and value network and eventually disrupts an existing market and value network, displacing established market leaders and alliances (e.g., the Ford Model T and mass production of lower-priced automobiles changed the transportation industry).2 Disruptive innovation may require an entirely different set of operating conditions and mindset than sustaining innovation, thereby creating near-term challenges to both the disrupted and the disrupter. The disruption, or upending, of the current ecosystem of a business is almost never embraced by “market insiders,” and the disrupters present difficulty initially in terms of lower gross margins, smaller target markets, and poor appeal of the disruptive product to customers, as measured by existing performance metrics. Only agile organizations that have the willingness to adapt have any chance of surviving the attack of the disrupters.

Some Historical Perspectives

Both sustaining and disruptive innovation are created by concerted efforts in experimentation, where the results may be unexpected success or failures but the learning paves the way for future success. A case in point is the HMS Mary Rose, one of the first British naval vessels to be fitted with broadside cannons and launched in 1512. It is considered to be a transitional point in naval warfare because coordinated volleys from all the guns on one side of a ship were possible for the first time in history. However, the ship was eventually sunk in battle, supposedly because its gun ports were left open when the ship was turning. Should the “broadside guns” concept be considered a failure? On the contrary, while the experiment was a failure, lessons from the experiment paved the way for major future improvements in naval warfare with lasting impact.

___________________

1 Clayton Christensen, The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail, Harvard Business School Press, Boston, Mass., 1997.

2 Ibid.

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

But why is it important to distinguish between sustaining and disruptive innovation? The reason is that efforts to focus on and foster sustaining innovation can actually stand in the way of and delay the significant value creation from disruptive innovation. For example, sustaining innovation and traditional thinking of naval staff continued to advance sail-powered naval vessels well into the 19th century, when in fact the future lay in steam-powered naval craft (e.g., HMS Dreadnought, 1906), which led to the disruption of traditional naval power.

Innovation Through Public-Private Partnerships: Some Historical Lessons

An important model for creating disruptive innovation is the public-private partnership (PPP) between the U.S. government and industry that launched the great American railroad and accelerated America to superpower status. While it can be argued that the companies that benefited received unfair advantages from the government, the end result was that the government had access to unprecedented transportation infrastructure that quickly accelerated the country to economic superpower status.

A similar model for PPP-driven innovation is currently taking shape in the commercial space industry under the auspices of NASA’s commercial orbital transportation services (COTS) program with existing and new entrants like Orbital ATK and Space X. While the United States dominated the commercial space launch market in 1980, with 100 percent market share, U.S. market share had vanished by 2002, simply because space launch costs were unaffordable to anyone but the U.S. government. With the launch of the COTS program, the United States has once again dominated the industry with 60 percent market share as of 2014. The reason may simply be that the government gave U.S. companies the freedom to exploit all commercial opportunities beyond its government contracts, and that allowed the companies to develop superior cost models that also created savings for the U.S. government. In addition to giving the contractors the “commercial upside,” some specific factors that drove the success of this new model were the following: fixed price contracts, a simple set of requirements, measurable objectives, competition, the customer’s ability and willingness to properly monitor technical progress, the customer’s ability to disengage from contractors when work performance did not meet objectives, and follow-on contracts from the government that sustained these companies.

Another PPP may be in the making for near- and deep-space exploration technology, with opportunities to create lower-cost space transportation infrastructure by private companies so that the materials needed to power deep-space craft may actually be sourced not on Earth but in space itself. Companies like Trans Astra, for example, are endeavoring to mine asteroids for water and carbon dioxide, some of the key ingredients of rocket propellants. Opening up opportunities and providing sustained support to new entrants with fresh thinking can in fact lead to a new set of disruptive capabilities in space, with potential for unprecedented value in many areas that are yet unimaginable.

Some Rules for Successful Innovation

Several key characteristics are critical to an organization’s ability to successfully create innovation, whether sustaining or disruptive, with most innovative organizations having a mix of both types of innovation.

  1. The mission has to be highly focused.
  2. The organization has to be flat, and lines of authority have to be clear (i.e., everyone knows who the boss is), so that decision making is rapid.
  3. Innovators have to be protected by senior leadership and relieved of distracting staff work, such as excessive documentation.
  4. The leader has to be an “innovator” in mindset, or else the other innovators will not follow him/her.
  5. The organization has to have the ability to hire (and fire) readily, based on a meritocracy, so that the best talent can be brought in and inserted into the right positions.
  6. The people should have the skills to communicate the new paradigm.
  7. The leadership has to encourage a culture of mission focus, not organizational focus, with incentives based on long-term rather than short-term performance.
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Final Thoughts

Disruptive innovation cannot be a luxury for the USAF enterprise, because if it focuses on sustaining innovation alone, it runs the risk of being disrupted by a competitor (in this case, a nation state). Disruptive innovation creates new paradigms that give unfair advantage to the disrupter by providing new and different ways of thinking. Such disruptive innovation requires safe spaces for innovators. Fostering a culture that embraces experimentation and innovation can create dividends in terms of disruptive potential, setting the USAF on a path to re-gaining its air dominance.

INNOVATION MODELS FROM THE VENTURE CAPITAL WORLD

LESSONS FROM VENTURE CAPITAL INDUSTRY: HOW INNOVATION AND RISK ARE MANAGED

Arif Janmohamed Partner, Lightspeed Venture Partners

Venture capital (VC) firms like Lightspeed from Silicon Valley have spearheaded several waves of disruptive changes in the technology industry, from enterprise infrastructure and mobile technologies to software-as-a-service (SaaS), through their judicious investments in and astute management of numerous startup companies, particularly in Silicon Valley.

While Silicon Valley startups are very different from large governmental organizations like the USAF enterprise, many of the lessons from the VC model may, nevertheless, be relevant to the Air Force’s experimentation campaign with regard to identifying and investing in the teams that work together, disrupt, innovate, and iterate toward success. In particular, the USAF could learn about how to think about innovation, how to organize teams to foster innovation, and how to cost-effectively take on large, expensive problems.

Recognize and Exploit Dramatic Innovation

Lightspeed Venture Partners is a global VC firm with a relatively small team of 18 investment professionals managing more than $4 billion in capital. The funds are invested over a 10-year investment horizon, deployed incrementally as needed to grow about 40 portfolio companies at any given time. Lightspeed typically invests in companies with product still at an early conceptual stage, or before customers have committed, which corresponds in Department of Defense (DoD) parlance to a technology readiness level (TRL) of 4 (TRL-4). The firm looks for companies with good “product-to-market fit” (i.e., customers want this product) and with the potential for large breakthroughs. Lightspeed’s partners then help the companies to mitigate technical risk (which is to be expected in potentially large breakthroughs) and/or market risk (i.e., Is there some indication that customers really want this?). The firm then follows a phased investment strategy, initially investing limited capital to address the problem cheaply and early (about $1 million to $10 million in the first 12-18 months). If the key questions are answered (and risk reduced), the company follows through on the funding commitment, to enable the creation of a “generally available” product, the build-up of the team, and/or the capture of beta customers ($5 million to $15 million over the next 18-24 months). Thus, Lightspeed has helped new companies to broadly disrupt the business of enterprise information technology—storage, servers, networking, cloud computing services, big data, SaaS, and so on—not only through creation of new business models, but also by leveraging the newest infrastructure and tools (e.g., cloud resources to build and test code) to get to a minimum viable product (MVP) sooner and to build the companies faster and cheaper.

Economics of the VC Model

The economics of the venture capital model are somewhat skewed, with 75 percent of the profits accruing to only the top 25 percent of the VC firms—that is, the gap between the best and the rest is large and is driven

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

by companies with true disruptive potential. Investments are typically made in the first 2 to 4 years of the fund’s life and they are “harvested” in the next 6 years. The firm undertakes an intensive vetting process with a final selection rate of only 0.4 percent.

The early-stage startup portfolio will have failures, but also has the potential to yield a few home runs that will more than pay for the rest. Typically, each early-stage investment fund will have some companies that will eventually go out of business (10-30 percent mortality rate); some that will do slightly better than break-even (returning 2-4 times the original investment within the 5- to 10-year investment horizon); and a handful that will significantly outperform, going public or being acquired at extremely high valuations (returning 10-30 times the original investment). Of course, it is very difficult to identify the winners in advance. So, rather than holding back on the number of investments and presuming some unrealistic confidence in being able to pick winners, the firm chooses to invest in a sufficiently large (yet manageable) number of companies to increase the odds of a home run, with clear “expectations of some failure” built into the model. This is the experimentation model in VC investing, supplemented by active company oversight and astute picking of teams with the highest potential for (and track record of) success.

The VC Process

The VC partnerships are typically small teams that are well aligned to the common goal of fund profitability. Therefore, they collaboratively make decisions yet compete for limited investment resources and exert most of their effort to find, justify, and make the big bets, because success requires making and winning the big bets. The partners therefore winnow more than 5,000 incoming investment leads to about 500, and then about 50 are sponsored by the individual partners and vetted by the firm as whole. With partners holding each other accountable, about 20 investments are made per year. In this portfolio view, the firm seeks to diversify investments but still holds out for the biggest and boldest opportunities in search of the home runs. With the recognition that risk and reward go together, the firm actively pursues risk reduction after the seed investment. It gets the portfolio companies to identify and reduce key technical risk, and to get customer feedback to see whether the customer really wants the product.

Alternate VC Models

Alternate VC models include the corporate venture capital model practiced by larger enterprises seeking to bring in external innovation to supplement their existing portfolio of goods and services. Analysis of investment value may not be as rigorous as at the pure VC firm, but the lesser rigor is offset by other strategic drivers such as reseller or original equipment manufacturer opportunity, “option” value (to have the right but not the obligation to acquire the company at a lower cost in the future), or even learning value (to learn about new markets as they evolve).

Characteristics of Winning Teams

Talent value in the team is perhaps most crucial. VC firms seek out the “10Xers”—that is, individuals whose output is 10 times that of their average peer. Examples include team founders who have made disproportionate contributions elsewhere. VC firms look for or assemble team members who have generally worked together successfully. They focus on teams of “doers”; no time for analysis/paralysis. Teams have to be in agreement, based on trust and honesty, and decision making in the best teams is typically driven by the leadership that is a “cult of one person.”

Freedom to Succeed Follows Freedom to Fail

VC firms encourage their portfolio companies to focus on speed, execution, agility, a willingness to risk failure, and succeeding by learning from failures. For example, the well-known tenets of Facebook (and by extension,

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×

Silicon Valley) are “Move fast and break things,” “Done is better than good,” and “What would you do if you weren’t afraid to fail?” The success of Facebook and others has encouraged a mindset of experimentation: put something out there (in front of a customer) and then iterate, instead of iterating to perfection in the absence of feedback. VC firms seek to foster a culture in which one asks, “If risk is expected and failure is accepted, what can you do?” and embrace entrepreneurs with bold visions who will be willing to pivot if their original vision is wrong. The mantra of the valley is “Build, measure, learn” (ref. Eric Reiss, The Lean Startup), “and if necessary, pivot, but pivot early and without fear.” Everyone is made to understand that there will be no serious repercussion for failure, provided one is honest in his/her assessment of opportunity and risk. Most importantly, the VCs emphasize that the opportunity cost of time is far greater than the opportunity cost of capital.

Some Recommendations
  • People. Build small, nimble, empowered teams that can pivot quickly. Find 10Xers: extraordinary talent requires extraordinary results.
  • Decision making. Avoid bureaucracy or analysis/paralysis in the decision-making process. Build, measure, and then honestly assess. If something is not working, cut bait early.
  • Managing risk. Innovation is, after all, a set of calculated risks. Teams should have bold visions and plan to disrupt big markets, but they should parse these big challenges into smaller manageable pieces and take incremental steps. The focus should be first on a MVP, to avoid over-committing early. Failure is ok and should be encouraged, but teams should fail fast. Teams should also iterate rapidly and constantly repeat the cycle of “measure, test, and repeat.” For government investments, playing it safe may save taxpayer money, but at the risk of losing time.
  • Celebrating success. Teams should celebrate incremental wins.
Great Innovation Examples and Their Key Features

Small nimble teams build iteratively toward a large bold vision. Apple’s iPhone design team initially had only 20 people and 100 at its peak. Amazon’s initial Cloud team had only 14 people but quietly created a $7 billion business.

Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 5
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 6
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 7
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 8
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 9
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 10
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 11
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 12
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 13
Suggested Citation:"3 Air Force Practices Versus Best Practices." National Academies of Sciences, Engineering, and Medicine. 2016. The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23602.
×
Page 14
Next: 4 Better Experimentation in the Air Force: Barriers and Levers »
The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop Get This Book
×
 The Role of Experimentation Campaigns in the Air Force Innovation Life Cycle: Proceedings of a Workshop
Buy Ebook | $9.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Workshop on the Role of Experimentation Campaigns in the Innovation Cycle was held in January 2016 to define and assess the current use of experimentation campaigns within the Air Force, evaluate barriers to their use, and make recommendations to increase their use. Participants at the workshop presented a broad range of issues, experiences, and insights related to experimentation, experimentation campaigns, and innovation. This publication summarizes the presentations and discussions from the workshop.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!