3

Experimentation for Innovation: Current Air Force Practice

Having summarized the best practices observed in highly innovative organizations, the committee now turns to investigating the nature of each of these practices as observed in today’s USAF.

DEFINING THE PROBLEM

The Valley of Death

In research on innovation, the so-called valley of death refers to the organizational commitment missing during the middle stages of the innovation life cycle as the innovation matures from a research project to an operational use. As was explained in Chapter 2, it typically takes some combination of push and pull to move an innovation across the valley of death. In the benchmark organizations studied, the push often came from the research and development community, while pull came from the organization’s need for a new or more cost-effective solution.

The committee discovered a very clear valley of death problem in the USAF, but when it came to finding the combination of push and pull needed to cross the valley, the committee found a situation that differs importantly from the highly innovative organizations it had studied. In a nutshell, the committee found some examples of the pull needed to get innovations across the valley of death, but it also found evidence of pushback from a powerful normal production organization.

The committee’s research led it to identify four imperatives at work within the USAF, all with implications for innovations and their ability to make it across the valley of death.

1. Innovation imperative—the place one is most likely to find technology push. A laboratory pushing to see its latest research and development (R&D) applied to the development of a new system would be a classic example of this imperative at work.
2. Operational imperative—the most likely home for operation pull. When warfighters clamor for a better technological solution to a problem they face in the field, the operational imperative is at work.
3. Economic imperative—another kind of pull associated with the search for more cost-effective solutions. In the past few years, the Air Force has been under tremendous pressure to find more cost-effective technological solutions, and this is evidence of the economic imperative at work.
4. Programmatic imperative—where a large part of the organization is focused on just fulfilling today’s mission. When an acquisition program seeks to keep to its schedule, or a logistics center works to the drumbeat of production, these are both examples of the programmatic imperative at work in what has been labeled the normal production organization.

While it often takes the combined efforts of push and pull to get a technology across the valley of death, research has consistently shown that forces like those associated with the programmatic imperative at work in the normal production organization are often more likely to push back against new technological solutions than to assist in pulling it across the valley of death.¹ This is especially true of the more disruptive innovations, because, by definition, disruptive innovations place at risk the ability of the organization to fulfill its programmatic imperative.²

The combination of push and pull is present when innovations are being offered as solutions to the operational or economic imperative; advocates of the new solution find ready partners in those looking for better operational or economic solutions. However, there is often passive—or even active—pushback against innovations from the part of the organization responsible for the programmatic imperative. It is a struggle to cross the valley of death even under the best of conditions, but when a large and powerful part of the organization offers passive or active resistance to the innovation, the challenge is often insurmountable. This is

___________________

¹ V. Govindarajan and C. Trimble, The Other Side of Innovation: Solving the Execution Challenge, Harvard Business Review, Boston, Mass., 2010.

² C.M. Christensen, 1997, The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail, Harvard Business School Press, Boston, Mass.

the situation limiting many of the experimentation-driven innovation efforts the Air Force desires. This is what Air Force leaders are sensing when they say it seems as if there is no space for experimentation and innovation in today’s Air Force.

And if there is no space for experimentation, how likely is it that there is space for the sorts of experimentation campaigns often needed to bring complex technological innovations to fruition?³

Smart Experimentation Campaigns

The committee has explained the importance of experimentation campaigns and described what constitutes a good one. Earlier, the committee detailed examples of experimentation campaigns (low observables and Surfing Aircraft Vortices for Energy [$AVE]) were both USAF examples. However, in the course of the study, the committee heard from several Air Force leaders who felt that the organization was suffering today from far too few well-designed and executed campaigns. During the interviews, numerous technologies (such as data to decision, directed energy, hypersonics, autonomy, and cyber warfare) were listed as key technology focus areas. While any one of these technologies has the potential to contribute substantially to future air supremacy, it was difficult to find anyone who could articulate the assumptions underlying work in these areas or set forth a coherent plan for experimentation campaign to test or inform these assumptions.

There is an effort to reboot campaigns as a normal way of life in the Air Force. The general approach is depicted in Figure 3-1, and the specifics entail pursuit of two experimentation campaigns in fiscal year 2017: Data to Decisions and Defeating Agile Intelligent Targets.

Both of these experimentation campaigns stemmed from the Air Superiority 2030 enterprise capability collaboration team (ECCT) work completed in 2016. To underscore USAF support of experimentation campaigns as well as future development planning efforts, including future ECCTs, the Air Force Chief of Staff and the Secretary of the Air Force signed the Charter for Air Force Capability Development on June 2, 2016 (see Appendix D, a complete reprint of the Air Force charter). The charter establishes governance and oversight authority for experimentation campaigns and ECCTs.

___________________

³ A good example of this is the NASA Solar Electric Propulsion Technology Application Readiness (N-Star), which was a NASA program that was stuck at technology readiness level (TRL) 4 for 20 years. The principle investigator (PI) on the program was committee member Joel Sercel from the Jet Propulsion Laboratory. The PI obtained funds for the program to jump the gap from TRL 4 to TRL 6. As a result they were able to perform a flight demonstration on Deep Space One under the New Millennium Program. The mission was successful and brought the program across the Valley of Death. The technology is presently TRL 9 and is in operational use on the Dawn 2 probe mission to the asteroid Ceres.

Further, as reported by committee’s point of contact at the office of the Assistant Secretary of the Air Force for Acquisition and Research (SAF/AQR), Jerry Lautenschlager:

No one thinks the experimentation campaigns described above will supply the scale and scope of experimentation and innovation needed for the USAF overall. However, they are tangible steps laying the foundation for what might be far greater

___________________

⁴ Jerry Lautenschlager, e-mail message to committee study director, July 19, 2016.

levels of experimentation in the future. The committee will explore options for further increasing the use of experimentation campaigns in Chapter 4.

Recall that Edisons and Edsels in Chapter 2 were terms the committee coined to capture the distinction between two different types of failures repeatedly encountered in its research. Although disappointing outcomes may result from a well-run experimentation campaign, Edisons are useful because they answer the question of whether or not something will work. Notwithstanding this, they are disappointing because the most common answer is that the idea tested did not work as hoped. Technological progress typically entails trying a lot of ideas before finding one that works, so Edisons are fundamental to good research and should be encouraged. Edsels, on the other hand, are the end result of no research and/or poorly executed or underutilized research, and they should be avoided.

The committee has observed⁵ that the USAF, as an organization, exhibits a fear of failure. Said in a more positive manner, the USAF is an organization that is tremendously interested in delivering good results. Either way, the net result is a strong organizational aversion to any type of failure. Consequently, the Air Force fails to distinguish between Edisons and Edsels. It is acceptable for the organization to reject the type of failure the committee is calling Edsels because these are the antithesis of good research, experimentation, and innovation, and they should be abhorred. However, in its efforts to avoid these failures, the organization has too often failed to distinguish between them and Edisons. But Edisons are fundamental to scientific progress—that is why so many innovative organizations operate with the mantra “fail fast.” If the USAF is going to achieve the level of experimentation and innovation its leaders’ desire, it must learn to distinguish between the two types of failures and encourage Edisons as the one best hope of avoiding Edsels.

Fortunately, the Air Force has a tremendous history of Edison-type progress it can draw upon. As just one example, consider the tolerance of Edisons that was evident in General John Jumper’s push to put Hellfire missiles on the Predator unmanned air vehicle (UAV).⁶ He knew from his earlier experience in U.S. Air Forces Europe (USAFE) and the Balkans that there was a need for something like a weaponized UAV. After pushing for this, the general was being briefed on two options, neither of which was very appealing to him. One was an accelerated 3-month effort for $1.5 million that would entail high technical risk and disrupt both Big Safari’s other Predator projects and Air Combat Command’s plans for

___________________

⁵ Committee observations are based on multiple interviews with individuals from the Air Force, other Services branches, government, industry, and academia. See Appendix C for a list of guests and speakers interviewed by the committee.

⁶ R. Whittle, Predator’s Big Safari, Mitchell Institute for Airpower Studies, Mitchell Institute Press, Arlington, Va., August 2011, https://higherlogicdownload.s3.amazonaws.com/AFA/ccb3323f-aa8a4ca0-8733-f03efbe5e361/UploadedImages/Mitchell%20Publications/Predator’s%20Big%20Safari. pdf.

operational test and evaluation. The second was a 9-month effort for $1.3 million with medium technical risk.

General Atomics was the contractor for the Predator, and the company’s president, Tom Cassidy, was sitting in the audience listening to the briefing. About three-quarters of the way through the briefing General Jumper turned to Cassidy and said, “What do you think about all this?”⁷

“Let’s go in your office,” Cassidy said, which they did. When they were alone, Cassidy told Jumper, “You give us 2 million bucks and 2 months and it’ll be a done deal.” General Jumper replied, “Done,” and the rest is history.

Note that General Jumper not only tolerated the risk of disappointing results associated with high technical risk but also committed additional funds to accelerate the program—even more than the most aggressive alternative originally given him. He was obviously more concerned about going to battle with Edsels than risking an Edison in trying to develop a better warfighting system. The Air Force has a proud tradition of taking risks in pursuit of groundbreaking, war-winning innovations. In Chapter 4, the committee will look at steps to be taken for the Air Force to get back on track.

EXPERIMENTATION LEADERSHIP AND ORGANIZATION

In its research on best practices, the committee discovered that highly innovative organizations are very consistent in having individuals assigned to the role the committee is calling the “Innovation Catalyst.” In short, the essential requirements of an effective Innovation Catalyst include strong and clearly identified ownership, with a sense of personal responsibility for driving experimentation and innovation, and a direct reporting line to the senior leadership.

In its research on current practices in the USAF, the committee found individuals acting as Innovation Catalysts only in isolated pockets such as Special Operations Forces Acquisition Technology and Logistics and the Rapid Capabilities Office. It did not find the equivalent of the sort of corporate-level chief technology officer it found in the C-suite⁸ of many highly innovative organizations. While instances of innovation were observed throughout the Air Force, the committee did not find the extensive fractal-like repeating pattern of Innovation Catalysts scattered across the larger organization it studied. Instead, it found there was widespread reliance on committees, groups, and processes to get things done. This may be appropriate for operating in the normal production part of an organization,

___________________

⁷ Ibid, p. 19.

⁸ The C-suite includes all those senior company executives who report directly to the chief executive officer of a company (e.g., the chief financial officer (CFO), the chief operating officer (COO), or the chief technology officer (CTO).

but it is entirely out of line with what was observed in the parts of highly innovative organizations responsible for developing new, and often disruptive, solutions.

The absence of clearly identified Innovation Catalysts is the single most glaring difference between the USAF today and the type of highly innovative organization it aspires to be. This absence of Innovation Catalysts is both a symptom and a cause that goes a long way toward explaining why senior Air Force leaders felt there was no space for experimentation-driven innovation in today’s Air Force. Closing this gap is a priority in the recommendations given in Chapter 4.

EXPERIMENTATION PROCESSES AND TOOLS

Just as it had when it looked for leaders playing the role of Innovation Catalysts, when it looked for the experimentation processes and tools it had found in highly innovative organizations, the committee found only isolated pockets of excellence. But, these are not enough to deliver the level of innovation the Air Force needs or desires. Consider three examples.

Experimentation Tools

Experimentation and prototyping were integral parts of the Air Force’s success in developing low observable aircraft, as was detailed in Chapter 2. However, it cannot be said that experimentation and prototyping are widely seen norms in today’s Air Force. The Defense Acquisition University is developing some basic online training on prototyping and experimentation, but there is no regular widespread training on these in the Air Force today. While it is easy to find use of any one of the classic tools of experimentation in the USAF today, their use lacks the breadth and depth needed to deliver adequate scale and scope of innovation. And, there is an even larger gap when looking at the frequency with which these tools were mixed and matched to carry out a multistage experimentation campaign.

Makerspaces

The Countermeasures Hands-On Project (CHOP) Shop at Kirkland Air Force Base is a good example of a makerspace in today’s USAF. CHOP brought together small teams of competitively selected junior officers for short-term (4-month) efforts to identify potential countermeasures to U.S. weapon systems and to build hardware prototypes. This opportunity was open to a small number of people (usually four) for a limited period of time. It blends research, experimentation, and making actual hardware systems for demonstration purposes, so it has many of the attributes of similar efforts observed in highly innovative organizations, but it is another isolated example and lacks the scale or scope needed for the USAF as a whole.

Partnerships

The USAF recognizes the opportunity to better utilize the Small Business Innovative Research (SBIR) program (Box 3-1) to further strategic partnerships between the Air Force and highly innovative smaller firms.

SAF/AQ and Air Force Materiel Command are presently working together to further this effort.⁹ Recognizing the challenge in transitioning successful innova-

___________________

⁹ D. Shahady, SBIR Program Manager, Air Force Materiel Command (AFMC), “Air Force SBIR/ STTR Program Overview Briefing,” presentation to the committee on May 19, 2016.

tion across the valley of death, the USAF is now committing 1 percent of the SBIR budget to accelerate SBIR transitions as part of the Commercial Readiness Program (see Box 3-1). As the Air Force’s SBIR program is being updated, an opportunity exists to align it more closely to experimentation campaigns. In this way, investments will be viewed not only as individual innovation projects but as part of a portfolio of new concepts that add to Air Force capabilities.

What these examples have in common is the isolated, temporary, limited nature of their use. The Air Force clearly knows how to run experimentation campaigns to develop advanced technology solutions, as evidenced by its work on low-observable aircraft. Yet this type of experimentation and prototyping appears to be more of an exception than a rule. Rather than open processes where all interested personnel are invited to participate, participation in the CHOP Shop examples is heavily filtered, allowing only a small number of competitively selected junior personnel to participate, and at the end of the competition or activity, those personnel are sent back to their respective units, where they resume business as usual. The once-in-a-lifetime nature of these efforts is a stark contrast to the type of ongoing, enterprise-wide innovation culture that is found in the other highly innovative organizations studied by the committee. And, while there are many examples of the power of partnering with small, innovative firms, the SBIR program in the Air Force only highlighted the need for greatly ramping up this type of partnering.

In short, it is clear that the Air Force does not need to discover what works to deliver more experimentation-driven innovation. Rather, the challenge is getting the organization to actually put to use what it already knows will work. This is a natural lead into a discussion of culture in today’s Air Force.

PEOPLE AND CULTURE

Air Force personnel currently receive mixed messages about experimentation-driven innovation. The committee interviewed speakers and panelists from the Air Force across science and technology, acquisition, and sustainment organizations about their perspective on challenges to experimentation and innovation. Their perception was that standardization and compliance with their associated formal processes and procedures are generally at odds with experimentation and innovation. The present culture is oriented to checklists and is perceived as intolerant of deviations and risk.

While there is talk about the need for experimentation and innovation, the message is drowned out by a combination of dwindling allocations of resources, organizational norms, and procedures that run counter to experimentation and innovation.

• The Air Force provides extremely limited training, mentoring, formal support, or instruction for innovators and experimenters. Instead, the educational focus is on courses from the Defense Acquisition University, which focus on complying with the formal acquisition system.
• Rather than having access to dedicated experimentation facilities, interested innovators must scrape together supplies to address emerging opportunities as they arise. The Air Force workforce generally has limited access to hardware, software, facilities, and funding related to experimentation and innovation. In fact, cadets at the Air Force Academy or undergraduates at engineering universities have easier access to experimentation tools and makerspaces than do Air Force lieutenants and captains. This means that upon commissioning, junior officers take a big step backwards in their ability to design and perform experiments.
• Similarly, the Air Force workforce does not have widespread access to training and education opportunities on the topic of experimentation and innovation. Personnel who are interested in learning about how to innovate or how to design and perform experimentation are left to their own devices to find training and educational opportunities.
• Leadership tends to pay more attention to compliance metrics than experimentation-oriented metrics. Promotions and incentives are oriented around standardization and process compliance, and the innovator runs the risk of not being promoted. Inspections by the Air Force Scientific Advisory Board and the Inspector General tend to focus on checklist compliance rather than on identifying and rewarding a sign of innovation and experimentation.
• Books, articles, and papers about experimentation best practices are not widely read or shared, while training and education focuses on learning the (very complex) acquisition process rather than on experimentation and exploration.

What should be systems enabling innovation are actually systems that disable innovation. For example, the committee repeatedly heard concerns about contracting being too constraining to effectively engage in experimentation. The individuals voicing these concerns contended that exploratory work [was] “fuzzy and messy” by nature but contracting procedures forced [innovation teams] to follow a production mindset that seemed at odds with an investigative mindset. Arguably, requirements and contract deliverables in an early-stage investigative program cannot be the same as in a mature development program, but the same contracting processes and requirements are often applied to both. Contracting and acquisition obviously affect technological innovation, but the committee’s interviews pointed to the Air Force’s human resource system as having the single strongest negative impact on innovation (Box 3-2).

While the examples presented in this report offer only a sample of the mixed messaging the committee heard during its research, it is sufficient to make the

point. As demonstrated in its various isolated pockets of success, the Air Force knows how to use experimentation to drive innovation. However, despite the clear need for greater innovation, the Air Force has in place a number of practices, procedures, norms, and systems that discourage experimentation and innovation. The result of this mixed messaging is a population of Air Force acquisition personnel who tend to be implementers and supporters rather than visionaries and innova-

tors. As explained by one young airman, “If you want us to be more innovative, a good place to start would be to stop finding ways to punish us when we try to do something different.”

This sentiment is a good lead in to the next chapter, in which the committee considers other steps the Air Force could take to close the gap between best practices in highly innovative organizations and current practices in today’s Air Force.