4

Findings and Recommendations for Closing Gaps

In Chapter 2, the committee discussed the most important and salient best practices it found in consistent use across the highly innovative organizations it studied. In Chapter 3, it discussed how virtually all of those practices were being implemented in isolated pockets across the USAF, but not with the scale or scope sufficient to address the tremendous need for experimentation and innovation across the Air Force as a whole. The committee concluded that the Air Force knows what needs to be done to use experimentation in driving innovation but is struggling to actually employ that knowledge in the widespread way needed.

In this chapter, the committee will look at what the Air Force might do to close the gaps between its current practices and the best practices seen in highly innovative organizations. As will be seen, most of these recommendations do not involve completely new ideas. Rather, they focus on getting greater use out of ideas the Air Force already understands and simply needs to put to better use. The discussion begins by summarizing key findings from the committee’s research. Next, it will summarize the most important steps the Air Force can take to close the gap between where it is today and where it needs to be tomorrow. Finally, it will close the chapter by linking its findings and recommendations to the relevant statement of task guiding the work.

FINDINGS

Finding 1: There is too little space, time, and funding for experimentation-driven innovation in today’s Air Force. The USAF has a rich heritage of innova-

tion through experimentation, but today, in the struggle between the priorities of normal production versus those of innovation through experimentation, normal production is winning. The Air Force needs a revival of innovation through experimentation, with emphasis on rapid prototyping, experimentation, and learning through failure (as in “big wins from small failures”). A few organizations, such as the Rapid Capabilities Office and the U.S. Special Operations Forces Acquisition, Technology and Logistics (SOF AT&L), have successfully created such environments, but these isolated pockets cannot function at the level sufficient to meet the total competitive threat facing the Air Force. This is not news—in fact, it is the reason why the committee’s research was commissioned in the first place. The committee is simply confirming the reality and significance of the problem.

Finding 2: Well-designed and executed experimentation campaigns are critically important drivers of innovation. A number of factors impact the overall level of innovation achieved by an organization, but experimentation is arguably the single most basic driver of innovation. There is an established body of literature on how to design and execute good experimentation campaigns, and the Air Force has enjoyed important historical successes with using experimentation campaigns to increase the speed and decrease the risks associated with developing and delivering new capabilities. However, today the Air Force lacks experimentation on the scale and scope needed for innovation to address an increasingly demanding set of missions in the face of increasingly tighter and tighter resources.

Finding 3: When it comes to experimentation, a fear of failure is crippling the Air Force today. Today’s Air Force work environment is so risk-averse that it discourages risk of failure even when there is an opportunity to learn through failure. Applying labels developed in earlier chapters, the organization fails to distinguish between two very different types of failures, which the committee calls Edisons and Edsels. The Air Force fears even relatively minor Edison-type failures and seeks to avoid and eliminate them even when they are the key to averting the much more harmful Edsel-type failures that are far greater threats to the Service’s ability to fulfill its mission.

Finding 4: Dedicated leadership in the form of “Innovation Catalysts” is the essential foundation on which innovation through experimentation must be built, a foundation largely missing in today’s Air Force. In the highly innovative organizations the committee studied, there are positions in which a single named individual is responsible for working with other senior leaders to maintain the strategic technical vision and for leading campaigns of experimentation and innovation to fulfill those visions. The committee calls these individuals Innovation Catalysts. The USAF, however, has only a few such individuals assigned to modest

roles in a few isolated pockets. Elsewhere, the organization seems to be relying on innovation led by committee-based processes. These are not as effective in producing the disruptive innovations critical to long-term strategic success as experimentation efforts led by focused and empowered leaders.

Finding 5: The tools and processes essential to Air Force success with innovation through experimentation are not in place. Except in a few notable areas, current Air Force processes and tools for experimentation and prototyping are ineffective at producing rapid innovation on the scale needed by the Air Force.

Finding 6: Metrics need to be carefully tailored to specific situations or they can do tremendous harm. To the extent the committee observed metrics at work, they were generally metrics in use by the normal production organization. These focus attention on consistent, uninterrupted output, and they provide a basis for pushing back against disruptive innovation. The committee did not find evidence of a countervailing set of metrics intended to drive experimentation and innovation across the USAF.

Finding 7: The culture in today’s Air Force is not one supportive of widespread experimentation, especially those leading toward disruptive innovations. In the highly innovative organizations studied by the committee, innovation is a byproduct of a reward structure and work environment in which innovations and innovators are valued as precious resources. In these organizations, leaders, peers, and organizational systems are all aligned to emphasize and encourage innovation through experimentation. While Air Force leaders are calling for more experimentation and innovation, they oversee an organization that makes both experimentation and innovation difficult and unattractive to many USAF personnel.

Consequently, the committee heard several presenters report that the Air Force currently has in place practices, procedures, norms, and systems that discourage experimentation and innovation. These practices create a mixed message. Two areas stand out as inhibitors: human resources and contracting (see the Chapter 3 section “People and Culture”). The committee repeatedly heard concerns about contracting being too constraining to effectively engage in experimentation—namely, requirements and contract deliverables in an early-stage investigative program cannot be the same as in a mature development program, but the same contracting processes and requirements are often applied to both. Nevertheless, while contracting presently inhibits technological innovation, the system most often perceived as having the single strongest negative impact on innovation in the USAF is its human resource system (Box 3-2).

The committee also found that the operational community is often an overlooked resource in the process of concept development and subsequent experimentation with new technologies. An example of how operators can enhance innovation and experimentation is the U.S. Navy submarine force’s initiative to reclaim acoustic superiority in the 1990s, during the post-Cold War period (Chapter 1). Submariners on the front lines were given the opportunity to participate and be change agents. When implemented, the operators’ participation contributed significantly to rapid and spectacular system performance achieved on early deployments. The program’s strongest advocates remain the operators in the field.

Finally, while the committee found that the contracting function is perceived to be an inhibitor to innovative solutions, the committee recognizes that it is an entirely different thing to suggest how the complex contracting function should be altered in this study. Likewise, the human resource function is also perceived to inhibit the hiring, incentivizing, and promotion of innovators. Innovation is only one objective of these important support functions, so it would be presumptuous to suggest altering them for the sake of innovation alone. Furthermore, the committee did not study these innovation and experimentation obstacles in particular, so any recommendations based on this finding would need to be based largely on prior experience rather than on new ideas discovered by the committee.

RECOMMENDATIONS

Recommendation 1: The Air Force should determine where it most critically needs innovation, and establish Innovation Catalysts to help drive experimentation and innovation in those areas. Based on what was observed in highly innovative organizations, an Innovation Catalyst can make important contributions at various levels of an organization. All the highly innovative organizations studied had a very senior leader working as an Innovation Catalyst at the highest levels of corporate leadership, usually with a title such as the corporation’s chief technology officer, but always as a key member of the organization’s C-suite. Beyond this, the role of Innovation Catalyst in larger organizations was replicated in a fractal-like pattern wherever innovation was critically important. This is not an investment or commitment to be taken lightly, but given the enormity and complexity of the USAF and the tremendous need to increase innovation simultaneously on several different fronts, it is reasonable to expect the position of Innovation Catalyst to be replicated at several key locations. The success of the isolated pockets of innovation already working in the Air Force is strong evidence that Innovation Catalysts will be effective at a variety of levels and locations across the organization.

Recommendation 2: Wherever they are established, the Innovation Catalysts should be directly linked to their senior leadership. All the highly innovative or-

ganizations studied had a corporate-level Innovation Catalyst, typically called the chief technology officer (CTO), but most of the highly innovative organizations also mirrored the corporate-level CTO with similar positions, working at a different level and scale, elsewhere across the organization. (The only exceptions to this were the relatively small organizations in the committee’s sample.) But regardless of where they were located, all the individuals working as Innovation Catalysts were engaged in developmental planning and the related experimentation campaigns spinning off of that developmental plan.

Figure 4-1 shows the connection between development planning and the experimentation campaigns.

In Figure 4-1 the committee endeavors to show the relationships among the new organizations established by the Charter for Air Force Capability Development in headquarters USAF, the Assistant Secretary of the Air Force (Acquisitions), and the Air Force Material Command (AFMC); the bottom-up operational community(an essential resource) in the process of concept development and

subsequent experimentation with new technologies; as well as collaboration with experimenters outside the USAF, including innovative government agencies like NASA and commercial entities, an often rich source of innovative concepts and technologies.¹

To be effective at this work, it is critical that the Innovation Catalysts have regular and direct access to their senior leadership. For example, if the Air Force established a position for a “corporate” CTO-type position at the top of the organization, the new CTO would report directly to the senior-most leaders in the organization. And, if a particular command created an Innovation Catalyst position, that person would report directly to the senior-most leaders in that command. It may also make sense that these lower level Innovation Catalysts have some sort of “dotted line” relationship to the senior-most, corporate-level Innovation Catalysts or the CTO, but until it is known where and how many Innovation Catalyst positions are to be established, it would be premature to address issues of reporting relationships between these offices.

Recommendation 3: Air Force leadership should move proactively to create organizational space for experimentation-driven innovation. One of the primary responsibilities of an Innovation Catalyst is to identify, manage, and protect (or fence off) strategically important initiatives leading to disruptive innovations. The committee has already seen that without this, the normal production organization will resist, and likely kill, the disruption innovations in favor of sustaining innovations. This is not to say that sustaining innovations are unimportant. In fact, in some situations, the Innovation Catalyst will need to provide oversight of

___________________

¹ On June 2, 2016, the USAF Chief of Staff and the Secretary of the Air Force signed the Charter for Air Force Capability Development (the complete document is reproduced in Appendix D.). The following are the first two paragraphs of the Charter:

program-related, sustaining initiatives. Balancing sustaining versus disruptive innovations is a critical part of the Innovation Catalyst’s job, and this balance would be reflected in the external/internal balance struck when the office is first established. Given the pendulum appears to have swung far in the direction of stressing normal production over disruptive innovation, the committee suspects that most Innovation Catalysts are needed to help shore up the level of disruptive innovation, so the logical choice will be to emphasize their independence from the normal production part of the organization, but this is a decision to be considered and made carefully, on a case-by-case basis. (Refer back to Chapter 2 and Figure 2-11 for more discussion of external, internal, and balanced approaches to organizing the office of the champion of innovation.) Finally, wherever they are located and however they are organized, the Innovation Catalysts would have authority to set priorities and would have discretionary control of a significant “innovation fund.” Without this, they are toothless tigers, an expensive bother unlikely to take much of a bite out of anything.

Recommendation 4: The Innovation Catalyst should establish a portfolio of proven management tools and disciplined approaches for experimentation based on established best practices. The committee has identified several of the best practices of highly innovative organizations it finds most salient to the Air Force. Beyond committee research, the Air Force can draw upon a useful body of knowledge captured in the extensive literature on managing innovation (see Box 2-1).The tools and processes about which the committee is most enthusiastic include these:

• Sandboxes. Protected space in which it is safe to experiment.
• Classic experimentation tools. Scenario planning, hypothesis testing, analysis, modeling, simulations, prototyping, and gaming.
• Makerspaces. Do-it-yourself space for discovery by hands-on building.
• Partnerships. Especially those involving end users and highly innovative contractors.

Recommendation 5: The Air Force should carefully and cautiously consider metrics, as bad metrics could quickly derail any effort to stimulate greater experimentation and innovation. The power of metrics is widely recognized, as is the danger of unintended consequences if the organization gets its metrics wrong. The Air Force seeks to create space within which innovation, especially disruptive innovation, can be protected from the normal production organization. But, metrics, by their very nature, lend themselves to use in managing in the normal production organization. Therefore, it would be all too easy to identify metrics that sound reasonable, but actually push the organization away from innovation

while strengthening Normal Production. Consider the case of a metric tracking the number of research and development (R&D) initiatives transitioning to programs of record. At first blush, this is a reasonable metric for evaluating the efficiency of the R&D process in terms of the amount of useful “product” it yields. However, if this metric were taken to its illogical extreme, as large organizations are wont to do, what would be the unintended consequence? It might well be that an organization seeking to maximize the efficiency of its R&D effort would cherry-pick safe projects offering only incremental advances on proven solutions. This is not the sort of outcome the Air Force is seeking, so it would need to be very cautious in identifying this type of metric. Given the complexity of identifying metrics that will not backfire, and the uncertainty of where the Air Force will focus its innovation efforts in the future, it would not be reasonable or responsible to identify the specific metrics at this time. What the committee can safely recommend is that metrics need to be established wherever an Innovation Catalyst is established, that these metrics need to reflect the particular strategies and organizational circumstances facing that particular Innovation Catalyst, and that any proposed metric be subjected to the simple acid test: What could be the unintended consequences if the organization takes this one metric to its illogical extreme?

Recommendation 6: Senior leaders should establish a clearer set of messages and incentives encouraging a culture of experimentation and risk-taking. These messages would strongly encourage “innovation” and cultivate “innovators” by rewarding them where appropriate and recognizing the virtues of this culture whenever possible. They would make a deliberate, strategic effort to leverage all four influence channels (Leadership, Peer Networks, Education and Training, and Enabling Systems) to foster a culture that embraces innovation through experimentation campaigns. As an example, the Air Force reached a tipping point in its appreciation for “space capabilities” as a warfighting multiplier during and after Desert Storm. One way it sought to enhance this appreciation across the entire Air Force was to create a course in Air and Space Basics for all new airmen coming into the Service and to create similar courses in all professional military education opportunities. A similar such step creating an awareness for the value (and limitations) of innovation, could go a long way toward changing the culture to an Air Force of innovators. Likewise, the guidance that promotion boards occasionally receive today concerning the value of “acquisition professionals” could be modified to ensure that innovation is recognized, appreciated, and, even, where appropriate, rewarded. In addition, the Air Force can examine the need for unique career paths for some innovators that could allow them to have successful careers rather than being punished, or not promoted, because of their unique contributions.

As pointed out earlier, the use of incentives to help drive innovation and experimentation is a valid leadership tool. The Air Force has numerous examples of

offering various incentives to show its support for addressing operational needs or to stimulate new concepts and technologies. A special example of this was the establishment of unique, highly selected organizations whose sole purpose was to define innovated solutions to problems. In the early 1990s, the Commander of Air Force Systems Command (AFSC) established a program it called High Gear to develop rapid, innovative, cost-effective capabilities to respond to problems identified by the operational major commands. System developers, testers, and, even supporters [financial/contracting/logistics] were motivated and first incentivized by being selected to man this High Gear team based on their proven ability to be innovative. Second, the results of their efforts were both recognized within AFSC, as well as the Operational Commands which benefitted from their activities. Even funding for these High Gear efforts were an incentive, since they generally came from a pool of funds from either the AFSC or MAJCOM, separate from other funding appropriations.

Another unique demonstration of using incentives were programs that addressed specific needs such as those meant to support morale, welfare, and, recreation (MWR). One previous commander at AFMC established a small amount of annual non-appropriated funds to support innovative, quick “projects” done by base personnel to address MWR needs on the Base. The incentive was to be selected to receive these funds by having the best ideas.

STATEMENT OF TASK

In the final section of this chapter, the committee reviews the statement of task guiding this study and links the tasks to key findings and recommendations. The points of overlap are summarized in Table 4-1 and discussed below.

1. Define the current state of practice of experimentation within the Air Force (AF). This study was commissioned because Air Force leadership thinks the service lacks the level of experimentation and innovation required for successful execution of the Air Force mission. The committee’s first finding, “There is too little space for experimentation-driven innovation in today’s Air Force,” confirms this idea. However, the committee also noted that there are important pockets of success that can be replicated and built upon. The committee found an inherent link between “innovation” and “experimentation.” As a result, an analysis of the current state of innovation in the Air Force was necessary to meet this task. This logic continues throughout the report as the committee found that addressing the current state of innovation was directly tied to addressing the current state of experimentation and experimentation campaigns. Its top recommendation is that Air Force leadership should identify the areas in which increased innovation is most important to the service and identify Innovation Catalysts, who will be

2. responsible for increasing experimentation and innovation in these critical areas (see Recommendation 1).

3. Assess the role of experimentation in the innovation life cycle and address how it can support the Air Force’s future technology requirements. As captured in the committee’s second finding, it is clear from an examination of the literature and a study of highly innovative organizations that well-designed and -executed experimentation campaigns are critically important drivers of innovation. In its report the committee has explained that a key responsibility of Innovation Catalysts is to oversee planning and execution of strong experimentation campaigns. But, if these campaigns are to support the Air Force’s future technology requirements, it is essential that the campaigns be carefully aligned with Air Force strategy and Developmental Planning. Figure 4-1 presents the committee’s vision of how experimentation campaigns are linked to the support of future technology requirements identified through the Developmental Planning process. To ensure the close linkage between experimentation and key future technology requirements, the committee’s second recommendation is that any part of the organization where Innovation Catalysts are needed and named be directly linked to the senior-most leaders of that part of the organization (see Recommendation 2).
4. Evaluate the role of risk management and experimentation in the innovation life cycle. The committee found the Air Force to be so risk-adverse that it operates today with a crippling fear of failure (Finding 3). This fear of failure extends fear of the disappointing results inherent in good trial and error learning through experimentation. But, accepting the failures that are part of learning in any good experimentation campaign is essential if the organization is to avoid the far greater risk of technological obsolescence. In fact, the primary goal of a good experimentation campaign is to manage risks by efficiently using the trial and error of experiments to rapidly improve the ratio of knowledge to assumptions. The Air Force does not need to be told this—the Service has a long history of success with using experimentation to reduce the set of unknowns by moving a technology along its life cycle of concept to fielded solution. But, as noted above, while the Air Force clearly knows how to do this, it still occurs in too few places (see Recommendations 4, 5, and 6).
5. Evaluate current legislative, organizational, or other barriers that limit the use of experimentation within the AF. As reported on p. 7, there is strong congressional impetus for increasing innovation in all the services. However, there are a number of barriers to accomplishing this goal in the Air Force. The committee’s Findings 1, 3, and 7 all touch on this point. Finding 1 points to the limited use of experimentation within the Air Force. Findings 3 and 7 suggest that a culture that fears failure and thus resists innovations, especially disruptive innovations, is the root of the problem. However, there are too many pockets of successful experimentation-driven innovation in the Air Force to say the Service lacks the

5. knowledge or skills required for innovation. Instead, the problem is that efforts to innovate, especially those required by disruptive innovations, are overwhelmed by the organization’s emphasis on what the committee has labeled Normal Production. The Air Force operates today in a world that demands more and more, even if it can afford less and less. Much of the organization is focused on the day-to-day demands for output. However, this focus has become the chief barrier to experimentation today. To address this, in Recommendation 3 the committee has recommended that the Service prioritize the areas where it most needs innovation and then move proactively to push back on the Normal Production organization to create space for experimentation and disruptive innovation. It has made specific recommendations intended to support this effort, such as naming Innovation Catalysts (Recommendation 1), strengthening the experimentation toolbox (Recommendation 4), careful use of metrics (Recommendation 5), and bringing about cultural changes (Recommendation 6).

6. Review and recommend best practices for incentivizing experimentation based on an assessment of comparable S&T organizations. The single largest block of information in this report addresses the best practices that were found in studying highly innovative organizations. The committee has grouped its findings from this work into three topics: leadership (see Finding 4), tools (see Finding 5), and culture (see Finding 7). (Each topic was given its own section in Chapter 2.) Based on the review of best practices in highly innovative organizations, the committee was able to identify corresponding recommendations for the Air Force:

• Leadership. Prioritize areas where innovation needs to be increased, and name Innovation Catalysts to drive this effort (see Recommendation 1.) The Innovation Catalysts provide the connection between senior leaders’ use of development planning the and experimentation campaigns (see Recommendation 2 and Figure 4-1) and they lead in creating space for innovation by providing balance against the demands of the Normal Production organization (see Recommendation 3).
• Tools. The committee’s study of highly innovative organizations identified many tools in use as best practices. In Recommendation 4, it identified “sandboxes,” classic experimentation campaign tools, makerspaces, and partnerships as best practices critically important to the Air Force. As discussed elsewhere in this section, to the extent metrics are seen as tools, the committee’s research on their use by highly innovative organizations points to the need to carefully tailor them to the particular goals and problems facing the organization (see Finding 6 and Recommendation 5).
• Culture. The committee’s research on best practices documented the considerable lengths to which highly innovative organizations go to incentivize experimentation and innovation as a way of life in their cultures. It identified four “influence channels” in use in these organizations, and Recommendation 6 suggests

6. Recommend metrics that can be implemented across the AF to monitor and assess the use and value of experimentation. The committee found heavy reliance on metrics to focus attention and effort in the Normal Production organizations of the Air Force. However, it did not find comparable use of metrics to drive innovation (see Finding 6). Its research failed to produce a uniformly accepted set of metrics in widespread use across the highly innovative organizations it studied. Instead, it found these organizations to move very cautiously in selecting metrics fitting their particular circumstances. This suggests there are no obvious all-purpose metrics for stimulating innovation, and the key to good use of metrics is that they be carefully selected to avoid the problems that can bring unintended consequences (Recommendation 5).
7. In addition, the committee will address any other factors deemed to be relevant, such as organizational structure or concepts of operation that could enhance the likelihood of successfully implementing a robust experimentation program within the Air Force acquisition community. As pointed out in Finding 7, many in the Air Force find it difficult to take risks, run experiments, and be innovative. This report has identified many causes of this and suggested many factors relevant to addressing the situation, such as matters of organizational structure and the concept of operations for Innovation Catalysts, both addressed in Chapter 2. The committee has also stressed throughout its report, however, that there is no one quick fix, no “silver bullet” solution for enhancing the likelihood of a robust experimentation program within the Air Force. The committee’s final finding, Finding 7, suggests that a broad range of factors have come together to form a culture that generally discourages risk-taking, experimentation, and innovation. Addressing this will require all six of the committee’s recommendations: dedicated Innovation Catalysts (Recommendation 1) closely linked to senior leadership (Recommendation 2) in their combined efforts to create more space for experimentation (Recommendation 3) using a proven portfolio of tools (Recommendation 4) and carefully constructed metrics (Recommendation 5). And, finally, as stated in Recommendation 6, all of this will need to be continually reinforced with clear messaging and incentives that are not in place today.

TABLE 4-1 Statement of Task and Corresponding Findings and Recommendations

NOTE: Acronyms are defined in the front matter.