National Academies Press: OpenBook

The Role of Experimentation in Building Future Naval Forces (2004)

Chapter: 2 Experimentation--What It Means

« Previous: 1 Introduction
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

2
Experimentation—What It Means

The Navy and the Marine Corps have embraced experimentation as a fundamental tool for force development. This chapter examines what it means to experiment in the context of military operations. In addition, it explores the methodology of experimentation and experimentation campaigns, including the application of spiral development. It also discusses the environment necessary to support a sound experimentation program. The chapter ends with a discussion of a few practical considerations for military experimentation.

WHAT IS AN EXPERIMENT?

Committee’s Working Definition

For the purposes of this report, the committee chose a relatively broad definition of the term “experimentation” in the military context: Military experimentation is a military activity conducted to discover, test, demonstrate, or explore future military concepts, organizations, and equipment and the interplay among them, using a combination of actual, simulated, and surrogate forces and equipment.

The definition highlights the fact that building future naval forces through experimentation means more than acquiring the equipment of the future. Building tomorrow’s Navy and Marine Corps means developing the doctrine, organization, training, materiel, leadership, personnel, and facilities (DOTMLPF) that together constitute the mission capability of a military force. If experimentation is to be useful, it must deal with all these elements of capability.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

Naval experimentation serves a purpose. It is not an end unto itself, nor is it merely a means for pursuing a few interesting ideas. It is intended to build future naval capabilities and must be so crafted. Experimentation must support learning what needs to be known by exploring the potential value of new systems and new ways of operating forces, so that leaders can make informed decisions about advancing the capabilities of tomorrow’s naval forces.

The committee recognizes that other definitions of experimentation are in use and plausible. Appendix C provides definitions of experimentation terms used in this report.

What Others Say an Experiment Is

There is surprisingly little agreement among definitions of the term “experimentation” in the military context. Joint Chiefs of Staff (JCS) Publication 1, the DOD’s authoritative source for standard definitions, does not define it. The U.S. Joint Forces Command’s glossary defines “joint experimentation” as the “application of scientific experimentation procedures to assess the effectiveness of proposed (hypothesized) joint warfighting concept elements to ascertain whether elements of a joint warfighting concept cause changes in military effectiveness.”1 Given actual practice, this definition is relatively narrow with respect to both the purpose of joint experimentation and its methods. U.S. congressional defense appropriation language calls for joint warfighting experimentation to be carried out “in field environments under realistic conditions against the full range of future challenges. . . .”2

In Code of Best Practice for Experimentation, co-author David Alberts, director of research and strategic planning of the Office of the Assistant Secretary of Defense, distinguishes among three types of experiments, each based on a separate meaning of the word “experiment” and each having a distinct purpose:

  • Discovery experiments are conducted “to determine the efficacy of something previously untried.”3 These experiments are similar in purpose to joint experimentation as defined in the USJFCOM glossary.

1  

See the Web site <www.USJFCOM.mil/about/glossary.htm>. Accessed August 8, 2003.

2  

Strom Thurmond National Defense Authorization Act for Fiscal Year 1999, P.L. 105-261, 112 Stat. 1920, Sec. 921 (October 17, 1998).

3  

David S. Alberts, Richard E. Hayes, John E. Kirzl, Leedom K. Dennis, and Daniel T. Maxwell. 2002. Code of Best Practice Experimentation, DOD Command and Control Research Program, Office of the Assistant Secretary of Defense (Networks and Information Integration), Washington, D.C., July, Ch. 3., p. 3-1. Available online at <http://www.dodccrp.org/>. Accessed October 7, 2003.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
  • Hypothesis-testing experiments are used to advance knowledge by “seeking to falsify specific hypotheses.”4

  • Demonstration experiments recreate known truth to “display existing knowledge to people unfamiliar with it.”5

As Alberts points out, the DOD and the Services conduct experiments of all three types. Congressional directives favor Alberts’s first two types over his third. The Defense Authorization Act of FY1999 stipulated that joint experimentation was to “investigate and test technologies and alternative forces and concepts.”6 More specific goals were provided in the Defense Authorization Act of FY2000—for example, improving interoperability, synchronizing technology fielding, and developing joint operational concepts.7 The law is silent on the question of demonstrating capabilities that are already understood for the purposes of achieving buy-in or sustaining momentum.

In the 1990s, the U.S. Army dubbed its Force XXI experimentation series advanced warfighting experiments (AWEs). The term was subsequently adopted by Congress and by others in the DOD. More recently, the DOD has favored the term “field experiments.” The USJFCOM glossary defines “field experiments” as “wargames conducted in the actual environment with military units and equipment.” It does not offer a separate definition for “warfighting experiments.”8 The 2001 Quadrennial Defense Review refers to “field exercises that incorporate experimentation.”9 Both of these sources hold that such exercises should emphasize the operational level of war rather than the tactical or strategic. Because the term “field experiment” is used today to mean the same thing that “warfighting experiment” meant 4 years ago, the committee has chosen to use the two terms interchangeably.

4  

David S. Alberts, Richard E. Hayes, John E. Kirzl, Leedom K. Dennis, and Daniel T. Maxwell. 2002. Code of Best Practice Experimentation, DOD Command and Control Research Program, Office of the Assistant Secretary of Defense (Networks and Information Integration), Washington, D.C., July, Ch. 3., p. 3-3. Available online at <http://www.dodccrp.org/>. Accessed October 7, 2003.

5  

David S. Alberts, Richard E. Hayes, John E. Kirzl, Leedom K. Dennis, and Daniel T. Maxwell. 2002. Code of Best Practice Experimentation, DOD Command and Control Research Program, Office of the Assistant Secretary of Defense (Networks and Information Integration), Washington, D.C., July, Ch. 3., p. 3-4. Available online at <http://www.dodccrp.org/>. Accessed October 7, 2003.

6  

Strom Thurmond National Defense Authorization Act for Fiscal Year 1999, P.L. 105-261, 112 Stat. 1920, Sec. 921 (October 17, 1998).

7  

National Defense Authorization Act for Fiscal Year 2000, P.L. 106-65, 113 Stat. 512 (October 5, 1999).

8  

See Web Site <http://www.USJFCOM.mil/about/glossary.htm>. Accessed August 8, 2003.

9  

Donald H. Rumsfeld, Secretary of Defense. 2001. Quadrennial Defense Review Report, Washington, D.C., September 30, p. 36.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

The committee settled on its relatively broad definition, which includes discovery and exploration, test, and demonstration10—all three of Dr. Alberts’s purposes—and espouses a relatively relaxed view of the scientific rigor with which experiments must be conducted.

Experimentation Includes a Spectrum of Activities

Experimentation is not limited to live events. Rather, it requires a spectrum of activities to advance the understanding of future concepts, organizations, and equipment and to demonstrate their benefits. These activities include studies and analyses, seminars and conferences, work by subject-matter experts, systematic interviewing of experienced officers, war games, and modeling and simulations, as well as live events in the field.

To amplify this point: The term “experimentation” as used in this study implies all activities in this spectrum, not just field experiments.

Figure 2.1 illustrates this spectrum. Any of the activities along the spectrum can be repeated at increasing levels of resolution as knowledge grows and systems and concepts take shape (hence the multiple boxes for each type of activity in the figure). Experimentation relies on studies and analyses as overarching activities to be conducted throughout the process, not just employed at the conclusion to explain the results of an individual live experiment. Studies and analyses include the development of theory and concepts to make sense of the whole; systems analysis and systems engineering for understanding systems of systems; empirical analysis to convert data into knowledge; and, ultimately, policy analyses to help decision makers choose among competing options.

Even field experiments, depicted in Figure 2.1 as a discrete component of experimentation, engage supporting activities. Not every experiment requires live play, wholly or even partially by fleet forces, although typically some level of forces is engaged. Even quite large experiments can be conducted using linked models and simulations.

Experimentation Campaigns Advance Understanding Systematically

Using experimentation for selecting, developing, and implementing future capabilities means invoking a full spectrum of activities systematically.

An experimentation campaign is a planned and cohesive, multiyear program of experimentation built on a series of experiments and related activities to

10  

It seems to be a common impression that “demonstrations” are biased against failures, in contrast to “experiments,” which allow failures. However, demonstrations have been part of the Navy’s fleet battle experiments, and some of these were not entirely successful. Demonstrations with mixed results have also occurred in experiments by the other Services and by the Department of Defense.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

FIGURE 2.1 Spectrum of experimentation activities (see discussion in text).

develop the knowledge needed to inform major decisions about future forces, explore the viability of potential or planned changes to forces or their capabilities, and/or confirm that planned development and directions will lead to capabilities that perform as expected.

An experimental campaign may be constructed to answer questions about one or several elements of DOTMLPF. It should be in alignment with the future vision of the Services and joint forces. The spectrum of activities includes seminars, work by subject-matter experts, studies and analyses, war games, and modeling and simulations, all of which usually precede and support actual field experiments. The field experiments themselves have varying objectives and sizes. A campaign typically uses smaller experiments that incrementally build to larger ones.11 A campaign must be flexible and may include the iteration of steps, regress to earlier stages, or skip a stage as results warrant. In fact, a campaign plan may be viewed as a “living document” subject to continual replanning.

11  

The size of various Service and joint experiments varies greatly: some experiments involve only a small number of lower-echelon units, whereas those conducted at the Joint Task Force level involve thousands of troops from all four Services. Both the Navy and the Marine Corps as well as the other Services conduct limited-objective experiments (LOEs), which focus on one or a few new capabilities in a single area. LOEs can cut across Services and geographic locations and can even include multinational involvement while remaining relatively small in scale. The Marine Corps makes extensive use of limited technical assessments (LTAs), experiments conducted to test the utility of a single technology or piece of equipment in some range of future tasks. By contrast, the Navy’s FBE series aims to explore multiple concepts simultaneously and to involve a substantial portion of a fleet, usually while the fleet is engaged in training exercises.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

To achieve objectives that embody dramatic departures from current capabilities, experimentation must be carried on continuously and systematically. No single experimentation activity is conclusive. Each experiment uses a sample of the surrounding environment that affects its outcome. The environments and scenarios adopted for an individual experiment are chosen from a range—often a very wide range—of possibilities. Generalizing to the numerous situations in which the findings may be applied is always risky. Experimentation campaigns reduce risk by addressing a spectrum of possibilities and build on successive activities systematically, with analyses done at every step.

A well-planned, prioritized experimentation campaign provides a framework for learning much of what needs to be known about new capabilities: whether they are desirable and feasible, how they compare with other options, and what risks will be involved in developing and fielding them. Campaigns can involve the validation and verification of models for such things as computer applications and the dynamic behavior of planners and resource managers. Structuring a campaign allows planners to proceed along multiple axes of investigation while organizing events around broad goals and objectives. It also introduces multiple decision points, both for experiment planning and for identifying and prioritizing the interlinked changes in forces, equipment, concepts, and organizations that are the main objective of an experimentation campaign.12 It is important to understand why certain aspects of experimentation activities fail to meet expectations and how things might work under different conditions, and to document these results and apply them in subsequent experimentation.

Thus, a well-structured campaign enables the knowledge gained from each experimentation activity to support and shape the succeeding activities. A campaign is definitive enough at each major step to inform decisions about future research and technology programs, acquisition efforts, organizational changes, and changes in operational concepts.

BUILDING CAPABILITIES THROUGH EXPERIMENTATION CAMPAIGNS

As noted above, individual experiments do not provide the answer to every capability question. In fact, they can result in false impressions about the desirability or feasibility of changes in fighting concepts, force structure, organizations, and equipment. This section highlights some of the limitations of relying too heavily on individual experiments or on simple threads of experimental activities and discusses how well-structured campaigns can mitigate these limita-

12  

David S. Alberts, Richard E. Hayes, John E. Kirzl, Leedom K. Dennis, and Daniel T. Maxwell. 2002. Code of Best Practice Experimentation, DOD Command and Control Research Program, Office of the Assistant Secretary of Defense (Networks and Information Integration), Washington, D.C., July, Ch. 3. Available online at <http://www.dodccrp.org/>. Accessed October 7, 2003.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

tions. While campaigns cannot explore every scenario, alternative, or excursion, they amplify understanding considerably.

Lack of Scalability

In a world of constrained resources, even a large, live experiment typically involves no more than a small portion of the total force that would ultimately be affected by any of the changes being explored. Thus, the lessons revealed in such a setting may not be scalable to a wider setting. This limitation can be particularly nettlesome for network-centric operations. For example, the performance of a surrogate communications network may support collaboration when only a few users are online, but be so poor as to make collaboration impossible for a larger number of linked operators. Analysts can sometimes rely on models and simulations to extrapolate from experiments to larger settings, but only if the models themselves are based on an understanding of the actual scalability of the variables involved. When cause-and-effect relationships are nonlinear and complex, gaining that basic understanding can require a sustained, iterative program of experimentation. The incremental structuring of activities within an experimentation campaign—such as building from smaller limited-objective experiments (LOEs) to larger FBEs and joint force experiments—can test scalability as well as the ability to integrate the partial results of the smaller experiments into the operation of large-scale forces. Iterative activities can highlight the problems and point toward solutions.

Transferability of Lessons Across the Force

Lessons learned in an experiment about one combination of ships, units, or locations may not apply to the next combination. Even an aggressive and carefully designed experimentation campaign will never be able to illuminate every such combination. As with scalability problems, a healthy dose of model-based analysis, simulation, and wargaming can help analysts to extrapolate the results from a few combinations to many.

Transferability of Lessons from Experiments to Wartime Operations

Safety concerns, resource constraints, artificially benign environments, and the absence of a real adversary make it impossible for experiments to reflect all of the actual conditions of military operations. As a result, lessons learned from experiments apply imperfectly to wartime operations. This limitation can be mitigated by including a strong, motivated, and creative opposing force in the experimentation program, allowing for the exploration of alternative paths when things do not go as planned. This limitation can also be mitigated through the modeling and simulation of various situations and degraded conditions.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

The Human Factor

Expanding the knowledge base by using experiments that involve people working together and making decisions in complex new situations will never be easy. As David Alberts points out,

The variety of applicable military contexts and the rich variety of human behavior and cognition argue for care…. Many innovations currently of interest, such as collaborative work processes and dispersed headquarters, have so many different applications that they must be studied in a variety of contexts. Others have organizational and cultural implications that must be examined in coalition, interagency, and international contexts.13

One problem in this as in any endeavor is that people can make mistakes. Human errors may be recognized or not. Recognized errors typically do not cause serious problems because the data that they affect can be analyzed separately or discarded. However, mistakes that go unrecognized can lead to false interpretations that destroy the validity of an experiment.

Another set of problems can be introduced when people play roles in an experiment. Research indicates that when complex decision making is involved, there may be differences in performance between people who hold the roles in reality and those who play their roles during experimental events. Bringing actual staffs into experiments obviates those differences, but using actual staffs is not always possible. When role-playing is involved, experiment planners and evaluators need to consider how such differences in decision-making style and skill might affect experimental outcomes.

More generally, different people respond to the same situation in different ways. Published job performance experiments by psychologists show large variations in performance among individuals conducting the same task.14 Furthermore, the same person may respond differently at different times, depending on training, experience in an experiment, or level of fatigue or stress. The effects of this limitation can be reduced by selecting the experimental operators randomly, including a large number of operators, establishing baseline and control groups, controlling for learning effects during experimentation, and training operators to uniform standards before an experimental event. In addition, human factors

13  

David S. Alberts, Richard E. Hayes, John E. Kirzl, Leedom K. Dennis, and Daniel T. Maxwell. 2002. Code of Best Practice Experimentation, DOD Command and Control Research Program, Office of the Assistant Secretary of Defense (Networks and Information Integration), Washington, D.C., July, Ch. 3., pp. 3-10 and 3-11. Available online at <http://www.dodccrp.org/>. Accessed October 7, 2003.

14  

For example, Frank L. Schmidt and John E. Hunter, 1983, “Individual Differences in Productivity: An Empirical Test of Estimates Derived from Studies of Selections Utility,” Journal of Applied Psychology, Vol. 68, No. 3, August, pp. 407-414, describes variations of two to one or higher in the ratio of productivity between 95th- and 5th-percentile performers.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

analysis can help to illuminate the differences among the operators who serve as experimental subjects, and modeling and simulation can improve understanding of how those differences might affect operational outcomes.

All of the potential problems described here argue for conducting multiple rounds of experiments to mitigate some of the limitations caused by complexity and the human factor.15

Unrealistic Surrogates

Because the equipment of the future has not been built yet, experiments typically must rely on commercially available surrogates or legacy systems modified to emulate the expected performance of postulated new systems. Such surrogates are needed to help people flesh out new operational concepts and to allow operators to explore the new concepts. However, they may be far from representing the actual capabilities that would be required in real future operations. For example, a surrogate may lack the physical hardening, the information security, the actual performance, or the capacity that would be needed in a fight against a real enemy. To explore the desirability of new concepts under such circumstances, experimenters typically tailor the experiment—for example, by constraining the conditions under which the opposing force can realistically operate. In the heat of enthusiasm, however, it is easy for both experimenters and decision makers to lose sight of the fact that surrogates, while necessary to carry out an experiment, in fact are poor substitutes for the equipment that would be needed to make the concepts real. As a result, experiments can lead to misunderstandings about how difficult it might or might not be in the future to acquire the new capabilities that the Navy and the Marine Corps are considering. Repeated, unvarnished discussion of the role and limitations of the surrogates by experimenters and advocates may help mitigate this class of limitations. Close coupling between the experimentation program and the research and technology base might also help.

Experimentation Campaigns in Summary

Single experiments are insufficient to provide the understanding needed to advance naval forces and their capabilities. Experimentation campaigns enable multiple axes of investigation, with a spectrum of activities that can be matched to the question posed. While no amount of foresight will prevent every surprise, a well-designed experimentation campaign can mitigate the limitations of single

15  

David S. Alberts, Richard E. Hayes, John E. Kirzl, Leedom K. Dennis, and Daniel T. Maxwell. 2002. Code of Best Practice Experimentation, DOD Command and Control Research Program, Office of the Assistant Secretary of Defense (Networks and Information Integration), Washington, D.C., July, Ch. 3. Available online at <http://www.dodccrp.org/>. Accessed October 7, 2003.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

experiments and simple threads of experimental events by combining repeated and persistent live experiments with modeling, analysis, wargaming, and other appropriate activities, building systematically upon the knowledge derived from each activity. Consequently, experimentation campaigns, not just experiments, are essential for building future naval forces.

SPIRALING IN EXPERIMENTATION

In this report the term “spiral” is applied to processes in four different contexts related to experimentation. Box 2.1 summarizes these briefly. This section discusses the importance of the well-known spiral development process and expands on spiral processes in experimentation campaigns.

Spiral Development

In 2001 and 2002, the DOD revised its acquisition directives to specify evolutionary acquisition as the preferred strategy for rapid acquisition of mature technology and spiral development as the means to implement it.16

Spiral development makes it possible to accelerate the delivery of capabilities that cut across multiple user communities, that involve rapidly changing technologies, and/or that have initially vague or uncertain requirements. This advantage makes spiral development promising for network-centric naval forces and is one of the reasons it was prescribed in the Sea Trial process17 and recommended by the Naval Studies Board’s report on network-centric naval forces.18 For capabilities that depend heavily on rapidly advancing computing and communications technologies, spiral development is essential if the military forces are not to lag adversaries that can acquire commercially available technological advances. Spiral development is not the preferred choice for every acquisition. For example, in the development of a new ship propulsion system, work on the initial concept and analyses of requirements that match user needs would likely benefit from a spiral process. But the detailed engineering design work entailed in

16  

DOD 5000 Series Resource Center. 2002. Department of Defense DOD Instruction 5000.2, “Attachment 2, Operation of the Defense Acquisition System,” Defense Acquisition University, Fort Belvoir, Va., October 30, Section 3.3. Available online at <http://dod5000.dau.mil/Memo50002Oct30.doc>. Accessed October 8, 2003.

17  

“Embracing spiral development, these technologies and concepts will then be matured through targeted investment and guided through a process of rapid prototyping and fleet experimentation.” ADM Vern Clark, USN. 2002. “Sea Power 21: Projecting Decisive Joint Capabilities,” U.S. Naval Institute Proceedings, Vol. 128, No. 10, October, pp. 32-41.

18  

Naval Studies Board, National Research Council. 2000. Network-Centric Naval Forces: A Transition Strategy for Enhancing Operational Capabilities, National Academy Press, Washington, D.C., p. 78.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

BOX 2.1
Spiral Processes in Four Contexts

The term “spiral” is applied to processes used in four different contexts in this report. Each form is applicable to experimentation.

  • Spiral exploration of broad concepts. A process that, within a given phase of inquiry, uses a variety of instruments (e.g., models, games, and experiments) to explore concepts broadly and uses the same instruments to adjust and iterate until a refined understanding of the issues has been obtained.

  • Spiraling in an individual experiment. A process that uses many experimental instruments (e.g., games, simulations, and focused experiments) sequentially in preparation for a subsequent complex individual experiment.

  • Spiraling in an experimentation campaign. A process that uses a spectrum of experimental events to investigate a complex problem incrementally; each spiral typically involves many events and complex experiments, and each subsequent spiral addresses problem complexity that is greater than and/or different from what preceded it (e.g., when new operational concepts for ground forces are studied at successive levels—company, then battalion, then brigade).

  • Spiral development. A process of evolutionary acquisition that iteratively develops a defined set of capabilities within each increment, providing the opportunity for interaction between the user, tester, and developer; refines requirements through experimentation and risk management; allows for continuous feedback; and provides the user with the best possible capability within each increment.1

1  

After several years in which multiple offices and organizations adopted their own definitions of spiral development, the acquisition leadership within DOD stipulated a single definition for both spiral development and evolutionary acquisition, which spiral development is meant to implement. According to this DOD definition, spiral development is “an iterative process for developing a defined set of capabilities within one increment. This process provides the opportunity for interaction between the user, tester, and developer. In this process, the requirements are refined through experimentation and risk management, there is continuous feedback, and the user is provided the best possible capability within the increment. Each increment may include a number of spirals. Spiral development implements evolutionary acquisition.” Quoted from memorandum from Under Secretary of Defense E.C. Aldridge, Jr., to the secretaries of the military departments and others, dated April 12, 2002, p. 2.

advancing immature technologies and mitigating technology and cost risk may be better handled through more traditional, deliberate development. Propulsion power-source design, such as new generations of gas turbines or turboelectric drives, involves the confluence of many technological advances—in materials, fluid flow management, controls, modes of transmitting power, and propulsion design (noncavitating propellers or water jets, for example)—all of which move ahead at different rates and achieve progress milestones at different times. Thus,

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

for some acquisitions, longer development times and major block enhancements may be the preferred method.

Spiraling in an Experimentation Campaign

By organizing a campaign of experimental activities into spirals, planners can ensure that integrated capabilities are explored at each step so that a coherent set of military capabilities is grown rapidly over a short period of time. With this approach, which works well in procuring large and complex systems and systems of systems, early expression of technical requirements is important. Every effort should be made to design the desired system correctly up front and to make it evolvable, with adaptations then achieved through spiraling. Often, “straw” (first approximation) requirements and designs used at the outset of a campaign are themselves the outcome of earlier experimentation.

Typically, the spirals within a campaign involve increasingly complex, individual live or simulated events aimed at gaining a refined understanding of the concepts and capabilities under consideration. Spirals may also be used to explore the sorts of scalability issues involved in moving from smaller to larger units.19 Alternatively, spirals may be used to break a broad area of inquiry or a complex set of decisions into more manageable chunks, with detailed exploration of multiple facets building over time to a broader set of integrated capabilities.

The spirals in a campaign are not simply a sequence of events. Rather, they support a continuous and integrated feedback process that engages all of the stakeholders (operators, developers, testers, architects, and others), with an analysis of risks and capabilities and with minispirals to facilitate backtracking in order to refine understanding and resolve issues identified during successive cycles. The spirals are undertaken with the intent, for example, of transitioning the resulting new equipment or systems into service if the outcome is successful, or of actually changing the force’s operating procedures; such a tangible outcome is the reason for the expense and effort attending the experimentation campaign.

Figure 2.2 illustrates an experimentation campaign that coevolves a mission capability package beginning with a straw DOTMLPF, which itself may have resulted from prior experimentation activities. Each campaign spiral is directed toward its own specific set of objectives, which are related to decisions about future forces. Each spiral requires its own system-of-systems architecture and integration efforts, which in turn are refined from spiral to spiral as experimental components of DOTMLPF grow in complexity. Within each campaign spiral, end-to-end testing and dry runs are conducted before major live events or simula-

19  

For example, the U.S. Army structured its experiments related to digitization of the battlefield to explore issues first at the company and battalion levels, then at the brigade level, and later at the division level.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

FIGURE 2.2 Spiraling in an experimentation campaign.

tions in order to bring to light problems in concepts, systems, organizations, doctrine, TTPs, and training.

In the campaign illustrated, the spirals and their culminating events should be defined so as to allow for rapid incremental development and, if the development is successful, for the folding in of the new capabilities—thus linking firmly into the spiral development/evolutionary acquisition model noted earlier. At culminating points throughout the campaign, mature, verified, and tested capability packages should be transitioned into new, fielded capabilities. Some packages may require new acquisition program starts; others may modify existing pro-

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

grams of record. Still others may require no changes in material but may instead involve changes in other elements of DOTMLPF. The less-mature concepts, organizational changes, and technologies are either set aside or moved into the next spiral of experimentation.

THE ENVIRONMENT FOR EXPERIMENTATION

As noted earlier, experimentation relies on a wide spectrum of experimental activities, from studies and analyses, seminars and conferences, war games, modeling and simulation, to live play events. Deriving the maximum benefit from experimentation requires an environment that supports all such activities.

An experimentation environment, as illustrated in Figure 2.3, provides a culture, committed leaders, and skilled personnel as well as infrastructure and tools. Box 2.2 expands on representative elements of such an environment, and these are briefly discussed in subsequent paragraphs.

Leadership and a Culture of Learning

Experimentation is naturally disruptive of the status quo. For experimentation to contribute to building future forces, senior leadership must support a free play of ideas that run counter to traditional culture and expectations. Because learning about ideas that do not succeed can be as important as learning about those that do, experimentation requires a culture that accepts negative results and occasional failures, coupled with an understanding that the learning itself is a successful outcome. A culture that rewards risk takers and encourages innovative behavior is key to capitalizing on experimentation. Equally important is the empowering of subordinates to make appropriate decisions, to work creatively, and to put forward ideas that may be unconventional and disruptive.

FIGURE 2.3 Experimentation environment. (See Box 2.2 for expanded list of elements.)

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

BOX 2.2
Major Elements of Experimentation Environment

Leadership and Culture

Information and Physical Infrastructure

Committed leaders

Networks

Cultures of learning

Information repositories

Incentives for risk

Architectural frameworks

Tolerance of negative results

Integration and test facilities

Empowerment

Training facilities

 

Places and platforms

Trained and Talented Personnel

Tools

Concept developers

Modeling and simulation

Systems analysts

Prototypes, surrogates, and so on

Operators

Artificial environments

Red-team cells

Data capture and dissemination

Support teams

 

Trained and Talented Personnel

Experimentation requires a commitment to providing talented personnel who are dedicated to experimentation. Each member of the experimentation cadre must be both educated and experienced in experimentation, in addition to having a discipline related to warfare.20 The mix of expertise required in any one experimental activity may vary depending on the nature of the activity, but core expertise includes concept developers, system analysts, operators, an opposing-force cell, and a support team of engineers, developers, testers, and trainers. Capable concept developers are essential, because without innovative new concepts to explore there can be no experimentation of a far-reaching nature. A substantive understanding of military operations plus a good imagination is required of these individuals. Systems analysts, typically working with the aid of models, are

20  

According to David Alberts, “The single most important consideration for those responsible for experimentation design, whether single experiments or campaigns, is to ensure current expertise is available to support the plan.” See David S. Alberts, Richard E. Hayes, John E. Kirzl, Leedom K. Dennis, and Daniel T. Maxwell, 2002, Code of Best Practice Experimentation, DOD Command and Control Research Program, Office of the Assistant Secretary of Defense (Networks and Information Integration), Washington, D.C., July, Ch. 6. Available online at <http://www.dodccrp.org/>. Accessed October 7, 2003.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

required to assess in detail the operational and technical feasibility of the concepts being developed.

When the concept under consideration has reached a degree of maturity, the experimentation cadre will design and execute experimentation activities to test it. This cadre should have a broad range of both operational and technical expertise—operational, for example, in order to develop scenarios and determine data collection needs, and technical to understand the necessary simulation capabilities and systems integration. General logistical support in experiment preparation and execution is also required. Analysts are also critically needed for planning, designing, and evaluating experimentation events.

Operators—i.e., the actual “players” in the experiment—must be familiar with the concept (or capability) being explored and with the use of the tools (e.g., simulations) being employed and must also be of a mind-set to explore and innovate. Playing against the operators will be an opposing force, or red-team cell. This independent body is necessary for exposing, in as honest and probing a manner as possible, potential flaws in the concept being explored.

Information and Physical Infrastructure

Most experiments will require network connectivity and capacity as well as computing power and data storage. In addition, they depend on the existence of accepted architectural frameworks (e.g., for simulation, the High-Level Architecture—Institute of Electrical and Electronics Engineers standards P1516, P1516.1, and P1516.2) for integrating simulation assets. The experiments not only will generate data but also will require much data, such as scenarios to drive the experimentation play. Having repositories of such information available and accessible can ease experiment planning and enable the extension of results to cases or scenarios that are not included in the live play.

Ultimately, a system that links platforms into a virtual environment is needed, to provide an appropriate level of realism in live experimentation conducted across a fleet. Such a system supports the controlled insertion of simulated threats and other scenario elements into the command and control environment. Embedding the system directly into training systems aboard each platform may help naval operators shift seamlessly between training and experimentation.

An experiment and any or all related activities must take place somewhere. Key facilities, such as those for simulations, for war games, for integration and testing, and for training, must be provided. Such facilities must be equipped with the basics of uninterrupted power, good lighting and ventilation, and suitable climate control, as well as with the space and support for any specialized equipment needed for specific activities. Equally critical is the timely availability of ships, test ranges, aircraft, and various platforms that are integral to the experiments—this need is particularly challenging for naval experimentation. Integration facili-

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

ties and environments are critical because integration of needed components to support experimentation is challenging,21 often encompassing systems, databases, sensors and weapons platforms, and networks, as well as various tools to support experimentation. Interoperability is often difficult to achieve, exacerbated by the relative immaturity of the concepts and capabilities that are being explored.

Tools

Many other types of tools are required to support experimentation, such as those for establishing artificial environments and for supporting actual prototypes and managing the use of surrogates. The interoperability and integration of tools are often challenging, requiring a defined framework and an appropriate integration environment, as noted above in the discussion on information infrastructure.

Models and simulations play an essential role in experimentation, given that physical platforms and facilities may not always be available or in fact may not yet exist. The models and simulations can range from largely computer-driven constructive models to virtual simulations in which human participants in platform simulators or at command-and-control terminals make real-time battle decisions.22 The constructive models can be of varying levels of resolution—for example, highly aggregated in support of a strategic-level war game, to quite detailed for examining urban operations. Ideally the models and simulations should form an interrelated family to serve and allow ready transitions among the different purposes (e.g., from analysis to wargaming to human-in-the-loop operations).

Several capabilities involved in experiments will be modeled or simulated individually and then linked with the larger models and simulations representing the overall military capability in the experiment. These include prototypes of new systems, surrogates needed because of a lack of availability of some systems, simulations, and artificial or model environments.

Experimentation activities can produce very large volumes of data. For example, a large virtual simulation can involve several thousand entities, with state data (position, velocity, and so on) on each entity being generated on the order of every second. Thus, mechanisms for data collection ranging from automated capture to human observers are necessary. In addition, sophisticated analysis

21  

For instance, see BG Steven Boutelle, USA, and Alfred Grasso, 1998, “A Case Study: The Central Technical Support Facility,” Army RD&A (now Army Acquisition, Logistics & Technology (AL&T) Magazine), March-April, pp. 30-33; and Annette J. Krygiel, 1999, Behind the Wizard’s Curtain: An Integration Environment for a System of Systems, National Defense University Press, Fort L.J. McNair, Washington, D.C., July.

22  

Constructive experiments use simulated forces in a simulated environment; virtual experiments use partial real forces in a simulated environment; field experiments use real forces in an actual environment.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

tools to aid analysts in interpreting the data are required, as are means to store and disseminate the information so derived.

PRACTICAL CONSIDERATIONS IN EXPERIMENTATION

As with other military activities, military experiments take place in the real world. This section discusses the practicalities of dealing with resource constraints and moderating the effects of less-disciplined methods of experimentation.

Contention for Resources

The most onerous practicalities for experimentation are those related to resources. In the U.S. military, money and personnel are typically spoken for years in advance. Dedicated funding for experimentation is essential, given the long lead times required for accomplishing not only campaigns but also individual experiments. Identifying the funding needed to transition the findings from experimentation to the field is very difficult, given the size of the requirement and its unanticipated impact on the programming cycle. Nonetheless, transition funding must be secured if the results of experimentation are to influence and transfer to force capabilities. This issue is addressed in some detail in subsequent chapters, but it should be noted here that the discussion of and contention for resources that accompany any change in a Service’s equipment, systems, or mode of operating significantly delay any decision to adopt the results of experiments. The decision cycle must start early. Therefore, in anticipation of success, planning for the transition should start at the same time that planning begins for an experiment or an experimentation campaign. The rationale for this step is elaborated more fully in Chapter 5, in which the progress and problems of Navy and Marine Corps experimentation to date are evaluated.

The competition for resources is exacerbated when the needs of military experiments are overlaid on military training exercises. In the Navy, major events on ships and at air training ranges are typically programmed 6 years in advance. Training schedules for individuals and units are tight even without additional duties. In response to such constraints, the military services and the joint community are increasingly using their training exercises as vehicles for experimentation, in essence tacking experiments onto exercises that units would perform in any case. This occurs so frequently that a military exercise is often characterized as an experiment.

Such opportunistic experimentation allows the Navy and Marine Corps to explore operational-level change that would be nearly impossible to examine using smaller, dedicated forces and allows limited resources, including time, money, and people, to be stretched.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

The objective of a military exercise, however, is to ensure and measure today’s readiness by training or evaluating military units23—quite different from the objective of experimentation, which is to explore and demonstrate future concepts, organizations, and equipment. Piggybacking experimentation on training exercises can mean accepting exercise conditions, forgoing a clear baseline against which to measure change, and limiting the repeatability of events. In some cases, these limitations can be mitigated through careful experiment design. In all cases, setting priorities and doing the best with the resources available are critical. The ramifications of these issues in the naval experimentation environment are discussed in more detail in subsequent chapters.

Overlaying naval experimentation needs with those of joint experimentation can also pose problems, as amplified in Chapter 4. Careful synchronization and alignment of experimental goals, assets, and schedules are necessary to make experiments useful to both the Department of the Navy and the joint community.

Less-Disciplined Field Experiments

Live-play events require greater improved discipline. The various briefings provided to the committee, as well as the views of individual committee members, precipitated a lively debate about the degree to which field experiments must hew to rigorous scientific experimentation procedures if they are to be worthy of the name. In general, those schooled in the physical or biological sciences argue for tighter standards of scientific rigor, including the unambiguous statement of hypotheses, rigorous experimental design, the development of a clear baseline against which to measure change, the establishment of control regimens, careful treatment of independent and dependent variables, repeatability, scalability, and so forth. Experts long engaged in field experiments typically favor more relaxed standards, in large measure because, in their experience, the more rigorous standards seemed unachievable in a field setting, especially for the larger-scale experiments.

If experimentation is to be a credible enabler of future forces, it must adhere to sufficiently rigorous methods. This requirement reinforces the need for a well-planned experimentation campaign that includes an appropriate complement of activities both preceding and following field experiments. Field events should

23  

The Department of Defense Dictionary of Military and Associated Terms (Joint Publication 1-02) defines an “exercise” as “a military maneuver or simulated wartime operation involving planning, preparation, and execution. It is carried out for the purpose of training and evaluation….” Joint Chiefs of Staff. 2002. Department of Defense Dictionary of Military and Associated Terms, The Pentagon, Washington, D.C., April 12 (amended through September 5, 2003), p. 189. Available online at <http://www.dtic.mil/doctrine/jel/new_pubs/jpl_02.pdf>. Accessed October 7, 2003.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×

continue to improve in rigor; however, it is also important that the full spectrum of activities augment field experiments to ensure the application of good scientific and analytical methods. This combination will convey a more coherent and incisive picture of the results of the fixed experiments, which enables discarding or substantiating and extending findings while determining the many factors that shape these live-play events.

Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 28
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 29
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 30
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 31
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 32
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 33
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 34
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 35
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 36
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 37
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 38
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 39
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 40
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 41
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 42
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 43
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 44
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 45
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 46
Suggested Citation:"2 Experimentation--What It Means." National Research Council. 2004. The Role of Experimentation in Building Future Naval Forces. Washington, DC: The National Academies Press. doi: 10.17226/11125.
×
Page 47
Next: 3 Experimentation--Past, Present, and Future »
The Role of Experimentation in Building Future Naval Forces Get This Book
×
Buy Paperback | $55.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Department of Defense is in the process of transforming the nation’s armed forces to meet the military challenges of the 21st century. Currently, the opportunity exists to carry out experiments at individual and joint service levels to facilitate this transformation. Experimentation, which involves a spectrum of activities including analyses, war games, modeling and simulation, small focused experiments, and large field events among other things, provides the means to enhance naval and joint force development. To assist the Navy in this effort, the Chief of Naval Operations (CNO) asked the National Research Council (NRC) to conduct a study to examine the role of experimentation in building future naval forces to operate in the joint environment. The NRC formed the Committee for the Role of Experimentation in Building Future Naval Forces to perform the study.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!