Panel on Modeling and Simulation

Committee on Technology for Future Naval Forces

Naval Studies Board

Commission on Physical Sciences,
Mathematics, and Applications

National Research Council

• Notice
• Panel
• Preface
• Contents
• Acknowledgments
• Executive Summary

NATIONAL ACADEMY PRESS
Washington, D.C. 1997

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.

This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce Alberts is president of the National Academy of Sciences.

The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering.

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.

This work was performed under Department of the Navy Contract N00014-96-D-0169/0001 issued by the Office of Naval Research under contract authority NR 201-124. However, the content does not necessarily reflect the position or the policy of the Department of the Navy or the government, and no official endorsement should be inferred.

The United States Government has at least a royalty-free, nonexclusive, and irrevocable license throughout the world for government purposes to publish, translate, reproduce, deliver, perform, and dispose of all or any of this work, and to authorize others so to do.




Copyright 1997 by the National Academy of Sciences. All rights reserved.




Copies available from:

Naval Studies Board
National Research Council
2101 Constitution Avenue, N.W.
Washington, D.C. 20418




Printed in the United States of America



---

PANEL ON MODELING AND SIMULATION

GEORGE F. CARRIER, Harvard University, Chair

PAUL K. DAVIS, RAND and the RAND Graduate School, Vice Chair

DONALD K. BLUMENTHAL, Gualala, California

RICHARD BRONOWITZ, Center for Naval Analyses

JOHN C. DOYLE, California Institute of Technology

DONALD P. GAVER, Naval Postgraduate School

DON E. HIHN, Charleston, South Carolina

RICHARD J. IVANETICH, Institute for Defense Analyses

JOHN P. LEHOCZKY, Carnegie Mellon University

DAVID L. McDOWELL, Georgia Institute of Technology

DUNCAN C. MILLER, Massachusetts Institute of Technology

DAVID R. OLIVER, Northrop Grumman Corporation

GABRIEL ROBINS, University of Virginia

BERNARD P. ZEIGLER, University of Arizona




Invited Participant

BEN P. WISE, Science Applications International Corporation




Navy Liaison Representatives

CDR THOMAS COSGROVE, USN, Office of the Chief of Naval Operations, N858D

CAPT JAY KISTLER, USN, Office of the Chief of Naval Operations, N6M




Consultants

LEE M. HUNT

SIDNEY G. REED, JR.

JAMES G. WILSON




Staff

RONALD D. TAYLOR, Director, Naval Studies Board

PETER W. ROONEY, Program Officer

SUSAN G. CAMPBELL, Administrative Assistant

MARY G. GORDON, Information Officer

CHRISTOPHER A. HANNA, Project Assistant




---

COMMITTEE ON TECHNOLOGY FOR FUTURE NAVAL FORCES

DAVID R. HEEBNER, Science Applications International Corporation (retired), Study Director

ALBERT J. BACIOCCO, JR., The Baciocco Group, Inc.

ALAN BERMAN, Applied Research Laboratory, Pennsylvania State University

NORMAN E. BETAQUE, Logistics Management Institute

GERALD A. CANN, Raytheon Company

GEORGE F. CARRIER, Harvard University

SEYMOUR J. DEITCHMAN, Institute for Defense Analyses (retired)

ALEXANDER FLAX, Potomac, Maryland

WILLIAM J. MORAN, Redwood City, California

ROBERT J. MURRAY, Center for Naval Analyses

ROBERT B. OAKLEY, National Defense University

JOSEPH B. REAGAN, Saratoga, California

VINCENT VITTO, Lincoln Laboratory, Massachusetts Institute of Technology




Navy Liaison Representatives

RADM JOHN W. CRAINE, JR., USN, Office of the Chief of Naval Operations, N81 (as of July 4, 1996)

VADM THOMAS B. FARGO, USN, Office of the Chief of Naval Operations, N81 (through July 3, 1996)

RADM RICHARD A. RIDDELL, USN, Office of the Chief of Naval Operations, N91

CDR DOUGLASS BIESEL, USN, Office of the Chief of Naval Operations, N812C1

PAUL G. BLATCH, Office of the Chief of Naval Operations, N911E




Marine Corps Liaison Representative

LtGen PAUL K. VAN RIPER, USMC, Marine Corps Combat Development Command




Consultants

LEE M. HUNT

SIDNEY G. REED, JR.

JAMES G. WILSON




Staff

RONALD D. TAYLOR, Director, Naval Studies Board

PETER W. ROONEY, Program Officer

SUSAN G. CAMPBELL, Administrative Assistant

MARY G. GORDON, Information Officer

CHRISTOPHER A. HANNA, Project Assistant




---

NAVAL STUDIES BOARD

DAVID R. HEEBNER, Science Applications International Corporation (retired), Chair

GEORGE M. WHITESIDES, Harvard University, Vice Chair

ALBERT J. BACIOCCO, JR., The Baciocco Group, Inc.

ALAN BERMAN, Center for Naval Analyses

NORMAN E. BETAQUE, Logistics Management Institute

NORVAL L. BROOME, Mitre Corporation

GERALD A. CANN, Raytheon Company

SEYMOUR J. DEITCHMAN, Institute for Defense Analyses (retired), Special Advisor

ANTHONY J. DeMARIA, DeMaria ElectroOptics Systems, Inc.

JOHN F. EGAN, Lockheed Martin Corporation

ROBERT HUMMEL, Courant Institute of Mathematical Sciences, New York University

DAVID W. McCALL, Far Hills, New Jersey

ROBERT J. MURRAY, Center for Naval Analyses

ROBERT B. OAKLEY, National Defense University

WILLIAM J. PHILLIPS, Northstar Associates, Inc.

MARA G. PRENTISS, Jefferson Laboratory, Harvard University

HERBERT RABIN, University of Maryland

JULIE JCH RYAN, Booz, Allen and Hamilton

HARRISON SHULL, Monterey, California

KEITH A. SMITH, Vienna, Virginia

ROBERT C. SPINDEL, Applied Physics Laboratory, University of Washington

DAVID L. STANFORD, Science Applications International Corporation

H. GREGORY TORNATORE, Applied Physics Laboratory, Johns Hopkins University

J. PACE VanDEVENDER, Prosperity Institute

VINCENT VITTO, Lincoln Laboratory, Massachusetts Institute of Technology

BRUCE WALD, Arlington Education Consultants




Navy Liaison Representatives

RADM JOHN W. CRAINE, JR., USN, Office of the Chief of Naval Operations, N81 (as of July 4, 1996)

VADM THOMAS B. FARGO, USN, Office of the Chief of Naval Operations, N81 (through July 3, 1996)

RADM RICHARD A. RIDDELL, USN, Office of the Chief of Naval Operations, N91

RONALD N. KOSTOFF, Office of Naval Research




Marine Corps Liaison Representative

LtGen PAUL K. VAN RIPER, USMC, Marine Corps Combat Development Command




RONALD D. TAYLOR, Director

PETER W. ROONEY, Program Officer

SUSAN G. CAMPBELL, Administrative Assistant

MARY G. GORDON, Information Officer

CHRISTOPHER A. HANNA, Project Assistant




---

COMMISSION ON PHYSICAL SCIENCES, MATHEMATICS,
AND APPLICATIONS

ROBERT J. HERMANN, United Technologies Corporation, Co-Chair

W. CARL LINEBERGER, University of Colorado, Co-Chair

PETER M. BANKS, Environmental Research Institute of Michigan

LAWRENCE D. BROWN, University of Pennsylvania

RONALD G. DOUGLAS, Texas A&M University

JOHN E. ESTES, University of California at Santa Barbara

L. LOUIS HEGEDUS, Elf Atochem North America, Inc.

JOHN E. HOPCROFT, Cornell University

RHONDA J. HUGHES, Bryn Mawr College

SHIRLEY A. JACKSON, U.S. Nuclear Regulatory Commission

KENNETH H. KELLER, University of Minnesota

KENNETH I. KELLERMANN, National Radio Astronomy Observatory

MARGARET G. KIVELSON, University of California at Los Angeles

DANIEL KLEPPNER, Massachusetts Institute of Technology

JOHN KREICK, Sanders, a Lockheed Martin Company

MARSHA I. LESTER, University of Pennsylvania

THOMAS A. PRINCE, California Institute of Technology

NICHOLAS P. SAMIOS, Brookhaven National Laboratory

L.E. SCRIVEN, University of Minnesota

SHMUEL WINOGRAD, IBM T.J. Watson Research Center

CHARLES A. ZRAKET, Mitre Corporation (retired)




NORMAN METZGER, Executive Director



---

Preface





This report is part of the nine-volume series entitled Technology for the United States Navy and Marine Corps, 2000-2035: Becoming a 21st-Century Force. The series is the product of an 18-month study requested by the Chief of Naval Operations, who, in a memorandum on November 28, 1995, asked the National Research Council to initiate through its Naval Studies Board a thorough examination of the impact of advancing technology on the form and capability of the naval forces to the year 2035. To carry out this study, eight technical panels were organized under the committee on Technology for Future Naval Forces to examine all of the specific technical areas called out in the terms of reference.

The study's terms of reference (Appendix A) asked for an identification of "present and emerging technologies that relate to the full breadth of Navy and Marine Corps mission capabilities," with specific attention to "(1) information warfare, electronic warfare, and the use of surveillance assets; (2) mine warfare and submarine warfare; (3) Navy and Marine Corps weaponry in the context of effectiveness on target; [and] (4) issues in caring for and maximizing effectiveness of Navy and Marine Corps human resources." The terms of reference went on to identify 10 technical areas for special attention. One involved modeling and simulation (M&S): "The naval service is increasingly dependent upon modeling and simulation. The study should review the overall architecture of models and simulation in the DoD (DoN, JCS, and OSD), the ability of the models to represent real world situations, and their merits as tools upon which to make technical and force composition decisions."

It was against this background that the Panel on Modeling and Simulation was constituted and asked to develop the present report. Upon reviewing the terms of reference and defining a feasible scope of work, the panel noted that recent documents (some of them produced after the terms of reference were created) already provide a reasonable architecture-level survey of the Defense Department's M&S, as well as a vision statement. In particular, the Office of the Secretary of Defense's (OSD's) Defense Modeling and Simulation Office (DMSO) has developed a substantial Master Plan for M&S, the purpose of which is to establish a common technical framework for DOD's M&S.¹ Given this body of existing material, the panel focused its efforts on key issues that have previously received little or insufficient attention. The objectives the panel set for itself, then, were (1) to clarify why the Department of the Navy leadership should care and be concerned about the substantive content and comprehensibility of M&S; (2)to assess what the Navy Department (and DOD) may need to do to benefit fully from the opportunities presented by M&S technology; (3)to clarify what M&S can and cannot be expected to accomplish in aiding decisions on technical, force-composition, and operations planning issues; and (4)to present priorities for M&S-related research.

The panel made no attempt to conduct a full survey of M&S relevant to the Department of the Navy. Much of the report deals with large-scale joint models such as those used in campaign planning, the evaluation of systems and new doctrinal concepts, or joint training—e.g., M&S such as the Joint Warfare System (JWARS) and the Joint Simulation System (JSIMS) systems now under development. The report has less to say about engineering- or engagement-level models, although it discusses the important role of simulation-based acquisition. Finally, this report is not a "forecast," nor does it lay out "roadmaps" for what should be done decade by decade for the next 40 years. Instead, the panel has chosen to focus on a chronic problem that took many years to develop and will take many years to deal with effectively—the lack of a good military-science research foundation on which to base the modeling and simulation that it so much depends on—and on priorities for remedying that problem over the years ahead.

Panel membership included experts in the research for and development and application of modeling and simulation, in both defense and nondefense domains. It also included experts in force planning; operations planning; applied mathematics, including probability and statistics; modeling and simulation theory; physics, including statistical mechanics; control theory; computer science; electrical engineering; operations research; gaming; and strategic planning.

The panel met eight times to receive briefings from Service and industry representatives, visit facilities, deliberate, and draft its report. It also participated in the three plenary meetings for the overall study. The first plenary meeting, in March 1996, established organization and a common starting point for the entire study. It included presentations by the Chief of Naval Operations and other high-level officials of the Navy Department, the other Services, the Defense Department, and industry. The subsequent plenaries were for drafting, comparison and integration across panels, the working out of cross-cutting issues, and synthesis (reflected primarily in Volume 1: Overview). The result follows. The report (which consists of a summary, the main report, and a set of appendixes) discusses modeling and simulation as a foundation technology for many developments that will be central to the Department of the Navy and Department of Defense over the next 3 to 4 decades.

The panel report is, of course, a product of the whole. However, the Vice Chair, Paul Davis, organized and led report preparation. He and Richard Ivanetich also compiled the panel's work and briefed it to study leadership along the way.




---

NOTE

¹See Defense Modeling and Simulation Office. 1995a. Department of Defense Modeling and Simulation (M&S) Master Plan, Office of the Under Secretary of Defense for Acquisition and Technology, Washington, D.C., October; Kaminski, Paul G., Under Secretary of Defense for Acquisition and Technology. 1996. "DMSO 'Modeling and Simulation,'" Keynote address at DOD Fifth Annual Industry Briefing, Alexandria, Va., May 22; and other materials—both formal and informal—available from the DMSO or the DMSO's World Wide Web site at http://www.dmso.mil.



---

Acknowledgments





The Panel on Modeling and Simulation is indebted to many people who provided briefings, scientific papers, or discussion time. It gives special thanks to Darryl Morgeson and Chris Barrett of Los Alamos National Laboratory, Jeff Grossman of the Naval Research and Development Laboratory (NRaD), Les Parrish of SPAWAR, Bill Stevens and Jeff Steinman of Metron, Inc., Henson Graves of Lockheed Martin, Tom Skillman of Boeing, Timothy Horrigan of Horrigan Analytics, CDR Dennis McBride, USN, of the Office of Naval Research, and Wayne Hughes of the Naval Postgraduate School. CAPT Jay Kistler, USN, was the panel's contact with the Navy, the study's sponsor. Both he and CDR McBride provided useful briefings and contacts.

The panel also acknowledges RAND's courtesy in supplying several of the figures used to illustrate concepts discussed in the report.



---

Contents





EXECUTIVE SUMMARY

1 INTRODUCTION


Background and Objectives
Reasons for the Department of the Navy to Be Interested in and Concerned About M&S
Getting Started: Some Definitions and Distinctions
Structure of This Report


2 TECHNOLOGICAL PROSPECTS FOR DOD'S M&S


Application Areas
Data on the Value of M&S for Acquisition, OT&E, and Training
M&S as a Cross-cutting Foundation Technology
Some Observations, Forecasts, and Images
Tools for Decision Support
Other M&S-related Forecasts


3 POTENTIAL FAILURES AND DISASTERS FOR DOD'S M&S


Broad Observations
Intellectual and Technological Infrastructure
Complex Systems and the Need for Humility


4 DEALING WITH AND IMPROVING DOD'S M&S


Significance of the Issues
The Multifaceted Nature of Model Quality
Uncertainty as a Core Reality in Building and Using Models
Approaches to Dealing with Uncertainty
"Doing Better" on Model Content: Need for Managerial Changes, Not Just Token Exhortation
Verification, Validation, and Accreditation


5 FOCUSING WARFARE RESEARCH AND IMPROVING M&S


Background
Prioritizing Warfare Subjects for Research
Desired Attributes of Research Programs


6 CREATING AND IMPROVING INTELLECTUAL AND TECHNOLOGICAL INFRASTRUCTURE FOR M&S KEY TECHNICAL PROBLEMS REQUIRING INVESTMENT


Hierarchically Integrated Families of Models
M&S Infrastructure
Repositories and Model Integration
Advanced Environments and High-level Languages for M&S
Recommendations on Joint Models
Recommendations for Research


7 CHALLENGES IN ASSIMILATING AND EXPLOITING M&S TECHNOLOGY


Traditional Challenges
Implications for the Navy
A Background of Leadership Aloofness from M&S
New Circumstances and the Need for Technology-driven Attitude Changes
Issues for the Department of the Navy


BIBLIOGRAPHY

APPENDIXES

A TERMS OF REFERENCE

B VIRTUAL ENGINEERING: TOWARD A THEORY FOR MODELING AND SIMULATION OF COMPLEX SYSTEMS
 John Doyle, California Institute of Technology

C  SIMULATION-BASED ACQUISITION
 Richard Ivanetich, Institute for Defense Analyses

D EXPLORATORY ANALYSIS
 Paul K. Davis, RAND and the RAND Graduate School

E MULTI-RESOLUTION MODELING AND INTEGRATED FAMILIES OF MODELS
 Paul K. Davis, RAND and the RAND Graduate School
 Bernard Zeigler, University of Arizona

F MODEL REPOSITORIES AND ASSEMBLY AND INTEGRATION OF MODELS
 Bernard Zeigler, University of Arizona
 Paul K. Davis, RAND and the RAND Graduate School

G COMPONENTS OF A THEORY OF MODELING AND SIMULATION
 Bernard Zeigler, University of Arizona

H AREAS OF RESEARCH IN MODELING AND SIMULATION
 Bernard Zeigler, University of Arizona

I COMBAT MODELING ISSUES
  Paul K. Davis, RAND and the RAND Graduate School
 Donald Blumenthal, Gualala, California
 Donald Gaver, Naval Postgraduate School

J PROBABILISTIC DEPENDENCIES IN COMBAT MODELS
 Paul K. Davis, RAND and the RAND Graduate School

K M&S-RELATED EDUCATION
 Donald Gaver, Naval Postgraduate School

L ACRONYMS AND ABBREVIATIONS



---

Executive Summary





OVERVIEW: CONCLUSIONS AND RECOMMENDATIONS

The Department of the Navy must take a new look at modeling and simulation (M&S):

• The very nature of warfare is changing, perhaps drastically. The U.S. style of war is becoming technologically complex and dependent on distributed and interconnected systems. Modeling and simulation will be core tools for planning and conducting warfare as revolutionary changes in military affairs take place, especially since intuition based on past wars will become less helpful over time.

• Indeed, independent of Navy and Marine actions, M&S will be deeply embedded within joint command-and-control systems. Without enhanced efforts, the Navy and Marine Corps will not understand the strengths or limits of such models and simulations, nor be proficient with them.

• M&S will also become a core feature of system development and acquisition, as is the case already in leading-edge civilian industry. Because of its centrality, M&S should be seen as an enterprise technology in itself—part of the revolution in business affairs that is now a key element of the Department of Defense's (DOD's) overall strategy.

While the future of M&S should be exceedingly bright, the Department of the Navy will not be able to exploit its potential unless it attends to serious and chronic shortfalls—the most important of which relate not to software, but to the quality and content of the underlying models. Dramatic advances are being made in DOD's M&S, but these advances are associated mainly with computer and software technologies. By contrast, too little attention has been focused on the content of the models themselves, or on the research base needed to create that content. Failure to address this shortfall will inevitably lead to less effective but more expensive combat forces and—quite possibly—serious operational failures. The escalating complexity of planned systems and operations creates profound integration challenges requiring superb M&S for success—and for the avoidance of downright failures.







FIGURE ES.1  Looking at M&S strategically.




All of this suggests that the Department of the Navy needs to make an attitude shift regarding M&S, which has never previously merited a high priority for leadership attention. Today, what is needed is a strategic commitment to exploiting M&S (Figure ES.1). This, of course, would lead to a strategy, policy, and investment actions. In this report the Panel on Modeling and Simulation identifies priorities for such matters. One priority involves the two principal joint simulation programs for training and analysis, the Joint Simulation System (JSIMS) and the Joint Warfare System (JWARS), respectively. Since first-generation versions will be quite imperfect and the systems may last 10 to 20 years, the panel recommends that the Department of the Navy take an active role to ensure that JSIMS and JWARS are produced as evolving systems that incorporate future research results. This is not simply a management issue, but rather something very challenging technically, since the architecture of the simulations must allow this evolution. Also, links must be created between the research and M&S worlds, links that do not now exist. And the research itself must be expanded substantially.

The research needed falls into two categories, warfare-area research and fundamental research. The former involves understanding the processes that take place in combat (e.g., those of littoral operations or long-range precision strike). This is military science, not programming, and commercial industry will not lead the way. The fundamental research needed involves theory and methodology that will make it possible to design and construct sound and reusable models and simulations that will also be comprehensible, flexible, and testable in specific contexts. Achieving these features will require major advances. It follows that the panel recommends that the Department of the Navy (working with other Services and DOD as appropriate) establish a robust but focused program in research, with both warfare-area research and research on fundamental theory and methods.

Unfortunately, research alone is not sufficient, and its fruits are often not harvested because researchers, M&S developers, warfighters, and other leadership figures are often disconnected. Accordingly, the panel recommends that the Department of the Navy establish processes that ensure early scientific review of models emerging from research, a competitive atmosphere in which "the market" of model users is both encouraged and assisted in constantly evolving their M&S to represent the best available knowledge (i.e., in assimilating improvements), and a general emphasis on quality, including the ability to represent uncertainty. Accomplishing this will require a multiyear commitment of senior leadership, because the baseline culture is very different from the one needed. Elements of a changed approach would include enhanced officer education and continual "beta testing" of models and simulations by organizations such as the war colleges and commands, testing that extends deeply into content, not merely software performance. However, such "operators" will need substantial assistance from the scientific community.

In addition to addressing the quality and content of models and simulation, the Department of the Navy needs to make investments that cut across usual organizational stovepipes and budget accounts. This relates to the promise of simulation-based acquisition. Accordingly, the panel recommends that the Department of the Navy treat simulation-based acquisition (SBA) as a key enabling technology with extraordinary long-term leverage and that it organize and invest consistently with that enterprise-technology view.




RICH OPPORTUNITIES FOR MODELING AND SIMULATION

Modeling and simulation (M&S) offers the promise of greatly enhancing future naval force capabilities and achieving major cost savings. To cite a few examples:

• Concept development. Next-century naval forces will require new operational concepts and force structures—in some cases radically different from current ones. Ours is an era of military ferment analogous to that of the 1920s or 1930s. M&S can help screen, design, and test concepts and force structures before irrevocable commitments to them are made. In principle, M&S could also be compelling enough to "force" needed changes of doctrine before disasters occur in war.

• Simulation-based acquisition (SBA). Representations of proposed system designs can be constructed and tested in simulated environments. These virtual prototypes can be used to refine system requirements and relate tradeoff and engineering decisions to these requirements. Subsequently, computer-based representations can be maintained as development and production occur, and as modifications are introduced throughout the life cycle. The results can be more affordable systems that are better attuned to an operator's needs, easier to assimilate, and easier to modify. The remarkable success of the Boeing 777's development merely tapped the surface of what will eventually be possible.

• Decision support. M&S can assist commanders in their planning for combat and other military operations. Key uses include developing and assessing proposed courses of action, mission planning and rehearsal, and dynamic situation assessment and adaptation in the course of battle. M&S-based decision support also has an important role in peacetime activities such as concept evaluation and resource allocation.

Models and simulations are being used for all of these functions today. Indeed, the breadth of M&S is enormous, as suggested by Figure ES.2. The best-known recent successes have been in training, but there is rapidly growing documentation on valuable applications throughout Figure ES.2's cube—as judged not only by those advocating M&S, but also by senior commanders. Documented examples of recent high-leverage payoffs from M&S are given in Chapter 2, but the future is more relevant in this study. It is especially significant for the future that M&S will be thoroughly embedded in command-and-control systems. There have already been numerous exercises in which differences between the real and the simulated have been blurred or made invisible to some participants. This will be increasingly the case as U.S. forces adopt the concepts sketched in the Joint Chiefs of Staff's Joint Vision 2010 (Shalikashvili, 1996).




THE POTENTIAL FOR FAILURES AND DISASTERS

Cautions Amidst Enthusiasm

Despite this bullish introduction and the fact that M&S will surely "take off" in the commercial sector, the potential of M&S for the Department of the Navy and DOD may not be realized in the foreseeable future. Some of the DOD's most important and expensive M&S efforts may fail or—perhaps worse—end up saddling the DOD components with mediocre, inflexible, and sometimes misleading tools that impede innovation and improvement of content. There is also the potential for disasters due to overdependence on M&S for images and predictions that appear more valid than they actually are (the "Spielberg" effect). This could cost lives and undercut military operations.







FIGURE ES.2  The scope of DOD's M&S activities. SOURCE: Adapted from Defense Modeling and Simulation Office (1996d), Figure 2-1.




One basic problem is that DOD's investments have been concentrated more on content-neutral computer and software technologies rather than on the content of the models.¹ On the technology side, much has happened, for example, object-oriented programming, high-performance computing, computer-aided design, and establishment of the communications protocols and infrastructure for everything from collaborative, distributed, multidisciplinary, simulation-based engineering design to large-scale distributed war games. The dramatic progress in technology will assuredly continue because of commercial developments and DOD efforts such as those of its Defense Modeling and Simulation Office (DMSO).




An Undernourished Knowledge Base

In contrast, there has been curiously little investment in the knowledge base determining the substantive content and quality of much M&S—particularly higher-level M&S needed for mission- and campaign-level work. It is an open secret, and a point of distress to many in the community, that too much of the substantive content of such M&S has its origin in anecdote, the infamous "BOGSAT" (bunch of guys sitting around a table), or stereotypical versions of today's doctrinally correct behavior. There is a need for focused research on the phenomena of combat and other military activities, both historical and prospective. This is the realm of military science.

Another shortfall in knowledge relates to theories and methods for conceiving, designing, and building models (as distinct from software). Symptoms of the problem are evident if one observes that DOD's M&S often consists of nothing more than the computer code itself: there is no separable documented "model" to be reviewed and improved, nor any way to readily understand the assumptions generating the simulation's behavior. This can hardly be a comfortable basis for decision support.




Complex Systems

The inherent complexity of the systems and force operations that DOD is attempting to simulate introduces new difficulties. Too many forecasts are extrapolating unreasonably from the Boeing 777 experience, and from M&S successes in weapon-system and small-unit training, to imagined M&S systems of extraordinary complexity. Recent failures such as the automated Denver airport baggage system and the Federal Aviation Administration advanced air-traffic-control system suggest the difficulties associated with reliably modeling and engineering complex systems. The military operations envisioned for future forces in the information era involve exceedingly complex systems.

Complexity is a multifaceted concept, but to appreciate some of what is involved here, the panel notes that planners, commanders, and engineers are most familiar and comfortable with systems (and models) that are primarily static, linear, and deterministic. However, automated and integrated military systems (the systems-of-systems approach) involve systems and models that are dynamic, nonlinear, and heterogeneous interconnections of mixed subsystems, with a much more sophisticated treatment of uncertainty, including uncertainties about the opponent's intentions and actions.

The panel's principal observation here is that dealing with such complex systems effectively is a decades-long task. In the meantime, there is need for humility, multiple approaches and competition, patience, and hedges. And even long-run success will require profound changes in the way M&S is conceived and designed, as well as an across-the-board attention to its content and validity—in a sense that suitably recognizes uncertainties. The panel describes what is needed below.




Assimilation of M&S Technology

Finally, there are the problems of assimilation and exploitation. M&S is an enabling technology, but its value is cross-cutting, and it has no natural single home. Nor should it, since—in a partial defense of stovepiping—the majority of work must be dictated by the needs of individual applications. However, whenever such a cross-cutting technology is introduced or its use expanded, there are organizational and managerial challenges. The key to success is often having a strategy embraced by the organization's leadership.

Against this background diagnosis, the panel has observations and suggestions in each of a number of subjects. In essence, the recommendation is for a concerted and long-overdue effort to improve the research base for the Department of the Navy's and DOD's models, and to ensure that the results of research are in fact incorporated. The panel's recommendations include priorities and suggestions for a strategy, not simply general funding of research.




RECOMMENDATIONS ON JOINT MODELS

Concerns

One useful focus for the Department of the Navy's thinking about M&S is the set of joint systems now in development (most prominently JSIMS and JWARS). Taken together, these worthy programs (including the Service components) have a price tag approaching $1 billion. It is DOD's intention that JSIMS and JWARS will become the core for all future joint work on training and analysis, respectively. Although this is unlikely (a wider range of models will probably prove necessary), it may indeed be that JSIMS and JWARS will dominate the joint M&S scene for the next 20 years. Thus, it is important to the Department of the Navy that naval forces be adequately represented. Otherwise, valuable training opportunities will be compromised and the Navy and Marines will suffer in the competitions about doctrinal changes, future missions, and force-structure tradeoffs. More generally, the quality of joint work will suffer.

Unfortunately, it is likely that first-generation versions of JSIMS and JWARS will not be satisfactory—even with heroic efforts and even though the products will have many excellent features. There will be major shortcomings with respect to both content and performance. Consequently, the panel recommends that the Navy insist that DOD and the program offices adopt open-architecture attitudes that will promote rather than discourage substitution of improved modules as ideas arise from the research and operations communities, and that they build explicit and well-exercised mechanisms to ensure that such substitutions occur.

This recommendation may seem uncontroversial, and it calls for no more than what some of the programs (notably JSIMS) are projecting on viewgraphs, but the history of DOD modeling has often been to produce relatively monolithic and inflexible programs. Further, there has been great DOD emphasis in recent years on avoiding alleged redundancies, collecting "authoritative representations," and exercising configuration control. The panel observes widespread frustration among analysts and other substantive users of models, who see DOD's M&S efforts as driven by civilian and military managers who think models are commodities to be standardized, who sometimes seem to value standardization more highly than quality (harsh words, but too important to be omitted), and who have given near-exclusive emphasis to software technology issues. They and the panel believe that M&S should instead be seen as organic, evolving, and flexible systems with no permanent shape (but with standardized infrastructure, including many component pieces).

In fact, the visionary technical infrastructure being promoted by OSD's Defense Modeling and Simulation Office (DMSO) (and software technologists) will permit the open system approach and will permit competition among alternative models (e.g., alternative representations of ballistic-missile defense, mine warfare, or command, control, communications, computing, intelligence, surveillance, and reconnaissance (C⁴ISR)). Thus, while it would be easy for JWARS, JSIMS, and other systems to end up as rigid monoliths, with the right architecture and organizational structure DOD can have its cake and eat it: it can have "standard configurations" while still making it easy for users to substitute model components as new ideas and methods emerge. An important but more subtle aspect of this infrastructure is connecting model evolution to the R&D and operational communities concerned with both current and futuristic doctrine; and, significantly, nurturing a competition of ideas and models. In that way the evolution will be more like survival of the soundest than like continuation of what has previously been approved.

The panel underlines the problem of incorporating research results when they exist because, at present, the communities that do research and the programming of models often do not communicate well and there is little pressure to assure that the "best" models are reflected in M&S. Indeed, there is much pressure to avoid changes.




Technical Attributes Needed in Joint Models

Against this background of concerns, the panel recommends that the Navy advocate an approach to joint-model development that has a long-haul view and an associated emphasis on flexibility. The groundwork should be in current model-building efforts for the following, which will be important in selected applications in the years ahead:

• Multi-resolution modeling, not only of entities, but also of physical and command-and-control processes, with the objective of building integrated models of families with different levels of resolution.

• Explicit decision models representing the reasoning and behavior of commanders and different levels, and reflecting in natural ways courses of action, plans, and the adaptations that commanders make in the course of operations.

• Diverse representations of uncertainty, including use of probability distributions (and, sometimes, alternatives such as fuzzy-set concepts), even in aggregate-level models.

• Systematic treatment of important correlations (e.g., the "configural effects" of mine warfare and air defense).

• Explanation capabilities linking simulated behavior to situations, parameter values, rules and algorithms, and underlying conceptual models.

• Mixed modes of play that are interactive, selectively interruptible (e.g., for only higher-level commander decisions), and automated. (The panel regards the option for human play as critical for analytic applications as well as training, and the option of closed play, e.g., of the opponent, as critical for training.)

• Testing of new doctrinal concepts requiring new entities, attributes, and processes.

• Different types of models. The systems should accommodate model types as diverse as general state-space and simple Lanchester equations, entity-level "physics-based" models, and agent-based models with emergent behaviors. They should employ such varied tools for such uses as statistical analysis, generation of response surfaces, symbolic manipulators, inference engines, and search methods (e.g., genetic algorithms). The models must be able to deal not just with old-style head-on-head attrition warfare, but also with maneuver warfare on a nonlinear battlefield in the information era, and with operations in urban sprawl. They must reflect different command-and-control concepts.

• Tailored assembly. The systems should facilitate tailored creation of models, including relatively simple M&S for specific applications. That is, one should conceive of JSIMS and JWARS as tool kits with rapid-assembly and modification mechanisms. Excessive complexity obfuscates and paralyzes.

In some respects, the last item is the most important. Given the breakthroughs in software technology over the last two decades, it is feasible (though not easy)—and essential—for major M&S efforts to be designed for frequent adaptation, specialization, and module-by-module improvement. One should think of assembling the right model, not taking it from the shelf whole. Further, it should be possible to discard or abstract complexities irrelevant to the problem at hand. Doing so runs directly counter to the common inclination to seek high resolution for everything, but tailored simplifications are crucial in applications—especially when they involve "exploratory analysis" over diverse situations and assumptions rather than point calculations. The type of analysis is crucial when uncertainties are large—as they often are. In any case, the need for simplicity is generally much better understood by those who have conducted studies or exercises, or designed decision-support systems, than by those who develop software.

The panel notes, however, that there are limits to what can be accomplished by assembly or composition. The Department of the Navy and DOD should be skeptical about the notion that a single system (e.g., JWARS or JSIMS) will prove useful to a wide range of communities. It is one thing to assemble and tailor components for one study rather than another, or for one exercise rather than another. It is a very different matter to have the same system and library of components support a broad range of different functions (testing, exercises, force planning, etc.). Viewgraphs postulating such versatility do not constitute an existence proof.




RECOMMENDATIONS FOR RESEARCH

Research in Key Warfare Areas

As noted above, there has been relatively little recent investment in understanding the phenomenology of military operations at the mission and operational levels. Much of the basis for related M&S is still programmer hypothesis and qualitative opinions expressed by subject matter experts. This has not always been so. During and after World War II, operations research worked from a rich empirical base, but now the United States is entering a period of nonlinear, parallel, information-era warfare for which the intuition of scientists, operations researchers, and warriors is insufficient and unreliable. Further, it will be relying on complex systems working as designed in multifaceted joint campaigns. Success may be much less tolerant of errors in concept and execution than in days past. Indeed, some of the doctrinal concepts under discussion will involve very high risks.

Given, then, that improvement of the research base is essential, how might it be accomplished? Rather than merely urging general support for research, the panel recommends a managerially focused approach with priorities and mechanisms for assuring relevance and assimilation. Accordingly, the panel recommends that the Navy and Marine Corps select a few high-priority warfare areas and create research programs to support them. These programs should be organized so as to ensure close ties to operational and doctrinal-development communities, and to relevant training and exercise efforts that could be mined as a source of empirical knowledge (e.g., as suggested in Figure ES.3, which would exploit emerging capabilities for distributed interactive simulation). This is a nontrivial and potentially controversial suggestion, since the long-standing tradition has been to avoid—and even prohibit—extensive data collection for use beyond those being trained. The costs of such efforts would be small in comparison with those for buying and operating forces, or even procuring large models. Although the Department of the Navy (and DOD) need to make up for past failures to invest adequately in research, this is a domain in which a total of $20 million to $30 million per year can accomplish a great deal.







FIGURE ES.3  Using exercises as a source of empirical data for M&S. SOURCE: Reprinted, by permission, from Davis (1995b). Copyright 1995 by IEEE.





As a first list of warfare areas for focused research, the panel recommends the following, which have some overlaps:

• Expeditionary warfare and littoral operations;

• Joint task force operations with dispersed forces;

• Long-range precision strike against forces employing countermeasures;

• Theater-missile defense, including counterforce and speed-of-light weapon options, against very large ballistic-missile and cruise-missile threats; and

• Short-notice, early-entry operations with opposition.

Each of the above warfare areas has major knowledge gaps that could be narrowed by empirical and theoretical research closely tied to the "warrior communities."

This report describes key attributes of research programs for such warfare areas. An overarching theme is the need to take a holistic approach rather than one based exclusively on either top-down or bottom-up ideas. A second theme is that the research should be seen as focused military science, not model building per se. This will determine the type and range of people involved, and also the depth of the work.

Two examples may be useful here. The first is the challenge of developing command-control concepts for highly dispersed Marine Corps forces operating in small units far from their ship-based support and dependent on a constellation of joint systems. The Marine Corps is studying alternative concepts in the Hunter/Warrior experiments. Such experiments need to be accompanied by systematic research and modeling of different types, perhaps including new types of modeling useful in breaking old mind-sets. It is plausible, for example, that cellular-automata models could help illuminate behaviors of dispersed forces with varying command-control concepts ranging from centralized top-down control to decentralized control based on mission orders. To its great credit, the Marine Corps is exploring such possibilities, opting to accept some "hype and smoke" in the realm of controversial complex-system research in exchange for new perspectives and tools useful in doctrinal innovation. While the panel does not believe such simplified models will prove adequate in the long run, they can be very helpful in developing new hypotheses.

A Navy example involves mine and countermine warfare. From prior research based on sophisticated probabilistic modeling accounting for numerous "configural effects" (i.e., effects of temporal and spatial correlations), it is known that effective strategies for mine-laying or penetrating minefields are often counterintuitive. By exercising such models and simulation-based alternatives in an exploratory manner (as distinct from answering specific questions), it should be possible to develop decision aids of great value in training, acquisition, and operations. Such aids should not, however, focus only on "best estimate" single-number predictions; they should instead provide commanders with information about odds of success, as a function of information. If the aids are to be useful, they must be informed by an intimate understanding of operational commanders' needs.




Recommendations on Fundamental Research

While many research activities are best driven by applications, other critical areas of M&S research require more fundamental research that might be sponsored by the Office of Naval Research (ONR), other Service analogues, the Defense Advanced Research Projects Agency (DARPA), and the Director of Defense Research and Engineering (DDR&E). In what follows the panel suggests particular subjects for fundamental research, divided into theory, advanced methodologies, and infrastructure (including tools).

Modeling Theory

The panel recommends that research on the following be given priority:

• Multi-resolution modeling, integrated families of models, and aggregation-disaggregation. Multiple-resolution modeling depicts phenomena at different levels of detail. In some cases a single model can operate at different levels of resolution with appropriate kinds of consistency. More often it is possible—although unusual and difficult—to design integrated families of models that can be mutually calibrated using information available in many forms and resolutions, and to do so with full recognition of statistical averaging issues. Such families of integrated hierarchical models would be invaluable in all application areas and would substantially improve validity, traceability, and the design of exploratory analyses. How to design and build such multi-resolution models or families, however, is a frontier problem in modeling theory. A related subject is often called aggregation-disaggregation. This often refers to distributed simulations in which, within the course of a simulation, some of the entities must be disaggregated and reaggregated (e.g., a Marine Corps company might have to be disaggregated to engage simulated entity-level opponents and then reaggregated to continue its maneuver). While this is relatively straightforward from a software perspective, there are deep questions concerning the consistency of behaviors at different resolution levels.

• Agent-based modeling and generative analysis. Some of the most interesting new forms of modeling involve so-called "agent-based systems" in which low-level entities with relatively simple attributes and behaviors can collectively produce (or "generate") complex and realistic "emergent" system behaviors. This is potentially a powerful approach to understanding complex adaptive systems generally—in fields as diverse as ecology, economics, and military command-control. A fundamental step in developing particular models and simulations is deciding which attributes and interactions to represent, and in what detail. This choice should be the one that most adequately describes the phenomena one is trying to observe, but that choice is often not known until the subsystems are connected and the simulation is run. Thus, methods should be developed to allow one to iterate on the choice of the initial representations of subsystem models, based on results of their use in interconnected systems.

• Semantic consistency. Phenomenological representations in different simulations need to interact with one another in distributed simulations. Such interaction is meaningful only if the representations are "semantically consistent," that is, if there is a shared understanding of what concepts and data "mean." This requires commonality of context and definition (or well-understood translations). The Navy should track related research, adding to it for special purposes. It should also support DMSO efforts to develop common models of the mission space (CMMS), which will assist in establishing semantic consistency in particular contexts and in developing integrated families of models.

Advanced Methodologies

The general task of developing and using models and simulations, and the particular activity of forming phenomenological representations, would be aided by methodological advances. Particular topics are as follows:

• Characterization of uncertainty. No matter how careful one is in preparing for a simulation, certain attributes and interactions will have some measure of uncertainty. Often, uncertainties dominate the problem. Methods to track the propagation of uncertainties should be developed since they can lead to large uncertainties in the output of the simulation. This is a particular challenge in heterogeneous, nonlinear dynamical systems, where uncertainties in components can interact in nonintuitive and unpredictable ways. The so-called "butterfly effect" in chaotic systems is a well-known popular example.

• Exploratory analysis under uncertainty. Running a simulation for one set of fixed conditions is generally not satisfactory since there are often large uncertainties throughout the system. Even normal sensitivity analysis on a one-variable-at-a-time basis does not suffice because of interaction effects. An important research area, then, is developing ways to use modern computer power to explore the space of simulation outcomes and to search for interesting regimes (e.g., regimes representing high or low risks for an operation or for especially profitable, or unacceptable, performance of a weapon). This research has implications for the design of models (some of it closely related to multi-resolution modeling), search engines, and visualization methods. It has even more profound implications for analysis and decision making because it encourages decision makers to ask not about best-estimate outcomes, which are often no more likely than very different ones, but rather about how outcomes of a strategy would be likely to vary as a function of the many assumptions in "scenario space." This can help by focusing attention on the need to avoid "dangerous regimes" in the course of operations, by focusing attention on the search for crucial information, and by emphasizing the need for both hedging and adaptability. This approach, of course, is quite different from the search for mythical optimality.

Infrastructure, Tools, and Supporting Technology

With regard to infrastructure, the following research areas are of particular significance:

• Intellectual infrastructure. Scientific and engineering disciplines typically have a mathematical language in which to frame and solve their problems—e.g., the use of calculus for disciplines as diverse as aeronautical engineering and chemistry. In contrast, there is no widely understood and adopted theoretical basis for M&S. To some extent, object-oriented modeling (not programming) is helping here, but in practice it usually deals with only some of the problems. While mathematics and systems theory can form a common language, modeling assumptions and their consequences tend to be domain-specific and implicit. Even worse, the only underpinning to many simulations is the computer code in which they are written. To help create the needed intellectual infrastructure, the Department of the Navy and DOD should cooperate with industry and universities in encouraging the development of theory and the promulgation of standard texts and case studies. DOD's adoption of software engineering methodologies (e.g., in the JWARS effort) is useful here. It may also be useful for the Department of the Navy, other Service components, and OSD to cooperate in developing "virtual centers" exploiting the World Wide Web, and in establishing additional peer-reviewed journals and scientific conferences overseen by research institutions.

• Object repositories and interface standards to enhance reusability and composability. Object-oriented technology admits the possibility of assembling major parts of simulations to meet the demands of a particular application from sets of stored objects representing entities and processes. Realization of this capability requires being able to manage large numbers of objects and to ensure consistency despite involvement of multiple developers. Such a capability could reduce costs in simulation development and allow flexibility in simulation application. A key of OSD's strategy in this domain is embodied in the high-level architecture (HLA), which establishes standards for M&S that may be used in federations employing distributed simulation. Despite controversy and anxiety about program costs, the HLA is a needed step in the direction of increased modularity and interoperability. It will have many long-term benefits. The Navy should support and exploit the HLA initiative—recommending modifications as needed.

• Explanation/traceability capability. This capability applies to all phases of the management process. For example, it would help document the source code with multimedia techniques so that one could understand the phenomena being represented, and it would help explain the results of a simulation by displaying the logic trail that led to the results. Realization of this capability would figure centrally in achieving the verification, validation, and accreditation (VV&A) of simulations, both in the formal sense and to the satisfaction of individual users. This capability is important for field commanders, managers, and engineers.

For example, commanders using M&S to assess courses of action may need to know the following: On what assumptions do the simulation outcomes depend critically; should those assumptions be modified and the simulations rerun; what if the component commanders are given contingent orders? The implications of having both comprehensible models and effective explanation/traceability capability are so significant for VV&A that the Navy should undertake more general efforts. The Navy should use commercial products where appropriate and should foster commercial development where necessary, since the capabilities required will have more general value (e.g., to computer-aided design). Such commercial coupling is necessary because development of these capabilities will be expensive.

• Other tools. Many other tools are badly needed. These include tools for (1) automated scenario generation and experimental design and (2) postprocessing and data analysis.

Research Is Not Enough: Planning to Incorporate Its Fruits

There is one further challenge associated with research: assuring that its products are recognized and used appropriately. This is challenging because the research, M&S-building, and user communities are reasonably distinct. Those building M&S often are only minimally acquainted with cutting-edge research in either the phenomenology of warfare operations or modeling methodology. Further, their sponsoring organizations are often more interested in the stability of M&S (and the related gargantuan databases) than in improvements of "theory," the rewards of which may be less than immediately tangible. This state of affairs probably continues because so little higher-level M&S is bounced against empirical realities. The panel has several suggestions here:

• Requiring documentation of "conceptual models" (as distinct from details of the implementing programs and databases),

• Providing scientific review of such models to advise the Navy about the quality of the models in relation to scientific knowledge and best practices in the community, and

• Redefining the JWARS and JSIMS programs to have a continuing component responsible for reviewing, sponsoring, and incorporating research results. The function should be one of nourishing military science, not merely administration or auditing. Major changes are needed if there is to be a resurgence of in-depth study of military phenomena and the kind of open scientific discussion and debate that will lead to top-quality M&S.





RECOMMENDATIONS ON ASSIMILATING
AND EXPLOITING M&S

The Need for Strategic Commitment

Finally, there are the organizational problems of assimilation and exploitation. As noted above, M&S is an enabling technology. However, whenever such a cross-cutting technology is introduced, there are organizational and managerial challenges. It is commonplace for the organization to measure the value of investments against the wrong yardsticks (e.g., saving money in narrow domains, as distinct from changing the very way the organization does business and improving effectiveness for mainstream missions). It is also common for investments to go awry because (1) the new technology is procured and used as an add-on without sufficient buy-in and influence by the organization's core responsibility and workers, (2) too much is done by committee without leaders and champions who understand the core business, or (3) the educational groundwork has not been laid. Despite much ongoing success, all of these M&S-related problems are visible in DOD's components, including the Navy and the Marine Corps.

Based on the history of technology assimilation and the specifics of the current situation with respect to M&S, the panel recommends that the Department of the Navy make a strategic commitment to the success of exploiting M&S. Such a commitment would have consequences for organizational structure and responsibility (although the panel makes no recommendations on such matters), investment mechanisms (e.g., assuring that investment funds are available without forcing program managers always to make tradeoffs within their own domains), and the establishment of clear policies and strategies that would make manifest the leadership's demand for constant improvements in "validity" (as understood in the context of sometimes-extreme uncertainties) and usefulness to decision makers. As discussed above, the panel believes that the appropriate strategy would place considerable emphasis on warfare areas and cross-cutting modeling challenges, rather than still more emphasis on computer and software technology. To put this more bluntly, if funding tradeoffs are needed within M&S budgets, then the panel recommends giving higher priority to research improving model content rather than programming or reprogramming of current models.

In addition to investing in research to improve the quality and content of models, the Department of the Navy must organize, plan, and invest strategically if it is to enjoy the potentially great benefits of simulation-based acquisition. In doing so, it should take a long view because, as in other aspects of M&S, there are substantial obstacles. Success will be evolutionary over a period of decades.




Education for Next-generation Officers

One element of a strategy should be increased education in M&S for next-generation officers. The effective exploitation of M&S depends on the experience, knowledge, and wisdom of its practitioners, hence upon their education. The panel recommends increased Navy investment in such education at all levels: for those who acquire and design M&S tools, and also for those who rely on them to guide acquisition, training, and operations. Some of the education should be in the form of enhanced master's and Ph.D.-level programs. Other aspects should include short courses tailored for officers needing refresher courses, technology updates, and preparation for next assignments involving M&S management.




---

NOTE

¹There are some partial counterexamples, of course: e.g., "knowledge acquisition" for the "semiautomated forces" (SAFOR) used in the Joint Countermine Operational Simulation (JCOS) and the Army's tactical training system, new modeling of C⁴ISR effects in JWARS, development of object models, and research on configural effects. Even in these cases, however, the work has often been more like "computer modeling" than establishing an empirical and theoretical research base.