Click for next page ( 357


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 356
11 Recommendations for Military- Sponsored Modeling Research T his report has reviewed the state of the art in individual, organi- zational, and societal (IOS) modeling and the ability of current modeling approaches to meet military needs; assessed the common pitfalls and problems associated with this type of modeling; and pointed out areas in which additional work is needed. This chapter summarizes the committee’s recommendations for advancing behavioral modeling capabili- ties to meet the military’s current and anticipated needs. There are many challenges in advancing the science of human behav- ioral modeling. The theory on which to base the models is often fragmented and incomplete, failing to specify key links that are needed to answer the questions of interest. Data for testing theories and models (or for deriving empirically based models) are also sparse and often lacking in detail for exactly those factors that are critical for the model. Because of the scale of many behavioral models, it is rarely possible to generate useful data from controlled laboratory experimentation (as, for example, is often possible for models of individual cognition and behavior). Furthermore, there are often no well-defined criteria for success in these modeling efforts and no widely accepted definitions and methods for validation of IOS models. Finally, the research and development efforts are being conducted in many different disciplines. Modelers currently use different types of data, at different levels of detail, to model different types of behavior in order to answer differ- ent kinds of questions. Little effort is devoted at present to comparison or integration of models from different perspectives. How, then, can this fragmented field best advance? Our recommenda- tions focus on cross-disciplinary information exchange and the compari- 5

OCR for page 356
5 RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH son and integration of models, structured around well-defined challenge problems and common datasets, with independent research thrusts recom- mended for those issues that are most critical. Recommendations fall into three broad categories: (1) large-scale, inte- grated cross-disciplinary research programs, focused around representative challenge problems and common datasets; (2) research in six independent areas that will advance the capabilities to address these integrated problems; and (3) multidisciplinary conferences, workshops, and other information exchange forums, with attendees to include not only model developers but also government program managers and military decision makers. INTEgRATED CROSS-DISCIPLINARy RESEARCH PROgRAMS We suggest the funding of multiple large-scale, multiyear research pro- grams that focus on comparing and, if appropriate, integrating models from different disciplines, different perspectives, and different levels of detail. This funding would provide incentives for researchers in diverse disciplines to work together on military-relevant problems. The goal would not be to pick the best model but rather to create a level playing field on which the capabilities of different approaches could be compared and the strengths of each assessed (see Gluck and Pew, 2005, for a description of a similar research program conducted for individual cognitive models). The ultimate goal is to move IOS modeling science forward through the process of com- parison, docking, and integration. It is essential for all participants in each program to focus on the same well-defined challenge problem instantiated in a common testbed and to use a common program dataset. At the heart of each program would be a representative problem that is critical for military operations, defined in detail. The five representative problems described in Chapter 2 provide a possible starting point for choosing the problems to be addressed. The definition of challenge problems is a difficult but essential step for the recommended IOS modeling research program, and it should be the first step in such a program. Initial grants should fund challenge problem devel- opment, and continuation of the program should be contingent on success in defining these problems. Operational users must be involved in defin- ing the challenge problems, and the criteria for modeling success should be clearly specified as part of problem definition. What type of model results—discovery, understanding, or forecasting—will be relevant for the problems being defined? What actions may be taken based on the model results? Criteria for model usefulness in the challenge problems should be clearly defined up front. The research teams for these efforts should be multidisciplinary, and the program team should also include military users with operational experi-

OCR for page 356
5 BEHAVIORAL MODELING AND SIMULATION ence in the domain for which the models are to be developed. These users will be ultimate judges of whether model results are useful (which we argue is the ultimate criterion for validation; see Chapter 8) and will provide advice on how the model results can best be presented for immediate com- prehension and relevance. The use of a common challenge problem and a common testbed will facilitate the docking of the different models for purposes of comparison. These integrated programs will encourage mutual education between modelers and operational users. Researchers will learn about the military domain and about user expectations. Users will learn about the scientific limitations to understanding of the basics of human behavior and what is feasible to represent in models (and implement in usable simulations) and will develop an understanding of the level of uncertainty associated with model forecasts or predictions. Results should be presented at workshops for program participants and other interested parties and at public confer- ences, published in the open literature for the research community at large, and presented “up the chain” to the program managers who rely on these models for operational, training, and mission rehearsal uses. INDEPENDENT RESEARCH THRuSTS In support of the integrated programs we recommend, we have identi- fied six independent areas in which research is needed. Progress in each of these areas could increase the ability to develop the integrated modeling capabilities that are needed to address military problems. In each area, we suggest the funding of multiple research teams approaching the work from multiple perspectives, with periodic workshops for researchers to exchange results. We also suggest that operational users as well as government pro- gram managers participate in these workshops to draw on their areas of expertise and to gain better insight as to model capabilities and limita- tions. The funding structure of the programs should support and enable the participation of individual researchers or smaller laboratories in both academic institutions and industry and not be limited to large institutions, as is often the case in collaborative projects supported by the Department of Defense (DoD). Thrust 1: Theory Development Models should be conceptually correct and grounded in the underlying fundamentals of what is known about individual human and group social behavior. However, current theory in this area does not answer all of the questions needed to structure models that address relevant issues. Basic research is needed for theory development, especially for the low-level social

OCR for page 356
5 RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH behaviors (e.g., choosing friends) that are the building blocks for larger scale social behavioral patterns (e.g., joining a terrorist group). Since affective states and traits represent a key component of individual motivation and play a critical role in interpersonal behavior and group and organizational deci- sion making, basic research in emotion and emotion-cognition interactions should be emphasized. This theory development work must involve multiple disciplines and perspectives with periodic workshops to exchange results. Theory development challenge problems should be defined to guide the work, but these can be nonmilitary and need not involve the level of military detail necessary for the integrated problems discussed above. A series of workshops should be conducted with researchers to identify key theory gaps. We recommend working backward from a set of operational problems (as defined for the integrated programs) to identify areas in which lack of theory is impeding modeling progress. These theory gaps can be used to define theory challenge problems. Academic institutions are key players for theory development, but they need information, incentives, and funding to address these theoretical issues. There is a need to educate academic researchers in military domains, establish conferences and journals in which their results can be presented, provide postdoctoral support, and provide funding that allows researchers to spend time learning about military domains in depth. Funding for gradu- ate students is a key part of this thrust as a cost-effective way to bring about shared understanding and progress. Thrust 2: uncertainty, Dynamic Adaptability, and Rational Behavior Models must deal with the inherent uncertainty (nondeterminism) and the dynamic adaptation (nonstationarity) that characterizes human behav- ior. Models must also be capable of modeling both rational and nonrational behavior. Basic research is needed in each of these areas. Issues include • How should models capture the “uncertainty-in-the-small” associ- ated with individuals and small groups? How can model structures and parameters capture this variability, and how much of this vari- ability must be included for the purposes of the model? • How should models capture the “uncertainty-in-the-large” associ- ated with populations and variations in population distributions? For example, to what extent should models be based on mean val- ues versus capturing effects from the tails of a distribution? How much variability must be included for the purposes of the model? • How can models capture adaptation and learning over time and in response to actions by others? For example, models of cultural

OCR for page 356
0 BEHAVIORAL MODELING AND SIMULATION groups often assume that cultural identity is static and unitary. In fact, people have multiple overlapping identities and allegiances that vary in their influences over behavior. How can these be cap- tured in a model, and how can one estimate the effects of actions and events on the primacy of these multiple allegiances as they affect decisions and actions? • What are the factors that contribute to rational, adaptive behavior and what factors induce behavior that appears nonrational? His- torically, emotions and affective factors have often been adduced to explain irrationality, but recent research in psychology and neuro- science has demonstrated that emotions also play a critical role in rational, adaptive behavior. Likewise, behaviors viewed as purely cognitive—including habit, bounded rationality, the range of beliefs unfamiliar to the observer, and ignorance, as well as behaviors with strong cognitive and affective components, such as fanaticism—can lead to what appears to be irrational behavior. Models of both rational and irrational behavior must therefore capture all the key factors—cognitive, affective, cultural, and contextual—that motivate and shape behavior of specific individuals in specific situations. Better techniques are needed for understanding the implications of diversity and variability for model-based sensitivity analysis. Combinatorial explosion of possible combinations of parameters is a challenge, and better automated technology is needed to put the model through its paces to explore the parameter space effectively and produce robust results. Thrust 3: Data Collection Methods The difficulty of obtaining data is an ongoing challenge for IOS model- ing. Research is needed to develop better data collection processes through field studies, experiments, and potentially by using massively multiplayer online games (MMOGs). Although a variety of ethnographic data collection techniques are cur- rently in use, they need to be better tailored to the needs of IOS models. For field data collection, it is necessary to bring modelers and data collectors together to develop data ontologies, joint specifications, and data collection methodologies and tools that are specifically tuned to IOS models. MMOGs are an untapped resource for collecting social and behavioral data on a large scale. We recommend the creation of an MMOG facility that could serve as a testbed for exploratory research and model testing and the funding of basic research to determine if MMOGs can be used to test, verify, and validate IOS models. We recommend that funding be put into developing the science of MMOGs rather than in developing additional

OCR for page 356
 RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH artificial worlds. The research agenda for this facility should be developed through workshops that convene both IOS modeling scientists and game experts. Note that funding MMOGs is a risky endeavor, with no guarantee that games that are useful for research purposes will find the widespread interest necessary for extensive data generation, but we think that the potential benefits outweigh the risks. Given the critical role of emotions and affective personality factors in organizational decision making and behavior, it is also important to enhance the current methods for collecting affective data. Emotions and moods are notoriously difficult to assess accurately, particularly in natural- istic and field settings. Yet recent progress has been made in using multi- modal approaches to affect assessment, including physiological monitoring and indirect assessment of these transient states via diagnostic tasks and performance tracking. We recommend that funding be allocated to the con- tinued refinement of these methods and to the development of standardized assessment instruments, particularly in naturalistic settings. Thrust 4: Federated Models It is a fundamental conclusion of the committee that no single model- ing approach can provide all the capabilities needed by DoD. We recom- mend a federated approach in which modeling components are created to be interoperable across levels of aggregation and detail. For example, a federated model might include a detailed representation of a few key individuals, linked to group-level models of different cultural groups and terrorist organizations, linked to geographic sector–level models of the level of unrest in a city. This approach is flexible and extensible, allowing the addition or subtraction of models at different levels of detail as needed for the problem to be addressed. Combining model components to create federated models in the sense being recommended is not simply a matter of specifying and using interface-level syntactic compatibility protocols. It requires deep semantic interoperability (i.e., theoretical consistency). To create semantic inter- operability, it is necessary to recognize that the links among components are themselves elements of the model. Components created at different levels of detail and for different purposes do not simply snap together to produce meaningful results. Assuming that the interface protocol issues will be solved by others (e.g., enterprise database developers), research is needed to answer the following questions: • What is the best way to ensure that the models being federated embrace compatible assumptions regarding concept abstractions,

OCR for page 356
 BEHAVIORAL MODELING AND SIMULATION entity resolution, time scale resolution (tempo), uncertainty, adapt- ability, docking standards, input-output semantics, etc.? • How should the components of the federated model be encapsu- lated, and which elements must be exposed to other components? • How should specific classes of models be linked (e.g., cognitive models to social network models)? • How can developers ensure dynamic extensibility? These issues are not unique to IOS modeling. In addressing them, IOS modelers should maintain awareness of research and development in model federation in the larger modeling and simulation community. Thrust 5: validation and usefulness Current verification, validation, and accreditation (VV&A) concepts and practices were developed for the physical sciences, and we argue that different approaches are needed for IOS models. Specifically, we recom- mend the use of a “validation for action” approach that assesses the use- fulness of a model for the specific purposes for which it was developed. Although promising work has been done in testing IOS models through triangulation among multiple types of expertise and multiple data sources, and some work has been done in docking different models for comparison, these approaches are not widespread. We recommend organizing a national workshop to agree on appropriate processes for VV&A of IOS models and to outline a roadmap for developing improved VV&A processes and standards. On the basis of the results of this workshop, we recommend that a DoD-wide authority develop and disseminate VV&A processes and standards for IOS models. These standards should be developed de novo, not as an adjunct to conventional VV&A standards. Basing model validation on the usefulness of the model for specific problems requires that model purposes be clearly stated by model users and clearly understood by model developers. This is an area in which mutual education is needed. We suggest that, as part of developing a VV&A stan- dard for IOS models, clear guidelines be developed for specifying model purpose. Thrust 6: Tools and Infrastructure for Model Building It is important to reduce the barrier to entry for developing models, modeling tools, frameworks, and testbeds. Scientists should be able to build and validate models without the large overhead currently associated with many DoD modeling and simulation investments. It should be possible to tailor existing models easily for specific purposes.

OCR for page 356
 RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH Sharing of IOS modeling knowledge across disciplines, as facilitated by the conferences and workshops recommended below, will support this goal. Work is also needed to develop an infrastructure for IOS modelers, including a national network of possible collaborators, common databases for model development and testing, and frameworks and toolkits for rapid model development. There is also a need for web-based repositories of information about existing models and, later, model components. To facilitate the development and use of shared ontologies and model components, funding must also be allocated to the refinement of existing markup and modeling languages, as well as the development of new lan- guages for particular domains or tasks. The limited data that exist for IOS models are often not accessible to model developers. We recommend the funding of national web-accessible data repositories that are open to researchers who seek to inform and test models. For militarily relevant domains in which some data are classified, we recommend an investment in automated tools to sanitize potentially sensitive military data. Often, the IOS models themselves are not readily accessible or even known to researchers or practitioners. Researchers are often unaware of efforts under way in DoD that are not reported on in conventional confer- ences and journals, and military developers are likewise unaware of progress being made in the research community. Or if they are, the typical user can face great difficulty in assessing the applicability of one approach or model over another, given their particular problem at hand. The occasional studies that attempt to survey the community and categorize development efforts and associated models, such as this one (see, e.g., Table 2-1 and Table 8-3) and its predecessor study (National Research Council, 1998), take small steps in this direction, but they are not meant to be exhaustive surveys and are only snapshots in time, become stale rather quickly, and fail to offer easy electronic access directly to the rich and evolving world of IOS models and associated simulations. We therefore recommend the development and maintenance of an online web-based catalog of general approaches, models, simulations, and tools. The notion is to develop something along the lines of the Defense Modeling and Simulation Office’s (DMSO’s) Modeling and Simulation Resource Repository (MSRR) at http://www.msrr.dmso.mil/, maintained by the Modeling and Simulation Information Analysis Center, or the clear- inghouse at Carnegie Mellon’s CASOS site (see http://www.casos.cs.cmu. edu). But to be effective, the envisioned site needs careful consideration in terms of the following: • Organization: The model ontology and site structure need to be carefully thought out, both from the researcher’s perspective (e.g.,

OCR for page 356
4 BEHAVIORAL MODELING AND SIMULATION foundational concepts underlying the particular model in the repos- itory) and from the user’s perspective (domain of application, limi- tations, simulation requirements, etc.). • Content: Considerations need to be given to what is maintained on the site, ranging from simple descriptive abstracts to full-fledged downloadable simulations and “read me first” instructions. • Currency: Once set up, the maintainers must devote effort to con- stantly updating the site, by tracking changes to existing models, adding new models that arise on the scene, and, certainly of equal importance, removing defunct models, or at least moving them to the archival section of the site to support historical surveys and the like. Failure to maintain currency will be the death knell of the repository, as it is with most websites today. One approach that should be considered is a Wikipedia-based model. • Usability: The site design needs to ensure ease of use for all autho- rized visitors, including contributors, users, and occasional viewers. Procedures need to be in place to vet content modifications or addi- tions, to support ease of navigation and internally searching for what the user is seeking, and to keep the site fresh and attractive to the larger community. It is clear that this cannot be a one-time effort like DMSO’s MSRR, nor an unfunded academic effort like Carnegie Mellon’s CASOS site. It needs significant startup funding and continued support over its lifetime. MuLTIDISCIPLINARy CONFERENCES AND WORKSHOPS A number of the issues and problems identified by the panel were the results of the failure of different disciplines to exchange information, or they resulted from misunderstandings among government funders of model development efforts, military users of models, and model developers. Because of the diversity of this group, there is no natural forum for them to exchange information, as there would be in conferences and journals for members of the same academic discipline or professional group. We there- fore recommend the organization of special-purpose workshops around the integrated research programs recommended above as well as workshops for the independent research thrusts described above. IOS modelers who are interested in working on military-relevant prob- lems need to be educated on: • The nature of the military decisions for which models are relevant and of the operational situations in which the decisions must be made.

OCR for page 356
5 RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH • Desired model functionality. • The most useful form(s) for presenting model results. • The value of work performed by others outside their discipline. • Feasible and appropriate VV&A approaches for IOS models. Operational users and managers need to be educated on: • The value of multidisciplinary approaches and the need for review of models from multiple perspectives. • The inherent uncertainty associated with IOS model predictions. • The value of models for sensitivity and trade-off analysis (versus the one right answer). • The design of virtual experiments to assess results over a range of conditions. • Reasonable definitions of validation for IOS models, feasible approaches for VV&A testing, and why these approaches differ from those used for physics-based models. The recommended workshops should involve model developers, opera- tional military users of the models, and government personnel who make funding decisions regarding model development. Issues to be discussed include methods for clearly specifying model purpose, criteria for judging the usefulness of models (i.e., what does it mean to validate a model), rea- sonable expectations for the certainty of model predictions, and methods for most clearly communicating model results. ROADMAP FOR RECOMMENDED RESEARCH The committee’s recommendations are based on the concept of use- driven research. As defined by Stokes (1997), use-driven research combines elements from both basic and applied research. Like applied research, use-driven research seeks to solve a practical problem—in this case, the development of IOS models that can serve military purposes. But, like basic research, it also asks “why” in a fundamental way—Why do some methods work and others not work? What are the principles that underlie success or failure? Figure 11-1 illustrates the major elements of a use-driven research program for IOS modeling. The process starts with challenge problem defi- nition, which includes a clear specification of the use to which a model is to be put. This specification should be based on the needs of the model users, expressed in terms that are meaningful to the IOS modeling com- munity. The challenge problem definition step is critical, and the funding of the remainder of the program should be contingent on its success. The

OCR for page 356
 BEHAVIORAL MODELING AND SIMULATION Can additional needs be met? Problem Definition Theory Data Development Collection Methods Model What additional data Is new theory needed? Development are needed? Federation Infrastructure Standards and Tools Model Has required interoperability What new tool been achieved? capabilities are needed? Validation Does the model meet the user’s needs? FIguRE 11-1 Elements of use-driven research for IOS modeling. 11-1.eps purpose of the model drives the theory to be applied, the data to be used, and the model development. Model development is made easier by model- ing tools and infrastructure and relies on federation standards to ensure the interoperability of model components. Once the model is developed it is validated by asking the question: Is the model useful for its intended purpose? As shown in Figure 11-1, the problem specification and model develop- ment process is cyclical. Based on the answers to the question “Is the model useful?” new models may need to be developed, new theory and new data (and new types of data) may be needed, and new interoperability standards, tools, and infrastructure may be required. Depending on the results, the problem itself may need to be redefined, clarified, or expanded. Figure 11-1 lays out the areas in which research and development are needed for IOS modeling and shows how they are interdependent. Fig- ure 11-2 organizes the suggested research areas into a roadmap that shows lines of activity and the interrelationships among them, repeating yearly in a cyclical fashion to advance the state of the art in meeting military IOS modeling needs. As in all use-driven research, the recommended activities start with a clear definition of the purposes to which IOS models are to be put. We recommend that the initial activity for the program (the first six months) be spent on developing a clear definition for selected representative challenge problems (the problems listed at the end of Chapter 2 can provide a starting point) in close collaboration with operational military users. Concurrent with problem definition, the first six months should be spent in developing

OCR for page 356
Year 1 Year 2 Year 3 Year 4 Problem Set 1 Problem Set 2 Problem Set 3 Problem Set 4 DEFINE CHALLENGE PROBLEMS Develop Datasets for Challenge Problems Select multiple MODEL approaches DEVELOPMENT Docking and Docking and Docking and comparison comparison comparison VALIDATION Theory Development 11-2.eps Research on Data Collection methods Development of Federation Standards Development of Infrastructure and Tools Workshops Workshops Workshops CONFERENCES landscape so that type can be a legible size Input for new Challenge Problems FIguRE 11-2 Roadmap for an IOS modeling research program.  NOTE: Only the first four years are shown.

OCR for page 356
 BEHAVIORAL MODELING AND SIMULATION datasets for these challenge problems. The challenge problems will provide common themes that tie together the diverse research and model develop- ment efforts. There is currently no single approach that is clearly dominant for IOS modeling. Our recommendation is to select and fund a number of modeling teams that take different approaches for each challenge problem. In addi- tion to the modeling teams, we recommend a series of specialized research thrusts focused on theory development, data collection methods, federation standards, and the development of infrastructure and tools. These thrusts will be aware of the challenge problems and will use the problems to focus their research, but their charter is broader and covers the entire field of IOS modeling. The modeling teams and the research thrusts will come together in a conference at month 6, to learn about the challenge problems and the data- sets associated with each problem. Conferences that involve the entire pro- gram will be scheduled yearly, with workshops for the individual research thrusts at the intervening 6-month intervals. The yearly conferences will also provide the forum for the presentation of new challenge problems, based on the results obtained in the prior year. At the end of year 1, the models that have been developed for the challenge problems will be presented and discussed at a validation work- shop, and docking and comparison activities will follow during the next 6 months, with results to be reported to the whole program at the yearly conference. These validation workshops should involve representative model users for each challenge problem. These users will assess the extent to which model results are useful for their intended purpose, as defined in the challenge problems. This process will repeat in subsequent years. As shown in Figure 11-1, the intention is that the results of the validation effort will inform all of the research thrusts as well as model development for the next cycle. Although not shown in the timeline, it is assumed that a concurrent effort will be focused on the development and maintenance of an online web-based catalog of general approaches, models, simulations, and tools, as described earlier. This will serve not only as a repository of current theories and models, but also as a common record of the results of the execution of the roadmap. The roadmap structure proposed in Figure 11-2 is intended to provide the field of IOS modeling with the common ground and forums for sharing information that will allow it to advance in a systematic way. Develop- ment and testing of models against a common set of challenge problems will avoid the current proliferation of specialized models for specialized purposes with no common framework for comparison and validation and therefore no foundation for scientific progress. Figure 11-2 shows the

OCR for page 356
 RECOMMENDATIONS FOR MILITARY-SPONSORED MODELING RESEARCH research cycle repeating over a four-year period, but we recommend that the program continue well beyond four years, with each year assessing the progress that has been made and increasing the complexity of the challenge problems based on the increasing capability of the modeling technology. New participants should be added to the funded programs and conferences each year, as new approaches and tools are developed and tested. REFERENCES Gluck, K.A., and Pew, R.W. (Eds.). (2005). Modeling human behavior with integrated cog- nitive architectures: Comparison, evaluation, and validation. Mahwah, NJ: Lawrence Erlbaum Associates. National Research Council. (1998). Modeling human and organizational behavior: Appli- cation to military simulations. R.W. Pew and A.S. Mavor (Eds.). Panel on Modeling Human Behavior and Command Decision Making: Representations for Military Simula- tions. Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. Stokes, D.E. (1997). Pasteur’s quadrant: Basic science and technological innovation. Washing- ton, DC: Brookings Institution Press.

OCR for page 356