Click for next page ( 272


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 271
8 Common Challenges in IOS Modeling T his chapter discusses broad issues and challenges that are encoun- tered across the range of individual, organizational, and societal (IOS) modeling approaches and methods, highlighting problems that need to be solved for these modeling approaches to be most useful for the military’s needs. We first describe issues of integration and inter- operability, the challenges that confront modelers and simulation develop- ers when they attempt to integrate multiple models and simulations, with the goal of making them interoperable—that is, able to use output from one model as input for another. Next we describe some of the challenges (and potential benefits) of developing and using modeling frameworks and tools that facilitate the development of IOS models. We then describe issues of model verification, validation, and accreditation (VV&A), issues that are especially challenging for the modeling of human behavior. Finally, we discuss some of the challenges posed by the data requirements of IOS models in light of the realities of the data and information available to model developers and users. In each section we note some potential solu- tions to the challenges. INTEgRATION AND INTEROPERABILITy In this section, we discuss the issues that confront modelers attempting to integrate models developed with different internal structures, at different levels of granularity, or with inconsistent inputs and outputs. The nature of the challenges requires that the discussion be quite technically sophisticated and use terminology and concepts that may be unfamiliar to many readers. 

OCR for page 271
 BEHAVIORAL MODELING AND SIMULATION We have tried to define some of the terms in footnotes, but a simplified discussion would not do justice to the subject matter. Model Interoperability: Incompatibilities and Functionality gaps1 There are several fundamental issues (and associated hard problems) that need to be addressed in undertaking the development of an inter- operable framework of IOS models. First and foremost is the problem of making existing or even new models interoperable, as these are developed independently (i.e., with no coordination) by different software design and development teams, in consultation with domain experts having vari- ous levels of skills and expertise. A very common approach is to build a wrapper around an existing model, thus converting it to an input-output (I-O) black box, or to provide an intelligent agent operating autonomously, which communicates with other models in the network. But this approach is likely to introduce other types of gaps and incompatibilities between models, some of which are identified in Table 8-1 and illustrated in Fig- ure 8-1. We discuss here the need to identify an overall methodology to fill these gaps, including various intelligent automated techniques, processes, and guidelines, as well as aid from human subject matter experts and ana- lysts whenever needed. Interface Incompatibility The first problem shown in Figure 8-1 (in the top row) concerns inter- face incompatibility between two models that either already exist or are being developed independently. If we intend to feed output from model A about a certain object X as input to model B, then some mismatch between the output and input may occur in terms of the assumptions about the numbers and types of X’s attributes. This is often straightforward but tedious to deal with, often merely involving translation from one descrip- tive framework to another (e.g., from numerical values—1, 2, 3, . . . —to “fuzzy” values—low, medium, high, . . .). A bigger problem ensues when different levels of resolution are used to represent the same object in two different models. If model A provides a high-resolution object representa- tion of X (e.g., a map, enemy force estimates) for model B, and model B needs a low-resolution representation (e.g., latitude/longitude of enemy center of gravity), then some aggregation process must be conducted, usu- 1 Much of the work described in this section on model integration and interoperability was performed by John Langton and Subrata Das at Charles River Analytics with support from the Air Force Research Laboratory, Information Directorate (AFRL/IF) under contract FA8750- 06-C-0076, and adapted from Langton and Das (2007).

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING TABLE 8-1 Gaps and Incompatibilities Between IOS Models Type Definition Interface Mismatch between the data types of different models or outputs of one model and inputs of another, e.g., real number vs. Boolean Ontological Different relationship structures, naming schemes, etc., in ontologies for different models Formalism Different logic and inferencing mechanisms and procedures for different models Subdomain Differing domains and dynamics between PMESII model dimensions, e.g., gaps economic vs. social SOURCE: Langton and Das (2007). About object of type X Interface Model A Model B Incompatibility Input Output Input Output Ontological Incompatibility Formalism Bel(X) p(X) Incompatibility Subdomain Gaps Economy Social FIguRE 8-1 Illustration of gaps and incompatibilities between IOS models. 8-1.eps SOURCE: Langton and Das (2007). ally based on one approximation method or another. The reverse process is much more difficult, going from a low-resolution output to a high- resolution input, since, in effect, missing input attributes have to be inferred or approximated and filled in. A number of approaches can be used to resolve the interface incompatibility. These are described in the section on interoperability recommendations below.

OCR for page 271
4 BEHAVIORAL MODELING AND SIMULATION Ontological Incompatibility The second problem illustrated in the figure is ontological incompatibility between models, which arises due to differing vocabularies and expressive power in their respective ontologies.2 Different teams of engineers and subject matter experts with a diverse range of expertise, knowledge, and cognitive capabilities independently creating models will inevitably develop and use dif- ferent underlying ontologies, which in turn will give rise to incompatibilities across models. Initially, one might suggest the development of a common ontology for the set of all possible models; however, many failed efforts in this direction make it clear that developing a universal ontological standard for model creation is impractical, if not theoretically impossible. Moreover, if models are to be built rapidly, analysts should ideally be free to use a model-building environment of their own choosing without assistance from knowledge engineers. The analysts should not be constrained by a predefined ontology to express their knowledge, which usually inhibits their expressive flow. Hence, rather than proposing to develop a common ontology for the model space, one approach is to focus on facilitating better mapping capa- bilities between differing ontologies. For example, there are tools that can map ontological terms from one domain to another by solving the problems of synonymy and polysemy;3 these clearly offer hope for translating differ- ing ontologies used in the models. In some cases of incompatibility between the underlying ontological structures of the models (e.g., semantic networks versus logical expressions), one domain can be mapped to another by pro- viding a more expressive ontological structure for one of the models (e.g., semantic networks can be mapped to first-order logical sentences). Therefore, some parts of the ontological incompatibility problem can be addressed via automated techniques. A number of approaches can be used to resolve the ontological incompatibility, described below. Formalism Incompatibility While ontological incompatibility creates problems due to multiple ways of designating an entity, the formalism incompatibility shown in Figure 8-1 is concerned with multiple ways of instantiating the object entity 2 An ontology, for the purposes discussed here, is “a systematic arrangement of all of the important categories of objects or concepts which exist in some field of discourse, showing the relations between them. When complete, an ontology is a categorization of all of the concepts in some field of knowledge, including the objects and all of the properties, relations, and functions needed to define the objects and specify their actions” (http://www.answers. com/ [accessed July 2007]). 3 Synonymy refers to one referent (concept) with several words that can denote it (plain English examples: big, large); polysemy refers to one word denoting multiple referents (plain English examples: break; park).

OCR for page 271
5 COMMON CHALLENGES IN IOS MODELING computationally represented in the model. For example, uncertainty can be expressed not only in terms of probability values, but also via various other formalisms, such as certainty factors, the Dempster-Shafer measure of beliefs (Shafer, 1976), and numerous other qualitative dictionaries. These are fundamentally incompatible with each other, both in terms of their underlying conceptual representation of uncertainty and probabilistic rea- soning, and in the sense of having different types of scales. Conversion between two such formalisms often requires deep understanding of the models and their formalisms, thus breaking the simple I-O black box idea of encapsulation. Specialization of formalism is often appropriate to map one approach to another. For example, probability theory is a special case of Dempster-Shafer theory that allows beliefs to be expressed only on singleton sets, facilitating development of a mapping from probability models into Dempster-Shafer models. Subdomain gaps If one wants to feed the output from a model in one domain to another, it will require an analyst or domain expert with knowledge of both domains to bridge the subdomain gaps. This is due not only to the ontological gaps between the domains being considered, but also to differ- ing dynamics between the domains. Addressing this problem requires the skills of experts from the respective domains or ideally ones who are expert in both domains. A number of approaches can be proposed to bridge such gaps, by high- lighting possible correspondences between concepts and variables across domains, described below. Recommendations are also made for more compre- hensive approaches that could be part of a long-term development effort. Figure 8-2 provides an illustration of model interoperability—focusing on political, military, economic, social, information, and infrastructure (PMESII)-related issues—with interactions among three layered models: one focusing on the social structure, one on the community infrastructure, and a third on the underlying information models, respectively from top to bottom. The infrastructure model in the middle models a stabilization and reconstruction operations (SRO) model, developed by the AFRL/IF, (Robbins, Deckro, and Wiley, 2005) using a system dynamics modeling approach (see Chapter 4), and captures a sequence of influences among variables, starting from the power supply at an electrical power substation. The generated power is fed into an industrial water plant, which produces water consumed by oil field work. An oil field produces crude oil to be refined by a refinery. Refined fuel is used to generate power, which in turn is supplied to various power substations, thus forming a loop. It is especially

OCR for page 271
High Level of Anger Level of Anger  Loss Medium Influence Diagram Among Population Among Population Low Model Fragment of Social Model in Town Aligned Refined Refined Sufficient Drinking Drinking with the USA Power Fuel Power Fuel Food Supply Water Water Drinking Power Water Refined Fuel Power Industrial Oil Oil Power Fragment of Substation Water Plant Field Refinery Generators Infrastructure Industrial Refined Crude Power Fuel Model Water High Voltage Power SRO Model Refined Fuel Terrorist Group A Leads Leader X Angers 8-2.eps Attr Attr Attr Concept Graph Fragment of Behavioral Model Information Model in Town with Terrorist Stronghold Aggressive Diplomatic Quick to Anger Attr Attr Attr Imminent Attack Causes in landscape view, smaller type is 5.73 pt Use of Threatening Calling for Inviting Suicide Observable Intelligence Phrases Jihad Bombers FIGURE 8-2 Interoperability of three different PMESII models. SOURCE: Langton and Das (2007).

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING difficult to reason with these types of graphs, containing such loops span- ning many variables, as it creates an additional burden for discounting the variables’ self-influence. The social model at the top of the figure captures the impact of these infrastructure-related variables on the society, using influence modeling technology (see Chapter 6). The model specifically captures the influence of the four variables of power, drinking water, refined fuel, and sufficient food supply on a variable representing the level of anger of the population in a town aligned with coalition forces. The dynamics of the social model are that short supply in any one of these three consumable products will increase the level of anger among the local population. In fact, if a terrorist organization became aware of the mid-layer SRO model sequence in the infrastructure, then the power substation would assume heightened impor- tance in the eyes of the terrorist strategists: an attack on a substation would not only cripple other services in the loop, but would also drive the senti- ment of the local population against the coalition. Note that the diamond box represents the expected mission utility in line with the level of anger. The utility (although difficult to quantify here) should go up when the anger level is down and vice versa. The behavioral information model at the bottom of the figure illus- trates how a model of a terrorist leader can be built using a concept graph approach (Sowa, 1984) in which concepts are represented by rectangles (e.g., [Person: Leader X] and [Behavior: Aggressive]), and conceptual rela- tions are represented by circles (e.g., has Attributes) and soft-cornered rectangles (e.g., Leads, Causes). An analyst can query such a model to determine who the terrorist leader is and the nature of the leader based on various observable intelligence. Such a leader X, who leads the terror- ist group A, can possess different types of behavior attributes, including aggressive, diplomatic, quick to anger, etc. If the leader is quick to anger and there are some stimuli to make the leader angry, then an attack on friendly targets may be imminent. One such stimulus would be coalition forces stopping the supply of oil to the region, as indicated by the link to the SRO model above. The key issue here is the interoperability among the models. Note that although an I-O connection has been made between the two variables Oil Refinery and Refined Fuel of the top two models, they are ontologi- cally incompatible as defined earlier. However, they can be made compat- ible by recognizing that the term “Oil” is synonymous with “Fuel,” and “Refined” and “Refinery” have a common base word. Another difficult compatibility problem is illustrated by the fact that there is no input for the variable Sufficient Food Supply in the social model, illustrating the inter- face incompatibility described earlier. One can envision, however, that this “sufficiency” concept could be automatically computed from the supply of

OCR for page 271
 BEHAVIORAL MODELING AND SIMULATION food previously recorded in available databases to bridge this last gap. A number of recommendations for resolving specific model incompatibilities and functionality gaps are provided below. More general approaches to resolving more than one of these gaps simultaneously are a current area of study (Langton and Das, 2007). Recommendations for Resolving gaps in Model Interoperability A number of approaches can be taken to maintain, adapt, and integrate diverse models in the context of the interoperability gaps just defined. Dealing with Interface Incompatibility Interface incompatibility generally refers to two or more models having different types of data for their inputs and outputs and thus not being able to interoperate without some form of data conversion. There are at least three types of interface incompatibilities: . I-O format incompatibilities: string versus binary, real versus integer, fixed versus floating point, numeric versus Boolean, incompatible scale, incompatible zero point, date-time format, color format. . Logical incompatibilities: number of I-O points (e.g., three out- puts versus four inputs—RGB to CMYK is a trivial example), I-O timing (e.g., fast output versus slow input). . Model persistence format incompatibilities: XML versus YAML, OWL versus RDF, etc. One way to deal with these issues is via a development interface that provides a basic set of translation functions that can learn from user interaction over time. A graphic user interface (GUI) would allow users to explicitly modify, add, and remove interface translation functions, as illustrated in Figure 8-3. Users could also specify these translation functions within an ontology or the XML schema of a model, based on specifica- tions derived, for example, from an evolved, global ontology. A full-scope GUI would then allow users to explicitly modify, add, and remove inter- face translation functions. A number of potential translation functions are described below in the context of the type of incompatibilities each addresses. Dealing with I-O Format Incompatibilities Many interface incompatibilities fall within this category, and most solutions can be resolved by some combination of the following:

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING About object of type X Model A Model B Input Output Input Output  X1  Y1   Contextual X  Y  Information 2  2 ⇒  ...  ...       Xm Yn   Y j = f ( X1i ,..., X i , C) k Develop an interface for encoding commonly used transformation functions Ex: PROJECT(X3), X1*X2+X3, min(X1, X2), gen(X3), fuzz(X3) FIguRE 8-3 Resolving interface incompatibility. 8-3.eps • Normalization: mapping any value to lie between 0 and 1 relative to its minimum and maximum possible values. • Weighting: scaling a value, typically in relation to other values. • Fuzzification: randomly generating a number to lie within some con- straining interval (e.g., some random number between 0.3 and 0.6). • Discretization: “binning” values according to their range and a range they must fall within—somewhat like rounding—sometimes taking their distribution into account (e.g., 0.5 within a range between 0 and 1 can be discretized to 1 for a range of only 0 or 1). XML schemas often exist to support model file persistence. These schemas define the elements of a model along with the possible values they can take on. XSLT can then be used along with a number of standard

OCR for page 271
0 BEHAVIORAL MODELING AND SIMULATION translation functions for integrating inputs and outputs of two models on relevant nodes or links. These functions can also be adapted according to user interaction over time. Dealing with Logical Incompatibilities In some cases, one model may have more outputs than another’s inputs or vice versa. When integrating models, we therefore need methods for addressing these situations. For an overabundance of values, we can simply use some form of aggregation. Again, a model schema or ontology can specify how this aggregation should be performed, or the user could specify this through the above-mentioned GUI. In the case in which we have only one value but must map to more than one, we can simply duplicate the value or partition it according to any context provided in the model schema or ontology. In some cases, the sample rate of inputs and outputs may differ. One way of dealing with this is through smoothing and resampling. Dealing with Model Persistence Format Incompatibilities In essence, this issue really mirrors the greater task of integrating models. The existence of a standard schema or ontology for different models would immediately resolve this issue. However, we cannot now depend on such a standard or on adherence to it. A partial solution may be to evolve or derive a standard schema or ontology. In either case, most effective solutions will entail the use of XML and XSLT for the translation of one model format to another. Dealing with Ontological Incompatibility Ontological incompatibility refers to two models having different struc- tures, including the entities they specify and the relationships between them. For instance, a rules system model may have several pairs of nodes connected by one link (precedent and consequent), whereas a Bayesian net typically has more of a tree structure. Nodes can have different names, graphs can be directed or undirected, and two models representing the same system can be at different resolutions and thus include a different number of nodes and links. The principal issue of this incompatibility is determin- ing which entities, nodes, or links in different models should map to one another for interoperation. Syntactic heuristics: The labels and descriptions of nodes and links in differing models can be compared on the basis of their raw string content. If these string components match, then the nodes or links may be a match

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING as well. For instance, “runway16” may map to “runway.” A threshold for how many characters must match to infer a string match must be specified. This type of matching can also include matching nodes/links based on the range, cardinality,4 and other attributes of their possible values. Semantic heuristics: Nodes and links from different models can be compared on the basis of the semantics of their labels, descriptions, and any other textual metadata specified in an XML file, XML schema, or ontology. Elements from different models that have a semantic similarity can then be mapped to one another for model integration. For instance, a node with the name “airport” in one model may be mapped to a node with the name “runway” in another model on the basis of the semantic similarity of their labels. Semantic similarity is determined by the relations between two words as derived from statistical usage, ontologies, thesauruses, dictionaries, etc. There are both service-oriented architectures and application program interface specifications for this purpose, including WordNet (Al-Halimi and Kazman, 1998) and Lexical Freenet (Beeferman, 1998). Relation mapping: Relation mapping can be used to address ontologi- cal incompatibility by mapping nodes from one model to nodes of another based on their relations (how they are connected) within their individual models. With this information, we can then suggest potential mappings between nodes of different models based on the similarity of their relations within their respective models. Consider the nodes α of model A and β of model B. Although these nodes may have very different names, they may have very similar relations. For example, both could influence five other nodes and be influenced by four other nodes. Based on their similarity, we may be able to deduce that these nodes can be mapped together for model integration. It is important to note that relations encompassing a node are not merely all of its incoming and outgoing links; they also include features identifying how the node affects any other nodes in the model. While this approach should rarely be used to draw links automatically, it could be used to make effective recommendations. Model node aggregation: Model aggregation can be used to address ontological incompatibility by identifying how sets of nodes in different models with differing cardinalities may be mapped to one another. It may be the case that a node α in model A maps to a subset of nodes N in model B, resulting in incompatible ontologies. For example, consider α to be the node airport and N to be the subset of nodes runway, plane, radar, and air traffic control. The question is, which nodes should airport be mapped to for model integration? We can use the semantic similarity of the labels on the nodes of N (e.g., interfacing with WordNet for ontological inference) 4 In mathematics, the cardinality of a set is a measure of the “number of elements of the set” (Wikipedia, see http://en.wikipedia.org/wiki/Cardinality [accessed Feb. 2008]).

OCR for page 271
 BEHAVIORAL MODELING AND SIMULATION tion accounts for more variance in recall and substitution errors than per- sonal attributes, such as gender, race, and age. In contrast to such well-developed conceptual frameworks, broad metaphors (brains as information-processing devices, organizations as cultures) are not really subject to verification or falsification. Whether or not they are used in a particular domain is likely to depend largely on face validity and established precedent. In evaluating the usefulness of a broad conceptual model, the yardstick is often not how well supported the model is, but how much interesting research it inspires. Even when a verbal model seems, in principle, to be subject to falsification, the underspecification of relations and processes often means that a rather broad array of different outcomes can be presented as consistent with the theory. As Harris (1976) noted in his paper entitled “The Uncertain Connection Between Verbal Theories and Research Hypotheses in Social Psychology,” theoretical terms often are not defined, boundary conditions are unspecified and, under various plausible interpretations of assumptions or conditions, several well-known theories include internal contradictions and inconsistencies (cited in Davis, 2000). validation of Cultural Models Cultural inventory models rely on ethnographic observation and are therefore both time-consuming to develop and highly subjective. Having multiple independent observers helps ameliorate the subjectivity problem, but it is expensive. Dominant trait models, such as the Hofstede dimensional models, can involve two sets of data. The first set is used to derive the dimensions. These can be validated by a number of different statistical methods, such as factor analysis. Once these are fixed, another set of data is obtained to score each new culture on the dimensions. These data have to be obtained from willing natives of the culture, and the data have to be updated over time because cultures change. validation of Cognitive Models While there is increasing emphasis on validation of cognitive architec- tures, validation remains one of the most challenging aspects of cognitive architecture research and development. “[Human behavioral representa- tion] validation is a difficult and costly process [and] most in the commu- nity would probably agree that validation is rarely, if ever done” (Campbell and Bolton, 2005, p. 365). Campbell goes on to point out that there is no general agreement on exactly what constitutes an appropriate validation of a cognitive architecture. Since cognitive architectures are developed for

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING a wide variety of reasons, there is a correspondingly wide set of validation (and evaluation) objectives and metrics and associated methods. Lack of established benchmark problems and criteria exacerbates this problem. validation of Cognitive-Affective Architectures In spite of the challenges associated with validation of emotion models and cognitive-affective architectures, progress is being made in the area. A promising trend in emotion modeling is the increasing emphasis on including evaluation and validation studies in publications. As is the case with cognitive architectures, no existing emotion models or cognitive- affective architectures have been validated across multiple contexts and a broad range of metrics. However, some important evaluation and valida- tion approaches and studies exist and are discussed in detail in Chapter 5. Cognitive-affective architecture validation has not yet reached the stage of systematic comparisons that is beginning to be used for their cognitive counterparts. However, given the recent emphasis on validation in the computational emotion research community, such studies are likely to be taking place in the near future. validation of Agent-Based Models Agent-based models (ABMs) are computational frameworks that permit the theoretical exploration of complex processes through controlled repli- cable experiments (see Chapter 6). In principle, these experiments could be run entirely with artificially generated initial conditions, parameter values, and functional forms. Nevertheless, their ultimate usefulness depends on the extent to which they prove capable of shedding light on real-world systems, that is, their ability to enhance understanding and guide decisions and actions. When validation of ABM frameworks is attempted, the validation is generally restricted to small areas of performance. A typical approach to validation is to run an experiment using an ABM framework, collect data from this experiment, statistically analyze the results to generate the response surface, and then contrast the response surface with real data. It is easy, even with only a few variables, to generate such a quantity of data from an ABM framework that there are no existing data with which to compare them, no existing statistical package can handle them, and most desktops cannot store them. Therefore, typically only small portions of the overall response surface can be estimated at once. The size of the analyzed response surface is thus often dictated by the user’s interests and the critical policy or decision-making questions at issue (i.e., the action domain and the scenarios relevant to that domain, as discussed above).

OCR for page 271
0 BEHAVIORAL MODELING AND SIMULATION ABM researchers have recently begun to explore promising new approaches to validation. For example, a number of them are now advo- cating iterative participatory modeling (IPM) as an effective way to incre- mentally achieve validation of the structural, institutional, and behavioral aspects of the complex systems they study. For an introductory exposition of IPM, see Barreteau (2003). The essential idea is to have multidisciplinary researchers join with stakeholders in a repeated looping through a four- stage modeling process: (1) field study and data analysis, (2) scenario discussion and role-playing games, (3) ABM development and implementa- tion, and (4) intensive computational experiments. The new aspect of IPM relative to more traditional participatory model- ing approaches is the emphasis on modeling as an open-ended collaborative learning process. The modeling objective is to help stakeholders manage complex problems over time through a continuous learning process rather than to attempt the delivery of a definitive problem solution. In addition, ABM researchers are also beginning to explore the poten- tial benefits of conducting parallel experiments with real and computational agents for achieving improved validation of their behavioral assumptions.11 A critical concern is how to attain sufficiently parallel experimental designs so that information drawn from one design can usefully inform the other. Recommendations for Developing and validating IOS Models We have argued that IOS models should be validated beginning with the purpose and then considering the action set, scenarios, and if-then relations in the specific situation. The committee makes a number of sug- gestions for modeling and simulations that will facilitate the validation of a specific model. Check with Multiple Experts Four different experts should examine an IOS model: the users of the model, the scenario experts, the if-then or domain experts, and the modelers themselves. Modelers cannot examine a model by themselves; they tend to focus on the verification with less emphasis on the purpose of the model. For an action model, the user is very important to check the relevance and feasibility of the action set. The scenario expert should examine the uncer- tainties and unknowns. Domain experts are particularly knowledgeable about the if-then relationships. However, their knowledge is not necessarily 11 See http://www.econ.iastate.edu/tesfatsi/aexper.htm for annotated pointers to ABM research on parallel experiments with real and computational agents; see also the survey by Duffy (2006).

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING framed in this manner, so some adjustment may be required. For example, domain experts know about “what is” and “what has been” but may be less certain about “what might be” outcomes. However, they are likely to point out errors in the models for what might be and limits of what is known. Each expert can contribute to the validation of an action model. It is unlikely that any single expert can ensure a valid action model alone. The structure and content of the model provide a template for a procedure by which multiple experts can validate different aspects of an integrated action model. Keep the Model as Simple as Possible for Its Purpose An IOS model does not have to be complex. Parsimonious models are preferred. The corn farmer action model is simple and does not capture the complexity of weather forecasting or the chemistry of fertilizers. But it is understandable and permits the farmer to make a decision and take action. Action models that are intuitively understandable to decision makers (trans- parent) are preferred. An action model that is disconnected from a deci- sion maker’s intuition and from concepts he or she is familiar with does not permit interplay between the decision maker and the model. In short, complicated, nonintuitive action models require decision makers to accept the implications of the models on blind faith. Action models should aid decision makers, not replace them. Examine “What Might Be” as Well as “What Is” “What is” should mimic the real world within limits. “What is” models are a basis for “what might be.” A model that has little or no correspon- dence with the real world is not likely to be relevant for what might happen. What might be is very important for action models—particularly in new situations (Burton, 2003). Many of the relevant action-scenario combina- tions have not been observed in the past. So the model must be relevant for action beyond what is or what has been to new situations. For example, it would be desirable if the illustrative village deployment action model could be used reliably in other similar situations, say for the withdrawal from a village as well as entry. But it is not likely that the model could be used to help plan an action to disarm a resistance cell. Presumably this would require a more detailed model of the functioning of the cell. Whether it would be desirable to develop one model to handle both entry and cell disarmament or two separate models would presumably depend on economies of scope—Is there anything to be gained by considering both issues jointly?—and on computational implementation costs. IOS models should be developed and examined beyond what is to what might be. At

OCR for page 271
 BEHAVIORAL MODELING AND SIMULATION the same time, it is important to examine the limits of the model and not use it in situations in which it might be inappropriate. As suggested above, simplicity is desirable, but it must be balanced so that the action model is useful for its purpose. IOS models are likely to forecast a range of possible outcomes, some more likely than others, and to incorporate many factors that are highly uncertain and, indeed, unknowable at the time the model is developed. How then can such models be validated? Popper, Lempert, and Bankes (2005) argue that models used to explore policy alternatives for an uncer- tain future should not be expected to yield predictions that can be tested but rather should be used to explore and compare possible outcomes under a variety of possibilities in order to select strategies that are robust—yielding the best overall results across a variety of possible futures. Postevent outcomes can also be used to evaluate models, although models are not necessarily incorrect if the actual outcome that occurred was not the one forecast to be most likely. Unlikely events do occur, and many IOS applications do not permit the replication that would generate a distribution of actual outcomes. A very useful approach would be to develop multiple models that take different perspectives and use different theories and data, merge their predictions to create zones of likelihood, and compare their forecasts with the actual outcomes (see Docking below). As with other validation approaches, the value of the model’s results depends on its intended use, so the degree to which forecasts need to correspond to reality will depend on the model’s purpose. use Model Touching for validation Model touching is comparison or juxtaposition of models. There are many ways to bring models together. Here is a list: • Bring experts (as described above) together to develop and examine the model. • Compare the action model with qualitative studies for the situation or domain. • Check with other studies that might be empirically based on data from the field or from experiments. • Compare with computational models that are based on field data. Docking. Docking is the bringing together of two models—a metaphor borrowed from space exploration. More precisely, docking is an evaluation of the extent to which two or more different models of the same action situation can be cross-calibrated so that they yield the same outcome (or outcome probability distribution) given the same contingency condition

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING (Axtell, Axelrod, Epstein, and Cohen, 1996). Docking goes beyond model touching to compare in more detail. It can provide a better understanding of the true connections relating the three key elements of an action model: the actions, the scenarios, and the possible outcomes resulting under each contingency condition (scenario-action pair). Docking gives confirmation that we have a reasonable understanding of an action situation, and that our conclusions are being driven by the intrinsic nature of the action situ- ation and not by idiosyncratic aspects of the model implementation. One possible approach is to compare how different models perform under the same benchmark action-scenario combination, which can provide insight into how different models define actions and how they structure if-then relationships. That is, for an action model, take the same action possibili- ties and the same unknown scenarios, then develop two separate if-then relationship models. Develop and compare the outcome tables for the two models. Are the outcomes the same? If not, why? One must go behind the model outcomes and examine the details of the models to understand their differences. Individuals who are expert in the subject are critical in judging the models and their value. Docking should involve experts throughout the process, as discussed above. Docking of multiple modeling approaches against common benchmark problems using a panel of expert judges has recently been used to provide considerable insight into individual cognitive performance models (Gluck and Pew, 2005). At this time, there is a need to develop benchmark scenario-action situ- ations that can be used to dock two or more models. This effort will involve action, scenario, and if-then experts. With these benchmarks, docking studies can add greatly to the development of action models. Given the current state of the art, the participation of experts in the docking process is essential. The next best step in validation is to support docking studies among experts who develop computation-based models. Automated machine docking of two or more models is a very high-risk endeavor at present. At a later stage of understanding, we may be able to develop a computationally based approach to the docking of models. But for now, experts and their judgment are mandatory. Triangulation. Triangulation goes beyond docking and involves examining the same action domain using an action model, an expert group using a qualitative approach, and reference to quantitative studies in the domain. An action model validated using multiple approaches is more likely to help the decision maker take actions that meet the purpose. However, a large number of triangulations are often possible. We do not know a priori what the best triangulation is for a given situation, but it is quite likely that a good triangulation will be situation dependent.

OCR for page 271
4 BEHAVIORAL MODELING AND SIMULATION Exploratory testing of robustness. Miller (1998) proposes active nonlinear tests for complex models to validate the model’s structure and robustness. In this approach, automatic nonlinear search algorithms probe for extreme outcomes that could occur within the set of reasonable model perturba- tions. This multivariate sensitivity technique can find places where a com- plex model “breaks,” that is, produces results that are outside a range of reasonable predictions. In summary, universal rules about what is the appropriate procedure for validating IOS models are not possible. However, we recommend the validation of models through a three-part triangulation process, based on the purpose of the model. Validation should involve (1) participation by multiple experts who can provide different perspectives on the action domain, the scenarios, and the if-then rules incorporated in the model; (2) docking of similar computational models against one another; (3) com- parison to qualitative and theoretical studies and previous quantitative results and exploratory testing for a range of outcomes. A good heuristic would be to begin with the experts as discussed above and move as quickly as possible to docking studies and exploratory testing. DATA ISSuES AND CHALLENgES Data can be used in two different ways in modeling. When models are developed inductively from data, the quality of the data is extremely important. In that case the data are broader in scope and limited only in a very general manner. For example, an anthropologist sees different things than an engineer in the same situation. For existing models, the data are prescribed by the model, and the quality of data is extremely important. Here again, the data yield values for the model parameters and make the model specific to a given situation and problem. The data requirements are driven by different modeling needs. For each situation, quality data are needed and are important to the usefulness of the model. This means that even the most promising, sophisticated, and elegant models may be severely limited or hampered by specific data needs and requirements. Thus, data issues are an essential component for assessing the ultimate success for model development, validation, and applications. A number of potential data factors need to be considered in the course of conceptualizing and developing models. These include but are not limited to the following. • Primary/secondary: Data may already exist (secondary) or may need to be collected (primary). Obviously, models using secondary source data have some advantages because they require little or no data collection. However, models using such forms of data may

OCR for page 271
5 COMMON CHALLENGES IN IOS MODELING be limited by the nature and quality of the data that exist. This might mean the model will be constrained by the type of data avail- able, and such constraints may limit the model’s ability to address important issues and problems. Models using primary sources of data have more flexibility, given that they can determine exactly what type of data needs to be collected. However, primary data collection involves its own set of limitations that are reflected in the factors described below. • Observable/nonobservable: Some data are directly observable, and this may facilitate ease of collection. Phenomena that are not directly observable may require more extensive efforts to uncover the necessary information (e.g., face-to-face interviews). • Distant/close: Some forms of data can be collected at a distance. This may involve the use of technology, such as cell phones or video links. However, other types of data require actually being there on the ground, as for face-to-face contact or interviews with subjects, respondents, or informants. • Representative/nonrepresentative: Often model assumptions require data to be collected or compiled in some specific manner. The best example of this is the explicit assumptions underlying classical parametric statistical models that require random samples from a population. There are other models that simply require units of analysis to be representative of a given theoretically important category of some type, and it may be the case that any unit of analysis fitting the categorical criteria will suffice. An important consideration is the extent to which units of analysis used in the model need to be derived by either probabilistic or nonprobabilistic methods (see Johnson, 1990). • Passive/active: This is related to some of the factors above in that some data can be collected casually or on the fly. Such data may still require being there but may require only documenting or record- ing naturally occurring events, conversations, or interactions. In contrast, more direct and active methods of data collection may be necessary and will involve, for example, actually interviewing individuals at events or interviewing them about given conversa- tions or interactions. • Tacit/explicit: Some forms of data require little interpretation or reading between the lines. Other types of data are implicit, and there is a need to make them more explicit. This is particularly true for some forms of human knowledge that are often tacit and may require specific types of elicitation interviewing techniques to extract the requisite information to be used in the model (Johnson and Weller, 2002).

OCR for page 271
 BEHAVIORAL MODELING AND SIMULATION There are certainly other important factors to be considered in terms of relating models to various data requirements. However, the factors described above potentially reflect impediments to the utility and validity of any proposed model. If, for example, models require data involving forms that are tacit, active, representative, close, nonobservable, and, of course, primary, then the data may be costly to obtain and may limit the models’ potential effectiveness given the data constraints. But this does not address in any way issues of data quality concerning reliability and validity. We can consider the factors above to reflect elements of how hard data might be to collect or obtain. Although some of these factors are related to issues of reliability and validity, they are not necessarily one and the same. Often the data that are the most difficult to collect (i.e., on-the-ground face-to-face interviews) are the data that have the most reliability and validity, whereas data that are the easiest to obtain (i.e., secondary source data) may be the most problematic. The extent to which one trusts the data will ultimately determine the extent to which one trusts model outcomes or predictions. In summary, even though quality data are extremely important, the operationalization of quality is different for the different demands of the model. One implication is that we need better quality data. Another impli- cation is that we need a better understanding of how we can model, describe, predict, and explain with less than quality data. This further sug- gests that a better notion is needed of what is meant by quality data for the various models and needs. REFERENCES AFRL/HE. (2002). Proceedings of the Eleventh Conference on Computer Generated Forces and Behavioral Representation, May, Orlando, FL. Al-Halimi, R., and Kazman, R. (1998). Temporal indexing through lexical chainin. In C. Fellbaum (Ed.), WordNet: An electronic lexical database (pp. 333–351). Cambridge, MA: MIT Press. Allen, J.G. (2004). Commander’s automated decision support tools. Briefing to Proposers’ Symposium for DARPA’s Integrated Battle Command Program, Dec. 15, Washington, DC. Axtell, R., Axelrod, R., Epstein, J.M., and Cohen, M.D. (1996). Aligning simulation models: A case study and results. Computational and Mathematical Organization Theory, (2), 123–141. Bachman, J.A., and Harper, K.A. (2007). Toolkit for building hybrid, multi-resolution PMESII models. (Final report #RI-RS-TR-2007-238.) Cambridge, MA: Charles River Analytics. Available: http://stinet.dtic.mil/cgi-bin/GetTRDoc?AD=ADA475418&Location=U2&do c=GetTRDoc.pdf [accessed Feb. 2008]. Barreteau, O. (2003). Our companion modelling approach. Journal of Artificial Societies and Social Simulation, (2). Available: http://jasss.soc.surrey.ac.uk/6/2/1.html [accessed April 2008].

OCR for page 271
 COMMON CHALLENGES IN IOS MODELING Beeferman, D. (1998). Lexical discovery with an enriched semantic network. In Proceedings of the Workshop on Applications of WordNet in Natural Language Processing Systems, Association for Computational Linguistics/Conference on Computational Linguistics (ACL/COLING 1998), Montreal, Canada. Available: http://www.lexfn.com/doc/lexfn. pdf [accessed Feb. 2008]. Box, G.E.P. (1979). Robustness in the strategy of scientific model building. In R.L. Launer and G.N. Wilkinson (Eds.), Robustness in statistics: Proceedings of a workshop. New York: Academic Press. Brinkley, J. (2003, May 14). AFRL/HE instruction #-0. Wright Patterson Air Force Base, OH: Air Force Research Laboratory/Human Effectiveness Directorate. Burton, R.M. (2003). Computational laboratories for organization science: Questions, validity and docking. Computational and Mathematical Organization Theory, (2), 91–108. Burton, R.M., and Obel, B. (1995). The validity of computational models in organization science: From model realism to purpose of the model. Computational and Mathematical Organization Theory, (1), 57–71. Campbell, G.E., and Bolton, A.E. (2005). HBR validation: Integrating lessons learned from multiple academic disciplines, applied communities, and the AMBR project. In K.A. Gluck and R.W. Pew (Eds.), Modeling human behavior with integrated cognitive archi- tectures: Comparison, evaluation, and validation (pp. 365–395). Mahwah, NJ: Lawrence Erlbaum Associates. Carley, K.M., and Svoboda, D. (1996). Modeling organizational adaptation as a simulated annealing process. Sociological Methods and Research, 5(1), 138–168. Davis, J.H. (2000). Simulations on the cheap: The Latané approach. In D.R. Ilgen and C.L. Hulin (Eds.), Computational modeling of behavior in organizations: The third scientific discipline (pp. 217–220). Washington, DC: American Psychological Association. Duffy, J. (2006). Agent-based models and human subject experiments. In L. Tesfatsion and K.L. Judd (Eds.), Handbook of computational economics, volume : Agent-based com- putational economics. Amsterdam, The Netherlands: Holland/Elsevier. Epstein, J.M. (2006). Remarks on the foundations of agent-based generative social science. In L. Tesfatsion and K.L. Judd (Eds.), Handbook of computational economics, volume : Agent-based computational economics. Amsterdam, The Netherlands: Holland/Elsevier. Fagiolo, G., Windrum, P., and Moneta, A. (2006). Empirical validation of agent-based models: A critical survey. Journal of Artificial Societies and Social Simulation, 0(2). Available: http://jasss.soc.surrey.ac.uk/10/2/8.html [accessed April 2008]. Fiske, A.P. (1992). The four elementary forms of sociality: Framework for a unified theory of social relations. Psychological Review, (4), 689–723. Gluck, K.A., and Pew, R.W. (Eds.). (2005). Modeling human behavior with integrated cog- nitive architectures: Comparison, evaluation, and validation. Mahwah, NJ: Lawrence Erlbaum Associates. Haefner, J.W. (2005). Modeling biological systems: Principles and applications. New York: Springer. Harper, K.A., Pfautz, J.D., Ling, C., Tenenbaum, S., and Koelle, D. (2007). Human and system modeling and analysis toolkit (HASMAT). (Final report #R06651.) Cambridge, MA: Charles River Analytics. Harris, R.J. (1976). The uncertain connection between verbal theories and research hypotheses in social psychology. Journal of Experimental Social Psychology, (2), 210–219. ITT Research Institute. (2001). Modeling and Simulation Information Analysis Center (MSIAC). Verification, validation, and accreditation (VV&A) automated support tools: A state of the art report, Part -Overview. Chicago, IL: Author. Johnson, J.C. (1990). Selecting ethnographic informants. Thousand Oaks, CA: Sage.

OCR for page 271
 BEHAVIORAL MODELING AND SIMULATION Johnson, J.C., and Weller, S. (2002). Elicitation techniques in interviewing. In J. Gubrium and J. Holstein (Eds.), Handbook of interview research (pp. 491–514). Thousand Oaks, CA: Sage. Judd, K.L. (2006). Computationally intensive analyses in economics. In L. Tesfatsion and K.L. Judd (Eds.), Handbook of computational economics, volume : Agent-based computa- tional economics. Amsterdam, The Netherlands: Holland/Elsevier. Kaelbling, L.P., Littman, M.L., and Moore, A.W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285. Langton, J.T., and Das, S.K. (2007). Final technical report for contract #FA50-0-C-00. Air Force Research Laboratory, May 16, Rome Research Site, Rome, NY. Marks, R.E. (2006). Validation and complexity. Working paper, Austrailian Graduate School of Management, University of New South Wales, Sydney. Miller, J.H. (1998). Active nonlinear tests (ANTs) of complex simulation models. Management Science, 44(6), 820–830. National Research Council. (1998). Modeling human and organizational behavior: Appli- cation to military simulations. R.W. Pew and A.S. Mavor (Eds.). Panel on Modeling Human Behavior and Command Decision Making: Representations for Military Simula- tions. Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. Popper, S.W., Lempert, R.J., and Bankes, S.C. (2005). Shaping the future. Scientific Ameri- can Digital, April. Available: http://www.sciamdigital.com/index.cfm?fa=Products. ViewIssue&ISSUEID_CHAR=8E3EADFC-2B35-221B-6A8455DA45AE8B50 [accessed Feb. 2008]. Robbins, M., Deckro, R.F., and Wiley, V.D. (2005). Stabilization and reconstruction opera- tions model (SROM). Presented at the Center for Multisource Information Fusion Fourth Workshop on Critical Issues in Information Fusion: The Role of Higher Level Informa- tion Fusion Systems Across the Services, University of Buffalo. Available: http://www. infofusion.buffalo.edu/ [accessed Feb. 2008]. Rouse, W.B., and Boff, K.R. (2005). Organizational simulation. Hoboken, NJ: John Wiley & Sons. Sageman, M. (2004). Understanding terror networks [web page, Foreign Policy Research Institute]. Available: http://www.fpri.org/enotes/20041101.middleeast.sageman. understandingterrornetworks.html [accessed August 2007]. Sageman, M. (2006). The psychology of Al Qaeda terrorists: The evolution of the Global Salafi Jihad. In C.H. Kennedy and E.A. Zillmer (Eds.), Military psychology: Clinical and operational applications (Chapter 13, pp. 281–294). New York: Guilford Press. Schelling, T.C. (1971). Dynamic models of segregation. Journal of Mathematical Sociology, , 143–186. Shafer, G. (1976). A mathematical theory of evidence. Princeton, NJ: Princeton University Press. Sowa, J.F. (1984). Conceptual structures: Information processing in mind and machine. Reading, MA: Addison-Wesley. Telvick, M. (Producer). (2005, January). Al Qaeda today: The new face of the global Jihad. Arlington, VA: PBS Frontline. Telvick, M. (2007). Al Qaeda today: The new face of the global jihad [web page, PBS Front- line]. Available: http://www.pbs.org/wgbh/pages/frontline/shows/front/etc/today.html [accessed August 2007]. U.S. Department of Defense. (1995). Modeling and simulation (M&S) master plan. (DoD #5000.59-P.) Washington, DC: Office of the Undersecretary of Defense for Acquisition and Technology.