Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Part III ADDRESSING UNMET MODELING NEEDS 337
10 Pitfalls, Lessons Learned, and Future Needs C hapters 3 though 9 summarized the state of the extensive work under way to develop a variety of individual, organizational, and societal (IOS) human behavioral models and how that work both contributes and falls short in solving representative military problems. In this chapter we take a step back from this detail and summarize the find- ings of the committee in the form of lessons learned and future needs if IOS models are to live up to their potential for delivering useful results. Given that most IOS models are in early phases of development, a clear set of best practices has not yet emerged. We can, however, identify some lessons to be learned from the initial approaches that have been taken and some of the pitfalls that have occurred on the road toward developing effective IOS models. Awareness of these pitfalls should help those develop- ing models to avoid wasting valuable time and effort relearning the same lessons. Avoiding these pitfalls will therefore improve the probability of success for new initiatives. When particular programs or efforts are mentioned as examples, this is not meant to suggest that those involved made choices that were known to be wrong at the time; in fact, many of the authors of this report have fallen into one or more of these pitfalls in our own modeling efforts. Hindsight often brings clarity, revealing that choices that seemed reasonable at the time have had undesirable results. Our goal is to support the field in gain- ing maximum benefit from the âtuition paidâ thus far in intellectual effort, hard work, and taxpayer money. For each pitfall we summarize the lessons learned, and on the basis of those lessons we identify the key needs to be 339
340 BEHAVIORAL MODELING AND SIMULATION met in order to move forward. Chapter 11 then presents our recommended plan to meet those needs. Pitfalls in Matching the Model to the Real World The following problems are created either by inattention to the real world being modeled or by unrealistic expectations about how much of the world can be modeled and how close a match between model and world is feasible. Model-Problem Mismatch Modelers should choose variables based on what theory and experience suggest will be most useful in characterizing the problem of interest. Poor characterization of the specific domain of problems to be addressed, failure to attend appropriately to the dictates of the problem domain, or failure to consult theory to assess which IOS variables are most likely to matter can lead to serious model-problem mismatches. Practical considerations of availability, for example, can lead modelers to select âoff the rackâ com- ponents just because they are available, even if they are inappropriate to the problem at hand. For example, using the Hofstede dimensions (see Cultural Models in Chapter 3) as generic representations of culture is inappropriate unless there are good theoretical reasons to believe that the specific dimensions chosen are relevant for behaviors that are important to the application. When modeling adversarial reasoning, specific cultural variations in infer- ence and dialectical reasoning may well be informative, while Hofstede dimensions such as masculinity-femininity are generally irrelevant. Another source of poor choices is the pull of familiarity. Those versed in game theory are often attracted to representations of culture as a dis- tribution of strategies for playing stylized games, such as the prisonersâ dilemma (see Game Theory in Chapter 5). For some applications, such as predicting the adversarial responses of enemy organizations to courses of action, game theory approaches to culture may be a good match. For other applications, such as predicting broader societal reactions, game theory approaches to cultures can lead to such problems as characterizing cultures as Ânecessarily moving toward or existing in an equilibrium state or assuming that conflict involves two parties. In reality, cultures are rarely in equilibrium; changes in technology, resources, and migration all impact cul- ture; and the Âimplications of conflict for a particular society rarely involve just two parties. This particular model-problem mismatch has resulted in misleading policy advice and a tendency to overlook major shifts in culture, resulting in policy makers and commanders being surprised by emergent
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 341 factors in the situations they face (see also Illusions of Permanence in this chapter). Another example of this pitfall comes from network research. Over the years, network researchers have used the same network properties (e.g., betweenness, centrality) as independent variables in a wide variety of dif- ferent contexts (see Network Models in Chapter 6). Yet there is little reason to believe that these properties are relevant for all applications for which network analysis may be informative. Even when correlations are found, it is difficult to design effective interventions and changes to a system because the variables are not necessarily explanatory. The key network properties relevant to prediction and control of any particular problem domain should generally be derived from the problem itself (Borgatti and Everett, 2006). Lessons Learned and Future Needs: The modeler should tie model choices to the application, which assumes that the application domain and the class of problems to be addressed are clearly specified. Subject matter experts in the application domain should lead or be represented on model- ing teams, or at the very least they should be extensively consulted regard- ing the appropriate choice of variables and assumptions for a particular problem domain. In general, a tighter connection is needed between model developers and the operational personnel who will use the models being developed. Shared understanding between developers and users should result in a clear specification of model purpose. Better communicationâincluding sharing of both theory and dataâis also needed across the many disciplines that may contribute to model specification. Based on the purpose of the model and the application domain, more collaborative cross-disciplinary efforts in an integrated community of interest are needed to ensure that model developers do not simply rely on the set of variables with which they are most familiar. All-Purpose Models That Ultimately Serve No Purpose Universal scope or âSwiss army knifeâ models attempt to solve, via large-scale software development, an entire set of wide-ranging concerns. In most cases, attempting to build universal scope models for the Depart- ment of Defense (DoD) has led to failure, to the loss of years of modeling and simulation effort, and to the expenditure of large amounts of money. The classic universal scope model, JSIMS (2 to 5 million lines of code and 6 years to develop), had the following scope (Bennington, 1995, p. 805): The mission of JSIMS is to develop a Joint Simulation System that will provide readily available, operationally valid synthetic environments for use by the CINCs, their components, other joint organizations and the
342 BEHAVIORAL MODELING AND SIMULATION Services. JSIMS has five major objectives: integrate the range of missions of the Armed Forces within a common modeling and simulation (M&S) framework that includes live, virtual, and constructive M&S capabili- ties: provide a training environment which will also accommodate space, transportation and intelligence requirements: establish a common simula- tion support structure which enables harmonious sharing of simulation resources, processes, and results among users; enable simulation users to readily create or access a simulation environment which supports their requirements: and enable joint simulation users to interact freely with elements of their command structure, supporting/supported organizations and other simulation centers or users. While the initial focus of JSIMS is joint planning and training activities, as the system matures, JSIMS will be available to the DoD community at large for the analysis of doctrine, organization, system and material alternatives. With such a large and broad scope, the likelihood for successful imple- mentation was, in retrospect, near zero. JSIMS ran from December 1995 to December 2002, at a development cost of $1.8 billion. In the end, DoD decided to fall back to smaller scale models and attempt to make those models interoperable (Office of the Secretary of Defense, 2004). Interoperability concerns in DoD also fell prey to the universal scope model syndrome with the high-level architecture (U.S. Department of Defense, 1996). In 1996, DoD decided that a big bang, Swiss army knife solution to the interoperability of models and simulations was the way for- ward for defense models and simulations. Instead of building an architec- ture that was dynamically extensible and semantically interoperable, DoD built a monolithic, black box piece of software that required everything to be defined ahead of time statically. The consequence is that, to make modeling and simulation (M&S) systems interoperable, the source codes of the systems must be modified and their definition files updated. For any subsequent M&S system to be integrated, the source code for all systems must be modified along with their definitional files. In retrospect, it is temptingly easy to build static interoperability solu- tions if most of the information transferred is physics-based. As one moves into the realm of modeling human and organizational behavior and begins to include cultural, network, emotional, cognitive, and psychological mod- els, one needs to build models as encapsulated smaller model components that can be dynamically linked together rather than trying to create one large source code component (Pratt and Henninger, 2002). â ynamically extensible means that the structure of the model allows new components to D be added without rewriting the source code. Semantically interoperable means that the lan- guage of the two models permits them to be put together in a meaningful and theoretically consistent way.
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 343 Lessons Learned and Future Needs: Monolithic, static approaches are inappropriate to IOS modeling. Flexible, adaptable components and semantically interoperable models will potentially do much to avoid this pitfall. In order for this to happen, advances are needed in federated model standards and architectures in order to allow different types of IOS models, at different levels of detail, to interoperate in meaningful ways. Verification, Validation, and Accreditation The DoD M&S community has always lived with the specter of veri- fication, validation, and accreditation (VV&A). We say specter because sometimes VV&A of a model is used to shut down further discussion and consideration of it, particularly if it has not yet gone through a VV&A process. VV&A is an important issue in IOS modeling, as in other types of modeling (see Burton, 2003, and Chapter 8 for an extended discussion of VV&A issues). With respect to the modeling of human and organizational behavior, however, rigorous VV&A (as it has been defined for validation of models of physical systems) is difficult if not impossible to fully achieve. VV&A for a model means the developers have verified that the model implements processes as intended, they have validated the model against empirical data, and they have accredited the model for use for particular cir- cumstances, usually for a particular service requirement. Early M&S efforts usually modeled physical properties exclusively, so verification consisted of being able to look at the source code and say, âyes, the source appears to implement the mathematics of the physical model.â For IOS models, there is no easy path to verification. One can look at the source code but cannot say âyes, the source appears to implement the matheÂmatics of the human or organizational modelâ because the techniques typically used for such models are code-based and not closed-form mathematics. Historically, models of physical phenomena have been validated by comparing the results of running the models with observations from the real world. If one builds a model of a tank being hit by a particular weapon, one can go out into the field and shoot that particular weapon at a tank and say âyes, the model is close to the results of the real worldâ and stamp the model as validated. For IOS models, it is typically not possible to validate the model against the real world in this way. For example, suppose one builds a network model of insurgency formation. One cannot then take real-world inputs into such a model so as to predict precisely what will happen next, as in the film âMinority Report.â At best, it is possible to run the model against historical data and see how well the model accounts for the observed events. Perhaps, at the end of such a validation, it will be possible to state that the model provides a possibility space or set of potential outcomes that are useful to consider in the analysis of the next course of action. Valid IOS
344 BEHAVIORAL MODELING AND SIMULATION models do not predict exactly what will happen in the future (see also Illu- sions of Permanence in this chapter) but rather provide a set of potential outcomes to consider. So when writing up the model usage document for such a model, it is feasible to state something like âthis model is useful for analyzing situations that have the following characteristics and will provide outputs that allow you to consider the set of things that may happen within the following limits.â Of course, this assumes valid inputs: data that are reasonably accurate, acceptably complete, and that match the requirements of the model (see Chapter 8 for a more detailed discussion of data issues). Accreditation is the final step in the VV&A process. Basically accredita- tion means that a sponsoring organization is willing to bless the model for a particular use, which generally occurs after someone has verified the model to an acceptable degree and some validation has been performed. Model accreditation is usually specific to a service office or the Office of the Sec- retary of Defense, and accreditation means that the office has determined that the model is sufficiently robust for some operational deployment. Many physics-based models have been accredited but, as far as the com- mittee knows, no models of human and organizational behavior have been accredited. Such models are perhaps too new to have yet made it through the accreditation paperwork process. However, we think that accreditation decisions for IOS models should be based on a better understanding and explication of the limitations and usages for such models, as well as a set of VV&A requirements that are appropriately tailored to the special nature of such models. Lessons Learned and Future Needs: Failure to appreciate the extent to which IOS models differ from physical models has led to inappropriate expectations regarding VV&A for IOS models. Rather than trying to apply an inappropriate VV&A process, an IOS model needs to be deployed with a strong set of guidelines that describe the limitations of the model and that remind users that the purpose of the model is not definite point predictions but rather indications regarding what possible outcomes will likely result from any particular course of action. Better standards are needed for IOS models, including appropriate VV&A guidelines. This report recommends an action validation approach (see Chapter 8) that requires a clear specification of the purpose of the model and validates the usefulness of the answers provided by the model against that purpose. We also recommend triangulation, in which mod- els are reviewed by multiple types of experts, compared with qualitative and theoretical studies as well as quantitative results, and similar models are compared with each other (docking). Appropriate IOS model valida- tion approaches need to be further developed and promulgated among model developers, users, and funding agencies through a widespread multiÂ disciplinary community of interest.
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 345 Problems in Designing the Internal Structure of a Model The following tactical design pitfalls are sometimes generated by unwarranted assumptions about the nature of the social, organizational, cultural, and individual behavior domain and sometimes by a failure to deliberately and thoughtfully match the scope of the model to the scope of the phenomena to be modeled. These pitfalls reveal the challenge of making wise choices of simplifying assumptions about the highly complex domain of IOS structure and behavior. Pitfall of Unvalidated Universal Laws Modelers who are accustomed to dealing with physical objects that behave according to well-known physical laws are especially prone to this pitfall. Comparable universal laws of human behavior and social structures have yet to be discovered, codified, and supported by empirical data. Even should they be discovered, it is unlikely that they could be rep- resented as closed-form equations. Furthermore, human behavior involves freedom of choice, and the results of the model themselves, if widely publicized, might affect those very behaviors that they were intended to forecast. Modelers fall into this pit when they model particular structures or processes in fixed form because they mistakenly believe that these struc- tures are universal. As an example, some modelers have subscribed to the notion that all evolved networks are scale-free (i.e., they have a degree distribution that is well described by a power law). However, because the behavioral capabilities of nodes in a network make a demonstrable differ- ence (people networks are different from gene networks), the data do not support the assumption of abstract commonalities across all networks. While the assumption of a scale-free network may well be warranted for particular types of networks and nodes, building this assumption in as a fixed feature of the model will limit its application in ways that may not be recognized by end users. Instead, the network structure should be treated as a model parameter. Lessons Learned and Future Needs: Beware of assumptions that any particular structure or process is universal in any IOS domain. Consult with subject matter experts to be sure empirical data provide very strong sup- port for any such claims before relying on them in designing a model. Set up the model so that users are explicitly reminded that they are making an assumption when they select a particular structure or process to represent a domain. A better integrated multidisciplinary community of interest in IOS modeling, with greater availability of empirical data and more extensive docking of alternative models around common applications, could protect
346 BEHAVIORAL MODELING AND SIMULATION against the unthinking and mistaken assumption that universal laws from other domains apply to IOS models. One-Dimensional Models Modelers should beware of inappropriately limiting themselves to a single independent variable and using it to account for an array of differ- ent processes and outcomes. For example, there is a tendency in network research to focus exclusively on structure as represented by a few network variables, while completely ignoring other information that is available about the nodes, the processes that are going on, and other contextual factors. Such models ignore possible influences, for example, the possibil- ity that the behavior of the nodes not only is influenced by the network structure, but also can alter that structure. Modelers may encounter this pitfall when operating under the sway of a strong structuralist position that views (network) structure as far more important than other variables, like culture or psychology. As a result, we often see standalone network models that do not incorporate cognitive, cultural, or other processes. Another example of a heavily structuralist approach that ignores process would be a model that uses Hofstedeâs Big Five personality structure (see Cultural Models in Chapter 3) as the sole predictor for a broad array of cognitive and behavioral outputs. We think the relative importance of structure should instead be treated as an empirical question, which can of course be investigated only in models that include more than one input variable. In any modeling enterprise, simplifying assumptions are necessary, and parsimony is an important scientific principle. However, the emphasis should be on parsimony for a purposeâfor example, to conduct a focused investigation of whether a particular input variable is plausibly related to an array of different outputs. The decision to exclude other candidate input variables should be based on careful deliberation rather than on unexam- ined assumptions. Lessons Learned and Future Needs: Focus is good; myopia is unwise. Better methods are needed to decide which variables are relevant for inclu- sion in a model. The specification of the variables to be included in a model should be based on a clear specification of the purpose of the model and, depending on that purpose, should take into consideration the judgment of multiple subject matter experts, theories drawn from multiple disciplines, empirical data if they exist, and prior work on similar problems. Com- parative studies are needed that address the same problem from multiple perspectives to determine which set of variables offers the most useful results.
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 347 Kitchen Sink Models IOS modelers who appreciate the complex nature of human and organi- zational behavior and who wish to avoid the pitfall just described may back themselves into a different pit by adding variables to a model in a hodge- podge fashion. Modelers may be especially vulnerable to this pitfall if they are operating outside their area of expertise (for example, people with no training in anthropology or cultural psychology attempting to model cul- ture), or not relying on strong theory for guidance. Modelers who are not well versed in a field will have little basis for choosing appropriate variables and will be especially vulnerable to suggestions to add this or that variable to increase realism. Sometimes the addition of variables is motivated by a desire to improve prediction by adding features and variables so that model output more closely matches a particular set of cases for which the modeler has data. This is actually postdiction (see Dibble, 2006, and Gauch, 2003, on postdictive versus predictive accuracy). The kitchen sink tactic is based on a misconception about the relation between model features and variables and about the modelâs ultimate usefulness for providing information about behavior in cases beyond those used for testing. Agent-based models of human and organizational dynamics are often suggested as an effective way to approach the IOS domain. However, the costs of developing, verifying, calibrating, and running complicated agent- based models can be extraordinarily high in relation to our ability to trust what we learn from them. Such a model may have so many degrees of free- dom that it often overfits to sample outcomes at the expense of providing an accurate characterization of the full population of potential outcomes that are important for effective insights and decisions. Predictive models are useful to the extent that they provide trustworthy insights and guidance about a particular population of potential outcomes. A related pitfall is pouring energy into model development, with endless tuning and adjustments, and never using the model in a rigorous fashion to generate insights, answers, or predictions about the probability of differ- ent plausible outcomes. What matters most is what new information can be learned from the model and to what degree and under what conditions what is learned can be trusted. In computational laboratories, research, developing, testing, refining, and calibrating a useful and trustworthy model should represent a modest fraction of the time, effort, and expense of put- ting the model through its paces to answer the important questions that motivated its development. Answers and insights are the ultimate goal, not the model itself. Lessons Learned and Future Needs: Models can become unwieldy when weighed down by a proliferation of features and variables. Strong theory and a clear specification of purpose should guide subject matter experts in
348 BEHAVIORAL MODELING AND SIMULATION the choice of features and variables to be included and excluded, based on the specific questions and problems to be addressed. Model development should not become an end in itself. As with the extreme parsimony pitfall of one-dimensional models, better methods, including comparative studies of alternative models for a common problem, are needed to determine which variables should be included in a model to generate the most useful results. Pitfalls in Dealing with Uncertainty and Adaptation The problems in this section are based on unrealistic expectations of how much uncertainty reduction is plausible in modeling human and orga- nizational behavior, as well as on poor choices in handling the changing nature of human structures and processes. Unrealistic Expectations A validated model of weapons delivery could be reasonably expected to predict the exact location where a bomb will fall when dropped from a specific height and location from an aircraft traveling at a specified speed and heading, with specified wind conditions, along with a trustworthy esti- mate of error in prediction based on likely measurement errors. Plugging in the numbers for the specified variables will supply the user with the desired prediction. It is, however, unrealistic to expect comparable model outputs when the outcome to be predicted is behavior by a human, an organization, a nation-state, or other social entity. We illustrate this problem using the behavior of an individual person, but the caveat also applies to predicting the actions of particular governments or other specific organizations. Models that purport to specify the exact actions of any given individual human being after plugging in a list of values (for example, nationality, group membership, genderâfor a nation, this might be values on a look-up table of cultural traits, estimated military strength, and known alliances) are misleading and seriously incomplete. Unrealistic expectations are often based on a misconception about what sort of prediction a human behavior model can actually produce. In most situations of interest, there is a range of plausible behaviors, and within the same situation, different people will behave differently, and the same person may also behave differently at different times. Rather than generating a single definitive prediction of behavior, a good human behavior model will instead identify the space of possible outcomes, give probability assessments for these behaviors, and specify some of the factors that could alter these probability assessments. This pitfall does not necessarily apply to targeted profiling of a particu- lar identified individual, when highly specific idiographic information about
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 349 that individual and a specified context for behavior are available. The data demands of such models are typically very high, however, and it remains plausible that even a very carefully profiled individual will do something completely unexpected. Hence, even for such profiles, predictions that are couched in terms of probabilities are more complete. For example, âJohn Doe is likely to do X, with probability estimate of 60 percent, but may do Y or Z instead (model estimate of 10 percent each) or take some other action not covered by the model (20 percent)â is a more informative and less misleading guide to planning and action than a point prediction: âJohn Doe will do X.â Unrealistic expectations can lead developers to reject a model as useless if postdictive accuracy is not very high. Yet any model that aids in deci- sion making and understanding and that measurably reduces uncertainty can have practical value. The primary contributions of some models are to suggest the space of possible outcomes, reduce the likelihood of surprise, and support systematic analysis. Bronowski (1953) discusses criteria for determining the usefulness of what one might learn. Users do need to know that they can trust what they are learning from the model, but it may be possible to support and test such trust without necessarily expecting the model to replicate observed outcomes in the real world, especially when modeling phenomena that are rare, infrequent, or otherwise nearly impos- sible to observe and compare with the model. For example, Cronbach and Glaser (1965) produced results that were useful for personnel selection and placement because they represented an improvement over the systems that were then in place. It was not the validity coefficient of the results that mat- tered, but the meaningful gain in prediction that they represented. Lessons Learned and Future Needs: When actions must be taken in social situations, IOS models can potentially be used to highlight the range of possible outcomes associated with each considered course of action, together with probability assessments clarifying the likelihood of these pos- sible outcomes. Point predictions are generally misleading and incomplete. The value of IOS models should be measured in terms of the reduction of uncertainty they achieve. Better methods are needed to define the inherent uncertainty in model results and communicate that uncertainty more clearly to users. Illusions of Permanence All models include variables, adjustable parameters, and constants. Even when a goal of modeling is to guide the choice of intervention intended to change structure and behavior (for example, to change organizational struc- ture or culture), which implies that the target of intervention is mutable, the feature to be modified is sometimes modeled as fixed. This strategy appears
350 BEHAVIORAL MODELING AND SIMULATION in models of culture (with culture treated as a static set of attributes) and in many social network models (with the network treated as fixed). This misleading approach encourages users of the model to overlook how the modeled structure may be changing in ways that dramatically alter the impact of an intervention and also forecloses the modeling of how a system or structure adapts and adjusts after an intervention. For example, using a model of terrorist networks to guide decision making about which members of a network to target for removal can mislead users if the network is modeled as a fixed structure from which nodes will be deleted, rather than as a dynamic network with a trajec- tory of change that will be altered by the deletion of a node, in ways that could either weaken or strengthen the effectiveness of the network. Models used to characterize adversary choices (e.g., game theory models) should explicitly allow the strategy of the adversary to change in response to (or in anticipation of) oneâs own strategy choices. Lessons Learned and Future Needs: When feasible, treat IOS structures as variables or as parameters that can be adjusted, rather than as hard- coded fixed attributes that can be altered only by rewriting the source code. Parameters and assumptions will change as a situation evolves, including adversary knowledge of the assumptions. Better methods are needed to build variability over time into models and to communicate the model results (with their accompanying uncertainty) to users. Problems in Combining Components and Federating Models The last three pitfalls we discuss arise from the way in which linkages within and across levels of analysis change the nature of system operation. They arise when creating multilevel models and when linking together more specialized models of behavior into a federation of models. Moving from Individual to Collective Action Social entities such as groups, organizations, and societies are made up of social beings. Yet many individual-based models do not include social capacities. Merely assembling such agents together into a group model will not enable the understanding of teams, the prediction of collective actions, or coordinated group decisions. To model the most rudimentary forms of social behavior, agents need the means to track the behavior of other agents and rules for adjusting their own attitudes or behavior accord- ingly. Depending on the application, the rules can be quite simple. Traffic models, for example, can model the interactive behavior of a collection of agents effectively by assuming that each agent acts to pursue an individual
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 351 goal (getting to a destination in a reasonable time without colliding with o Â thers) and chooses among possible actions based on the presence, position, and density of other agents, who are also trying to get to their preferred destinations. Collective action, however, such as group decision making, requires further rule structures that specify how agents communicate and coordinate their preferences (see Voting and Social Decision Models in ChapterÂ 6). Models that represent changes in attitude or behavior based on social influ- ence need to incorporate rules for how social influence operates. In deciding what social capacities need to be explicitly modeled, rel- evant theory should be consulted. In modeling crowds, for example, social science theories (see Conceptual Models in Chapter 3) suggest that changes in behavior are driven either by a weakening of normative regulation or by emergent norms that become salient to crowd members. While flocking models of crowds that treat human beings as analogous to birds and fish may well be useful in capturing some aspects of crowd behavior (particu- larly a crowd in flight), such models are unlikely to be adequate to inform interventions designed to strengthen social structures that help prevent a crowd from becoming a mob. Moreover, a flock of birds is not the same as one big bird; crowds of people do not necessarily behave like one big person. Models of large organizations should draw on the extensive existing theory and research on how organizational levels are defined and how they relate to each other (Klein and Kozlowski, 2000). Lessons Learned and Future Needs: For social dynamics to operate in multiagent systems, social capacities of agents, such as communica- tion, must be explicitly modeled. For collective action, collective struc- tures such as rules, norms, and social decision schemes are needed. More work is needed to determine the level of detail at which individuals and groups need to be modeled in order to provide useful results. Comparative s Â tudies are needed that examine the contribution of models at different levels of granularity to a common challenge problem, and better methods are needed to link models of individuals to models of larger groups and organizations. Using Collective Attributes to Predict Individual Action Just as modeling problems arise in moving from the individual to the collective, inferences made in the opposite direction also pose special problems. Incorporating cultural information in models of human behavior is a positive step toward explicitly modeling the heterogeneity of people. Modelers need to keep in mind that modeling people from the same culture as homogeneous is also a simplifying assumption. First, the boundaries of nation-states are not necessarily the appropriate cultural boundary. In rela-
352 BEHAVIORAL MODELING AND SIMULATION tively homogeneous nations, such as Japan, the nation-state boundary may well be a good choice. For multiethnic countries, such as Iraq and Afghani- stan, tribal boundaries based on ethnic identities, such as Kurd or Pashtun, may be more appropriate. Second, people in the same group also exhibit considerable variability. The extent to which shared culture results in more or less homogeneous behavior depends strongly on the situation and the type of behavior involved. Third, people have multiple social identities and belong to multiple groups of different sizes, all of which have cultural norms and practices that shape behavior. Membership in a group such as the military, for example, may influence individual behavior more power- fully than membership in a national and ethnic group, so that soldiers in two countries may behave more similarly in a large variety of domains than soldiers in either country compared with civilians belonging to the same nation and ethnic group. Finally, one must also be aware that characteristics of a higher level of unit of analysis (macro indices) may not be characteristic of behavior at a lower level of analysis (individuals at the micro level). For example, members of a rioting crowd may smash windows, set vehicles alight, and violently attack innocent bystanders even if hardly any of the individuals involved would behave that way on their own. Lessons Learned and Future Needs: Be aware of the limits and bound- ary conditions that apply in predicting individual behavior from informa- tion about groups, organizations, and cultures. Behaviors vary in how strongly they are regulated by cultural norms; people belong to multiple groups, all of which have cultural features; and the unit boundary used for modeling may not be the most appropriate one (see Cultural Modeling in Chapter 3). Better methods are needed to represent variable and shifting cultural identities, and comparative studies are needed to assess the benefits of modeling cultural affinities dynamically in providing useful results for a representative challenge problem. Assemblage of Parts Recognizing the problems inherent in universal scope models (see above), many in the modeling community have embraced the goal of linking together component models (which may focus, for example, on a specific aspect of human affect or on culture) to create a more comprehensive IOS model. The logic is for subject matter experts to build the parts separately and eventually snap the component models together to yield the complex behavior of the whole. The need for creating such federated models is a fundamental challenge. It is not possible to build a large universal model without a federation of models. So the challenge is to develop systematic ways to federate models so that the federated result is valid for its own purpose.
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 353 Good federated models require (1) an understanding of the purpose of the federated model, which might require a deep understanding of the problem domain; (2) a good understanding of the individual federated components; and (3) assessment of the validity and limitations of the rela- tionships between the individual federated components and the resulting federated model. In creating such federated models, modelers need to be aware that straightforward snap-together assembly will yield sensible results only when assumptions of additivity and functional independence are tenable, and they often are not. Complex system analysis has shown repeatedly that the connections among the federated components are themselves components to be modeled. The nature of these connections is part of the structure that yields the behavior of the whole. Moreover, the internal structures of the federated components might themselves need to change dramatically when an additional component is connected into the federated model. An example from cognitive modeling will help illustrate the problem. In the early days of cognitive science, many believed that it would be possible to simply piece together separate models of reasoning, auditory processing, visual processing, memory, etc., to build a reasonable model of the human. However, it became clear that due to complex interactions, a more holistic approach was needed. Separate models could be connected together as components of a federated model only if the connections themselves were included as part of the federated model and if the internal structures of the components were adapted to the presence of these connections. The same is true for complex social modeling. Along with models of individuals, the nature of links among the actors and the connections of individuals with larger level units, such as groups and organizations, need to be modeled to yield adequate models of both individual behavior in social context and the behavior of social entities, such as groups and nation-states. As the complexity of models and federations of models grows, it may create the need for âwrappersâ that help human beings understand the implications and dynamics of the models. New analytic components, per- spectives, and tools will be needed to support understanding and use. The complex interactions that are typical of social science models, as discussed above, will make this a challenging area for research. Federation also has implications for the VV&A process (see Chapter 8). A federated model formed by combining two models that have previously been individually validated should not be automatically viewed as vali- dated; the federated model must be validated on its own (Burton, 2003). Lessons Learned and Future Needs: When linking component models, appropriate theory needs to guide the modeling of the linkages as a new component in the resulting federation of models. Systems of systems theory
354 BEHAVIORAL MODELING AND SIMULATION (see Systems Analysis in Chapter 4) can help guide the process of federation, and standards are needed for validating the federation itself. Standards, guidelines, methods, and architectures are needed to improve the state of the art in model federation, addressing semantic interoperability issues that go beyond simple syntactic interoperability. Issues to be addressed include the compatibility of definitions and levels of abstraction, time scale resolu- tion, and treatment of uncertainty in the models to be federated. Summary of Future Needs Social, cultural, and organizational modeling is a complex, emerging science with roots in many different disciplines: psychology, sociology, eco- nomics, anthropology, systems theory, and computer science, among oth- ers. The advancement of a scientific field typically requires that researchers maintain awareness of each otherâs work and build on each otherâs results. The multidisciplinary nature of IOS modeling, however, has created a frag- mented field with researchers in different disciplines often unaware of each otherâs relevant work and failing to make use of relevant existing theory and data. In order for the field to advance, researchers need better frame- works and forums in which to compare, discuss, and evaluate their results. The field currently features a multitude of complex models created using different data and different theories to address different problems, making comparative analysis nearly impossible. Common datasets and challenge problems are needed in order to learn which modeling approaches and sets of variables are most useful for specific types of problems. It seems clear that no single model or approach will meet everyoneâs needs. There is no single right model and probably will never be. The com- mittee thinks that a federated modeling approach, in which different models at different levels are linked together and component submodels can be swapped in and out, are promising for attacking complex IOS modeling problems. Considerable research needs to be done to make this federated vision a reality, however. Standards, architectures, methods, and tools are needed to lower the barriers for developing, linking, and validating feder- ated models. Different modeling purposes require different types of models. In the committeeâs judgment, the purpose of the model should drive the appropri- ate variables to be included in the model. To do this successfully requires a clear specification of model purpose and criteria for usefulness for that purpose, which in turn requires that model developers work closely with the eventual users of the model. The committee also recommends validation for action, in which the purpose of the model drives its validation criteria. IOS models cannot be validated âin generalââthey must be validated for a specific use. Research
PITFALLS, LESSONS LEARNED, AND FUTURE NEEDS 355 is needed with a cross-disciplinary community of interest to establish and promulgate accepted standards for validation of IOS models. Triangulation methods that combine expert judgment, qualitative results and theoretical work, and quantitative results should be further refined and more widely used. Common challenge problems and datasets are needed to facilitate docking of models for comparative purposes. Finally, models of human beings and their individual and collective behaviors must necessarily include a large amount of inherent uncertainty. This uncertainty is not a flaw of the model and cannot be designed out of the model. Human behavior is dynamic and adaptive over time, and it is impossible at the moment (and into the foreseeable future) to make exact predictions about that behavior. What is needed are ways to estimate the probability of plausible outcomes and express those estimates in ways that are clear and meaningful to model users, who can then judge whether the results meet their needs. It is important also to avoid raising expectations about the capabilities of IOS models beyond what they can realistically deliver. References Bennington, R.W. (1995, May). Joint simulation system (JSIMS): An overview. In Proceedings of the IEEE 1995 National Aerospace and Electronics Conference (NAECON 1995) (pp. 804â809). Available: http://ieeexplore.ieee.org/iel3/3912/11342/00522029.pdf Â[accessed Feb. 2008]. Borgatti, S.P., and Everett, M.G. (2006). A graph-theoretic framework for classifying centrality measures. Social Networks, 28(4), 466â484. Bronowski, J. (1953). The common sense of science. Cambridge, MA: Harvard University Press. Burton, R.M. (2003). Computational laboratories for organization science: Questions, validity and docking. Computational and Mathematical Organization Theory, 9(2), 91â108. Cronbach, L.J. and Glaser, G.C. (1965) Psychological tests and personnel decisions, second edition. Urbana: University of Illinois Press. Dibble, C. (2006). Computational laboratories for spatial agent-based models. In L. ÂTesfatsion and K.L. Judd (Eds.), Handbook of computational economics, volume 2: Agent-based computational economics. Amsterdam, The Netherlands: Holland/Elsevier. Gauch, H.G., Jr. (2003). Scientific method in practice. Cambridge, England: Cambridge U Â niversity Press. Klein, K.J., and Kozlowski, S.W.J. (Eds.). (2000). Multi-level theory, research and methods in organizations: Foundations, extensions, and new directions. San Francisco: Jossey-Bass. Office of the Secretary of Defense. (2004). Training capabilities: Analysis of alternatives, final report, volume 1. Washington, DC: Author. Pratt, D., and Henninger, A. (2002). A case for micro-trainers. In Proceedings of Interservice/ Industry Training, Simulation and Education Conference (I/ITSEC), Orlando, FL. U.S. Department of Defense. (1996). DoD high level architecture baseline approved. (Report #537-96.) Washington, DC: Author.