Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 22
--> 3 A Methodology for Creating Human Behavior Representations The Role of Psychology Scientific psychology has more than a century's accumulation of data, theory, and experience in research concerning basic human abilities. Many of these results are so useful in practical domains that they have disappeared from psychology and become integrated into technology. For example, the design of high-fidelity audio equipment is based on precise measurements of human auditory abilities collected many years ago (Lindsey and Norman, 1977). Similarly, a variety of other practical domains have been making use of other aspects of psychological research. The development of practical human behavior representations for military simulations is especially exciting because it presents an opportunity to construct and apply comprehensive models of human abilities that span the various subareas of psychology. The resulting synthesis of results and theory will be not only practically useful, but also a stimulus to a broader and deeper theoretical integration that is long overdue. In addition to a century of research, psychology also has about three decades of experience with computational theories of human ability. Prior to this time, most psychological theory was expressed as verbal descriptions of mental processes whose implications were hard to define because of the lack of precision. The rise of information-processing theory in psychology after World War II was helped considerably by applying a metaphor: humans process information in a manner analogous to computer systems. Information is acquired, manipulated, stored, retrieved, and acted on in the furtherance of a given task by distinct mechanisms. The metaphor was taken further in the 1970s by expressing theories
OCR for page 23
--> and models of mental processes in terms of computer programs. By writing and running the programs, researchers could explore the actual implications of a theoretical idea and generate precise quantitative predictions from the theory. The Role of Sociology and Organizational Science Similarly, sociology and organizational science have almost a century of data, theory, and experience in research concerning the behavior of groups of humans. Many of these results are also useful in practical domains and so have disappeared from these fields and become integrated into operations research techniques and best management practices. For example, shop floor allocation procedures derived from early work on scientific management. The development of practical applications of human behavior representations is exciting because it presents an opportunity to construct and apply comprehensive models of units that span distributed artificial intelligence, organizational science, sociology, small group psychology, and political science studies of power. Following from early work in cybernetics, an information-processing tradition arose within sociology and organizational science. This movement arose more or less in parallel with the one in psychology. However, unlike the movement in psychology, which concentrated on how the individual human acquired, manipulated, stored, retrieved, and acted on information, the movement in sociology and organizational science concentrated on how cognitive, temporal, physical, and social constraints limited the acquisition of information and the consequent actions taken by individuals and groups. The part of this tradition that focused on temporal, physical, and social constraints became known as structural theory. Sociologists and organizational theorists further found that who you know and interact with, and what you think others think of you, affect your opinions, attitudes, and actions. Social information-processing theory, and the various mathematical models of exchange and influence, developed from this research. Many social and organizational theories are expressed as verbal descriptions of institutional, social, and political processes. As with such descriptions of psychological theories the implications of these processes are hard to determine particularly for dynamic behavior. The reason it is difficult to derive a consistent set of predictions from these verbal models for dynamic behavior is in large part because the behavior of units is extremely nonlinear, involves multiple types of feedback, and requires the concurrent interaction of many adaptive agents. Humans, unassisted by the computer, are simply not good at thinking through the implications of such complexity. In addition to almost a century of research, sociology and organizational science have about four decades of experience with computational theories of unit-level behavior. Most of these computational models developed from work in information processing, social information processing, and structural theory.
OCR for page 24
--> By writing and running these computational programs, researchers can explore the actual implications of theoretical ideas and generate quantitative predictions for unit-level behavior. Also, such models can be used to examine the impact of alterations in group size on the resultant outcome. The Necessity of Situation-Specific Modeling The panel notes that we are a long way at the present time from having a general-purpose cognitive model that can be incorporated directly in any simulation and prove useful. However, the field has developed far enough that simulations incorporating known models and results of cognition and behavior will greatly improve present efforts by the military, if—and only if—the models are developed and exactly tailored to the demands of a given task and situation. At the present state of development of the field, it is probably most useful to view a human operator as the controller of a large number of programmable components, such as sensory, perceptual, motor, memory, and decision processes. The key is the idea that these components are highly adaptable and may be tuned to interact properly to handle the demands of each specific task in a specific environment and situation. Thus, we may think of the system as a framework or architecture within which numerous choices and specializations must be made when a given application is required. A number of such architectures have been developed and provide examples of how one might proceed, although the field is still in its infancy and it is too early to recommend a commitment to any one architectural framework. Examples include the SOAR architecture (Laird et al., 1986, 1987; Newell, 1990), the ACT1 series (Anderson, 1976, 1983, 1993), and EPIC2 (Meyer and Kieras, in press; Kieras and Meyer, in press). The panel cannot overemphasize how critical it is to develop situation-specific models within whatever general architecture is adopted. The situations and tasks faced by humans in military domains are highly complex and very specific. Any effective model of human cognition and behavior must be tailored to the demands of that situation. In effect, the tailoring of the model substitutes for the history of training and knowledge by the individual (or unit), a history that incorporates personal training and military doctrine. Some may find it disappointing that a general-purpose human model cannot be produced that can be utilized in a variety of quite different situations, but no such model is likely to be available in the near future. At the unit level, several computational frameworks for representing teams 1 ACT (Adaptive Control of Thought) is a theory of learning and problem solving used to build intelligent tutors. 2 EPIC (Executive-Process/Interactive Control) is a cognitive architecture for computational modeling of human performance. It represents a synthesis of results on human perceptual/motor performance, cognitive modeling techniques, and task analysis methodology.
OCR for page 25
--> or groups are emerging. These frameworks at worst supply a few primitives for constructing or breaking apart groups and aggregating behavior. At best, they facilitate the representation of formal structure, such as the hierarchy or team, the resource allocation structure, the communication structure, and unit-level procedures that will be inherited by all team members. These frameworks supply only a general language for constructing a model of how human groups perform tasks and the coordination and communication necessary for pursuing those tasks. Representing actual units requires filling in these frameworks with details for a specific team, group, or unit and for a specific task. A Methodology for Developing Human Behavior Representations To develop human behavior representations, the panel suggests that the Defense Modeling and Simulation Office encourage developers to employ a systematic methodology, which should include the following steps. First, they should employ interdisciplinary teams. Second, they should review alternatives and adopt a general architecture that is most likely to be useful for the dominant demands of the specific situation. Third, they should review available unit-level frameworks and support the development of a comprehensive framework for representing the command, control, and communication structure. (The cognitive framework adopted should dictate the way in which command, control, and communication procedures are represented.) Fourth, they should review available documentation and seek to understand the domain, its doctrine, procedures, and constraints in depth. They should prepare formal task analyses that describe the activities and tasks as well as the information requirements and the human skill requirements that must be represented in the models. They should prepare unit-level task analyses that describe resource allocation, communication protocols, skills, and so forth, for each subunit. Fifth, to prepare estimates of the parameters and variables included in the model that are unconstrained by the domain or procedural requirements, they should use behavioral research results from the literature, procedural model analysis, ad hoc experimentation, social network analysis, unit-level task analysis, field research, and, as a last resort, expert judgment. Finally, they should systematically test, verify, and validate the behavior and performance of the models at each stage of development. We also encourage government representatives of the military to work with researchers to define the incremental increase in model performance as a function of the effort required to produce that performance. In the sections that follow we elaborate the four most important of these methodological recommendations. Employ Interdisciplinary Teams For models of the individual combatant the development teams should
OCR for page 26
--> include cognitive psychologists and computer scientists who are knowledgeable about the contemporary literature and modeling techniques. They should also include specialists in the military doctrine and procedures of the domain to be modeled. For team-, battalion-, and force-level models, as well as for models of command and control, teams composed of sociologists, organizational scientists, social psychologists, computer scientists, and military scientists are needed. In this way, the resultant simulations will make effective use of the relevant knowledge and many (partial) solutions that have been developed in cognitive psychology, artificial intelligence, and human factors for analyzing and representing human behavior in a computational format. Similarly, employing sociology, organizational science, and distributed artificial intelligence will ensure that the relevant knowledge and solutions for analyzing and representing unit-level behavior will be employed. Understand the Domain in Depth and Document the Required Activities and Tasks The first and most critical information required to construct a model of human behavior for military simulations is information about the task to be performed by the simulated and real humans, in terms of the procedures, strategies, decision rules, and command and control structure required for performing the task. This information does not exist in the science of psychology; it is knowledge that exists only in the military domain. For example, under what conditions does a combat air patrol pilot engage an approaching enemy? What tactics are followed? How is a tank platoon deployed into defensive positions? As in the SOAR/IFOR work, military experts have to supply information about the desired skilled behavior that the model is to produce. The form in which this information is collected should be guided by the computational structure that will encode the tasks. The first source of such information is military doctrine. Doctrine is ''fundamental principles by which military forces guide their actions in support of national objectives'' (U.S. Department of the Army, 1993). Behavioral representations need to take account of doctrine (U.S. doctrine for BLUFOR, non-U.S. doctrine for OPFOR). On one hand, doctrinal consistency is important. On the other hand, real forces deviate from doctrine when there is good reason to do so. They may deviate because of a lack of training or knowledge of the doctrine. Or they may deviate on their own initiative, to violate an enemy's expectations. Since doctrine is defined at a relatively high level, there is much room for variability in behavior even while maintaining consistency with it. The degree of doctrinal conformity that is appropriate and the way it is manifest for a given model will depend on the goals of the simulation. Conformity to doctrine is a good place to start in developing a human behavior representation, because it is written down and agreed to by organizational
OCR for page 27
--> management. However, reliance on doctrine is not enough. First, it does not provide the task-level detail required to create a human behavior representation. Second, just as there are both official organizational charts and informal units, there is both doctrine and the ways the jobs really get done. There is no substitute for detailed observation and task analysis of real forces conducting real exercises. The Army has a large-scale project to develop computer-generated representations of tactical combat behavior, such as moving, shooting, and communicating. These representations are called Combat Instruction Sets (CIS). According to the developers (IBM/Army Integrated Development Team, 1993), each CIS should be: described in terms of a detailed syntax and structure layout, explicit in its reflection of U.S. and OPFOR tactical doctrines, explicit in how the CIS will interface with the Semi-Automated Forces simulation software, and traceable to doctrine. Information input for Army CIS development comes from written doctrine and from subject-matter experts at the various U.S. Army Training and Doctrine Command schools who develop the performance conditions and standards for mission training plans. The effort includes battalion, company, platoon, squad, and platform/system-level behavior. At the higher levels, the Mission, Enemy, Troops, Terrain, and Time (METT-T) evaluation process is used to guide the decision-making process. The CISs, like the doctrine itself, should provide another useful input to the task definition process. At the individual level, although the required information is not part of psychology, the process for obtaining and representing the information is. This process, called task analysis and knowledge engineering, is difficult and labor-intensive, but it is well developed and can be performed routinely by well-trained developers and analysts. Similarly, at the unit level, although the required information is not part of sociology or organizational science, the process for obtaining and representing the information is. This process includes: unit-level task analysis, social network analysis, process analysis, and content analysis. The procedures are difficult and labor-intensive, often involving field research or survey collection, but they can be performed routinely by well-trained researchers. At the individual level, task analysis has traditionally been applied to identify and elaborate the tasks that must be performed by users when they interact with systems. Kirwan and Ainsworth (1992:1) define task analysis as: A methodology which is supported by a number of specific techniques to help the analyst collect information, organize it, and then use it to make judgments or design decisions. The application of task analysis methods provides the user with a blueprint of human involvement in a system, building a detailed picture
OCR for page 28
--> of that system from the human perspective. Such structured information can then be used to ensure that there is compatibility between system goals and human capabilities and organization so that the system goals will be achieved. This definition of task analysis is conditioned by the purpose of designing systems. In this case, the human factors specialist is addressing the question of how best to design the system to support the tasks of the human operator. Both Kirwan and Ainsworth (1992) and Beevis et al. (1994) provide detailed discussions of a host of methods for performing task analysis as part of the system design process that can be equally well applied to the definition of human behavior representations in military simulations. If the human's cognitive behavior is being described, cognitive task analysis approaches that rely heavily on sophisticated methods of knowledge acquisition are employed. Many of these are discussed by Essens et al. (1995). Specifically they report on 32 elicitation techniques, most of which rely on interviewing experts, asking experts to make judgments and categorize material, or reviewing and analyzing documents. Descriptions of the physical and cognitive tasks to be performed by humans in a simulation are important to guiding the realism of the behavior representations. However, developing these descriptions is time-consuming and for the most part must be done manually by highly trained individuals. Although some parts of the task analysis process can be accomplished with computer programs, it does not seem possible that the knowledge acquisition stage will be automated in the near future. Consequently, sponsors will have to establish timing and funding priorities for analyzing the various aspects of human behavior that could add value to engagement simulations. At the unit or organizational level, task analysis involves specifying the task and the command and control structure in terms of assets, resources, knowledge, access, timing, and so forth. The basic idea is that the task and the command and control structure affect unit-level performance. Task analysis at the unit level does not involve looking at the motor actions that an individual must make or the cognitive processing in which an individual must engage. Rather, it involves laying out the set of tasks that the unit as a whole must do in order to achieve some goal, the order in which those tasks must be accomplished, what resources are needed to accomplish them, and which individuals or subunits have the necessary resources. A great deal of research in sociology, organizational theory, and management science has been and is being done on how to do a task analysis at the unit level. For tasks, the focus has been on developing and extending project analysis techniques, such as PERT (Program Evaluation and Review Technique) charts and dependency graphs. For the command and control structure, early work focused on general features such as centralization, hierarchy, and span of control. Recently, however, network techniques have been used to measure and distin-
OCR for page 29
--> guish the formal reporting structure from the communication structure. These various approaches have led to a series of survey instruments and analysis tools. There are a variety of unresolved issues, including how to measure differences in the structures and how to represent change. Instantiate the Models Specific data are required to fill in the models. Ideally, the model with its parameters specified would already be incorporated into an architectural framework, along with the more general properties of the human information-processing mechanisms. Parameters for selected sensory and motor processes can and should be obtained from the literature. However, many human behavior representations are likely to include high-level decision-making, planning, and information-seeking components. For these components we are still seeking to define suitable underlying structures, and general models at this level will require further research. However, in many cases, the cognitive activities of interest should conform to doctrine or are highly proceduralized. In these cases, detailed task analyses provide data that will permit at least a first-order approximation of the behavior of interest. Sometimes small-scale analytic studies or field observations can provide detailed data suitable for filling in some aspects of the models, for example, the time to carry out a sequence of actions such as positioning, aiming, and firing a rifle, targeting and launching a missile. Some of these could be readily measured, whereas others could be approximated without new data collection by using approaches based on time and motion study predictions methods from industrial engineering (Antis et al., 1973; Konz, 1995); Fitts' law (Fitts and Posner, 1967); or GOMS3 (John and Kieras, 1996; Card et al., 1983). These could then be combined with estimates of perceptual and decision-making times to yield reasonable estimates for human reaction times to be incorporated into military simulations. It is inevitable that some data and parameter requirements cannot be met by the literature or by modeling and analysis and for which it would be too expensive to conduct even an ad hoc study. In those cases, the developer should rely on expert judgment. However, it is the panel's impression from workshops and conference attendance that expert judgment is often viewed as the primary source. We emphasize that it should be the alternative of last resort. Much of the modeling of human cognition that will be necessary for use in human behavior representations—particularly those aspects of cognition involv- 3 GOMS (Goals, Operators, Methods, and Selection Rules) is a relatively simple methodology for making quantitative estimates of the performance times for carrying out well-structured procedural tasks.
OCR for page 30
--> ing higher-level planning, information seeking, and decision making—is not yet available and will require new research and development. At the same time, these new efforts can build productively on many recent developments in the field, some of which are discussed in the next chapter. Verify and Validate the Model According to the Defense Modeling and Simulation Office, validation is defined as "the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model" (U.S. Department of Defense, 1996). Different types of models require different levels of validation. The degree of precision needed is guided by the characteristics of the model and its intended use. For example, some large models have too many parameters to test the entire model; in these cases, an intelligent testing strategy is needed. Sensitivity analysis may be used to provide guidance as to how much validity is needed as well as to examine the contributions of particular models and their associated costs. Carley (1996) discusses several types of models, including intellective and emulation models. Emulation models are built to provide specific advice, so they need to include valid representations of everything that is critical. Such models are characterized by a large number of parameters, several modules, and detailed user interfaces. Intellective models are built to show proof of concept or to illustrate the impact of a basic explanatory mechanism. Simpler and smaller than emulation models, they lack detail and should not be used to make specific predictions. Validation can be accomplished by several methods, including grounding, calibration, verifying, and harmonizing. Note that we are including verification as a subset of validation. Grounding involves establishing the face validity or reasonableness of the model by showing that simplifications do not detract from credibility. Grounding can be enhanced by demonstrating that other researchers have made similar assumptions in their models or by some form of ethnographic analysis. Grounding is appropriate for all models, and it is often the only level of validation needed for intellective models. Calibration is used to tune a model to fit detailed real data. This is often an interactive process in which the model is altered so that its predictions come to fit the real data. Calibrating a model occurs at two levels: at one level, the model's predictions are compared with real data. At another level, the processes and parameters within the model are compared with data about the processes and parameters that produce the behavior of concern. All of these processes are relevant to the validation of emulation models. Verification is used to determine the validity of the model's predictions relative to real data. Generally this is accomplished through statistical or graphical comparisons between the model's results and those in the real world—a key requirement is the availability of real data obtained under comparable conditions.
OCR for page 31
--> In verification the focus is on results. Furthermore, in the verification process, the model is not altered. Harmonization is a set of techniques for determining the theoretical adequacy of a verified computational motel. That is, it is an approach used to show that the theoretical assumptions in the model are well grounded in the real world. It is important to point out that validation is a labor-intensive process that often requires a team of researchers and several years to accomplish. It is recommended that the work be conducted by trained investigators other than those who develop the computational models. In the military context, the most highly validated models are physiological models and a few specific weapons models. Few individual combatant or unit-level models in the military context have been validated beyond the calibration stage—in fact, many have only been grounded. Validation, clearly a critical issue, is necessary if simulations are to be used as the basis for training or policy making. Large models cannot be validated by simply exhaustively examining the predictions of the model under all parameter settings and contrasting that behavior with experimental data. Basic research is therefore needed on how to design intelligent artificial agents for validating these models. Many of the more complex models can be validated only by looking at the trends that they predict. Additional research on statistical techniques for locating patterns and examining trends needs to be done, and standardized validation techniques that go beyond those currently used need to be developed. This may in part involve developing sample databases against which to validate models at each of the levels. Sensitivity analysis may be used to distinguish between parameters of a model that influence results and those that are indirectly or loosely coupled to outcomes. Finally, it may be useful to set up a review board for ensuring that standardized validation procedures are applied to new models and that new versions of old models are docked against old versions (to ensure that the new versions generate the same correct behavior as the old ones).
Representative terms from entire chapter: