6
Basic Research for Human Interactions with Air Force IS&T Systems

The concept of human-system interactions (HSI) includes the traditional field of human-computer interactions (HCI) but also encompasses the coordinated and purposeful interactions of several or many humans with complex systems and the interactions of teams of humans mediated through systems. The committee recommends that AFOSR focus on HSI because it is essential to the successful operation of complex IT-based systems and to the accomplishment of network-enabled operations. There is strong historical precedence for this area of research. One example of clear importance to the Air Force is the use of cockpit simulators for training and rehearsing. However, the increasing pervasiveness of interactive complexity in IS&T systems argues for a broader interaction, much beyond the computer interfaces and human-machine synergy. Examples are interactions in real-time virtual and simulated environments and in the collaborative control of UAVs. Clearly, such capabilities are both team-focused and network-enabled. An ultimate goal of HSI research would be to enable machines (or algorithms) to perform most if not all of the complex data manipulation, correlation, computation, and automatic data reduction—and even some decision making—leaving humans to perform the most critical judgments that cannot be accomplished by algorithms. Furthermore, HSI should help humans to interact with one another in cooperative tasks where multiple humans are part of the system.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs 6 Basic Research for Human Interactions with Air Force IS&T Systems The concept of human-system interactions (HSI) includes the traditional field of human-computer interactions (HCI) but also encompasses the coordinated and purposeful interactions of several or many humans with complex systems and the interactions of teams of humans mediated through systems. The committee recommends that AFOSR focus on HSI because it is essential to the successful operation of complex IT-based systems and to the accomplishment of network-enabled operations. There is strong historical precedence for this area of research. One example of clear importance to the Air Force is the use of cockpit simulators for training and rehearsing. However, the increasing pervasiveness of interactive complexity in IS&T systems argues for a broader interaction, much beyond the computer interfaces and human-machine synergy. Examples are interactions in real-time virtual and simulated environments and in the collaborative control of UAVs. Clearly, such capabilities are both team-focused and network-enabled. An ultimate goal of HSI research would be to enable machines (or algorithms) to perform most if not all of the complex data manipulation, correlation, computation, and automatic data reduction—and even some decision making—leaving humans to perform the most critical judgments that cannot be accomplished by algorithms. Furthermore, HSI should help humans to interact with one another in cooperative tasks where multiple humans are part of the system.

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs CHALLENGES POSED BY HUMAN INTERACTIONS WITH AIR FORCE IS&T SYSTEMS Scope of the Challenge In the Air Force, there are many instances where one or more humans interact with one or more IS&T systems. These include systems that are distributed not only among different platforms but also perhaps across geographical and organizational boundaries, most often with strict security and service reliability constraints such as near-real-time or time-critical services. Added complexity comes about because both the humans and the systems might be interacting with one another in additional ways that are disconnected from the task being analyzed. New paradigms such as publish-subscribe architectures are being investigated to cope with some of the challenges. What sorts of information, architecture, and format should be used to achieve desired effects, and how can designers and users estimate the uncertainties and internalize context and caveats associated with each option? Assuming the right information is available at the right time and in the right form (e.g., as text or images), what techniques will enable the user to make the best use of it? How can what-if simulations be considered and evaluated? Such complex capabilities might require integrated and synchronized multimodal interfaces (visual, aural, and/or haptic) to capture the high dimensionality of a system of sensors and actuators in the battlefield. Research into HSI should shed light on the usability of the (same) information in a battlefield command-and-control situation relative to the different perspectives (ranks) of the users and the different degrees of granularity (detail). In other words, one must understand and characterize the most likely and useful level of complexity for each potential user, from the warfighter to the commander, so that the complexity of the information can be tailored to provide an optimal amount of information for battlefield decision-making—not a paucity of data, but not data overload either. Visualization (interactive visualization in particular) is an important topic. Because this subject is receiving much attention as part of DARPA’s Command Post of the Future program, AFOSR should work to complement the DARPA investment. The issue of how to ensure security pervades the topic of HSI—in particular because human behavior regarding security has a great influence on the effectiveness of embedded security protocols. User behavior can reinforce or undermine security systems, and we do not know enough about how the interface influences those behavioral choices. The most frequent cause of system failure is human error, and even under the best of circumstances (good training, no stress, well-designed interface system,

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs simple tasks), human operators have been shown to have a finite (non-zero) rate of placing a system outside its operational allowed boundaries.1 Usability and error-free HSI design may work at cross purposes in certain cases. For example, the Therac-25 medical radiation device allowed operators to bypass error messages and apply treatment even if lethal dosages were being used. In interface design, unusual system circumstances must be prominently flagged. It is well known that a high level of stress increases the errors that may cause security or system failures. A modicum of system sensor display redundancy (from different sensors) also gives the user greater confidence in the validity of critical information. HSI research could lead to improved design and evaluation of IT-based systems to achieve better usability, safety, and security simultaneously. Much of the research on influence operations goes beyond IS&T and requires fundamental research into behaviors and how they can be affected. To this base must be added research into information interpretation and presentation, personnel training, and modeling and simulation for influence operations, building on what is known about cultural and behavioral factors. As an example, characterization and recognition of normal and abnormal behavior, in general, would help in surveillance at all levels. Characterizing those behaviors requires ongoing research in social sciences, and automatic recognition of the identified characteristics is an ongoing challenge for IS&T. Power consumption—concerns about which were noted in Chapter 3—may also be improved through HSI research into how users interact with mobile devices with small displays whose display component consumes a significant proportion of the device’s energy. While hardware advances allow for the dimming of displays and the use of screen savers to lower power consumption, the possibility of selective dimming of display areas necessitates a better understanding of which display regions to dim and when and of which display effects may be acceptable to the users or allowed by different operations (e-mail, for example, will have different criteria on dimming than video streaming). Research efforts could be shared by industries that are also seeking power-efficient displays. HSI challenges also arise when using commercial off-the-shelf hardware and software, which when merged into complex systems may present the user with a confusing, even contradictory, set of interfaces. HSI challenges certainly present in specialized equipment interfaces such as those used by the remote operators of UAVs. Such applications drive the development of context-aware, cross-modal analysis techniques that 1   Barry Kirwan, A Guide to Practical Human Reliability Assessment, Taylor and Francis, Ltd.: London (1994).

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs provide input to automated systems. In other cases, the analysis techniques can assist human decision makers in sorting through huge amounts of information. In such cases, data-mining and machine-learning techniques can simplify complex multidimensional data. How to represent such information in multimodal form to facilitate its use is an important area of research. New approaches for the display and rendering of multimodal signal sets, such that important context-dependent features are highlighted and subtle features are not obscured, will be necessary to provide context-aware, user-oriented interfaces that allow effective decisions to be made in complex situations. Such system interfaces must allow users to probe deeper layers to capitalize on uniquely human analysis capabilities. Finally, two issues related to the sharing of information should be kept in mind when planning the HSI research program: DOD and the services have a range of geographically distributed R&D efforts in HSI and modeling. Because the HSI challenge is so huge, it is important that these resources and accomplishments be shared, striving for commonalities of icons, network services, standards, interoperability, response time, scalability, and so on. Some key concerns with respect to information sharing in battlefield situations include scalability, allocation and management of bandwidth, message and resource priority allocation policies, time order of the processing steps used in publish-subscribe systems, intelligence needed to establish logical and semantic relations on the information objects, potential deadlocks, etc. These concerns are closely related to the shift to network-enhanced operations, where multiple sites and organizations need to share common information on a timely basis. Interfaces for Air Force Decision Makers There is a general need within the Air Force for decision-support tools providing more effective interfaces between decision makers and the voluminous information available to them. Over the years, various software packages have been developed to support decision-making. One of the simplest and most useful is the spreadsheet, but others such as critical path methods and the Program Evaluation and Review Technique (PERT) have also found a place. However, most real-world problems involve uncertainty. Whether we are dealing with scientific issues, engineering problems, or capabilities of individuals, we make decisions based on imprecise assumptions and incomplete and time-varying data. Decision-making under uncertainty is usually studied as an offshoot of probability

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs theory, merged with ideas of risk. In principle, decision theory offers a set of structured procedures that assist decision makers. Classical decision theory is most widely studied in a static setting in which the data are given and then analyzed and acted upon. In the real world—and especially in Air Force information operations and influence operations—the inputs to the decision often change before the solution can be implemented. Thus, one needs to pay attention to techniques that lend themselves to rapid, perhaps approximate, solutions and techniques for rapidly updating given solutions when small parts of the problem change. Stochastic programming methods could be useful in some probabilistic settings, although problems involving military logistics are often too large to be treated successfully using these methods. There is also the need to develop software that incorporates Bayesian inference, although a completely resolved Bayesian inference model might not be realistic given the computational resources likely to be required. As a consequence, statistically sound heuristics could be developed and explored to extend computational feasibility to much larger problems. As a model one might think of greatly extending the functionality of spreadsheet software by combining it with packaged statistical algorithms and an appropriate user interface designed to help users formulate questions and display the probabilistic information associated with the results. Part of any effective system of this type would be effective optimization algorithms, because every nontrivial what-if question requires optimization. (See Chapter 3 for some discussion of optimization, dynamic programming, and game theory in complex systems.) It would be desirable for this type of research to develop new classes of statistical models for complex phenomena that (1) are rich enough to capture relevant physical laws and other known constraints in large classes of problems of interest to the Air Force and (2) have exploitable structure that allows the development of new methods for information extraction and for the development of bounds on achievable performance for problems that previously defied practical solution. Another example where better decision support is needed is in control systems that involve humans as essential parts of the decision chain. Technology—especially IT, collaborative work analysis, and communication technology—can provide such decision makers with better local and global awareness, e.g., about the weather and other environmental conditions, including other aircraft and machines in the environment. The research community has been quite successful in modeling and predicting the behavior of machines interacting with relatively well-understood or well-measured environments, including those with uncertainty and some bounded nonlinearity. But people in the loop make decisions that are highly nonlinear and not always predictable, and therefore their

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs inclusion creates a greater modeling challenge. In addition, the Air Combat Command told the committee that it was increasingly concerned with winning the local population’s hearts and minds as the most effective means of achieving military goals, and in such a context, improved understanding of people can be more important than improved models of equipment. To address all of these people-dependent modeling challenges, there is a great opportunity to devise research experiments that instrument people and their environment with cameras, microphones, stress and pressure sensors, and other measurement devices, perhaps collecting data via wireless technology. Such measurements would offer objective observations about people’s behavior under varied conditions, which could help in its modeling. One specific need is HSI systems that make the control of UAVs more efficient. Currently, Global Hawks and Predators require as many as 25 people for each operation: at least three ground-based pilots, one for each 8-hour shift, and various support personnel to launch and capture UAVs, process collected data, etc. While some attention has been paid to the design of ground control units, significant improvements could probably be made using state-of-the art knowledge of HSI design and collaboration. However, significant advances in intelligent control, HSI methods, and other areas of interactive-collaboration research will be needed to move toward the goal of having one person control a single UAV, with even more research being required before one person could control multiple UAVs. Machine Learning to Support HSI System components that adapt and learn from their environment or from coaching by a human supervisor would open an amazing number of doors, from intelligent control systems to intelligent decision aids. Machine learning is a critical technology for eventually enabling software-intensive systems to (1) adapt and extend their functionality in response to specific training and (2) make intelligent inferences either from general human guidance or through reasoning based on interaction with the system’s domain. One particularly important application area for intelligent systems would be immersive environments for training and exercise support. This will require advances in modeling and simulation, eventually yielding virtual worlds that can be used to support training, exercises, and critical decisions about system development and acquisition. All too often modeling and simulation are underused because technical difficulties prevent reuse and combination.

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs Simulation as a Design and Training Tool for HSI Another challenge for human interactions with complex IS&T systems is the need for realistic, validated simulations that allow humans to design systems and learn about their management, operations, and vulnerabilities. Design options and capabilities of systems as complex as a network-enabled battlefield force or a publish-subscribe system with 100,000 users can only be explored via simulation. The software engineering and validation and verification aspects of such simulations are known to be challenging. Less well known is the need for the proper design and analysis of experiments run via simulation. Without attention to this need, complex simulations often lead to inaccurate or imprecise results. Moreover, the widespread use of simulation for Air Force training presents more challenges owing to the presence of a human in the loop. The simulation modeling of human behavior is not well developed or understood, though efforts along these lines (e.g., models of crowd behavior) are ongoing. In a training mission, the vagaries of human behavior make careful attention to statistical design and analysis even more critical. Another area where the potential of simulations is underutilized is input/output sensitivity analysis. The simulation model is not only for obtaining predictions but also for investigating which inputs are important and which are not (often of equal interest). The same statistical design/analysis methods for “production” runs can be used for sensitivity analysis to elucidate things like main effects, interactions, and so on. This leads rather naturally to validation and verification, because validation relies on statistical comparisons between the outputs from the simulated system and those from the real system. A rigorous simulation procedure offers the potential for objective and rational validation. And a good simulation system can also be a good training tool for users. BASIC RESEARCH RECOMMENDATIONS FOR HSI FOR AIR FORCE IS&T SYSTEMS Some of the most valuable basic research efforts that would address the challenges in this chapter are listed below. The committee recommends that these topics be addressed in an interdisciplinary manner with goals that could enhance multiple capabilities (e.g., network security, simulation, information operations). Some of this research would seem to be appropriately conducted collaboratively with other branches of the military or with the National Science Foundation. The research should strongly articulate basic design principles, goals, and evaluation of human interfaces with complex IS&T systems. The committee recommends a focus on the following areas:

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs Behavioral models for individuals, groups, and organizations. Specification of an information operations testbed and experimental metrics for evaluating its effectiveness in distributed environments that preserve realistic situation awareness. Better understanding of mixed-modality methods (systems that understand speech, gestures, sound, touch, etc.) for imparting information and effecting control. Research should aim to illuminate the minimum speech capability (or iconography, haptic stimuli, etc.) needed under varied conditions. Design of communication methods that combine text, speech, and visual modalities appropriate to the information being communicated. Understanding user needs that have implications for power usage (and energy efficiency) in displays, such as how users scan areas of a display. Exploration of the theoretical tools used to design, implement, and field complex, time-critical information systems. Topics include the development of large-scale deterministic, stochastic, convex and nonconvex, dynamic, and integer-constrained mathematical programming; algorithms for large-scale Bayesian analysis, including approximate techniques, partial evaluations, and update algorithms; game theory; multiscale modeling and multiscale analysis; and formal approaches to data mining, knowledge representation, and reasoning. Development of better tools for automated reasoning and inference under uncertainty. Such tools can draw on diverse techniques, such as data mining and knowledge discovery in databases; models that learn from data; information retrieval; automated diagnosis and decision support; and machine learning. The goal is to enable users to quickly navigate complex data sets, to set up and analyze decision trees or Bayesian networks, to understand the effects of decision rules, and so on. The paradox of automation is that the more automation is added to aid humans, the more work the humans have to do to understand both the automation and the task(s) which the automation is designed to support. To counter this, specific research is needed to enhance models of users’ capabilities to interact with and understand systems via different modalities; to automate acquisition and enhancement of information fusion models; to create models of users that can be trained from instruction or in exercises with humans; and experimentation in domains of Air Force relevance. Interactive system components that can adapt and learn from their environment or from human supervision should be investigated.

OCR for page 70
Basic Research in Information Science and Technology for Air Force Needs Appropriate methods include indexing, retrieving, and reusing past problem-solving episodes; encoding problem-solving methods from instruction or observation; explanation-based methods; inferring or suggesting improved problem-solving methods; enhancing the efficiency of algorithms based on past problem solving; inductively inferring regularities using, for example, graphical models (Bayesian networks); reinforcing learning methods; understanding emergent behavior; transferring lessons learned in one domain to another domain; using multiagent teams to learn how groups collaborate to solve problems; integrating multiple machine learning techniques; developing toolkits and frameworks that integrate and facilitate adoption of learning methods and provide guidance on the best methods to use in different situations; and experimenting in domains of Air Force relevance. Research on the modeling and simulation of scenarios of importance to the Air Force, including more rapid and cost-effective construction of domain-faithful models of information and influence operations; support for the authoring of scenarios and their translation into the models and databases required for simulations; and software tools to allow users to tailor and understand the behavior and functionality of software components.