National Academies Press: OpenBook

Basic Research in Information Science and Technology for Air Force Needs (2006)

Chapter: 6 Basic Research for Human Interactions with Air Force IS&T Systems

« Previous: 5 Basic Research for Air Force Information Management and Integration
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

6
Basic Research for Human Interactions with Air Force IS&T Systems

The concept of human-system interactions (HSI) includes the traditional field of human-computer interactions (HCI) but also encompasses the coordinated and purposeful interactions of several or many humans with complex systems and the interactions of teams of humans mediated through systems. The committee recommends that AFOSR focus on HSI because it is essential to the successful operation of complex IT-based systems and to the accomplishment of network-enabled operations. There is strong historical precedence for this area of research. One example of clear importance to the Air Force is the use of cockpit simulators for training and rehearsing. However, the increasing pervasiveness of interactive complexity in IS&T systems argues for a broader interaction, much beyond the computer interfaces and human-machine synergy. Examples are interactions in real-time virtual and simulated environments and in the collaborative control of UAVs. Clearly, such capabilities are both team-focused and network-enabled. An ultimate goal of HSI research would be to enable machines (or algorithms) to perform most if not all of the complex data manipulation, correlation, computation, and automatic data reduction—and even some decision making—leaving humans to perform the most critical judgments that cannot be accomplished by algorithms. Furthermore, HSI should help humans to interact with one another in cooperative tasks where multiple humans are part of the system.

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

CHALLENGES POSED BY HUMAN INTERACTIONS WITH AIR FORCE IS&T SYSTEMS

Scope of the Challenge

In the Air Force, there are many instances where one or more humans interact with one or more IS&T systems. These include systems that are distributed not only among different platforms but also perhaps across geographical and organizational boundaries, most often with strict security and service reliability constraints such as near-real-time or time-critical services. Added complexity comes about because both the humans and the systems might be interacting with one another in additional ways that are disconnected from the task being analyzed. New paradigms such as publish-subscribe architectures are being investigated to cope with some of the challenges. What sorts of information, architecture, and format should be used to achieve desired effects, and how can designers and users estimate the uncertainties and internalize context and caveats associated with each option? Assuming the right information is available at the right time and in the right form (e.g., as text or images), what techniques will enable the user to make the best use of it? How can what-if simulations be considered and evaluated? Such complex capabilities might require integrated and synchronized multimodal interfaces (visual, aural, and/or haptic) to capture the high dimensionality of a system of sensors and actuators in the battlefield.

Research into HSI should shed light on the usability of the (same) information in a battlefield command-and-control situation relative to the different perspectives (ranks) of the users and the different degrees of granularity (detail). In other words, one must understand and characterize the most likely and useful level of complexity for each potential user, from the warfighter to the commander, so that the complexity of the information can be tailored to provide an optimal amount of information for battlefield decision-making—not a paucity of data, but not data overload either. Visualization (interactive visualization in particular) is an important topic. Because this subject is receiving much attention as part of DARPA’s Command Post of the Future program, AFOSR should work to complement the DARPA investment.

The issue of how to ensure security pervades the topic of HSI—in particular because human behavior regarding security has a great influence on the effectiveness of embedded security protocols. User behavior can reinforce or undermine security systems, and we do not know enough about how the interface influences those behavioral choices. The most frequent cause of system failure is human error, and even under the best of circumstances (good training, no stress, well-designed interface system,

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

simple tasks), human operators have been shown to have a finite (non-zero) rate of placing a system outside its operational allowed boundaries.1 Usability and error-free HSI design may work at cross purposes in certain cases. For example, the Therac-25 medical radiation device allowed operators to bypass error messages and apply treatment even if lethal dosages were being used. In interface design, unusual system circumstances must be prominently flagged. It is well known that a high level of stress increases the errors that may cause security or system failures. A modicum of system sensor display redundancy (from different sensors) also gives the user greater confidence in the validity of critical information. HSI research could lead to improved design and evaluation of IT-based systems to achieve better usability, safety, and security simultaneously.

Much of the research on influence operations goes beyond IS&T and requires fundamental research into behaviors and how they can be affected. To this base must be added research into information interpretation and presentation, personnel training, and modeling and simulation for influence operations, building on what is known about cultural and behavioral factors. As an example, characterization and recognition of normal and abnormal behavior, in general, would help in surveillance at all levels. Characterizing those behaviors requires ongoing research in social sciences, and automatic recognition of the identified characteristics is an ongoing challenge for IS&T.

Power consumption—concerns about which were noted in Chapter 3—may also be improved through HSI research into how users interact with mobile devices with small displays whose display component consumes a significant proportion of the device’s energy. While hardware advances allow for the dimming of displays and the use of screen savers to lower power consumption, the possibility of selective dimming of display areas necessitates a better understanding of which display regions to dim and when and of which display effects may be acceptable to the users or allowed by different operations (e-mail, for example, will have different criteria on dimming than video streaming). Research efforts could be shared by industries that are also seeking power-efficient displays.

HSI challenges also arise when using commercial off-the-shelf hardware and software, which when merged into complex systems may present the user with a confusing, even contradictory, set of interfaces. HSI challenges certainly present in specialized equipment interfaces such as those used by the remote operators of UAVs. Such applications drive the development of context-aware, cross-modal analysis techniques that

1  

Barry Kirwan, A Guide to Practical Human Reliability Assessment, Taylor and Francis, Ltd.: London (1994).

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

provide input to automated systems. In other cases, the analysis techniques can assist human decision makers in sorting through huge amounts of information. In such cases, data-mining and machine-learning techniques can simplify complex multidimensional data. How to represent such information in multimodal form to facilitate its use is an important area of research. New approaches for the display and rendering of multimodal signal sets, such that important context-dependent features are highlighted and subtle features are not obscured, will be necessary to provide context-aware, user-oriented interfaces that allow effective decisions to be made in complex situations. Such system interfaces must allow users to probe deeper layers to capitalize on uniquely human analysis capabilities.

Finally, two issues related to the sharing of information should be kept in mind when planning the HSI research program:

  • DOD and the services have a range of geographically distributed R&D efforts in HSI and modeling. Because the HSI challenge is so huge, it is important that these resources and accomplishments be shared, striving for commonalities of icons, network services, standards, interoperability, response time, scalability, and so on.

  • Some key concerns with respect to information sharing in battlefield situations include scalability, allocation and management of bandwidth, message and resource priority allocation policies, time order of the processing steps used in publish-subscribe systems, intelligence needed to establish logical and semantic relations on the information objects, potential deadlocks, etc. These concerns are closely related to the shift to network-enhanced operations, where multiple sites and organizations need to share common information on a timely basis.

Interfaces for Air Force Decision Makers

There is a general need within the Air Force for decision-support tools providing more effective interfaces between decision makers and the voluminous information available to them. Over the years, various software packages have been developed to support decision-making. One of the simplest and most useful is the spreadsheet, but others such as critical path methods and the Program Evaluation and Review Technique (PERT) have also found a place. However, most real-world problems involve uncertainty. Whether we are dealing with scientific issues, engineering problems, or capabilities of individuals, we make decisions based on imprecise assumptions and incomplete and time-varying data. Decision-making under uncertainty is usually studied as an offshoot of probability

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

theory, merged with ideas of risk. In principle, decision theory offers a set of structured procedures that assist decision makers.

Classical decision theory is most widely studied in a static setting in which the data are given and then analyzed and acted upon. In the real world—and especially in Air Force information operations and influence operations—the inputs to the decision often change before the solution can be implemented. Thus, one needs to pay attention to techniques that lend themselves to rapid, perhaps approximate, solutions and techniques for rapidly updating given solutions when small parts of the problem change. Stochastic programming methods could be useful in some probabilistic settings, although problems involving military logistics are often too large to be treated successfully using these methods. There is also the need to develop software that incorporates Bayesian inference, although a completely resolved Bayesian inference model might not be realistic given the computational resources likely to be required. As a consequence, statistically sound heuristics could be developed and explored to extend computational feasibility to much larger problems. As a model one might think of greatly extending the functionality of spreadsheet software by combining it with packaged statistical algorithms and an appropriate user interface designed to help users formulate questions and display the probabilistic information associated with the results. Part of any effective system of this type would be effective optimization algorithms, because every nontrivial what-if question requires optimization. (See Chapter 3 for some discussion of optimization, dynamic programming, and game theory in complex systems.) It would be desirable for this type of research to develop new classes of statistical models for complex phenomena that (1) are rich enough to capture relevant physical laws and other known constraints in large classes of problems of interest to the Air Force and (2) have exploitable structure that allows the development of new methods for information extraction and for the development of bounds on achievable performance for problems that previously defied practical solution.

Another example where better decision support is needed is in control systems that involve humans as essential parts of the decision chain. Technology—especially IT, collaborative work analysis, and communication technology—can provide such decision makers with better local and global awareness, e.g., about the weather and other environmental conditions, including other aircraft and machines in the environment. The research community has been quite successful in modeling and predicting the behavior of machines interacting with relatively well-understood or well-measured environments, including those with uncertainty and some bounded nonlinearity. But people in the loop make decisions that are highly nonlinear and not always predictable, and therefore their

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

inclusion creates a greater modeling challenge. In addition, the Air Combat Command told the committee that it was increasingly concerned with winning the local population’s hearts and minds as the most effective means of achieving military goals, and in such a context, improved understanding of people can be more important than improved models of equipment. To address all of these people-dependent modeling challenges, there is a great opportunity to devise research experiments that instrument people and their environment with cameras, microphones, stress and pressure sensors, and other measurement devices, perhaps collecting data via wireless technology. Such measurements would offer objective observations about people’s behavior under varied conditions, which could help in its modeling.

One specific need is HSI systems that make the control of UAVs more efficient. Currently, Global Hawks and Predators require as many as 25 people for each operation: at least three ground-based pilots, one for each 8-hour shift, and various support personnel to launch and capture UAVs, process collected data, etc. While some attention has been paid to the design of ground control units, significant improvements could probably be made using state-of-the art knowledge of HSI design and collaboration. However, significant advances in intelligent control, HSI methods, and other areas of interactive-collaboration research will be needed to move toward the goal of having one person control a single UAV, with even more research being required before one person could control multiple UAVs.

Machine Learning to Support HSI

System components that adapt and learn from their environment or from coaching by a human supervisor would open an amazing number of doors, from intelligent control systems to intelligent decision aids. Machine learning is a critical technology for eventually enabling software-intensive systems to (1) adapt and extend their functionality in response to specific training and (2) make intelligent inferences either from general human guidance or through reasoning based on interaction with the system’s domain.

One particularly important application area for intelligent systems would be immersive environments for training and exercise support. This will require advances in modeling and simulation, eventually yielding virtual worlds that can be used to support training, exercises, and critical decisions about system development and acquisition. All too often modeling and simulation are underused because technical difficulties prevent reuse and combination.

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

Simulation as a Design and Training Tool for HSI

Another challenge for human interactions with complex IS&T systems is the need for realistic, validated simulations that allow humans to design systems and learn about their management, operations, and vulnerabilities. Design options and capabilities of systems as complex as a network-enabled battlefield force or a publish-subscribe system with 100,000 users can only be explored via simulation. The software engineering and validation and verification aspects of such simulations are known to be challenging. Less well known is the need for the proper design and analysis of experiments run via simulation. Without attention to this need, complex simulations often lead to inaccurate or imprecise results. Moreover, the widespread use of simulation for Air Force training presents more challenges owing to the presence of a human in the loop. The simulation modeling of human behavior is not well developed or understood, though efforts along these lines (e.g., models of crowd behavior) are ongoing. In a training mission, the vagaries of human behavior make careful attention to statistical design and analysis even more critical.

Another area where the potential of simulations is underutilized is input/output sensitivity analysis. The simulation model is not only for obtaining predictions but also for investigating which inputs are important and which are not (often of equal interest). The same statistical design/analysis methods for “production” runs can be used for sensitivity analysis to elucidate things like main effects, interactions, and so on. This leads rather naturally to validation and verification, because validation relies on statistical comparisons between the outputs from the simulated system and those from the real system. A rigorous simulation procedure offers the potential for objective and rational validation. And a good simulation system can also be a good training tool for users.

BASIC RESEARCH RECOMMENDATIONS FOR HSI FOR AIR FORCE IS&T SYSTEMS

Some of the most valuable basic research efforts that would address the challenges in this chapter are listed below. The committee recommends that these topics be addressed in an interdisciplinary manner with goals that could enhance multiple capabilities (e.g., network security, simulation, information operations). Some of this research would seem to be appropriately conducted collaboratively with other branches of the military or with the National Science Foundation. The research should strongly articulate basic design principles, goals, and evaluation of human interfaces with complex IS&T systems. The committee recommends a focus on the following areas:

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
  • Behavioral models for individuals, groups, and organizations.

  • Specification of an information operations testbed and experimental metrics for evaluating its effectiveness in distributed environments that preserve realistic situation awareness.

  • Better understanding of mixed-modality methods (systems that understand speech, gestures, sound, touch, etc.) for imparting information and effecting control. Research should aim to illuminate the minimum speech capability (or iconography, haptic stimuli, etc.) needed under varied conditions.

  • Design of communication methods that combine text, speech, and visual modalities appropriate to the information being communicated.

  • Understanding user needs that have implications for power usage (and energy efficiency) in displays, such as how users scan areas of a display.

  • Exploration of the theoretical tools used to design, implement, and field complex, time-critical information systems. Topics include the development of large-scale deterministic, stochastic, convex and nonconvex, dynamic, and integer-constrained mathematical programming; algorithms for large-scale Bayesian analysis, including approximate techniques, partial evaluations, and update algorithms; game theory; multiscale modeling and multiscale analysis; and formal approaches to data mining, knowledge representation, and reasoning.

  • Development of better tools for automated reasoning and inference under uncertainty. Such tools can draw on diverse techniques, such as data mining and knowledge discovery in databases; models that learn from data; information retrieval; automated diagnosis and decision support; and machine learning. The goal is to enable users to quickly navigate complex data sets, to set up and analyze decision trees or Bayesian networks, to understand the effects of decision rules, and so on.

  • The paradox of automation is that the more automation is added to aid humans, the more work the humans have to do to understand both the automation and the task(s) which the automation is designed to support. To counter this, specific research is needed to enhance models of users’ capabilities to interact with and understand systems via different modalities; to automate acquisition and enhancement of information fusion models; to create models of users that can be trained from instruction or in exercises with humans; and experimentation in domains of Air Force relevance.

  • Interactive system components that can adapt and learn from their environment or from human supervision should be investigated.

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×

Appropriate methods include indexing, retrieving, and reusing past problem-solving episodes; encoding problem-solving methods from instruction or observation; explanation-based methods; inferring or suggesting improved problem-solving methods; enhancing the efficiency of algorithms based on past problem solving; inductively inferring regularities using, for example, graphical models (Bayesian networks); reinforcing learning methods; understanding emergent behavior; transferring lessons learned in one domain to another domain; using multiagent teams to learn how groups collaborate to solve problems; integrating multiple machine learning techniques; developing toolkits and frameworks that integrate and facilitate adoption of learning methods and provide guidance on the best methods to use in different situations; and experimenting in domains of Air Force relevance.

  • Research on the modeling and simulation of scenarios of importance to the Air Force, including more rapid and cost-effective construction of domain-faithful models of information and influence operations; support for the authoring of scenarios and their translation into the models and databases required for simulations; and software tools to allow users to tailor and understand the behavior and functionality of software components.

Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 70
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 71
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 72
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 73
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 74
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 75
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 76
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 77
Suggested Citation:"6 Basic Research for Human Interactions with Air Force IS&T Systems." National Research Council. 2006. Basic Research in Information Science and Technology for Air Force Needs. Washington, DC: The National Academies Press. doi: 10.17226/11529.
×
Page 78
Next: 7 Priorities in Basic IS&T Research for the Air Force »
Basic Research in Information Science and Technology for Air Force Needs Get This Book
×
Buy Paperback | $41.00 Buy Ebook | $32.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Air Force is developing new force capabilities appropriate to an emerging array of threats. It is clear that advances in information science and technology (IS&T) are essential for most of these new capabilities. As a consequence, the Air Force is finding it necessary to refocus its IS&T basic research program to provide stronger support for reaching these goals. To assist this effort, the AFOSR asked the NRC for a study to create a vision and plan for the IS&T-related programs within the Office’s Mathematics and Space Science Directorate. This report provides an assessment of basic research needs for Air Force systems and communications, software, information management and integration, and human interactions with IS&T systems. The report also offers a set of priorities for basic IS&T research, and an analysis of funding mechanisms its support.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!