3

Human Sciences: Simulation and Training Technology

INTRODUCTION

The Panel on Human Sciences at the Army Research Laboratory conducted its review of ARL’s Simulation and Training Technology Center (STTC) at Orlando, Florida, on June 18-20, 2013. This chapter evaluates that work, recognizing that it represents only a portion of ARL’s portfolio of core competency in human sciences technology.

Broadly stated, the mission of STTC is to enhance readiness through research and development of applied simulation technologies for learning, training, testing, and mission rehearsal. The goal is to understand and opportunistically integrate the human science implications of these activities in order to optimize the behavior of individual soldiers and of small units or teams. One of the goals is to understand and develop immersive technologies that are both operationally effective and cost effective for training and mission rehearsal, and to transition the next generation of modeling and simulation technologies to the future.

The STTC research program has four components:

1. Adaptive Training Technologies Program. Its objective is to design, develop, apply, and assess artificially intelligent agent technologies—such as adaptive tutoring and virtual human tools and methods—to enhance training effectiveness and reduce costs. An important emphasis is on developing tools that allow others—researchers, instructional designers, training developers, and trainers—to efficiently author new training modules so that artificial tutors can be created for different training needs as they arise. The effort to develop a broadly applicable framework has produced the STTC’s generalized intelligent framework of tutoring (GIFT), which is a useful tool for the development and evaluation of intelligent training systems.

2. Synthetic Natural Environments Program. Its objective is to enhance physics-based synthetic environment modeling, with a particular emphasis on dynamic effects such as changes in weather and lighting.

3. Training Applications. The objective of this work is to create and test physical models and software-based simulations for diverse training applications such as medical triage, dismounted soldier operations, or training on specific ground platforms.

4. The Advanced Distributed Simulation Program. Its objective is to conduct R&D work in the area of software design to create a common core infrastructure and toolset to enable distributed and collaborative services in modeling and simulation. The long-term goal is to perform R&D aimed at integrated software architecture to support modeling and simulation across the Human Research and Engineering Directorate (HRED) and beyond.

This assessment is the Board’s first in-depth review of the STTC since its merger with the ARL’s HRED in 2010. During the 2011-2012 review cycle, several STTC programs were presented as part of a broader review at the ARL HRED at Aberdeen Proving Ground, Maryland. The Board’s most recent 2011-2012 report observed that the integration of the STTC into HRED creates great opportunities for human factors influence on STTC products and STTC enhancements of traditional HRED endeavors.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 32
3 Human Sciences: Simulation and Training Technology INTRODUCTION The Panel on Human Sciences at the Army Research Laboratory conducted its review of ARL’s Simulation and Training Technology Center (STTC) at Orlando, Florida, on June 18-20, 2013. This chapter evaluates that work, recognizing that it represents only a portion of ARL’s portfolio of core competency in human sciences technology. Broadly stated, the mission of STTC is to enhance readiness through research and development of applied simulation technologies for learning, training, testing, and mission rehearsal. The goal is to understand and opportunistically integrate the human science implications of these activities in order to optimize the behavior of individual soldiers and of small units or teams. One of the goals is to understand and develop immersive technologies that are both operationally effective and cost effective for training and mission rehearsal, and to transition the next generation of modeling and simulation technologies to the future. The STTC research program has four components: 1. Adaptive Training Technologies Program. Its objective is to design, develop, apply, and assess artificially intelligent agent technologies—such as adaptive tutoring and virtual human tools and methods—to enhance training effectiveness and reduce costs. An important emphasis is on developing tools that allow others—researchers, instructional designers, training developers, and trainers—to efficiently author new training modules so that artificial tutors can be created for different training needs as they arise. The effort to develop a broadly applicable framework has produced the STTC’s generalized intelligent framework of tutoring (GIFT), which is a useful tool for the development and evaluation of intelligent training systems. 2. Synthetic Natural Environments Program. Its objective is to enhance physics-based synthetic environment modeling, with a particular emphasis on dynamic effects such as changes in weather and lighting. 3. Training Applications. The objective of this work is to create and test physical models and software-based simulations for diverse training applications such as medical triage, dismounted soldier operations, or training on specific ground platforms. 4. The Advanced Distributed Simulation Program. Its objective is to conduct R&D work in the area of software design to create a common core infrastructure and toolset to enable distributed and collaborative services in modeling and simulation. The long-term goal is to perform R&D aimed at integrated software architecture to support modeling and simulation across the Human Research and Engineering Directorate (HRED) and beyond. This assessment is the Board’s first in-depth review of the STTC since its merger with the ARL’s HRED in 2010. During the 2011-2012 review cycle, several STTC programs were presented as part of a broader review at the ARL HRED at Aberdeen Proving Ground, Maryland. The Board’s most recent 2011-2012 report observed that the integration of the STTC into HRED creates great opportunities for human factors influence on STTC products and STTC enhancements of traditional HRED endeavors. 32

OCR for page 32
The report suggested that the STTC and HRED increase their focus on human factors in training and continue to integrate STTC technical competencies with HRED skills in human factors research. While some progress toward this goal was evident in the present assessment, the merger of STTC with HRED needs to be taken to the next level, with greater emphasis on integration of human sciences. The design and development of effective training systems is an inherently interdisciplinary enterprise necessitating an early and balanced collaboration of computer science and human science inputs. The installation of the STTC commander as deputy director of HRED is a commendable, vital step to integrating two strong organizational capabilities and cultures to the benefit of both ARL and the Army. ACCOMPLISHMENTS AND ADVANCEMENTS The STTC mission is to improve training efficiency and effectiveness through technology. The projects briefed reflect an ongoing emphasis on training technology for the dismounted soldier that is arguably the most difficult technical challenge and the area with greatest need in the Army today and the foreseeable future. STTC is tackling a number of very challenging technical problems in training technology such as how to make tutoring systems adaptive to individual learners, how to best manage instructional experiences, how to make synthetic entities behave more intelligently in training simulations, and how to make training simulations more interoperable. Many areas pursued by the synthetic natural environments technology group have great potential to set standards that other programs (and other Services) can follow. These areas include simulation in the cloud and terrestrial databases. The advanced distributed simulation (ADS) group therefore has a unique opportunity to generate guidelines or protocols for future developments across the DoD in these areas. For example, documenting the methodology, process, definitions, performance metrics, and validation tests for running simulations in the cloud, and identifying standards and technology to accomplish this, would establish a standard approach for the community. The STTC work is significant and valuable in specific application domains (e.g., simulations of battlefield medical situations), as well as in the design of general tools for simulation (e.g., the generalized intelligent framework for tutoring [GIFT]) to make it possible for others to rapidly create new training modules for areas with new content. Laudable progress has been made to date in the development of GIFT and in the incorporation of this framework within the computer game Virtual Battle Space II being used by the Army for training. OPPORTUNITIES AND CHALLENGES General Opportunities and Challenges The Role of Human Sciences Training simulation and human behavior representation are inherently inter-disciplinary R&D domains that would clearly benefit from the integration and balancing of insights and the connection of ideas from the diverse perspectives of the computer sciences and the human sciences. It is evident that STTC has its strong suit in computer sciences. The paucity of the human science research presented to the review panel made it equally evident that, as it stands, STTC is weak in this area—for example, cognitive and perceptual psychology and human factors—and is losing the potential benefits that often come from the fusion of alternative technical frameworks and approaches driven by human sciences. For example, human sciences identify a wealth of variables and data in the areas of sensation and perception (visual, auditory, tactile, olfactory); attention; problem solving and decision making; learning, motivation; 33

OCR for page 32
emotion and mood, social factors, human workload; human factors/ergonomics; psychometrics; individual differences; and cognitive modeling—all relevant to immersive learning, adaptive tutoring, performance in synthetic environments, and assessment of performance. The merger of STTC with HRED was a very well-conceived decision by Army management that needs to play out more effectively than it has to this point. The main challenge lies in the integration of STTC into HRED in a manner that is useful to both. The human sciences, available more generally at HRED, need to be considered early in the requirements phase and then throughout the design of simulation and training systems. The training technologies being developed have significant implications for humans and their ability to acquire task-critical skills and to become proficient performers. It is difficult to design training software without an appropriate understanding of the instructional objectives and human response to the training environment. For example, how can one know how much fidelity is needed in different aspects of the simulation without understanding the impact of fidelity on learning and retention? Other vital areas that would benefit from considering human sciences include expertise and the elicitation of expert knowledge, identification of training objectives, feedback, team cognition and learning, student and expert models, learning processes, transfer, and retention. Insular Community As described above, the technical challenges faced by STTC are multidisciplinary. Though there is a valuable concentration of simulation and training expertise in Orlando, STTC needs to engage and leverage the broader national and international simulation and training research communities as well. A large component of the STTC R&D staff appears to be bred and trained by a handful of local institutions. As a result, they appear to be somewhat insular with respect to the communities of practice from which they draw. The ARL STTC team needs to consciously recruit at all levels from outside “Team Orlando.” Constraining recruiting and hiring to the military, local industry, and local universities will lead to isolation from ideas current in the broader S&T community. While there is evidence of senior leadership participating in occasional international conferences, wider engagement of the research staff with the broader national and international research community through publications, invited speakers, scholars in residence, conferences, internships, and postdoctorate positions is warranted. The problems being tackled by STTC are large, complex, important, and currently relevant to the modeling and simulation community. However, there are also many researchers working on these issues in academia and in other military laboratories outside the STTC environment. STTC researchers need to clearly define and focus their efforts within the broader scientific community, identifying precisely where they expect to advance the state of the art. For the original research presented, it was not clear where the research fit (or was framed) within the broad base of existing scientific literature. Although some work was described as applied research, the value case was not established with respect to specific Army training applications and problems. STTC is in a unique position to enable collaboration between military-subject-matter experts, trainers, and trainees with computer scientists and simulation specialists. Developing training technology with specific users in mind and developing an in-depth understanding of their needs (as was done by STTC in the medical simulation area) would lead to more focused research and more useful results. Some of the ways to accomplish this might include applying human factors models of task requirements and human capabilities/limitations, identifying relevant individual differences, testing prototypes, soliciting feedback from trainees and subject-matter experts, and collaborating on design and testing. It is important to maximize opportunities to engage with military-subject-matter experts and users at training centers, battle labs, and similar venues. While current budgetary and policy constraints are self-evident, external engagement needs to be a management priority. 34

OCR for page 32
Relationship with the Institute for Creative Technology The STTC is the government program manager for a University Affiliated Research Center, the Institute for Creative Technology (ICT) at the University of Southern California. The ICT receives 100 percent of the STTC basic research (6.1) funding and needs to be integral in addressing some of the STTC’s basic research questions. It was not clear, for example, how STTC requirements are communicated to ICT or how ideas feed down from ICT to STTC. This is obviously a relationship that can be exploited to broaden perspectives in both directions: people closer to the user community and people closer to research. The ICT is, however, assessed by a separate Army assessment board, and the ARLTAB was not asked to review the ICT. Publications STTC’s output of peer-reviewed publications is low. Aside from the work presented in the area of adaptive tutoring, which identified a half dozen reports in venues other than Defense-related conferences, STTC presenters identified almost no publications yielded from the work presented. Publications are more than a simple indicant of quality. Active publication systemically drives quality in S&T cultures and organizations. Engagement in the publication process subjects the research to rigorous review, generally improving it and increasing quality. Furthermore, publications encourage project completion deadlines, polished results, and thoughts about the next steps in R&D. They are also a venue to increase visibility in the research community. It is important that STTC aim to publish in high-quality venues—for example, in top journals and at conferences that require full papers, where each paper is reviewed by two to four outside reviewers (not on the program committee). Its papers could be indexed in catalogues and collections that are available to students and researchers (e.g., IEEE Explore and ACM Digital Library). Observations on Individual Projects Immersive Learning and Intelligent Tutoring The goal of the Learning in Intelligent Tutoring Environments (LITE) group is to develop an adaptive, computer-based tutoring system that selects optimal instructional strategies to meet the specific learning needs of individuals or teams; assesses trainee attributes (e.g., progress, behaviors, and physiology); uses these attributes to classify states and predict learning outcomes (e.g., performance, skill acquisition, and retention); and then adapts the instruction to influence learning. This is clearly an important and challenging problem for the STTC, and the general approach the LITE group has taken to the problem seems to fit well with the broader mission of the STTC. The approach is to develop a flexible, modular system that can be used by individuals who are not computer experts to support a broad range of different point-of-need training challenges. There are some challenges for this broad area. First, the complexity of the problem being tackled almost certainly requires a multidisciplinary approach. As discussed earlier, there was little evidence of expertise in the areas of human cognition, attention, motivation, emotion, or perception outside those listed as external “advisors” to the program. These areas of expertise need to be intimately involved in all stages of this type of research. Secondly, it appears that little progress has been made on the immersive aspect of tutoring. From the presentations it was obvious that a large investment in effort has already been made in investigating a few specific factors such as voice-of-God (i.e., a computer voice, not identified with a virtual team member or instructor, provides feedback to the trainee) versus socially grounded tutoring, windowed versus embedded tutoring, and navigation by mouse versus joystick. However, it was not clear why these particular factors have taken on such importance in light of the many 35

OCR for page 32
other factors that are more likely to enhance the immersive experience. Among these would be sensory inputs (full field of vision, auditory stimuli, tactile stimuli, and, for some applications, olfactory and gustatory stimuli) that provide cues and feedback relevant to a trainee’s decisions or actions and that facilitate the transfer of training to real-world situations. Another challenge and area of concern was the lack of attention to the issue of readiness for general and online training. The assessment of learner readiness, temperament, and personality has been shown to be a critical step in the pedagogical development of training materials; given the normal and expected diversity in learner backgrounds, it deserves to be a priority. Adaptive Tutoring Research The adaptive tutoring research initiative is focused on tutoring technologies that can equal or exceed skilled classroom instruction or skilled human tutors. The focus on tutoring technologies tailors learning activities to the state of the learner or group of learners using adaptive machine-learning methods. Overall, the vision for this research is very ambitious, and the issues being addressed are very important, such as modeling cognitive and affective states, instructional management, and the rapid development of expert models. The research group has apparently reviewed a wide range of existing work in intelligent tutoring, is building on what exists, and seems to be grounded in both empirical and theoretical work. The research studies presented were focused on motivational issues related to stress or boredom that might be translatable to dismounted individuals in complex and team missions. What were not clearly stated was how the R&D agenda is being driven by specific Army training needs and how some of the ICT work on training is being incorporated. The domain of adaptive tutoring research is enormous. The research program would benefit from a tactical evaluation and staging of subgoals. For example, to what degree should the focus be on tutoring individuals in specific skills versus tutoring and training platforms for multiple agents or teams? The issues, solutions, and products could be quite different. Discussion with the STTC group indicated that the priority was on team training. However, the product cited as very successful was a mobile improvised explosive device (IED) trainer—primarily for the training of individuals using preprogrammed training regimens. Another issue is the degree of simulation fidelity required for successful adaptive tutoring. The demonstrations implicitly assumed that very high visual fidelity is desirable and will improve effectiveness of training. However, high-fidelity visual displays may not be the limiting aspect of tutoring environments where joysticks or other simplified tools of action are used to, for example, replicate the physical movements of the agents. If the idea is to train sequences of behavior or logical decision trees of a mission plan, then perhaps visual fidelity, which may be too expensive, could be reduced in favor of focusing on training properties more critical to transfer to real missions. This is yet another illustration of the need for integrating more human sciences into the R&D effort. A third issue is whether to focus on training for general situations or for preparatory rehearsals for actual missions. Training for general scenarios might be very useful—for example, to practice responses to low-probability event structures that would infrequently be encountered in a real situation yet must be among the trained repertoire for robust performance. In this case, there is little necessity for integration of real 3D terrain mapping or urban mapping of specific locations. On the other hand, if the goal is a preparatory rehearsal for a mission, software support that is effective in integrating real-world mapping data may be quite important. Effective adaptive tutoring is likely to increase the interplay between computer science approaches and human science specialists in relation to the delivery of sensory inputs, the biosensing of the emotional or motivational state of the individual, learning theory, and adaptive testing. It would be 36

OCR for page 32
useful to identify several high-priority, prototypical applications as benchmarks for testing development. The project must incorporate human testing and evaluation early in the design stage when deciding about where to spend resources in the development of technological systems so as to maximize human learning and performance. Overall, this group is cognizant of the need to integrate knowledge from the existing literature and has done a fine job engaging the academic side of intelligent tutoring at well-selected conference venues that is likely to provide them far more rigorous reviews than DoD conferences. It has been effective in establishing a Board of Advisors made up of distinguished subject-matter experts from industry and academia. These advisors can serve them well if effectively used as critical and constructive reviewers of their ongoing and planned research. Overall, this team exhibited a strong sense of teamwork and passion for the research as well as an attitude strongly supportive of the development of technical skills and the education of junior scientific members. Generalized Intelligent Framework of Tutoring The GIFT framework is a laudable effort to establish a general framework for intelligent tutoring. The GIFT demonstration illustrated a computational platform early in development that ambitiously aims to provide a structured environment for the authoring and managing of individual and complex multi- actor team tutoring by individuals with less specialized technical and programming skill. The framework was outlined, but no details were presented on its significant components, such as the expert or student models or the selections of instruction. A wide range of sensors and the limitations of those sensors were discussed. However, there was no justification for the sensors that were selected with respect to their relevance to learning the task at hand. For example, is a full-motion platform that enables the trainee to experience walking in the game justified if a joystick is adequate for what the trainee needs to experience for the task at hand? GIFT has potential to become a useful and important framework for embedding results from the center’s research and solving some of the Army’s training problems. It is also early enough in development to reconsider its design priorities—that is, to consider implementing a truly adaptive assessment and training system within a pedagogical model that could better drive the system in the direction of the tutoring, training, and simulation problems, which are the highest priority of the Army. An alternative might be to consider studying and integrating truly adaptive aspects of existing tutoring products in domains such as mathematics and knowledge space theory. These applications map dependencies between target skills or knowledge (skill A generally precedes skill B) and use probabilistic testing to first assess current knowledge and performance and then to determine what might best be trained next. The knowledge spaces of the military training applications may be “flatter” and less complex than domains in mathematics, but they might still exhibit strong content structure. Knowledge structure approaches, if applicable, could provide important information about readiness to learn subskills that could be critical in effective use of time on training. Trainee performance data may provide validation of content structure analyses and need to be considered an important tool in product development. Further, to the extent that many aspects of military training have already been codified in procedures and sequences of training activities, there may already exist starting structures for defining the relevant content domains. Explicit Feedback with Game-Based Training Current game environments provide only implicit feedback. To enable explicit feedback, STTC has embedded intelligent tutoring functions in games through embodied pedagogical agents (EPAs). This represents an effective use of an existing technology investment (Virtual Battlespace 2 [VBS2]) to perform studies and collect data. Since the VBS2 gaming system is becoming a more prevalent in the 37

OCR for page 32
Army, integrating GIFT with it is a good strategic move for future tutoring, because trainees will have a common platform they can use for a range of tasks. The integration of GIFT enables them to collect information in real time for after-action review and to correct trainees in real time so they do not repeat erroneous behavior. The EPA study presented was an excellent example of the use of the human sciences to design intelligent tutors. The study was grounded in social cognitive theory and was a simple, well-designed test of the increased efficacy of embodied agents in providing feedback. EPAs were used as instructors or team members from different sources (e.g., GIFT or embedded in the game). The idea of GIFT turning a player in the game into a tutor appears to be a very natural way for a trainee to learn—that is, from other teammates (albeit virtual). In this study, simple verbal feedback was about as effective as embodied agents feedback—simplifying the demand characteristics of instructional game design in contrast to the literature, which suggests a larger impact for embodied sources. However, it may still be that an actual embodied agent (e.g., a military expert) could be more effective in, say, transfer, suggesting the need for further study. Modeling Learner Mood in Real Time Through Biosensors for Intelligent Tutoring Improvements This study is based on the idea that if the stress levels of individuals could be tracked and classified in real time, then training regimens could be monitored to advance training without triggering a stress response. The study included a number of biomarker measurement devices, including sonar (for distance measurements), a device for measuring pupillary response of the eye, a heart rate sensor, a higher-end electroencephalogram (EEG) system, and an inexpensive, 14-channel, simple EEG system. Trainees completed tasks and periodically self-reported their stress state. Several data-mining techniques, including simple regression and several machine-learning clustering techniques, were tested for their ability to learn to classify the biosensor data using the trainee rating as the target. This is essentially a test of the success of several different classification algorithms. The results had yet to be tested with the usual forms of classification cross-validation, nor had the quality of classification been determined with different subsets of biosensor inputs, highly relevant if the aim is to classify trainee reactions based on small numbers of inexpensive sensors. The development of simple biosensor-based assessments of stress has the potential to make useful contributions to monitoring trainees. The project would benefit from the inclusion of human science specialists in physiological response monitoring for EEG and heart rate and pupillary dilation to place the data mining of biosensor measures for classification within large literatures related to human physiological responses. Synthetic Natural Environments The research programs in synthetic natural environments are focused on computer technical developments that will generate large-scale, realistic, immersive environments. The researchers cite the need to integrate 3D information about environments and the detailed information required to simulate urban warfare environments in high resolution. The goal is to incorporate and simulate weapons and machine impacts in these environments, which requires sophisticated models of the physical world. For example, models of building strengths and stressors in the urban environment are necessary to implement realistic consequences of simulated interventions. 38

OCR for page 32
Real-Time Dynamic Physical Effects in Army Synthetic Environments The project demonstration in this area clearly illustrated the need for faster computations in order to create useful environments with realistic physical effects. There are interesting elements to this work. One is the ability to adjust fragment grid size to achieve performance. Another is accepting (for now) simpler models (without drag) to achieve subsecond results with full knowledge that more powerful processors will allow more faithful models in the future. However, the approaches presented did not take advantage of current and newer hardware architectures, such as graphics processing units (GPUs). All the work presented was based on traditional central processing unit algorithms. STTC needs to advance its research in this area toward the more general trend in the computing community if it wants to remain a leading research group in this discipline. Terrain Generation It was indicated that the current state of the art of terrain database generation is costly, slow, and complex and not up to the challenge posed by the need to dynamically represent high-resolution urban terrain. The ambitious challenge addressed by this STTC effort is the assumed need for effective and low- cost representations of terrestrial line of sight; dismounted and mounted maneuver; weapons effects; close air support; intelligence, surveillance, and reconnaissance (ISR); and communications. However, it was not fully evident that the fundamental requirements of tactically relevant ground force simulation are best served by the stated objectives: to minimize cost, complexity, storage, and licensing fees. Perhaps it would have been more illuminating if the case had been based on a critical evaluation of contemporary practice against the inherent interoperability requirements of live, virtual, constructive, and gaming simulation. Cited plans for future research included rapid database generation for mission planning, exploitation of comprehensive feature extraction from LiDAR 1 and hyperspectral imagery, and real-time processing of aerial- and ground-collected data. There was no evidence of STTC expertise in this area; STTC needs to pursue aggressive evaluation and exploitation of contemporary geospatial data sources such as Urban Feature Data Level 3, Specialized Urban Topographic Data Store, and Multinational Geospatial Co-production Program products from the National Geospatial-Intelligence Agency or the Homeland Security Infrastructure Program. Similar to the presentations on real-time dynamic physical effects in Army synthetic environments, there was little discussion about how newer hardware architectures could help with this work. The underlying methods were grounded in more conventional computing and data management approaches. Human Representation STTC reported on work to improve the physical representations of humans—for example, gestures and movements in distributed simulation, which are a serious shortcoming in current systems. However, the progress achieved since 2010 was not made evident. The reported work with Soar, which is aimed at enhancing the intelligence of agents in order to enable consistent social cultural behaviors, is commendable. 2 However, the degree of social and behavioral realism for synthetic forces continues to be a challenge and a concern. The complex challenge of modeling creditable tactical behavior in computer-generated forces requires ongoing collaboration and 1 LiDAR (also written Lidar or LIDAR) is a remote sensing technology that measures distance by illuminating a target with a laser and analyzing the reflected light. The term "lidar" comes from combining the words light and radar. 2 Soar is a cognitive architecture used to model different aspects of human behavior. 39

OCR for page 32
engagement between military subject-matter experts, computer scientists, and human scientists. In particular, there needs to be explicit guidance from cognitive and behavioral scientists on the level of realism and fidelity in training applications. For example, in training simulations that involve interaction of avatars with foreign agents, emphasis on the facial expressions, pupil dilation, body language, and other features are paramount; if resources are limited, it is important to scale back other less critical features of the environment. Future research areas in synthetic natural environments cited by STTC included improved immersive capabilities involving acoustics and tactile feedback support (haptics). Distributed simulation systems as early as SIMNET achieved considerable perceptual stimuli from dynamic acoustic cues (e.g., munitions effects, vehicle operations). Such auditory inputs are critical for monitoring and maintaining situation awareness. Background sounds are sometimes a critical cue. For example, lack of normal activity played a big role in alerting experienced soldiers to potential terrorist threats in Afghanistan and Iraq. STTC’s stated intention to pursue investments in localized acoustic, haptic, and olfactory stimuli is appropriate. Advanced Distributed Simulation Research Overall, the challenges addressed by the Advanced Distributed Simulation (ADS) research group are of great importance to the simulation community and the Army training mission. This group has very strong skills and backgrounds in computer science and software engineering, which gives it a great foundation for designing and building stable, well-engineered simulation systems. Computing technology is changing rapidly, and the ADS group needs to track and leverage leading advances in industry because it will not be able to accomplish the needed advances on its own. The STTC group has developed some important strategic relationships with the information and communications industry (e.g., Intel). Moreover, the shared computing infrastructure in Research Park is a great resource to leverage for managing costs. The ADS group has a good appreciation of existing standards, commercial off-the-shelf (COTS) technology, and open source tools that it can leverage to enable it to focus on new developments that are needed. Examples of this include modeling and simulation (M&S) standards from the Simulation Interoperability Standards Organization, virtualization, cloud computing infrastructure, and others. However, keeping up with technology evolution will be a continuing challenge. For example, there was no mention of work with GPUs or of any intent to implement parallelization or to leverage multicore processors. Also, there was no mention of big data, analytics, or computational social science, all of which are relevant to the work the group is pursuing. It could be that partners in Research Park with whom it is collaborating already cover these technologies. Having a technology roadmap would help ADS to determine which partners to leverage for those technologies important to achieving program objectives. Overall, the ADS group appears to be staying reasonably abreast of research in the simulation community. This gives it an appreciation of open technology issues that need to be addressed, such as the behavior of semiautomated forces, simulation in the cloud, and realistic locomotion in virtual environments. The group’s publications appear to be mainly in connection with nonacademic, nonrefereed conferences. It needs to branch out to academic-type conferences, where the review process is more considered and thorough. This will also expose it to a broader range of work in the M&S and computing community beyond that being accomplished in Orlando and at the University of Central Florida. 40

OCR for page 32
The Science Behind Executable Architecture Systems Engineering The Executable Architecture Systems Engineering (EASE) project is tackling some very difficult and important problems in training simulation interoperability and is focused creating an easy-to-use environment for running simulations in the cloud. While STTC is not developing any of these simulations, it is developing the approach and methodology to easily set up and run the simulation. While this is recognized as an important problem to resolve, the group did not present clear evidence that the approach taken can succeed given the scale and complexity of the experience provided by real training environments. The presentation’s high level of abstraction made it difficult to determine how much of the work is focused on infrastructure and how much on simulation interoperability issues. For example, simply having an infrastructure that enables simulation to communicate is not the same as semantic interoperability. It appears that this has been initially designed for a very limited, known set of simulations to simplify the overall problem. However, it was not clear whether there is intent to expand the set of simulations to lesser-known systems, and, if there is, whether the project’s methodology is capable of addressing these. One of the features of the proposed system is to provide users with a simple interface where they can answer questions about what they are trying to accomplish (e.g., purpose, scenario), after which the simulations are automatically set up in the cloud. One such question proposed for presentation to the user concerned the necessary level of fidelity. Given how fidelity is defined and mapped to a given simulation, it could be very challenging for a user to know what fidelity is needed from a training perspective. This area of research opens for STTC a real and strong opportunity to lead future developments in synthetic natural environments by using its research efforts to generate guidelines and protocols on how simulation software needs to be developed, aimed at the long-term goal of an integrated infrastructure in modeling and simulation. Training Applications Live Training Research Objectives This project was focused on potential technological improvements in the widely used laser tag live training systems. Challenges exist, for example, with respect to the extension of firing trajectories from direct line of sight and improvements in positioning information relative to the target. Current weaponry increasingly uses nonlinear trajectory targeting that requires computational solutions rather than direct sensing. A related goal is to more clearly understand which parts of targets have been hit, and this requires extended sensor arrays and damage models that might predict survivability of human agents or functionality of larger-scale equipment such as tanks or vehicles. The importance of this work to the Army was well communicated, particularly with respect to the importance of identifying and avoiding friendly fire incidents. Future developments stemming from this project could provide identification information for friendly agents and a processing system that would support large-scale, complex, multi-agent situations. These projects were tightly coupled with widely used training technologies, and the described projected improvements in next-generation modules and sensor arrays involve fairly straightforward, though important, technology upgrades. This technological development track needs to incorporate human science and human factors input to effectively address questions such as, Which functions should be assigned to the human and which should be automated? When should human judgment override computer-aided decision? How should such decisions be communicated? Answers to such questions will provide boundary conditions for any new systems. 41

OCR for page 32
Next-Generation Common Multiple Integrated Laser Engagement System This work deals with the specific issue of weapons’ targeting precision using laser propagation. The presentation focused on the laser propagation and signal detection science and did not address the human user (shooter and victim) element of the system or how this might play into training. The research team comprises engineers without representation from the human factors perspective. This was evidenced by the quest to achieve realism without addressing the assumptions underlying its value to training. Dismounted Soldier Training Research Objectives The objective of this advanced technology development (6.3-level R&D) program is to support TRADOC Warfighter Objectives to improve virtual immersion, locomotion, and avatar intelligence for training dismounted soldiers. STTC is addressing important challenges for the effective use of mixed reality, including the volume and weight of backpack needed to do the computation for the training, locomotion using joystick interfaces, and comfort issues with heads-up displays. These systems raise many questions about what aspects are necessary to simulate from a training effectiveness perspective. For example, what are the implications of not representing with adequate fidelity the auditory, tactile, or olfactory sensory cues in training and rehearsal activities? Improving the intelligence and realism of synthetic entities (e.g., opposing force) in simulations and training games is an important issue and has generated a great deal of research. As it did for the other projects it presented, STTC conveyed the assumption that realism is a reasonable goal for its own sake and worth the cost. A huge issue from the human sciences perspective is this: What level of realism suffices in time of high stress, particularly tempo-related stress, where high quality may not be possible owing to sensor and computational constraints such as dynamically changing terrain? Will the current warfighters insist on videogame visual quality in a training system if training stress can be raised to approximate that of action in the field? How important is simulating more realistic motor performances, fatigue, or stress through general levels of noise and other inputs? What is good enough in which situations? Medical Simulation for Training Research Objectives The medical simulation technologies used in training for medical interventions provided a model case for front-end analysis of downstream application needs and a model of the use of technology to provide suitable training and general exposure to individual medical personnel. This “full recurrent cycle” process—applying, sequentially and recursively, assessment, planning, implementation, and reassessment—seemed a model that could be extended to other research programs. Several of the demonstrations entailed very detailed renderings of hypothetical environments. This raised interesting questions about whether these are necessary for effective training and whether they might constrain training to these detailed representations and, in effect, reduce the transfer of training. Evidence was not presented to permit determination of whether joystick interfaces are sufficient or whether more emphasis should be placed the on natural sensing of motions and their translation into virtual reality environments. The STTC work in this area appeared strongly linked to actual needs of the medical instruction community. Researchers said that they had had to find the gaps in needs and knowledge themselves. Of particular note are the efforts this group made to observe at the Multiple Amputation Trauma Trainer at the Army Medical Center in order to better capture the relevant parameters for their simulation scenarios. As a result, they developed strong ties to the medical training community, understood its needs (e.g., instructors’ overload), and developed training systems that have been successfully transitioned to field use for medical training. 42

OCR for page 32
Use of Holography in Medical Training This project examined the requirements for holographic representations of the human body for medical training. This is a new and potentially expensive technology that needs to be weighed against alternative technologies (e.g., motion-based animations of human body and brain systems now available in inexpensive applications for cell phones or iPads) that could prove as effective as or even more effective than holography from a cost and training perspective. Incorporating odor into medical training simulations is a relevant consideration, because numerous studies have revealed that unexpected or unpleasant odors can be a significant distraction in the field. Prior training that includes exposure to malodors that could be encountered is a way to immunize the warfighter or medic against the loss of concentration or attention as well as to minimize the likelihood that the odors can become associated with stress and later serve as triggers for trauma. There is also significant benefit in augmenting training by multisensory information beyond the medical domain, because warfighters can encounter unusual, unexpected, or unpleasant odors in a variety of deployment settings. Virtual Locomotion Concepts and Metrics Study The stated objective of the work is to achieve natural and humanlike locomotion in virtual space. The study team seemed well abreast of the current state of this technology and related research, and it has been able to effectively create a complex, real experimental condition against which to compare the virtual techniques. The experimental conditions included a good selection of movement gestures, including jumping, squatting, crawling, and climbing; the data show that the encumbrance of real gear makes some gestures difficult, just as it makes some real locomotion hard. It is well documented in the literature that users want to be able to move in any direction while gazing in a different direction. Locomotion techniques that do not allow this separation of view direction and movement direction are, in a sense, noncompliant with respect to best practices and ought not be included in testing designed to find the most natural locomotion technique. High-Fidelity Character Autonomy for Virtual Small-Unit Training This work was presented using realistic dynamic behaviors and intelligent decision making based on situational context and the decisions made by other players. They are using the Joint Simulation Bus developed by ONR to allow connectivity with different game engines. The game environment is a potential constraint limited by the product’s behavior (i.e., the conceptual model established for the game world). Another potential constraint was the selection and use of a legacy system and architecture that were originally developed based on different assumptions and technology than are currently available. Therefore, it was not clear how this product would benefit training. OVERALL TECHNICAL QUALITY OF THE WORK STTC has historic strength in computer science and engineering, and it has developed a number of successful technology-enhanced training products. The ARLTAB’s assessment recognized these accomplishments and also examined how the performance of the STTC might be improved by integrating additional scientific expertise, exposure to new or alternative scientific approaches, and tactical consideration and staging of project goals. Overall, STTC has a clear and substantive mission with many important and unique objectives. The Orlando-based leadership and scientific research groups exhibited a high level of professionalism, 43

OCR for page 32
commitment to high technical standards in their projects, and a broad appreciation of their role in enhancing military and human outcomes. The esprit de corps and desire to integrate innovative and effective research strategies were notable in both research teams and leadership. Investment in professional development and training of junior scientists was a priority of the program. The STTC is an excellent research unit that embodies high technical standards and strong operational attitude. Much of the work presented represents an interesting intersection of M&S with computer science. In this context computer science is broader than software development: It includes artificial intelligence, intelligent systems, parallel and distributed systems, social media, online learning communities, digital media and gaming, big data and analytics, computing performance, networking, graphics, and visualization. Many research groups at STTC tend to focus on a specific problem domain and simply use computer science disciplines to create better simulations. Owing to the breadth of work being done at STTC, the staff members are in a unique position to establish themselves as one of the leading research groups working at the intersection of M&S and computer science. While basic research is done by ICT, STTC can (and needs to make the effort to) still integrate research from ICT and others, as well as innovate in the application of these technologies into real-world programs. This would require achieving a broader view of the intersection of computer science and M&S and creating a technology roadmap and strategy for how to accomplish this goal. The research presented by STTC would, to varying degrees, benefit from integrating human science experts (e.g. human factors and cognitive scientists and social psychologists) into the research teams. For example, one of the goals of the simulation technology group is to push the envelope of simulator fidelity, whether it is the number of live users using the system or the physical realism of the system. Some of these questions are mostly technical problems, but in many instances they could benefit from human science experts. For example, knowledge about human perceptual limitations could steer simulator designs and, in turn, not waste bandwidth pushing unnecessary data. Conversely, it could also reveal areas where computational resources should be concentrated (e.g., in auditory realism). Related to this is the idea of satisficing in simulator design. That is, what level of fidelity is adequate for satisfactory training to occur? Overall, some of the areas could benefit from a systems-engineering approach that views the simulator not just as hardware and software but as a complete system of interdependent elements that include human operators. The incorporation of human science approaches and human science professionals in the research teams and in the specification of internal standards could enhance the human science expertise of the researchers, augment functionally engaged scientific advisory arrangements, and expand training opportunities for developing senior and junior scientists. A strong related suggestion is that STTC work to improve the quality of its user study designs, including design of the data analysis. The studies it conducts are costly, and it is vital to be certain that they are both gathering sufficient data on the critical variables of interest and performing well-designed statistical analyses. Technologists are not trained experimenters and will likely benefit from expertise identifying proper independent and dependent variables, measures, and study design elements such as between- or within-subject, needed sample size, and elimination (or mitigation) of confounders. This is another area where closer engagement by human science researchers can pay off, because these subject- matter experts are generally well-trained in experimental design and statistical analysis of both quantitative and qualitative data. 44