The Panel on Human Sciences at the Army Research Laboratory (ARL) conducted its review of ARL’s Simulation and Training Technology Center (STTC) at Orlando, Florida, on June 18-20, 2013; its review of ARL’s translational neuroscience program at Aberdeen Proving Ground, Maryland, on June 11-13, 2013; and its review of the soldier performance and human systems integration programs at Aberdeen Proving Ground, Maryland, on July 8-10, 2014.
Simulation and Training Technology
Broadly stated, the mission of STTC is to enhance readiness through research and development of applied simulation technologies for learning, training, testing, and mission rehearsal. The overall goal is to understand and opportunistically integrate the human science implications of these activities in order to optimize the behavior of individual soldiers and of small units or teams. One of the constituent goals is to understand and develop immersive technologies that are both operationally effective and cost effective for training and mission rehearsal, and to transition the next generation of modeling and simulation technologies to the future.
The STTC research program has four components:
- Adaptive training technologies program. Its objective is to design, develop, apply, and assess artificially intelligent agent technologies—such as adaptive tutoring and virtual human tools and methods—to enhance training effectiveness and reduce costs. An important emphasis is on developing tools that allow others—researchers, instructional designers, training developers, and trainers—to efficiently author new training modules so that artificial tutors can be created for different training needs as they arise. The effort to develop a broadly applicable framework has
-
produced the STTC’s generalized intelligent framework of tutoring (GIFT), which is a useful tool for the development and evaluation of intelligent training systems.
- Synthetic natural environments program. Its objective is to enhance physics-based synthetic environment modeling, with a particular emphasis on dynamic effects such as changes in weather and lighting.
- Training applications. The objective of this work is to create and test physical models and software-based simulations for diverse training applications such as medical triage, dismounted soldier operations, or training on specific ground platforms.
- Advanced distributed simulation program. Its objective is to conduct R&D work in the area of software design to create a common core infrastructure and tool set to enable distributed and collaborative services in modeling and simulation. The long-term goal is to perform R&D aimed at integrated software architecture to support modeling and simulation across the Human Research and Engineering Directorate (HRED) and beyond.
This assessment is ARLTAB’s first in-depth review of the STTC since its merger with the ARL’s HRED in 2010. During the 2011-2012 review cycle, several STTC programs were presented as part of a broader review at the ARL HRED at Aberdeen Proving Ground, Maryland. The ARLTAB’s 2011-2012 report observed that integration of the STTC into HRED would create great opportunities for human factors influence on STTC products and STTC enhancements of traditional HRED endeavors. The report suggested that the STTC and HRED increase their focus on human factors in training and continue to integrate STTC technical competencies with HRED skills in human factors research.
While some progress toward this goal was evident in the present assessment, the merger of STTC with HRED needs to be taken to the next level, with greater emphasis on integration of human sciences. The design and development of effective training systems is an inherently interdisciplinary enterprise necessitating an early and balanced collaboration between computer science and human science.
The installation of the STTC commander as deputy director of HRED is a commendable, vital step to integrating two strong organizational capabilities and cultures to the benefit of both ARL and the Army.
Translational Neuroscience
The goal of the translational neuroscience (TN) program is to integrate modern neuroscience with human factors, psychology, and engineering to enhance the understanding of soldier function and behavior in complex operational settings. TN is a unique and important effort whose objectives, if successfully accomplished, could be a game changer for soldier and mission effectiveness.
TN has concentrated its efforts on three research thrusts:
- Brain–computer interaction (BCI) technologies. Enable mutually adaptive brain–computer interaction technologies for improving soldier–system performance.
- Real-world neuroimaging. Assess those aspects of brain function that can be usefully monitored outside the laboratory setting and assess and/or develop the technologies that are best suited for this purpose.
- Brain structure–function couplings. Translate knowledge of differences in individuals’ brain structure and function to understand and predict differences in task performance.
From 2009 through 2013, the TN group made significant gains in publication rates and quality (from 16 publications in 2009 to 44 in 2013, including an increase in publications in peer-reviewed journals
from 6 in 2009 to 17 in 2013); numbers of postdoctoral researchers (from 2 in 2009 to 11 in 2013); outreach and mentoring activities (from none in 2009 to 7 in 2013); and level of external funding (from $730,000 in 2009 to $10,750,000 in 2013). On these measures, the group has attained a level found in neuroscience groups at many first-tier academic institutions.
Soldier Performance and Human Systems Integration
The elements of soldier performance that were assessed were these:
- Sensory perception. The goal of research in this area is to understand the perceptual requirements of interpreting unaided and aided visual, auditory, and tactile signals in complex, dynamic, militarily relevant environments.
- Physical and cognitive performance. The goal in this research area is to investigate the physical and cognitive burdens on soldier performance and to develop design guidance for mitigating negative effects.
- Human systems integration (HSI). The goal of research in this applied area is to develop and enhance human performance modeling tools that enable early, cost-effective insertion of HSI criteria into acquisition requirements.
ACCOMPLISHMENTS AND ADVANCEMENTS
Simulation and Training Technology
The STTC mission is to improve training efficiency and effectiveness through technology. The projects briefed reflect an ongoing emphasis on technology for training the dismounted soldier, which is arguably the most difficult technical challenge and the area of greatest need in the Army today and for the foreseeable future.
STTC is tackling a number of very challenging technical problems in training technology, such as how to make tutoring systems that adapt to individual learners, how to best manage instructional experiences, how to make synthetic entities behave more intelligently in training simulations, and how to make training simulations more interoperable.
Many areas pursued by the synthetic natural environments technology group have great potential to set standards that other programs (and other Services) can follow. These areas include simulation in the cloud and terrestrial databases. The advanced distributed simulation (ADS) group therefore has a unique opportunity to generate guidelines or protocols for future developments across the DoD in these areas. For example, documenting the methodology, process, definitions, performance metrics, and validation tests for running simulations in the cloud, and identifying standards and technology to accomplish this, would establish a standard approach for the community.
The STTC work is significant and valuable in specific application domains (e.g., simulations of battlefield medical situations), as well as in the design of general tools for simulation (e.g., the generalized, intelligent framework for tutoring [GIFT]) to make it possible for others to rapidly create new training modules for areas with new content. Laudable progress has been made to date in the development of GIFT and in the incorporation of this framework into the computer game Virtual Battle Space II, used by the Army for training.
Translational Neuroscience
Over the past 6 years the TN group has received consistently high marks from ARLTAB and has been repeatedly cited as a model for how a new group can effectively be developed at a government research laboratory.
Publication rates and journal quality have continued to rise to impressive levels and represent a very significant accomplishment. While further improvement in the quality of journal publication is urged, the TN group’s productivity is on a par with what might be expected from recognized academic institutions working in the domain.
The TN group has successfully attracted more outside funding and now enjoys a level of external support that matches that of most first-tier academic institutional groups in neuroscience. This is an outstanding accomplishment.
The number of postdoctoral fellows in the program originating from first-tier academic institutions has grown considerably. Overall, the program now employs an impressive group of early-career scientists. The consistent investment in the growth of access to intellectual capital is exemplary.
In 2012, HRED completed the renovation of its Aberdeen headquarters. This renovation included the construction of the new MIND laboratory for the neuroscience group. Although somewhat smaller than would be ideal, this lab is an excellent state-of-the-art facility that is well built and well equipped. It is likely to prove to be an excellent facility for the group in the years to come.
Brain–Computer Interaction Technologies
While ARL’s program in brain–computer interaction (BCI) technologies is only a few years old, it has made significant strides in fundamental research and in the development of applications. ARL has carved out an appropriate niche in the BCI community and is well positioned with a clear emphasis on enabling practical BCI systems for soldier support. The decision to explore the integration of other sensing modalities—for example, electrocardiography (ECG), electromyography (EMG), galvanic skin response (GSR), and eye movements—into EEG-based BCI applications is conceptually strong and innovative. Overall, BCI has the potential to lead theoretical and practical breakthroughs in achieving maximum application performance with minimum invasiveness.
The applications and demonstrations shown indicated clear evidence of innovative fundamental and systems-based research. In particular, the integration of rapid serial visual presentation (RSVP) with the RAVEN system (RSVP-based adaptive virtual environment with neural processing) represents a significant fundamental advance. The cross-validation of performance estimates from two tasks (driving and RSVP) is a significant achievement, showing the generalizability of the work across multiple BCI applications. The application of transfer learning to train EEG classifiers using data from previous subjects is innovative and appropriate and has the potential to significantly reduce individual-specific training time needed for BCI systems. The use of sliding windows in hierarchical discriminant component analysis (HDCA) to deal with temporal variability in BCI responses is well considered and solves a significant practical problem in the performance of these systems.
Real-World Neuroimaging
The real-world neuroscience group outlined a project to develop novel EEG measurement technologies and supporting algorithms. The project goal is to design systems for specific tasks rather than a single system that can work in all contexts. Some substantive directions of the work were described,
among them to overcome real-world limitations for use of the electrode system so that it works with hair, slips on and off easily without significant setup, and has high enough sensitivity to capture the signals necessary for the specific task. Overall, project goals and potential applications were clearly articulated and progress to date was illustrated by demonstrations.
Brain Structure–Function Couplings
Finite-Element Head Model
The finite-element head model project involves a rigorous method for incorporating anisotropic properties of biological tissue into a finite-element model for use in the development of head protective systems. This is an important goal for which the neuroscience group appears to be well qualified. The group evaluated this model using real brain-pressure data sets from a cadaver study conducted several years ago. The results demonstrated that the model was considerably more accurate than had been expected. A suggested future direction would be to model the diffusion tensor imaging (DTI) abnormalities from the postmortem brains of individuals with blast or concussive injuries. The effects of an intact, living vascular system, cerebrospinal fluid channels, and interstitial pressure gradients might yield a very different outcome than is seen in impact-damaged cadavers.
Functional Connectivity Project
The TN group appears to have made significant headway in assessing several tools for causal modeling in EEG data. These are state-of-the-art tools that have not been well tested or validated for EEG use anywhere else, and the TN group is doing a solid job of identifying the strengths and limitations of these approaches. Given the central role EEG plays in HRED’s translational portfolio, undertaking this validation is an excellent use of resources. The approach is sound, and the tools seem to be undergoing reasonable validation.
Phase Synchronization Tools in Electroencephalography
The TN group has made good progress in assessing tools for measuring and identifying phase synchronization in EEG data. These are tools that have not been well tested or validated anywhere else. Given the central role EEG plays in the TN portfolio, this project reflects an excellent use of resources. The approach is sound, limitations of the tools seem to be well identified by the research, and methods for maximizing EEG signal in a fieldable device are being developed.
Decision Making in the Field Project
Using advanced psychological models of decision making in a simulated checkpoint screening task, the TN group is assessing the degree to which mission or task biases can be adjusted by instruction and incentives. The work is of very high quality and Army-relevant. It will likely provide important data about how soldiers and officers take mission instructions into consideration and how effectively they can adapt their behavior to the needs at hand, and it might even offer insight into training effectiveness. In the future, the project needs to be expanded to disentangle expertise effects from task difficulty effects, and it also needs to allow for assessing the difference between soldiers and civilians in these kinds of tasks.
Soldier Performance and Human Systems Integration
In general, significant gains were evident in publication rates, numbers of postdoctoral researchers, and collaborations with relevant peers outside ARL. Awareness and effective leverage of the pertinent research literatures were clearly in evidence in most areas. The research work environments were impressive in terms of their unique and advanced technology capabilities to support research. The research staff were confident and passionate about their work, and, in particular, ARL has done a commendable job by hiring clever postdoctoral researchers who are self-motivated and excited about the research. It was also evident that ARL has acted on earlier ARLTAB suggestions to good effect (e.g., by initiating a seminar series featuring external and internal speakers). Overall these are outstanding accomplishments and mark a visible advance over prior years.
Sensory Perception
The research is conducted in three major thrust areas: fundamental sensory capabilities of the soldier; methods, devices, and technologies for aiding perception; and advanced approaches for augmenting perception.
Overall, the presentations and posters were clear and focused and benefited noticeably from an organized approach and a consistent template for conveying the information, which facilitated the panel’s ability to process the information and ask relevant questions.
Progress has been made in each of the three thrust areas. Specific achievements include (1) an empirical characterization of the probability of detection of low-contrast, moving images in the far visual periphery; (2) the completion and report of several studies evaluating the effects of headgear and hearing protection on sound localization; (3) the completion and report of studies on listener perception of small arms fire; and (4) the completion and report of studies investigating unconventional modes of communication (bone conduction and the tactile modality). Important future work is being planned to understand how soldiers recognize and interpret relevant operational and battlefield sounds in complex natural scenes.
In general the research design of the reported studies appears appropriate for the specific goals for which they were intended, and the psychophysical methods employed, if not always the most accurate or efficient, appear at least adequate. Some of the work has been published in premier scientific journals (e.g., Pollard et al., 2013,1 and Tran et al., 20132), a good indicator of the quality and significance of the research.
On the whole, the research questions addressed by the reported studies were well motivated and in keeping with the general charge of this group. Each question also gained significance by its ability to be identified, more or less directly, with a broader research question aggressively pursued in the scientific research literature.
The research greatly benefited by the superb facilities made available to ARL researchers, most notably, the environment for auditory research (EAR) and the computer-assisted virtual environment (CAVE). The research also appeared to benefit from reported collaboration with investigators at several universities across the nation.
_______________
1 K.A. Pollard, P.K. Tran, and T.R. Letowski, 2013, A free-field method to calibrate bone conduction transducers, Journal of the Acoustical Society of America 133(2):858-865.
2 P.K. Tran, T.R. Letowski, and M.E. McBride, 2013, The effect of bone conduction microphone placement on intensity and spectrum of transmitted speech items, Journal of the Acoustical Society of America 133:3900-3908.
Physical and Cognitive Performance
The presentations and posters were very well prepared, organized, and executed. Generally, the programs in this area were addressing practical problems and issues relevant to the Army mission. The experiments were generally well designed according to solid experimental psychology principles. They included good quantitative biomechanical measurements (e.g., projects on the effect of physical load and environment on soldier performance and on the effect of weapon recoil on marksmanship) and some good examples of competent and sophisticated data analysis (e.g., the project on estimating soldier ground reaction forces using activity monitor acceleration). The research staff were very enthusiastic and seemed excited about the idea of doing translational research that has practical outcomes. Scholarly productivity was evidenced by some important publications in academic journals of the just-mentioned projects.
Human–Systems Integration
Improved Performance Research Integration Tool (IMPRINT) is an important human–systems integration (HSI) tool and a signature program of ARL and has had widespread use and considerable continuity for more than 20 years. Since the last ARLTAB review, ARL’s HSI group has collaborated with systems engineers, which is necessary for the continued advancement of this tool.
HSI researchers have been working on several mobile applications that show great promise. Of particular note is the Work Activity Observer, which is a mobile application that can be downloaded for free. This appears to be an easy-to-use tool that holds promise for the collection of collaboration data for applications outside the military.
There are two postdoctoral researchers planned within the HSI team, and they, like the other postdoctoral researchers hired by ARL, appear to be highly energetic and enthusiastic. The expertise of these researchers complements that of the existing HSI team.
HSI researchers have leveraged expertise from external institutions, which helps them in areas in which they may lack strength. Of particular note is the current collaboration with Dr. Matthews of the University of Central Florida, a renowned expert in human performance and stress.
Simulation and Training Technology
General Opportunities and Challenges
The Role of Human Sciences
Training simulation and human behavior representation are inherently interdisciplinary R&D domains that would clearly benefit from the integration and balancing of insights and the connection of ideas from the diverse perspectives of the computer sciences and the human sciences. It is evident that STTC has its strong suit in computer sciences. The paucity of the human science research presented to the review panel made it equally evident that, as it stands, STTC is weak in this area—for example, cognitive and perceptual psychology and human factors—and is losing the potential benefits that often come from the fusion of alternative technical frameworks and approaches driven by human sciences. For example, human sciences identify a wealth of variables and data in the areas of sensation and perception (visual, auditory, tactile, olfactory); attention; problem solving and decision making; learning;
motivation; emotion and mood; social factors; human workload; human factors/ergonomics; psycho-metrics; individual differences; and cognitive modeling—all relevant to immersive learning, adaptive tutoring, performance in synthetic environments, and assessment of performance.
The merger of STTC with HRED was a very well conceived decision by Army management that needs to play out more effectively than it has to this point. The main challenge lies in the integration of STTC into HRED in a manner that is useful to both. The human sciences, available more generally at HRED, need to be considered early in the requirements phase and then throughout the design of simulation and training systems. The training technologies being developed have significant implications for humans and their ability to acquire task-critical skills and to become proficient performers. It is difficult to design training software without an appropriate understanding of the instructional objectives and human responses to the training environment. For example, how can one know how much fidelity is needed in different aspects of the simulation without understanding the impact of fidelity on learning and retention? Other vital areas that would benefit from considering human sciences include expertise and the elicitation of expert knowledge, identification of training objectives, feedback, team cognition and learning, student and expert models, learning processes, transfer, and retention.
Insular Community
As described above, the technical challenges faced by STTC are multidisciplinary. Though there is a valuable concentration of simulation and training expertise in Orlando, STTC needs to engage and leverage the broader national and international simulation and training research communities as well. A large component of the STTC R&D staff appears to be bred and trained by a handful of local institutions. As a result, they appear to be somewhat insular with respect to the communities of practice from which they draw. The ARL STTC team needs to consciously recruit at all levels from outside “Team Orlando.” Constraining recruiting and hiring to the military, local industry, and local universities will lead to isolation from ideas current in the broader S&T community. While there is evidence of senior leadership participating in occasional international conferences, wider engagement of the research staff with the broader national and international research community through publications, invited speakers, scholars in residence, conferences, internships, and postdoctorate positions is warranted.
The problems being tackled by STTC are large, complex, important, and currently relevant to the modeling and simulation community. However, there are also many researchers working on these issues in academia and in other military laboratories outside the STTC environment. STTC researchers need to clearly define and focus their efforts within the broader scientific community, identifying precisely where they expect to advance the state of the art. It was not clear where the original research presented fit (or was framed) within the broad base of existing scientific literature. Although some work was described as applied research, the value case was not established with respect to specific Army training applications and problems.
STTC is in a unique position to enable collaboration between military-subject-matter experts, trainers, and trainees with computer scientists and simulation specialists. Developing training technology with specific users in mind and developing an in-depth understanding of their needs (as was done by STTC in the medical simulation area) would lead to more focused research and more useful results. Some of the ways to accomplish this might include applying human factors models of task requirements and human capabilities/limitations, identifying relevant individual differences, testing prototypes, soliciting feedback from trainees and subject-matter experts, and collaborating on design and testing. It is important to maximize opportunities to engage with military-subject-matter experts and users at training centers,
battle labs, and similar venues. While current budgetary and policy constraints are self-evident, external engagement needs to be a management priority.
Relationship with the Institute for Creative Technology
The STTC is the government program manager for a University Affiliated Research Center, the Institute for Creative Technology (ICT) at the University of Southern California. The ICT receives 100 percent of STTC basic research (6.1) funding and needs to be integral in addressing some of the STTC’s basic research questions. It was not clear, for example, how STTC requirements are communicated to ICT or how ideas feed down from ICT to STTC. This is obviously a relationship that can be exploited to broaden perspectives in both directions: people closer to the user community and people closer to research. The ICT, however, is assessed by a separate Army assessment board, and the ARLTAB was not asked to review the ICT.
Publications
STTC’s output of peer-reviewed publications is low. Aside from the work presented in the area of adaptive tutoring, which identified a half dozen reports in venues other than DoD-related conferences, STTC presenters identified almost no publications contributed by the work presented. Publications are more than a simple indicant of quality. Active publication systemically drives quality in S&T cultures and organizations. Engagement in the publication process subjects the research to rigorous review, generally improving it and increasing quality. Furthermore, publications encourage project completion deadlines, polished results, and thoughts about the next steps in R&D. They are also a means to increase visibility in the research community.
It is important that STTC aim to publish in high-quality venues—for example, in top journals and at conferences that require full papers, where each paper is reviewed by two to four outside reviewers (not on the program committee). Its papers could then be indexed in catalogs and collections that are available to students and researchers (e.g., IEEE Explore and ACM Digital Library).
Observations on Individual Projects
Immersive Learning and Intelligent Tutoring
The goal of the Learning in Intelligent Tutoring Environments (LITE) group is to develop an adaptive, computer-based tutoring system that selects optimal instructional strategies to meet the specific learning needs of individuals or teams; assesses trainee attributes (e.g., progress, behaviors, and physiology); uses these attributes to classify states and predict learning outcomes (e.g., performance, skill acquisition, and retention); and then adapts the instruction to influence learning. This is clearly an important and challenging problem for the STTC, and the general approach the LITE group has taken to the problem seems to fit well with the broader mission of the STTC. The approach is to develop a flexible, modular system that can be used by individuals who are not computer experts to support a broad range of different point-of-need training challenges.
There are some challenges for this broad area. First, the complexity of the problem being tackled requires a multidisciplinary approach. As discussed earlier, there was little evidence of expertise in the areas of human cognition, attention, motivation, emotion, or perception outside those listed as external advisors to the program. These areas of expertise need to be intimately involved in all stages of this
type of research. Secondly, it appears that little progress has been made on the immersive aspect of tutoring. From the presentations it was obvious that a large investment in effort has already been made in investigating a few specific factors such as “voice of God” (i.e., a computer voice not identified with a virtual team member or instructor provides feedback to the trainee) versus socially grounded tutoring, windowed versus embedded tutoring, and navigation by mouse versus joystick. However, it was not clear why these particular factors have taken on such importance in light of the many other factors that are more likely to enhance the immersive experience. Among these would be sensory inputs (full field of vision, auditory stimuli, tactile stimuli, and, for some applications, olfactory and gustatory stimuli) that provide cues and feedback relevant to a trainee’s decisions or actions and that facilitate the transfer of training to real-world situations. Another challenge and area of concern was the lack of attention to the issue of readiness for general and online training. The assessment of learner readiness, temperament, and personality has been shown to be a critical step in the pedagogical development of training materials; given the normal and expected diversity in learner backgrounds, it deserves to be a priority.
Adaptive Tutoring Research
The adaptive tutoring research initiative is focused on tutoring technologies that can equal or exceed skilled classroom instruction or skilled human tutors. The focus on tutoring technologies tailors learning activities to the state of the learner or group of learners using adaptive machine-learning methods. Overall, the vision for this research is very ambitious, and the issues being addressed are very important, such as modeling cognitive and affective states, instructional management, and the rapid development of expert models. The research group has apparently reviewed a wide range of existing work in intelligent tutoring, is building on what exists, and seems to be grounded in both empirical and theoretical work. The research studies presented were focused on motivational issues related to stress or boredom that might be translatable to dismounted individuals in complex and team missions. What were not clearly stated was how the R&D agenda is being driven by specific Army training needs and how some of the ICT work on training is being incorporated.
The domain of adaptive tutoring research is enormous. The research program would benefit from a tactical evaluation and staging of subgoals. For example, to what degree should the focus be on tutoring individuals in specific skills versus tutoring and training platforms for multiple agents or teams? The issues, solutions, and products could be quite different. Discussion with the STTC group indicated that the priority was on team training. However, the product cited as very successful was a mobile improvised explosive device (IED) trainer that was primarily for training individuals to recognize IEDs using preprogrammed training regimens.
Another issue is the degree of simulation fidelity required for successful adaptive tutoring. The demonstrations implicitly assumed that very high visual fidelity is desirable and will improve effectiveness of training. However, high-fidelity visual displays may not be the limiting aspect of tutoring environments where joysticks or other simplified tools of action are used to, for example, replicate the physical movements of the agents. If the idea is to train sequences of behavior or logical decision trees of a mission plan, then perhaps visual fidelity, which may be too expensive, could be reduced in favor of focusing on training properties more critical to transfer to real missions. This is yet another illustration of the need for integrating more human sciences into the R&D effort.
A third issue is whether to focus on training for general situations or for preparatory rehearsals for actual missions. Training for general scenarios might be very useful; for example, the practice of responses to low-probability event structures that would infrequently be encountered in a real situation yet must be among the trainee’s repertoire for robust performance. In this case, there is little necessity
for integration of real three-dimensional (3D) terrain mapping or urban mapping of specific locations. On the other hand, if the goal is a preparatory rehearsal for a mission, software support that is effective in integrating real-world mapping data may be quite important.
Effective adaptive tutoring is likely to increase the interplay between computer science approaches and human science specialists in relation to the delivery of sensory inputs, the biosensing of the emotional or motivational state of the individual, learning theory, and adaptive testing. It would be useful to identify several high-priority, prototypical applications as benchmarks for testing development. The project must incorporate human testing and evaluation early in the design stage when deciding about where to spend resources in the development of technological systems so as to maximize human learning and performance.
Overall, this group is cognizant of the need to integrate knowledge from the existing literature and has done a fine job engaging the academic side of intelligent tutoring at well-selected conferences that are likely to provide far more rigorous reviews than DoD conferences. The group has been effective in establishing a board of advisors made up of distinguished subject-matter experts from industry and academia. These advisors can serve them well if effectively used as critical and constructive reviewers of their ongoing and planned research. Overall, this team exhibited a strong sense of teamwork and passion for the research as well as an attitude strongly supportive of the development of technical skills and the education of junior scientific members.
Generalized Intelligent Framework of Tutoring
The GIFT framework is a laudable effort to establish a general framework for intelligent tutoring. The GIFT demonstration illustrated a computational platform early in development that ambitiously aims to provide a structured environment for the authoring and managing of individual and complex multiactor team tutoring by individuals with less specialized technical and programming skill. The framework was outlined, but no details were presented on its significant components, such as the expert or student models or the selections of instruction. A wide range of sensors and the limitations of those sensors were discussed. However, there was no justification for the sensors that were selected with respect to their relevance to learning the task at hand. For example, is a full-motion platform that enables the trainee to experience walking in the game justified if a joystick is adequate for what the trainee needs to experience for the task at hand?
GIFT has potential to become a useful and important framework for embedding results from the center’s research and solving some of the Army’s training problems. It is also early enough in development to reconsider its design priorities—that is, to consider implementing a truly adaptive assessment and training system within a pedagogical model that could better drive the system in the direction of the tutoring, training, and simulation problems, which are the highest priority of the Army.
An alternative might be to consider studying and integrating truly adaptive aspects of existing tutoring products in domains such as mathematics and knowledge space theory. These applications map dependencies between target skills or knowledge (skill A generally precedes skill B) and use probabilistic testing to first assess current knowledge and performance and then to determine what might best be trained next. The knowledge spaces of the military training applications may be flatter and less complex than domains in mathematics, but they might still exhibit strong content structure. Knowledge structure approaches, if applicable, could provide important information about readiness to learn subskills that could be critical in effective use of time on training. Trainee performance data may validate content structure analyses and need to be considered an important tool in product development. Further, to the
extent that many aspects of military training have already been codified in procedures and sequences of training activities, there may already exist structures for starting to define the relevant content domains.
Explicit Feedback with Game-Based Training
Current game environments provide only implicit feedback. To enable explicit feedback, STTC has embedded intelligent tutoring functions in games through embodied pedagogical agents (EPAs). This represents an effective use of an existing technology investment (Virtual Battlespace 2 [VBS2]) to perform studies and collect data. Since the VBS2 gaming system is becoming more prevalent in the Army, integrating GIFT with it is a good strategic move for future tutoring, because trainees will have a common platform they can use for a range of tasks. The integration of GIFT enables them to collect information in real time for after-action review and to correct trainees in real time so they do not repeat erroneous behavior.
The study on EPAs presented was an excellent example of the use of the human sciences to design intelligent tutors. The study was grounded in social cognitive theory and was a simple, well-designed test of the increased efficacy of embodied agents in providing feedback. EPAs were used as instructors or team members from different sources (e.g., GIFT or embedded in the game). The idea of GIFT turning a player in the game into a tutor appears to be a very natural way for a trainee to learn—that is, from other teammates (albeit virtual). In this study, simple verbal feedback was about as effective as embodied agents feedback. This simplified the demand characteristics of instructional game design in contrast to the literature, which suggests a larger impact for embodied sources. However, it may still be that an actual embodied agent (e.g., a military expert) could be more effective in, say, transfer, suggesting the need for further study.
Modeling Learner Mood in Real Time Through Biosensors for Intelligent Tutoring Improvements
This study is based on the idea that if the stress levels of individuals could be tracked and classified in real time, then training regimens could be monitored to advance training without triggering a stress response. The study included a number of biomarker measurement devices, including sonar (for distance measurements), a device for measuring pupillary response of the eye, a heart rate sensor, a higher-end electroencephalogram (EEG) system, and an inexpensive, 14-channel, simple EEG system. Trainees completed tasks and periodically self-reported their stress state. Several data-mining techniques, including simple regression and several machine-learning clustering techniques, were tested for their ability to learn to classify the biosensor data using the trainee rating as the target. This is essentially a test of the success of several different classification algorithms. The results had yet to be tested with the usual forms of classification cross-validation, nor had the quality of classification been determined with different subsets of biosensor inputs, highly relevant if the aim is to classify trainee reactions based on small numbers of inexpensive sensors.
The development of simple biosensor-based assessments of stress has the potential to make useful contributions to monitoring trainees. The project would benefit from the inclusion of human science specialists in physiological response monitoring for EEG and heart rate and pupillary dilation to place the data mining of biosensor measures for classification within large literatures related to human physiological responses.
Synthetic Natural Environments
The research programs in synthetic natural environments are focused on computer technical developments that will generate large-scale, realistic, immersive environments. The researchers cite the need to integrate 3D information about environments and the detailed information required to simulate urban warfare environments in high resolution. The goal is to incorporate and simulate weapons and machine impacts in these environments, which requires sophisticated models of the physical world. For example, models of building strengths and stressors in the urban environment are necessary to implement realistic consequences of simulated interventions.
Real-Time Dynamic Physical Effects in Army Synthetic Environments
The project demonstration in this area clearly illustrated the need for faster computations in order to create useful environments with realistic physical effects. There are interesting elements to this work. One is the ability to adjust fragment grid size to achieve performance. Another is accepting (for now) simpler models (without drag) to achieve subsecond results with full knowledge that more powerful processors will allow more faithful models in the future. However, the approaches presented did not take advantage of current and newer hardware architectures, such as graphics processing units (GPUs). All the work presented was based on traditional central processing unit algorithms. STTC needs to advance its research in this area toward the more general trend in the computing community if it wants to remain a leading research group in this discipline.
Terrain Generation
It was indicated that the current state of the art of terrain database generation is costly, slow, and complex and not up to the challenge posed by the need to dynamically represent high-resolution urban terrain. The ambitious challenge addressed by this STTC effort is the assumed need for effective and low-cost representations of terrestrial line of sight; dismounted and mounted maneuver; weapons effects; close air support; intelligence, surveillance, and reconnaissance; and communications. However, it was not fully evident that the fundamental requirements of tactically relevant ground force simulation are best served by the stated objectives: to minimize cost, complexity, storage, and licensing fees. Perhaps it would have been more illuminating if the case had been based on a critical evaluation of contemporary practice against the inherent interoperability requirements of live, virtual, constructive, and gaming simulation.
Cited plans for future research included rapid database generation for mission planning, exploitation of comprehensive feature extraction from lidar3 and hyperspectral imagery, and real-time processing of aerial- and ground-collected data. There was no evidence of STTC expertise in this area; STTC needs to pursue aggressive evaluation and exploitation of contemporary geospatial data sources such as Urban Feature Data Level 3, Specialized Urban Topographic Data Store, and Multinational Geospatial Co-production Program products from the National Geospatial-Intelligence Agency or the Homeland Security Infrastructure Program. Like the presentations on real-time dynamic physical effects in Army synthetic environments, there was little discussion about how newer hardware architectures, computing methods, and management approaches could help with this work.
_______________
3 Lidar is a remote-sensing technology that measures distance by illuminating a target with a laser and analyzing the reflected light.
Human Representation
STTC reported on work to improve the physical representations of humans—for example, representation of human gestures and movements in distributed simulation, which are a serious shortcoming in current systems. However, the progress achieved since 2010 was not made evident.
The reported work with Soar, which is aimed at enhancing the intelligence of agents in order to enable consistent social cultural behaviors, is commendable.4 However, the degree of social and behavioral realism for synthetic forces continues to be a challenge and a concern. The complex challenge of modeling creditable tactical behavior in computer-generated forces requires ongoing collaboration and engagement between military subject-matter experts, computer scientists, and human scientists. In particular, there needs to be explicit guidance from cognitive and behavioral scientists on the level of realism and fidelity in training applications. For example, in training simulations that involve interaction of avatars with foreign agents, emphasis on the facial expressions, pupil dilation, body language, and other features is paramount; if resources are limited, it is important to scale back other less critical features of the environment.
Future research areas in synthetic natural environments cited by STTC included improved immersive capabilities involving acoustics and tactile feedback support (haptics). Distributed simulation systems as early as SIMNET achieved considerable perceptual stimuli from dynamic acoustic cues (e.g., munitions effects, vehicle operations). Such auditory inputs are critical for monitoring and maintaining situation awareness. Background sounds are sometimes a critical cue. For example, lack of normal activity played a big role in alerting experienced soldiers to potential terrorist threats in Afghanistan and Iraq. STTC’s stated intention to pursue investments in localized acoustic, haptic, and olfactory stimuli is appropriate.
Advanced Distributed Simulation Research
Overall, the challenges addressed by the advanced distributed simulation (ADS) research group are of great importance to the simulation community and the Army training mission. This group has very strong skills and backgrounds in computer science and software engineering, which gives it a great foundation for designing and building stable, well-engineered simulation systems. Computing technology is changing rapidly, and the ADS group needs to track and leverage leading advances in industry because it will not be able to accomplish the needed advances on its own. The STTC group has developed some important strategic relationships with the information and communications industry (e.g., Intel). Moreover, the shared computing infrastructure in Research Park is a great resource to leverage for managing costs.
The ADS group has a good appreciation of existing standards, commercial off-the-shelf technology, and open source tools that it can leverage to enable it to focus on the new developments that are needed. Examples of this include modeling and simulation (M&S) standards from the Simulation Interoperability Standards Organization, virtualization, cloud computing infrastructure, and others. However, keeping up with technology evolution will be a continuing challenge. For example, there was no mention of work with GPUs or of any intent to implement parallelization or to leverage multicore processors. Also, there was no mention of big data, analytics, or computational social science, all of which are relevant to the work the group is pursuing. It could be that partners in Research Park with whom it is collaborating already cover these technologies. Having a technology roadmap would help ADS to determine which partners to leverage for those technologies important to achieving program objectives.
_______________
4 Soar is a cognitive architecture used to model different aspects of human behavior.
Overall, the ADS group appears to be staying reasonably abreast of research in the simulation community. This gives it an appreciation of open technology issues that need to be addressed, such as the behavior of semiautomated forces, simulation in the cloud, and realistic locomotion in virtual environments. The group’s publications appear to be mainly in connection with nonacademic, nonrefereed conferences. It needs to branch out to academic-type conferences, where the review process is more considered and thorough. This will also expose it to a broader range of work in the M&S and computing community beyond that being accomplished in Orlando and at the University of Central Florida.
The Science Behind Executable Architecture Systems Engineering
The executable architecture systems engineering (EASE) project is tackling some very difficult and important problems in training simulation interoperability and is focused on creating an easy-to-use environment for running simulations in the cloud. While STTC is not developing any of these simulations, it is developing the approach and methodology to easily set up and run the simulation.
While this is recognized as an important problem to resolve, the group did not present clear evidence that the approach taken can succeed given the scale and complexity of the experience provided by real training environments. The presentation’s high level of abstraction made it difficult to determine how much of the work is focused on infrastructure and how much on simulation interoperability issues. For example, simply having an infrastructure that enables simulation to communicate is not the same as semantic interoperability. It appears that this has been initially designed for a very limited, known set of simulations to simplify the overall problem. However, it was not clear whether there is intent to expand the set of simulations to lesser known systems or, if there is, whether the project’s methodology is capable of addressing them.
One of the features of the proposed system is to provide users with a simple interface where they can answer questions about what they are trying to accomplish (e.g., purpose or scenario), after which the simulations are automatically set up in the cloud. One such question proposed for presentation to the user concerned the necessary level of fidelity. Given how fidelity is defined and mapped to a given simulation, it could be very challenging for a user to know what fidelity is needed from a training perspective.
This area of research opens for STTC a real and strong opportunity to lead future developments in synthetic natural environments by using its research efforts to generate guidelines and protocols on how simulation software needs to be developed, aimed at the long-term goal of an integrated infrastructure in modeling and simulation.
Training Applications
Live Training Research Objectives
This project was focused on potential technological improvements in the widely used laser tag live training systems. Challenges exist, for example, with respect to the extension of firing trajectories from direct line of sight and improvements in positioning information relative to the target. Current weaponry increasingly uses nonlinear trajectory targeting that requires computational solutions rather than direct sensing. A related goal is to more clearly understand which parts of targets have been hit, and this requires extended sensor arrays and damage models that might predict survivability of human agents or functionality of larger-scale equipment such as tanks or vehicles.
The importance of this work to the Army was well communicated, particularly with respect to the importance of identifying and avoiding friendly fire incidents. Future developments stemming from this
project could provide identification information for friendly agents and a processing system that would support large-scale, complex, multiagent situations.
These projects were tightly coupled with widely used training technologies, and the described projected improvements in next-generation modules and sensor arrays involve fairly straightforward, though important, technology upgrades. This technological development track needs to incorporate human science and human factors input to effectively address questions such as: Which functions should be assigned to the human and which should be automated? When should human judgment override a computer-aided decision? How should such decisions be communicated? Answers to such questions will provide boundary conditions for any new systems.
Next-Generation Common Multiple Integrated Laser Engagement System
This work deals with the specific issue of weapons’ targeting precision using laser propagation. The presentation focused on the laser propagation and signal detection science and did not address the human user (shooter and victim) element of the system or how this might play into training. The research team comprises engineers without representation from the human factors perspective. This was evidenced by the quest to achieve realism without addressing the assumptions underlying its value to training.
Dismounted Soldier Training Research Objectives
The objective of this advanced technology development (6.3-level R&D) program is to support Training and Doctrine Command (TRADOC) Warfighter Objectives to improve virtual immersion, locomotion, and avatar intelligence for training dismounted soldiers. STTC is addressing important challenges for the effective use of mixed reality, including the volume and weight of backpack needed to do the computation for the training, locomotion using joystick interfaces, and comfort issues associated with heads-up displays. These systems raise many questions about which aspects need to be simulated from a training effectiveness perspective. For example, what are the implications of not representing with adequate fidelity the auditory, tactile, or olfactory sensory cues in training and rehearsal activities?
Improving the intelligence and realism of synthetic entities (e.g., opposing force) in simulations and training games is an important issue and has generated a great deal of research. As it did for the other projects it presented, STTC conveyed the idea that realism is a reasonable goal for its own sake and worth the cost. A huge issue from the human sciences perspective is this: What level of realism suffices in time of high stress, particularly tempo-related stress, where high quality may not be possible owing to sensor and computational constraints such as dynamically changing terrain? Will the current warfighters insist on videogame visual quality in a training system if training stress can be raised to approximate that of action in the field? How important is simulating more realistic motor performances, fatigue, or stress through general levels of noise and other inputs? What is good enough in which situations?
Medical Simulation for Training Research Objectives
The medical simulation technologies used in training for medical interventions provided a model case for front-end analysis of downstream application needs and a model of the use of technology to provide suitable training and general exposure to individual medical personnel. This “full recurrent cycle” process—applying, sequentially and recursively, assessment, planning, implementation, and reassessment—seemed a model that could be extended to other research programs.
Several of the demonstrations entailed very detailed renderings of hypothetical environments. This raised interesting questions about whether these are necessary for effective training and whether they might constrain training to these unnecessarily detailed representations and, in effect, reduce the transfer of training. Evidence was not presented to permit determination of whether joystick interfaces are sufficient or whether more emphasis should be placed on the natural sensing of motions and their translation into virtual reality environments.
The STTC work in this area appeared strongly linked to actual needs of the medical instruction community. Researchers said that they had had to find the gaps in needs and knowledge themselves. Of particular note are the efforts this group made to observe at the Multiple Amputation Trauma Trainer at the Army Medical Center in order to better capture the relevant parameters for their simulation scenarios. As a result, they developed strong ties to the medical training community, understood its needs (e.g., instructor overload), and developed training systems that have been successfully transitioned to field use for medical training.
Use of Holography in Medical Training
This project examined the requirements for holographic representations of the human body for medical training. This is a new and potentially expensive technology that needs to be weighed against alternative technologies (e.g., motion-based animations of human body and brain systems now available in inexpensive applications for cell phones or iPads) that could prove as effective as or even more effective than holography from a cost and training perspective.
Incorporating odor into medical training simulations is a relevant consideration, because numerous studies have revealed that unexpected or unpleasant odors can be a significant distraction in the field. Prior training that includes exposure to malodors that could be encountered is a way to immunize the warfighter or medic against the loss of concentration or attention as well as to minimize the likelihood that the odors can become associated with stress and later serve as triggers for trauma. There is also significant benefit in augmenting training by multisensory information beyond the medical domain, because warfighters can encounter unusual, unexpected, or unpleasant odors in a variety of deployment settings.
Virtual Locomotion Concepts and Metrics Study
The stated objective of the work is to achieve natural and humanlike locomotion in virtual space. The study team seemed well abreast of the current state of this technology and related research, and it has been able to effectively create a complex, real experimental condition against which to compare the virtual techniques. The experimental conditions included a good selection of movement gestures, including jumping, squatting, crawling, and climbing; the data show that the encumbrance of real gear makes some gestures difficult, just as it makes some real locomotion hard. It is well documented in the literature that users want to be able to move in any direction while gazing in a different direction. Locomotion techniques that do not allow this separation of view direction and movement direction are, in a sense, noncompliant with respect to best practices and ought not to be included in testing designed to find the most natural locomotion technique.
High-Fidelity Character Autonomy for Virtual Small-Unit Training
This work was presented using realistic dynamic behaviors and intelligent decision making based on situational context and the decisions made by other players. STTC is using the Joint Simulation Bus
developed by the Office of Naval Research (ONR) to allow connectivity with different game engines. The game environment is a potential constraint due to the product’s behavior (i.e., the conceptual model established for the game world). Another potential constraint came from the selection and use of a legacy system and architecture that had originally been developed based on different assumptions and technology than are currently available. Therefore, it was not clear how this product would benefit training.
Translational Neuroscience
Many of the challenges in the TN program represent opportunities to do more of what the group is already doing.
Brain–Computer Interaction Technologies
BCI technology based on neural recordings has emerged over the past 15 years as an important subfield of both TN and bioengineering. This is evidenced by the dramatic increase in the number of publications and presentations related to BCI techniques in neuroscience journals and conferences.
ARL’s mission focus on healthy soldiers poses an interesting challenge for BCI research, since the common goal of most BCI research is to treat or assist patients with sensory, motor, or cognitive disabilities. Hence, instead of using BCI technology for control as most clinically relevant BCIs do, the TN group has focused its efforts on the detection of mental states such as fatigue (state-based BCIs) or of external events such as relevant targets in a visual scene (event-based BCIs). Because their goal is to assist healthy soldiers in the field, the ARL researchers have resorted to the only currently viable noninvasive technology, namely, EEG electrodes placed on the scalp.
The BCI program is largely predicated on the assumption that computers and humans have complementary strengths (e.g., processing throughput versus higher-level reasoning and reliability and objectivity versus flexibility and situation awareness) and that hybrid systems leveraging BCI have the potential to achieve performance levels that neither a computer nor a human could achieve individually. This is a compelling notion, but it is not fully clear how the ongoing efforts and future plans specifically leverage, demonstrate, and validate it. There is the opportunity to do so, for example, by comparing the RAVEN system performance using RSVP-based BCI against state-of-the-art machine vision and automatic target recognition (ATR) algorithms. Such a comparison would show that the flexibility and situational awareness of humans greatly contribute to a computer system’s ability to detect and recognize targets and other items of interest.
While the performance estimate cross-validation work is technically impressive, it is important to note that these systems are effectively detecting performance (often with significant delay) rather than predicting it in a temporal context. This may not be inherently problematic, given that some applications may benefit from BCI-based performance detection if performance assessment is not possible through other means. The group’s claims to have the ability to predict poor performance before it begins to set in (e.g., early detection of fatigue that may lead to poor performance) need to be tempered. In point of fact, the current BCI system cannot determine if the brain is in a state that leads to poor performance or if the brain is just reacting to the poor performance (but see Baldassarre et al., 2012).5 The goal of this
_______________
5 A. Baldassarre, C.M. Lewis, G. Committeri, A.Z. Snyder, G.L. Romani, and M. Corbetta, 2012, Individual variability in functional connectivity predicts performance of a perceptual task, Proceedings of the National Academy of Sciences 109(9):3516-3521.
research is ambitious and laudable and will necessitate a deeper understanding of the brain response and/or a demonstration of true temporal performance prediction.
Given that the BCI technology program is still in its infancy, the advancements in individual projects are impressive. However, the emphasis of the presentations on such projects made it difficult to appreciate the long-term vision of the program and the nested fundamental research questions and application goals that will be addressed as the program matures.
Reliability has become an important concern in the BCI field as is evidenced by the Defense Advanced Research Projects Agency’s (DARPA’s) program in reliable neural interfaces. For BCI systems to become widely used in clinical and nonclinical applications, it is crucial that electrodes continue to record stable signals from relevant brain areas for at least months and ideally for many years. This is a particularly serious problem for invasive electrodes, where foreign body reactions and the intracranial environment can affect signal quality and stability. For noninvasive EEG electrodes, the exact electrode placement can vary slightly every time the EEG cap is removed and reattached. The TN group needs to investigate whether its EEG-based BCI systems that have been calibrated for a particular subject can continue to detect relevant states and events over weeks to months without resorting to recalibration of the decoding algorithm.
It is widely assumed that field potentials recorded from different EEG electrodes provide partially redundant information. Redundancy is important because it can buffer these BCI systems from catastrophic failure and allow for graceful degradation. On the other hand, redundancy may allow BCI systems to transmit data from a smaller number of electrodes, thereby reducing the bandwidth requirements, which may be particularly relevant for wireless transmission. The ARL needs to explore the degree of redundancy in its systems by examining state- and event-detection performance as a function of the number of electrodes used, much as researchers who use invasive electrodes perform “neuron-dropping” analyses.
On a related note, it would be useful to explore the spatial organization of information content across the scalp. Are there certain cortical regions that provide more accurate detection information than others? Is the information distributed evenly across the scalp? For example, there is evidence from single-cell recordings in nonhuman primates that neurons in the inferior temporal (IT) cortex modulate their firing rates to targets that are to be searched for in a complex scene in a visual search paradigm. Responses are enhanced in IT neurons that “prefer” the target, whereas responses are suppressed in neurons that do not prefer the target. Do EEG electrodes located over the temporal lobe provide better target detection capabilities than electrodes over the occipital, parietal, or frontal lobes? This suggestion to exploit the redundancy of multielectrode signals can be viewed as an alternative to source location, discussed above.
The applications of the BCI systems that were presented were limited to state- and target-detection requiring no more than 1 bit of information. It might be useful to explore opportunities to extract richer information content than can be available with EEG alone or together with other biosensor technologies. For example, would it be possible to detect multiple levels of fatigue, attention, and arousal? Could EEG systems be used to detect states associated with subjects’ ability to acquire information or to learn? Could the RSVP target detection system be expanded to detect more than one target class? For example, an operator might be looking for two different types of aggressors in a visual scene and respond differently to each.
Although their target detection system based on RSVP is quite impressive, it will be important to validate it against several control conditions to ensure that the improvement in search speed is attributable to target detection from the EEG system. One control condition would be to compare search speed using randomly sorted images that are not sorted via the EEG system. Another control condition would be to compare search speed using different machine vision and automatic target recognition algorithms
to perform the sorting of images with potential targets instead of the EEG system. Such comparisons would help validate the larger claim that hybrid human–computer systems, leveraging their complementary strengths, can perform better than either system alone.
Real-World Neuroimaging
The TN effort to develop nonproprietary dry electrodes is a very challenging area wherein a breakthrough could significantly impact medical EEG, human factors, neuroeconomics, and neuromarketing and likely lead to important new applications. The integration of electrode technologies with thoughtful statistical analyses for the purpose of artifact detection and classification could bring important and valuable contributions.
The TN group has designed a very sensible balance of projects in the portfolio and has organized a strong international collaboration to help achieve the group’s goals. Among these goals are the following:
- Phantom head development. The work in support of the EEG phantom presents a good opportunity to perform standardized testing. The goal of having a real-world system requires the group to do viable real-world testing that extends to different levels of sweating and motion.
- High-risk dry-electrode project. Given the focus on dry-electrode pads as a key impediment, it would be useful to conduct analyses of the computational and energy requirements for a real-world system. For example, how much energy, processing power, communications capability, and data storage will be required? Furthermore, will new algorithms be required to handle the additional challenges of the real-world environment?
The current-generation dry-electrode system is in an early stage of development. The published time series from the electrode that was initially provided did not include measurements of brain waves. Fortunately, the panel was updated with sample brain wave recordings collected in real time. These indicated that while the overall stability of this dry electrode is impressive, the signals are very small. They detect large artifacts, and it is not evident that cortical potentials are being picked up.
The TN group needs to compare the scalp measurements of these electrodes with other active dry EEG electrode systems (e.g., Gtec medical engineering) to ascertain the advantages and disadvantages of different approaches. Should there be significant differences in the scalp data, say in the signal-to-noise of averaged event-related potentials measured using the different dry electrodes and wet electrodes, the group might further make measurements of impedances and signals, humidity and perspiration testing (i.e., salt bridges), electromagnetic interference, and a standard 10-20 system for comparison of topography with the wet system to check for antenna effects from high-impedance electrodes.
The group needs to continue to consider alternative electrode types and analysis and signal enhancement methods that can reduce artifacts from electrode movement. Measures for evaluating signal quality need to be developed, and the TN group needs to ensure it is aware of and understands the lessons learned from prior work in this area.
Relevance to Protection from Traumatic Brain Injury
While the ARL is focused on the performance and protection of healthy individuals (it has noted that medical conditions are outside its mission), the problems that the group does focus on are relevant to performance and life-threatening situations that are commonly encountered by Army personnel and that are poorly understood. For example, a major threat to the performance of military personnel, during
peacetime as well as wartime, is traumatic brain injury (TBI), particularly mild traumatic brain injury. Reports of soldiers recently returned from combat in Iraq found that 22.8 percent had sustained a TBI and that most of these were mild. The TN group has the potential to model and predict which areas of the brain are most susceptible to various mild TBIs and can in turn use these data to help guide the design of protective gear to militate against these injuries.
Obviously the work by TN could be of significant value with respect to identifying areas of TBI. For example, moving this work beyond impacts that might result from a blunt object striking a forehead could lead to techniques for identifying areas of brain damage when the injury is not detectable by routine imaging.
Soldier Performance and Human–System Integration
The portfolio of research is much applied and is very relevant to current Army needs. These needs could be better balanced by the addition of basic science efforts that get ahead of the current requirements. Current applied research often depends on past fundamental research. Therefore, strategic investments in basic science today can provide the options for responding to future requirements. Perhaps the relationships built through customer-driven work can be leveraged for support of the science needed to solve emerging and potential future problems.
In a similar vein, one would hope that there is an appreciative yield from past investments in research that is sufficiently mature for transition. However, the linkage between the research and applied functions of ARL human sciences is not apparent. For example, the Army’s manpower and personnel integration (MANPRINT) program could provide an accessible transition path for maturing human-science-based technologies—for example, from the soldier performance and equipment advanced research (SPEAR) and EAR facilities.
ARL could better capitalize on its historic role over many decades of investing and innovating in military human sciences and engineering. It is generally accepted in the research and development field that past investments in fundamental research have led to important applications that were not anticipated at the time that the work was undertaken. It is quite probable that current proposed game changers—for example, programs sponsored by the Defense Advanced Research Projects Agency (DARPA) and the Iron Man suit program—have been enabled by specific advances gained from ARL’s legacy of research and technology investments. An exercise that identifies and maps these linkages could be a worthwhile exercise for proposed new investments in fundamental science.
Generally, the ARL portfolio, as presented in this review, represents good, solid research, development, and applications. However, there were no individual efforts that stood out as especially innovative and exceptional.
Sensory Perception
The reported increase in use of the EAR facility by external parties involves some limited collaborative efforts between ARL staff and external investigators.
With a few exceptions, evidence of progress was given in the form of ARL internal reports, papers from meeting proceedings, and papers in preparation or in review. This group needs to be challenged to publish more of its work in mainstream, peer-reviewed scientific research journals, which is probably the best indicator of the quality and significance of the research.
The paucity of publications outside of ARL internal reports also causes the research of this group to appear somewhat insular. Reinforcing this view is the apparent lack of collaboration among researchers
within the group. As described, the work on vision is largely conducted independently of the work on audition, and the work on audition is largely independent of the work on tactile modality. While this is a natural requirement of certain specific research questions, the broader charge to this group requires significant interaction among these sub-specialties. It is also worth noting in this regard that the reported collaboration with investigators outside ARL often involved no more than participation on a student’s dissertation committee.
The restrictions on conference attendance (except for postdoctoral fellows) impair the ability of the working scientists from gaining prepublication knowledge of what is being done in their field or from obtaining important feedback on their current efforts. A seminar series bringing in outside speakers shows progress toward solving this problem; however, attendance at either a generalist or specialist meeting can, in a few days, expose the scientist to dozens of new discoveries and ideas, well ahead of publication in the literature.
Several projects were led by engineers who did not have formal training in human experimentation, and this significantly affected the quality and interpretability of the work. Experimental psychologists (or experimental researchers from allied fields) need to be part of the design process from the start, either as team members or as internal consultants.
Given the complexity of the charge given to this group, future progress will almost certainly require a significant multidisciplinary, collaborative approach involving individuals with expertise in the areas of engineering, physics, psychology, and the social sciences. The ARL’s open campus initiative appears directed to this goal, and the sensory perception researchers may stand to benefit by doing what they can to take advantage of this initiative.
There is expressed interest in and there have been efforts toward integrating other sensory modalities into this group (e.g., olfaction). Given the complexity of the environment surrounding the modern warfighter and the wealth of current knowledge regarding multimodal interactions, these efforts need to be amplified; to this end the team is encouraged to seek out experts in areas that are not represented or are underrepresented at HRED. The efforts of a group working on multisensory cybernetics appears to hold promise for progress in this area, but it will be important that they are integrated at early stages with the researchers conducting fundamental sensory studies.
The International Multisensory Research Forum provides a platform for scientists from around the world who are interested in how different senses interact and how their input is integrated to communicate with one another. There are numerous laboratories training graduate students and postdoctoral researchers in this area, and it would be beneficial for ARL to consider hiring a few such individuals who could be positioned to work with and across the auditory-visual-tactile (and eventually, olfactory) research groups.
Physical and Cognitive Performance
Understanding the Effects of Physical Load
A number of high-quality research projects are being conducted in this area. Participating researchers are enthusiastic and feel that they are making a difference in improving soldier performance. There are a number of issues that are not currently addressed, but the team is well aware of the challenges ahead. For example, the portable measurement system used to estimate the ground reaction force can generate a huge amount of data that makes it impossible to analyze data manually and effectively. There also appear to be some limitations and concerns with the use of the CAVE for testing: tasks performed in the CAVE facility suffer from more errors and poorer performance than tasks performed on the computer
or in the real environment. This was most apparent in the presentation describing a study on the effect of physical load and environment on soldier performance, and it raises questions about the utility of a two-dimensional (2D) display for extrapolating effects to real-world soldier environments.
One major obstacle in conducting physical load experiments is securing soldiers to participate in the research program. Although the group claims to have maximized the use of available soldiers for this purpose, the low participation rate makes it difficult to understand the entire spectrum of the soldier population in terms of age and gender. There are alternatives recommended to mitigate this effect. For example, if soldiers have other duties and cannot come to the laboratory, maybe a portable laboratory could be designed that allows researchers to acquire the data in the field. Additionally, attempts could be made to conduct research in parallel to gather as much data as efficiently as possible using the limited available soldiers. For example, estimation of vertical ground reaction force from accelerometers can be completed in parallel with measurements of horizontal ground reaction force, thereby enabling one set of experiments to obtain data for both projects.
Under the current Army command system, medical-injury-related research projects are not part of the ARL mission. Peak performance and injury can sometimes be separated by a thin margin. For example, a tired soldier may not be able to effectively use his/her muscle to protect his/her bones and joints, and so it becomes easier to sustain an injury. Setting such constraints could waste resources when trying to identify the source of injury. It can be imagined that the current group can only exercise a soldier to a fatigue stage, while the medical research group may push slightly further to identify the threshold of injury. The Army would benefit from more collaboration between ARL researchers and Army medical teams to reduce waste and identify physical markers as indicators for injury prevention.
In order to disseminate the knowledge gained in the current group, several outreach activities are being done with other teams within the Department of Defense (DoD) and with international organizations. The list of activities is impressive and deserving of continued encouragement.
Understanding the Effects of Cognitive Load
Soldier performance as a product of biodynamics and cognition is an important way forward. To succeed using this paradigm, it would be very helpful to have stronger theoretical frameworks from cognitive science for developing hypotheses and for generalizing results to new situations. While highly focused experiments are useful for solving practical problems, it is difficult, if not dangerous, to generalize to other situations and problems without the guidance of theory to identify how new factors interact. The experiments described during the review were narrowly defined by customer-driven military applications; more basic research is needed to prepare for solving future problems.
Therefore, the research program needs to effectively leverage theories from cognitive science and higher level cognition (e.g., decision sciences). In particular, two areas could be useful for theoretical guidance. One is the work on decision models that provides a theoretical basis for predicting the relations among choice accuracy, decision time, and confidence. This refers to the extensive work on what are called sequential sampling and accumulation models of decision making (also known as random walk or diffusion models of decision making). The second theoretical area that is important to consider is perception action models of motor movement that arise from the ecological psychology work. It would also be worthwhile to integrate more work on team decision making.
More broadly, the program needs a bigger and stronger vision for organizing the next generation of advances, and the researchers need continued encouragement to be ambitious and attempt more dramatic improvements—for example, bigger advances in the improved performance research integration tool (IMPRINT) program and more empirical validation of the integrated system predictions.
Equipment and facilities available to the group are very good and are fully utilized for ongoing projects. However, such facilities are also available in a number of university settings, and the group may want to consider the acquisition of high-end facilities to establish its exceptional quality as a leading laboratory. For example, the motion analysis system could be equipped with even faster speed of motion data acquisition to better understand impact. Also, augmenting the 2D virtual reality with a three-dimensional (3D) system could be very useful for advancing research in this program.
Human–System Integration
The HSI group needs to develop a clear scientific vision and charter that clarifies responsibilities and delineates a path for future research. This vision would best be developed in parallel with the current mission of addressing customer needs by identifying opportunities where specific customer needs can drive more general research questions.
There appears to be a gap in expertise in key areas within HSI. For example, the HSI group could benefit from expertise in theory and modeling in both cognition and complex systems. In general, some of the work could greatly benefit from a stronger link to current research. Some research areas (e.g., adaptive interfaces) have evolved tremendously over the past 5 years; keeping up to date on ongoing research is vital to avoid unnecessary duplication. For example, the work on development of the next generation of adaptive interfaces seemed to be quite preliminary and conducted in a very insular manner. It does not appear to take into account numerous recent studies on adaptive interfaces.
It is important that HSI researchers continue to leverage the expertise of postdoctoral researchers, other ARL colleagues, and other research institutions to fill gaps in the training and knowledge of their research staff.
ARL has the opportunity to be on the forefront of the research in this area, and, for the most part, the researchers are doing very interesting work. However, the current portfolio of projects within the HSI area may be too customer-driven. ARL could leverage this applied work and/or fund companion projects to advance the science base for HSI as well as broaden the impact of the work beyond the immediate customer. Specific customer needs will often have broader research questions embedded within them, and effort is needed to identify and address them to the extent possible within project and funding constraints.
In the past, ARL has been a leader within the military services in advancing HSI, specifically in the development of the MANPRINT tools and techniques. Continuing to invest in this area can help the HSI group to be on the forefront of development of modeling and analytical tools to support effective human systems integration. In particular, there are opportunities to leverage the specific applied work they are engaged in, so as to (1) advance the behavioral science that is needed to inform HSI issues (e.g., innovations in adaptive interfaces; tasks that may increase the likelihood of injuries); (2) perform the fundamental work needed to establish the validity of the models and tools they are developing; and (3) expand the impact of the work within and beyond their immediate Army customers (e.g., via development and validation of more generic models or guidelines).
IMPRINT
A compelling scientific vision is needed for the IMPRINT tool. The existing work appears to be piecemeal and customer-driven. IMPRINT is an important tool that has the potential to contribute significantly to the HSI community. For instance, there are currently very few quantitative tools for HSI that can provide detailed information on cognitive performance. To realize its potential, however, the
IMPRINT group needs to broaden its base of expertise beyond systems engineers and human factors scientists to include additional interdisciplinary collaborators, based on the topic areas being addressed.
The IMPRINT tool needs to have the capability to measure and model cognitive performance beyond speed, accuracy, and cognitive load. There is a specific need for more detailed models of cognitive performance in many potential areas of application. For example, modeling the decision to switch from an M4 to an M14 weapon could clearly benefit from understanding the cognitive requirements of tasks beyond mental workload (e.g., decision making, situation awareness, and communication). Many tasks, like this one, are predominantly cognitive in nature, and modeling them using IMPRINT in its present form will be limiting.
As part of the vision for IMPRINT, it would be useful to identify the most critical future research areas and develop a plan for validation for an overall model, not just separate plug-in modules. Studies that focus on improving IMPRINT and testing the validity of each component as well as the overall model would be very useful. Likewise, it might be useful to identify and address the more general research issues associated with federating models that contain different levels of detail and different views of a system—for example, IMPRINT and unified modeling language models. Given the limited resources within HSI to conduct more fundamental research in this area, bringing external experts together for an IMPRINT workshop could help chart a way forward for both future research and overall model validation.
In the project presentations researchers did not provide an obvious link between the traditional HSI tools (e.g., IMPRINT) and the survivability research (e.g., the operational-requirements-based casualty assessment [ORCA] model), but in discussions, several HSI researchers were able to elaborate the connection. More specifically, many injuries are related to the specific tasks, and HSI can contribute to this knowledge by identifying the tasks that have the greatest impact on performance.
The HSI group could collect and document successful case studies that show measurable impacts of their work with citations to other tools and human sciences research. For example, there is a paper from the Coast Guard that won a best paper award (the 2012 David Meister Award at the Human Factors and Ergonomics Society Meeting) with the use of IMPRINT, and such success stories could help showcase the value of IMPRINT.
The HSI group is commendable for continuing to refine and improve the usability of its models and tools such as IMPRINT and for using these tools to support specific Army applications, but the research being conducted could have broader impacts.
OVERALL TECHNICAL QUALITY OF THE WORK
Effective human system performance is essential to Army mission effectiveness, and ARL’s investment in quality research and development in the human sciences has potential for significant impact on the present and future Army.
The natural tension between the comfortable technology pull of customer-driven work and the disruptive potential of innovating through technology push needs to be better balanced toward impacting the Army of the future.
The core quality of the research presented, the capabilities of the leadership, the knowledge and abilities of the investigators, their productivity, and proposed future directions are continuing to advance on a positive vector. The facilities are, for the most part, state-of-the-art, and, in general, the researchers demonstrated outstanding and effective leverage and collaboration with the broader scientific communities.
Simulation and Training Technology
STTC has historic strength in computer science and engineering and has developed a number of successful technology-enhanced training products. The ARLTAB’s assessment recognized these accomplishments and also examined how the performance of the STTC might be improved by integrating additional scientific expertise, exposure to new or alternative scientific approaches, and tactical consideration and staging of project goals.
Overall, STTC has a clear and substantive mission with many important and unique objectives. The problems being tackled by STTC are large, complex, and important and have huge potential value to Army mission readiness and effectiveness. Technical problems include identifying, for example, how to develop tutoring systems that are adaptive to individual learners, how to best manage instructional experiences, how to make synthetic entities behave more intelligently in training simulations, and how to make training simulations more interoperable. STTC researchers are pushing the state of the art of simulation in the cloud and protocols for advanced distributed simulation (ADS). The ADS group has a unique opportunity to lead future developments across the DoD in these areas. The STTC work is demonstrably significant and valuable in specific application domains (e.g., simulations of battlefield medical situations), and the design of general tools for simulation (e.g., the generalized intelligent framework for tutoring [GIFT]) makes it possible for others to rapidly create new training modules for new content areas. Laudable progress has been made to date in the development of GIFT and in the incorporation into this framework of the computer game Virtual Battle Space II, now being used by the Army for training.
The Orlando-based leadership and scientific research groups exhibited a high level of professionalism, commitment to high technical standards in their projects, and a broad appreciation of their role in enhancing military and human outcomes. The esprit de corps and desire to integrate innovative and effective research strategies were notable in both research teams and leadership. Investment in professional development and training of junior scientists was a priority of the program. The STTC is an excellent research unit that embodies high technical standards and strong operational attitude.
Much of the work presented represents an interesting intersection of M&S with computer science. In this context it is important that computer science be recognized as broader than software development: It includes artificial intelligence, intelligent systems, parallel and distributed systems, social media, online learning communities, digital media and gaming, big data and analytics, computing performance, networking, graphics, and visualization. Many research groups at STTC tend to focus on a specific problem domain and simply and selectively use computer science disciplines to create better simulations. Owing to the breadth of work being done at STTC, the staff members are in a unique position to establish themselves as one of the leading research groups working at the intersection of M&S and computer science. While basic research is done by the ICT, STTC can still (and needs to make the effort to) integrate research from ICT and others, as well as innovate in the application of these technologies into real-world programs. This would require achieving a broader view of the intersection of computer science and M&S and creating a technology roadmap and strategy for how to accomplish this goal.
The research presented by STTC would, to varying degrees, benefit from integrating human science experts (e.g., human factors and cognitive scientists and social psychologists) into the research teams. For example, one of the goals of the simulation technology group is to push the envelope of simulator fidelity, whether it is the number of live users using the system or the physical realism of the system. Some of these issues are mainly technical, but in many instances they could benefit from human science experts. For example, knowledge about human perceptual limitations could steer simulator designs and, in turn, not waste bandwidth pushing unnecessary data. Conversely, it could also reveal areas where computational resources should be concentrated (e.g., in auditory realism). Related to this is the idea
of satisficing in simulator design. That is, what level of fidelity is adequate for satisfactory training to occur? Overall, some of the areas could benefit from a systems-engineering approach that views the simulator not just as hardware and software but as a complete system of interdependent elements that includes human operators.
The incorporation of human science approaches and human science professionals in the research teams and in the specification of internal standards could enhance the human science expertise of the researchers, augment functionally engaged scientific advisory arrangements, and expand training opportunities for both senior and junior scientists.
A strong related suggestion is that STTC work to improve the quality of its user study designs, including design of the data analysis. The studies it conducts are costly, and it is vital to be certain that they are both gathering sufficient data on the critical variables of interest and performing well-designed statistical analyses. Technologists are not trained experimenters and will likely benefit from expertise identifying proper independent and dependent variables, measures, and study design elements such as between- or within-subject, needed sample size, and elimination (or mitigation) of confounders. This is another area where closer engagement by human science researchers can pay off, because these subject-matter experts are generally well trained in experimental design and statistical analysis of both quantitative and qualitative data.
Translational Neuroscience
Overall, the quality of the research presented, the capabilities of the leadership, the knowledge and abilities of the investigators, their scientific productivity, and proposed future directions are impressive. The work is well aligned with the clear and substantive mission to move neuroscience from the laboratory to real-world military settings—that is, from the bench to the battlefield. The TN group conducts high-quality neuroscience research that is routinely validated by its publication in recognized, peer-reviewed journals and is on a par with work at a good university neuroscience department.
The TN program is a unique and important effort that is tackling key technology bottlenecks to moving neuroscience from the laboratory to the field. For example, they are exploring the integration of other sensing modalities—ECG, EMG, GSR, and eye movements—into EEG-based BCI applications; approaches to overcome real-world limitations for use of the electrode system so that it works with hair, slips on and off easily without significant setup, and has high enough sensitivity to capture the signals necessary for specific tasks; and the development of nonproprietary dry electrodes, which, if successful, could significantly impact medical EEG, human factors, neuroeconomics, and neuromarketing and could lead to important new applications.
The group leadership is highly effective and qualified, and there is a palpable energy and enthusiasm in the strong mix of early-career and mid-career scientists. The facilities are, for the most part, state-of-the-art, and the group demonstrated impressive leverage of and collaboration with the broader scientific communities at universities, industry, and other government laboratories.
Soldier Performance and Human Systems Integration
The quality of the research presented, the capabilities of the leadership, the knowledge and abilities of the investigators, and proposed future directions continue to improve. The facilities are, for the most part, superb, and collaboration with the broader scientific community is generally good. Overall, ARL has the opportunity to be on the forefront of the research, development, and applications in these areas for the DoD and, for the most part, is doing interesting work. The work is generally solid, but perhaps
a bit closer to turning the crank than being out on the cutting edge in the areas of interest. This is, no doubt, a function of the customer-driven nature of the projects that constitute much of the portfolio. ARL could leverage a portion of these assets for support of higher risk fundamental science that advances the state of knowledge and/or developments aimed at leading the way with challenging innovations.
