3
Human Research and Engineering Directorate
The Soldier Systems Panel visited the Human Research and Engineering Directorate (HRED) at Aberdeen Proving Ground, Maryland, on May 15-18, 2011, and June 19-22, 2012, and examined written materials provided by HRED. During those visits, the panel received briefings on aspects of HRED work, mostly in the 6.1 (basic research) and 6.2 (applied research) categories. This chapter provides an evaluation of that work, recognizing that it represents only a portion of HRED’s portfolio.
Overall Organization
Research in HRED is currently organized around two large, major laboratory programs: the Human Dimension program and the Simulation and Training Technology program. This chapter is organized along the seven research thrusts within the Human Dimension program:
1. Sensory performance. The sensory performance thrust is concerned with the sensory performance of individual soldiers and small teams as they carry out tasks that are critical to their military roles. The research is focused on the interaction of the dismounted soldier with his/her equipment and environment. Examples might include studying the impact of protective gear on the perceptual ability to detect, identify, and localize threats while also maintaining general situational awareness and the effects of devices intended to enhance natural sensory capabilities (for example, night vision devices).
2. Physical and cognitive performance interaction. Soldiers carry heavy (more than 100 lb) loads for extended periods of time (for example, 72-hour operations), often over rough terrain and while being inundated with large amounts of information. The physical and cognitive perfor-
mance interaction thrust investigates the effects of the interaction of cognitive load and physical and cognitive stress on the soldiers and on their performance.
3. Translational neuroscience. As Army operations enter battlefields that are more dynamic and complex, it will become increasingly critical to build translational capability into the design and development of systems that capitalize on the soldier’s neurocognitive abilities to meet the demands of these environments, ensuring mission effectiveness and maximizing soldier survivability. The goal of the translational neuroscience thrust is to enable system designs that are consistent with brain function, taking into account its limitations and exploiting its potentials, to maximize soldier performance.
4. Social-cognitive network science. The soldier on the modern battlefield is increasingly part of a network of humans and machines in rich communication with one another. Decision making is increasingly distributed and dispersed. This reality poses a set of questions about presentation and consumption of information that is the domain of the social-cognitive network science thrust area. These human and human-machine questions are related to questions about the structure and function of networks more generally. Consequently, the primary discussions of this thrust area will be found in the chapters of this report that discuss the Computational and Information Sciences Directorate and crosscutting research areas.
5. Human-robot interaction. An increasing number of unmanned, robotic systems with various degrees of autonomy are being fielded. Humans need to control these devices and to plan their activities. In addition, humans receive and interpret input from robotic devices. Optimal (or even merely adequate) use of these devices requires an understanding of human-robot interactions. Such interactions were reviewed as part of the autonomous systems enterprise work within the Vehicle Technology Directorate. Accordingly, this research thrust is examined in the chapter covering the autonomous systems enterprise.
6. Human-systems integration (HSI). The human-systems integration thrust assesses the ability of soldiers to work with new systems prior to their deployment. Much of this work is HRED’s contribution to the Manpower and Personnel Integration (MANPRINT) program, a formal Army program with the goal of ensuring that most if not all human dimensions are accounted for in the design, development, procurement, and life cycle management of all Army materiel systems. A leading HSI tool for MANPRINT work is IMPRINT (the Improved Performance Research Integration Tool), a modeling system based on discrete event simulation. IMPRINT is one of several tools used by HRED to produce early, cost-effective evaluation of constraints that can inform the determination of requirements for new systems.
7. Opportunity-driven human factors research. HRED scientists have the expertise to address novel problems and questions as they arise in the field. The opportunity driven research thrust is the home for research directed at specific problems faced by soldiers in the field or by equipment development programs. Opportunity driven research is typically customer funded and occurs throughout HRED.
8. Simulation and training technology. Simulation and training technology is a major laboratory program with five research areas: adaptive and intelligent training technologies, synthetic environments, immersive learning, training applications, and advanced distributed simulation. Training applications are domain-specific research areas and include medical, dismounted soldier, and ground platform training. The laboratory’s primary physical location is in Orlando, Florida, and it has a strong relationship with the Institute for Creative Technology at the University of Southern California.
The HRED mission in 2012 can be separated into three large parts. First, there is research intended to improve soldier performance in the future. This work involves research on humans and their capabilities and on human-machine interactions. The most basic and speculative aspects of HRED research fall within this area. Second, because HRED is a center of human factors expertise, it is in a position to evaluate the integration of soldiers and systems within the Army. Importantly, it can help to shape requirements during the acquisition process to ensure adequate consideration of the human dimension. Research within this customer-service aspect of HRED’s work is devoted to improving assessment tools such as IMPRINT. Third, with the inclusion of the Simulation and Training Technology Center (STTC), HRED is assuming an important role in improving the training of soldiers through the use of artificial training environments. STTC’s research involves the development of new simulation tools such as generalizable frameworks that speed development of specific training and simulation tools.
The wide scope of these research areas poses a central challenge to HRED. Given limited resources of time, personnel, and funding, what specific topics should HRED tackle? Of course there are always more topics of interest than can be addressed. Like other research labs, some of HRED’s choices are driven by funding. The ability to perform basic research is dependent on the continued availability of basic research (6.1) funds from the Army.
The Soldier System Panel was charged to assess the scientific and technical quality of the Army Research Laboratory’s (ARL’s) research and development related to its soldier systems mission and to examine how HRED’s work compares to similar work being done externally. As a result, the panel focused its attention on questions about the HRED research that might have an impact on the broader scientific community. This focus is not meant to denigrate specific work performed by HRED to solve specific problems with specific systems. Such work is a vital part of HRED’s mission, and the challenge lies in producing synergies between basic research and specific problem-solving work. By its nature, not all basic research will produce results that can be transitioned to Army-specific problems in the field. However, basic research can produce new classes of solutions. The answer to a customer’s human factors question may not be publishable in the scientific literature. However, a workforce, encouraged to think as basic scientists, may find that a specific task raises new topics for basic research. HRED’s leadership is tasked with steering a course than enables HRED to strike a balance between its basic research and specific, problem-solving missions.
CHANGES SINCE THE PREVIOUS REVIEW
The appointment of a new HRED Director in January 2011 has provided stability to HRED management. A senior scientist in neuroscience has also been appointed, which represents an important milestone in the development of the translational neuroscience thrust in HRED.
In terms of facilities, the most significant change has been the opening of the Soldier Performance and Equipment Advanced Research (SPEAR) facility. This facility, with its instrumented obstacle, cross-country courses, and biomechanics laboratory, gives HRED important new capabilities for measuring soldier performance under realistic conditions. With regard to staffing, the recruitment of early-career, clever scientists at a time of little or no growth is commendable.
A previous report noted that some research areas were governed by a clearer scientific vision than were others.1 During the past 2 years, HRED leadership has made it a priority to develop a clear vision for each of the research thrust areas. Although a work in progress, this decision reflects a noticeable change since the previous review.
There is increased evidence of good contacts with the broader scientific community through presentations at meetings and publications in the peer-reviewed literature. Moreover, the quality of presentations has generally improved, suggesting a steady improvement in the research. Additionally, the background and hypotheses motivating individual research projects are more clearly stated, and the connection to scientific research beyond HRED is documented more frequently.
The presentation of data and the use of statistics have been improved, although statistically underpowered studies and questionable statistical analyses remain. A more subtle issue is the distinction between statistical significance and scientific or practical significance. It is possible to have statistically reliable results that are, nevertheless, of little consequence, which has not been clearly recognized. At the directorate-wide level, this represents a locus of intervention, where senior, experienced researchers may be able to provide useful mentoring to less seasoned staff.
Quite a few projects seem to be stand-alone studies; that is, they are initial experiments of what would be a series of studies if substantial publication were to be the result. This may also be a place where scientific mentoring by senior researchers would be helpful. The data being gathered by the opportunity-driven research represents an opportunity to close the loop between basic and applied research and soldier system problem solving. To support this effort, the hiring of more senior, experienced scientists should be complemented by the better use of available field operations reports, and even medical reports, to prioritize studies and plan how they should be executed.
Changes Since the Previous Review
The most notable positive change in the sensory performance area has been the progress toward a strategic vision. The most notable challenge is the need to develop a program of basic and applied research that implements such a vision.
Accomplishments and Advancements
In its past two reports, the ARLTAB stated that the sensory performance area lacked clear direction.2,3 Although the direction is still not entirely clear, progress is visible. The sensory performance thrust focuses on the individual soldier and on small teams. The thrust’s main areas of work are in auditory and visual performance. There is a clear understanding that HRED’s work should concentrate on soldier-critical sensory performance capabilities, including such issues as the ability to detect threats (e.g., improvised explosive devices [IEDs]), interpret the intentions of bystanders, and communicate without
___________________________
1National Research Council. 2010. 2009-2010 Assessment of the Army Research Laboratory. Washington, D.C.: The National Academies Press, p. 33.
2National Research Council. 2010. 2009-2010 Assessment of the Army Research Laboratory. Washington, D.C.: The National Academies Press, p. 43.
3National Research Council. 2010. 2007-2008 Assessment of the Army Research Laboratory. Washington, D.C.: The National Academies Press, p. 34.
being seen or heard. All of these goals should be accomplished in a sensory environment shaped by the soldier’s gear (e.g., protective equipment).
The sensory performance portfolio consists of three types of research: (1) basic and applied research to understand sensory-perceptual capabilities, (2) human factors research to understand the impact of specific equipment on performance, and (3) developmental work to examine new approaches to sensory enhancement. Although the primary focus is on the optimization and enhancement of our perceptual capabilities, efforts are under way to develop methods of manipulating perceptual information, reducing or altering the perceptual information available to others, and thus allowing the soldier to operate with stealth capability. More recently, a refinement of this vision has appeared under the rubric of “owning the environment.” Here, the emphasis is on use of stealth and deception as force multipliers. Through better equipment and better understanding of the perceptual capabilities of friend and foe alike, it may be possible to reduce threats to our soldiers while increasing their effectiveness. Although in an early stage of study and not yet giving rise to a specific research program, the concept of using stealth and deception to own the environment is an interesting direction for work in sensory performance. The sensory performance group’s state-of-the-art Environment for Auditory Research (EAR) facility enables very sophisticated basic and applied auditory research.
The clearest examples of accomplishments in this area are human factors projects, such as work on the effects of military headgear on auditory performance. The wide range of helmet designs results in differing effects on the ability to detect and localize sounds. The EAR facility is well-suited to conducting research in this area, with the eventual goal of developing a predictive model that can influence future designs. This is a good example of a scientific project that could transition back to an evaluation tool like IMPRINT.
Another application of basic sensory science to an Army setting is the work on the ability to recognize different types of small arms by the sounds they make when fired. Although it is less clear whether this application will result in a generalizable piece of new scientific knowledge or an obvious transition to a practical application, soldiers will benefit greatly from improved ability to identify what is being fired in the vicinity.
Further examples of potentially useful human factors and assessment work within this area involve the comparison of different interfaces for a missile defense system and the study of oculomotor indicators of workload in the complex setting of a helicopter cockpit.
Opportunities and Challenges
Impressive progress has been made in the sensory performance area, but many significant challenges remain, including the following:
• The vision for sensory performance research is still a work in progress. For example, it is not clear whether “owning the battlefield” will replace earlier formulations or simply be added to the mix. HRED understands that its strategic vision in this regard is still evolving.
• Once in place, there needs to be a major effort to align the strategic vision with the work. There will be legacy projects that will end, and new research areas that will need to be developed.
• HRED’s best work falls under the category of human factors evaluation. Although the evaluations of various off-the-shelf systems can provide an important service to the Army, as a general rule, they do not connect to progress in basic science. Given HRED’s laboratory capabilities, one would like to see a research loop in which an evaluation task raises basic science questions. Those basic science questions should be studied in a rigorous manner, and the results should be
published in the open scientific literature whenever possible. In addition, those results, along with other findings from the basic science community, could be used to support more applied research to shape the development of the next generation of systems. All of the pieces of this circuit can be found in the sensory performance area, but, as noted, most of the success seems to lie in the first—evaluation—step, and there is little evidence of programmatic coordination between these steps.
• Some of the work, although well-intentioned, is not methodologically strong. For example, an interesting project on IED detection made good use of anecdotal accounts from soldiers in the field but never managed to convert those accounts into hypothesis-driven research with the statistical power to draw conclusions.
• Several projects have not been influenced by the relevant scientific literature. For example, a project on detection of muzzle flashes in the periphery would have benefited from more consideration of the vast clinical literature on detection of targets in peripheral vision.
• Overall, the empirical work is dated; that is, it is more typical of psychophysical work conducted 20 years ago. There is room in the basic science work for greater use of naturalistic stimuli and field data derived from opportunistic research, more consideration of the role of attention, and more modern theoretical constructs.
• The state-of-the-art EAR facility continues to be underutilized and lacks a world-class program of basic research. Here, and perhaps elsewhere (for example, in the area of visual perception and attention), it might be worth considering bringing in a senior researcher from academia on a temporary basis (e.g., via the Intergovernmental Personnel Act [IPA] mechanism) or permanent basis to help shape such a program.
The previous four items present an opportunity to reach out to the broader community in studies of sensation, perception, and attention. Funding vehicles such as the Army Research Office (ARO), multidisciplinary university research initiative (MURI) system, or the Collaborative Technology Alliance (CTA) vehicle could be used to foster productive relationships between those working in the basic sensory performance world and those interested in application of research to Army problems.
Overall Quality of Research
In general, the strongest work in the sensory performance area is customer-driven evaluation of equipment. The group is not yet a force in basic or applied research on sensory performance.
PHYSICAL AND COGNITIVE PERFORMANCE INTERACTION
Changes Since the Previous Review
There have been two notable changes since the previous review. The first is an evolving focus on the interaction of physical and cognitive factors in soldier performance. The second is the completion of the SPEAR facility, giving this research thrust a particularly strong set of tools for research. Current physical and cognitive performance interaction research consists of three thrust areas aimed at quantifying the effects of soldier equipment on physical and cognitive performance and their interaction: (1) Developing new measurement methods/devices for collecting cognitive and physical performance data in operationally relevant environments, (2) Employing traditional and new metrics to quantify the
effects of soldier equipment on soldier performance, and (3) Applying the new metrics to understanding the effects of physical and cognitive load on soldier performance over longer duration activities.
Accomplishments and Advancements
Biomechanics has long been at the core of HRED’s mission to examine the physical limits on soldier performance. HRED has developed a very strong set of facilities for studying soldier performance under various approximations of the real-world situation. Field studies are extremely valuable for their realism but are not well suited to well-controlled, basic research studies. For such studies, HRED has multiple research venues for examining the interaction of physical and cognitive stress on soldier performance. These facilities include first-person gaming facilities for isolating the effects of cognitive stress (such as the C4ISR laboratory) and a state-of-the-art biomechanics laboratory for investigating the physical effects of load. The Tactical Environment Simulation Facility (TESF) is a fully immersive simulator that allows research subjects to walk through and interact with their environment. The M-Range Shooter performance research facility is a live-fire range capable of providing real-time high-fidelity data on marksmanship performance. The newest facility, the Soldier Performance and Equipment Advanced Research (SPEAR) facility (opened in the summer of 2012) combines HRED’s biomechanics laboratory, outdoor obstacle course, and a WiFi-networked, 2.5-mile cross-country course that goes through the woods of Aberdeen Proving Ground. This array of facilities allows HRED a high level of experimental control if needed (i.e., in the C4ISR or biomechanics lab) and a high level of operational relevance if needed (i.e., in the SPEAR).
With these tools in hand, the task becomes deciding where to focus research effort. HRED has identified a niche that it is, perhaps, uniquely suited to study—the interaction of physical and cognitive stress on soldier performance. HRED notes that not only are soldiers required in modern missions to carry very large physical loads (in excess of 100 pounds), but also they need to do so in ambiguous, yet threatening environments (such as patrolling potentially hostile city streets), while making split second, life or death decisions. Few research facilities are capable of supporting the study of this combination of factors; this capability is a strength of the physical and cognitive performance interaction research thrust.
Because considerable effort has been devoted to the development of these new facilities, it is not surprising that a significant portion of recent work has involved validation of devices including the omni-directional treadmill, the force plate treadmill, and aspects of marksmanship measurement.
During the past year, the group has started to demonstrate how it can use its tools in work that moves beyond validation. For example, if the laboratory has multiple ways to test the effects of physical and cognitive load, then does it matter if you test in the real world or in a simulator? The answer from one recent study is that this does matter and needs to be taken into account. World-class, one-of-a-kind facilities notwithstanding, another recent project showed how to make clever use of off-the-shelf tools such as the Microsoft Kinect sensor and the Sony Move system.
The work on hand carriage of load is a good example of the sort of classic “bread and butter” biomechanics work at the center of HRED’s mission to understand the demands placed on soldiers.
Opportunities and Challenges
The physical and cognitive performance interaction research group has a vision for a research program that exploits its unique and impressive set of available facilities. The challenge chiefly lies in implementing that vision while responding to more traditional requests for support, which is recognized in the current research plan. The group plans to continue to employ traditional methods to quantify the
effects of specific equipment on soldier performance. At the same time, it plans to develop new methods to study both cognitive and physical data and to do so in the more realistic environments that are now available. The goal is to use these new metrics to study and understand how physical and cognitive loads impact physical and cognitive performance in activities over an extended period of time.
This group is in a good position to make progress in the next few years. It faces some challenges, many of which it has identified itself:
• Because the world of possible research is vast, the group needs to be strategic in choosing its work. Possibly these choices should be driven by closer contact with infantry and armor operations. More generally, continuing broad collaborative contacts with both the Army and the broader scientific community will help to assure that the research program does not become insular.
• It is obvious that research of this sort requires a steady stream of human volunteer participants. Less obvious is where those human subjects will come from. This is a general HRED issue, but it is particularly salient in the physical and cognitive performance area, because the specialized equipment and facilities require that subjects go to Aberdeen. The group has made some excellent efforts at outreach to different groups of potential subjects, in effect trading participation for benefits that those groups can obtain from the use of HRED facilities. Nevertheless, this is a chronic problem that deserves continuing thought by HRED management.
• Related to both of the preceding points, the group should continue its efforts to study female soldiers, because the patterns of response to physical and cognitive stressors are likely to be different in male and female populations. Male and female may not be the only grouping that should be considered. The potential for other relevant subgroups underlines the need for an ability to recruit substantial numbers of research subjects.
• Here and elsewhere in HRED, more systematic thinking about what constitutes a significant result would be valuable. There are two relevant senses of significance. The first is statistical significance; HRED researchers have been improving markedly in their statistical analyses of data. The second is the significance of the finding to the Army and/or scientific community. Sometimes (as in public health research), very small effects can be both statistically and practically significant. Other times (as occurs quite frequently in cognitive research), it is possible to achieve statistical significance without having much scientific or practical impact. When planning for research in the physical and cognitive performance area, it would be valuable to try to determine how large an effect would be needed to be interesting as either a basic or applied result.
• Some studies appear to terminate without discussion of further research avenues. Ideally, clever, successful, single experiments would give rise to more programmatic series of experiments.
Overall Quality of Research
The research of the physical and cognitive performance interaction research group is generally of high technical quality. To date, its scope has been somewhat limited, but its work may be viewed as preliminary to a successful, coherent program of research based on a developing vision and an excellent suite of deployed experimental platforms.
Changes Since the Previous Review
Two changes are noteworthy. First, the program is now referred to as “Translational Neuroscience” rather than simply “Neuroscience.” This is not mere jargon, but reflects a continuing refinement of HRED’s goals in neuroscience. The goal of the translational neuroscience program at HRED is to integrate modern neuroscience with human factors, psychology, and engineering to enhance our understanding of soldier function and behavior in complex operational settings. Neuroscience is a vast field, and it has been imperative for HRED to concentrate its efforts on specific problems with potential Army relevance. The second notable change is the arrival of a senior scientist in neuroscience, who is also an ARL Fellow and whose background brings expertise in computational methods of signal analysis and neural modeling to HRED.
Accomplishments and Advancements
Over the past 2 years, translational neuroscience has made impressive strides at HRED. At the time of the previous review, the neuroscience group was in its promising infancy. Progress during the past 24 months, driven by strong leadership, provides a model for how a new group should be developed at a government research laboratory. During the past 5 years, ARL’s neuroscience group has grown into the Department of Defense’s (DoD’s) largest internal nonmedical translational neuroscience research effort. A central task during this period has been to refine the neuroscience group’s vision in a way that allows it to be Army-relevant while continuing its trajectory toward being a neuroscience laboratory on a par with strong university research programs.
To develop this Army-relevant, translational research capability, the group has concentrated its efforts on three internal research thrusts:
1. Brain-computer interaction technologies. What kinds of neurotechnologies have potential for broad DoD and civilian impact? Where is basic research required before translational research will be possible?
2. Real-world neuroimaging. What aspects of brain function can be usefully monitored outside of the laboratory setting? What are the technologies that are best adapted for this purpose? For example, would it be practical to record aspects of EEG from a soldier as he/she is driving on a mission? Would there be reliable biomarkers for stress or fatigue in the signals?
3. Individual differences and neurocognitive performance. Assessing the different capabilities of soldiers is important if those soldiers are to be optimally matched to tasks. What is the role of neuroscientific measures in assessing individual differences?
There are a palpable energy and enthusiasm among the strong mix of early-career (HRED has added more than 10 postdoctoral researchers) and mid-career scientists. They seem to work well together and seem quite well connected with academic neuroscience research labs. As noted, above, a senior researcher has now been added to this mix. Connections to the broader scientific community include partnerships with more than 20 national and international universities and companies through programs including the Cognition and Neuroergonomics Collaborative Technology Alliance (C&N CTA) and the Institute for Collaborative Biotechnology (ICB).
The neuroscience group is to be commended for its excellent publication record. While concerns about publication have been raised in the review of other HRED groups, the neuroscience group has
maintained an excellent publication record that would be appropriate at a good academic lab. The group has published at least 20 journal articles and 10 technical reports over the past 2 years. Moreover, the group is publishing in strong, peer-reviewed journals.
An example of the type of work that defines this group is that on detection and classification of artifacts in EEG signals. Typical EEG research is performed in quiet rooms with stationary subjects who have been recruited into the EEG study. If one anticipates recording EEG from soldiers who are moving through a stressful environment and who have other things on their mind besides data quality, then those standard laboratory conditions will not apply. The recording environment introduces substantial artifacts into the EEG. The goal of this research is to detect and then acknowledge and/or remove the artifacts so that the underlying signals can be interpreted. Although not presented as a definitive solution to this very substantial problem, the group’s work represents a good start on developing methods that would be sufficiently computationally simple to permit online use. The expertise of the new senior scientist in modeling EEG data should be valuable in projects of this sort.
Opportunities and Challenges
In the case of the translational neuroscience program, many of the opportunities and challenges consist of recommendations to do more of what the group is already doing. The translational neuroscience group should:
• Continue to build outside collaborative networks.
• Continue to grow internally, by adding expertise in computational modeling.
• Work to develop methods to remove as well as detect EEG artifacts.
• Incorporate neural source modeling (generator localization methods).
• Consider multiple measures of functional connectivity (e.g., phase coherence, phase lag, and Granger causality).
Other groups within HRED and elsewhere in the Army are doing relevant neuroscience. Improving communication with those groups would assist this group in its work. The translational neuroscience group seems to be the obvious locus of leadership for all HRED-relevant neuroscience research.
Although the arrival of the senior scientist has associated benefits, there is an implicit challenge to integrate a senior investigator into HRED in a manner that will forward his goals as well as the goals of the translational neuroscience group.
Overall Quality of Research
The translational neuroscience group’s research is on a par with work in the university community. This work is at the level of a good, university neuroscience department. This is not to say that each project will transform either fundamental or applied neuroscience research. However, the group conducts high-quality neuroscience research that is routinely validated by its publication in good, peer-reviewed journals.
SOCIAL-COGNITIVE NETWORK SCIENCE
Changes Since the Previous Review
During the past 2 years, the social-cognitive network science group has worked to refine its vision. Concretely, this has involved a reduction from five to two thrusts in sociotechnical network operations and network-enabled cognition and an increased effort to align work with that of the ARL’s network science program. Consequently, in 2012 this area was examined as part of a review of the ARL network science program, managed by the Computational and Information Sciences Directorate. Discussion of that review can be found in Chapter 2 of this report.
Accomplishments and Advancements
The social-cognitive network science group is working on a significant set of Army-relevant problems. Soldiers will be increasingly making decisions as part of a dispersed and distributed network. Optimizing performance in these settings is an important goal. Moreover, the way in which soldiers interact with networks is a topic that is clearly related to HRED’s core interests in human-systems interactions. Connecting more closely with the broader network science endeavor can serve to keep everyone focused on the reality of the human element in most of these networks. There is a growing network aspect to the core HRED mission of human-systems integration. Some aspects of the research program have great promise. For example, the ability to study networks during field exercises is an opportunity that is unavailable to most other scientists, and the group has made some interesting progress with this approach.
Opportunities and Challenges
The following are central concerns with the work in this area:
1. The vision for the group remains a work in progress. Social-cognitive network science (like any of the HRED research thrusts) is a vast topic, and choices should be made about where to devote research effort and where to build staff expertise. HRED’s answer to this concern has not crystallized for the social-cognitive network science area.
2. Concerns about the quality of the research in this area remain. Good work is being conducted, but too many presentations of the work suffer from problems such as:
a. Unclear goals. Why was this specific problem deemed worth study? What is the theoretical framework?
b. Weak statistical analysis. This comment is based on observations during the 2011 briefing. Social-cognitive work was not briefed to the panel in 2012, and so this report does not present evaluation of progress on statistical analysis that may have occurred.
c. Some studies seemed underpowered. For example, sweeping conclusions cannot be based on comparisons of two teams that differ from each other on a variety of variables.
d. Many studies seem to be one-shot studies that do not build a cumulative body of knowledge.
With these challenges come opportunities. The refinement of the vision of this group may make it clear that the group needs to bring in new expertise. Addressing the challenge of network science may require staff with backgrounds different from what is currently available in HRED. Some of this expertise can come from external partners, for example, via the CTA process. It will be important for this group to interact closely with the external researchers working on related issues.
Overall Quality of Research
The overall quality of the research in this area is uneven. There is some fine work but much work will have low impact, either because it is methodologically weak or because it does not appear to be part of a systematic program of work. Working more closely with the opportunity-driven researchers could assure that the work in social-cognitive networks is well grounded and relevant to field operations.
Changes Since the Previous Review
Incorporation of the impact of social and cultural influences into human-systems integration is a new departure for this group.
Accomplishments and Advancements
HSI represents one of HRED’s core competencies—providing the Army with assessments of how humans will work with new systems. The HSI program works to develop tools to model human performance and to evaluate performance tradeoffs that will occur when factors such as cost, size, weight, user interface, and other factors are changed. By using effective human performance modeling tools early in the acquisition process, potential performance problems can be identified and resolved sooner resulting in better-designed systems for the warfighter. This work requires a combination of knowledge of the Army materiel acquisition system and of the constraints on human performance. Ideally, if effective human performance modeling tools are used early in the acquisition process, potential performance problems can be identified at this early stage, and expensive, potentially dangerous problems can be avoided. HSI analysis might, for example, offer informed opinions about the numbers of crew required for a vehicle, or it might determine whether increasing the armor on that vehicle reduces the ability to navigate in an unacceptable manner.
Currently, three main research thrusts support the HSI group’s evaluation mission:
1. Development of tools. The HSI goal in tool development is to develop or improve tools that allow early evaluation of human-systems integration. MANPRINT is the Army’s programmatic implementation of HSI. MANPRINT is a set of human factors requirements intended to optimize total system performance, reduce life cycle costs, and minimize risk of soldier loss or injury by ensuring a systematic consideration of the impact of materiel design on soldiers throughout the system development process. IMPRINT is HRED’s primary tool supporting MANPRINT. HRED describes IMPRINT as a dynamic, stochastic, discrete-event network modeling tool designed to help assess the interaction of warfighter and system performance throughout the system life cycle—from concept and design to field testing and system upgrades.
2. Analysis methodology. Early, rapid, cost-effective MANPRINT analyses are the traditional heart of the HRED HSI enterprise. HRED works to improve this process by increasing the coordination among the research, development, testing, and warfighter communities. The goals include ensuring that MANPRINT analyses occur early enough and that the results and recommendations arising from those analyses are tracked by the program manager in an effective manner.
3. Socio-cultural behavioral modeling. This third thrust area is relatively new for the HSI group. Research is aimed at understanding and modeling cognitive aspects of socio-cultural influences on soldier and/or commander decision making and communication. Models of decision making
and of communication should account for socio-cultural influences. The work is intended to produce guiding principles for presentation of socio-cultural information.
IMPRINT is the crown jewel of HRED’s HSI efforts. It represents a mechanism for applying human factors science through modeling. Its continued development and use constitute a strength of the program. IMPRINT development can be monitored at the IMPRINT website (http://www.arl.army.mil/www/default.cfm?page=445). There may be a gap in IMPRINT in the form of inadequate coverage of cognition and perception. If so, given the nature of today’s Army tasks, this is a critical gap.
The use of HRED’s HSI expertise can be seen in work such as that on progressive insertion of human figure modeling into acquisition. Good human figure modeling tools are of obvious use in the design of systems that humans are going to use. However, models for Army use are not going to be simple, off-the-shelf civilian models. Civilians do not need to enter and exit vehicles wearing body armor, hydration packs, extreme cold weather gear, chemical protective equipment, or other gear. Designing for the classic 5th and 95th percentiles may be misleading if the population is not unimodal. HRED’s effort to improve models of soldiers is a cost-effective way to improve the design of systems during the acquisition process. The research has provided effective comparisons between model-predicted outcomes and actual outcomes.
Opportunities and Challenges
Research in the human-systems integration area is uneven. Much of the work, such as that described above, is solid—part of an important effort to insert human factors into the acquisition process. Other work is less impressive. For example, the presentation of the work on multiple simultaneous task experiments revealed the types of problems that afflict some of the weaker work in this area. This project was designed without adequate attention to the existing literature on, for example, effect of cognitive load or dual task interference. An ad hoc task design was neither realistic enough to be as compelling as applied research nor basic enough to be likely to generate generalizable basic research findings. The statistical analysis was weak, and, in the absence of theory, it was unclear how these results could be used to predict or estimate performance in any other task. The intent here is not to single out one study, but to point out recurring problems with the design, analysis, and interpretation of studies.
The new social-cultural thrust faces another set of challenges. The study of emerging work in social, cultural, and behavioral modeling is interesting and quite appealing but elicits serious programmatic reservations. The problem area is vast. To make any serious progress in this area, HRED would need to devote significant resources to the endeavor. It would need to assign multiple investigators to the problem, some of whom would have to come from outside HRED because the requisite expertise does not exist in house. That decision support is a good endpoint for the study has not been convincingly justified. HSI has dipped a toe into these interesting waters. It either needs to plunge in with a serious investment of resources or leave the topic to others.
As previously noted, it is possible that cognitive and perceptual variables are treated too lightly in IMPRINT. If this is the case, then it would be valuable to address this deficiency. Doing so represents an opportunity for HRED to bring together investigators who have been developing physical and perceptual/cognitive models in other domains for the purpose of providing a more robust human model of soldiers than is available in the existing IMPRINT model.
Overall Quality of Research
As indicated above, the picture in this area is uneven. Much of the work is very solid, high-quality human factors work, employing a range of tools, including IMPRINT. Other work is deficient in conception and/or execution.
Changes Since the Previous Review
Since the previous review, opportunity-driven research (ODR) has been elevated to the status of a research thrust. This development is very positive. The value of careful management and nurturing of this pathway cannot be overstated. ODR can be a central aspect of HRED’s mission to enhance the ability of soldiers to work with Army systems, to neutralize threats to those soldiers, and to increase affordability. Moreover, successful response to problems that present in the ODR context can enhance the credibility and perceived relevance of HRED. This, in turn, seems likely to build external advocacy for the Army investment in HRED research.
Accomplishments and Advancements
HRED frames ODR as having a close relationship to the HSI/MANPRINT work, described above. Although the HSI/MANPRINT effort is targeted toward intervening early in the acquisition process, MANPRINT issues can be identified during usability testing or through reports received from the field. When there is no previous research to provide a solution to a problem, an opportunity arises for research. This problem-solving approach to MANPRINT issues is an excellent opportunity for HRED to demonstrate its unique talents. This sort of problem-inspired research and development seems central to the HRED niche. ODR enhances the credibility of the science program and should be an incubator for further research.
ODR also fulfills a soldiers’ advocacy role when issues identified in the field rise to the level of science rather than yield mere complaints or ad hoc fixes. Much of the ODR work is conducted in HRED’s 20 field elements, although it does occur elsewhere in HRED, which makes sense because the field elements are HRED’s forward positions. The field-element-based work can be the first step in work that would be later developed at Aberdeen.
By its nature, ODR is not systematic in the way that a program such as translational neuroscience would be. Consequently, it is not possible to comment on ODR as a whole based on reports of a few programs. However, in general, the projects that were presented bring good human factors research to interesting problems of obvious relevance. There is potential here that is not being fully exploited.
Opportunities and Challenges
Current HRED doctrine is that the goals of ODR are to fill critical knowledge gaps and to support the solution of real-world problems for a specific group of end users. As such, it does not consist of large lines of research geared toward creating a new generalizable body of knowledge. That ODR does not involve large lines of research is unassailable. But to conclude that ODR does not lead to generalizable knowledge limits its potential impact. In some cases, ODR research could lead to important, larger lines of research with implications for the Army as well as potential to generate good fundamental research.
An example is the work on investigating takeoff and landing tasks in a brown-out degraded visual environment. This was a very fine presentation on an important problem. Obviously, it is difficult for pilots to land a helicopter if they cannot see. Relatively abstract symbology about altitude and other factors related to landing is useful, but symbology that exploits pictorial depth cues can enhance situation awareness and improve performance. Clearly, if this work prevented one helicopter crash it would have been more than worth the investment. However, there would be no reason to stop at this point. The specific symbology made use of some depth cues in one configuration. This was a good first try, but could the display be improved further? HRED has a sensory performance group that should have the expertise to ask the more general questions about displays of this sort. Here is an example of a problem that could be brought from the field element into HRED Aberdeen for more systematic study. Improvements could be suggested and then transitioned back to the field element for testing.
As noted elsewhere within HRED, many projects are often done in response to some specific customer request and terminate without discussion of further research avenues. In some cases, this ending may be appropriate. However, in other cases, an opportunity may be missed. The specific request may require only a quick study and a good-enough solution. HRED scientific leadership should monitor ODR projects for opportunities to bring research from the field elements to Aberdeen, with the goal of producing better than good-enough solutions to the specific problem. Such solutions might have impact beyond the originating problem.
The project on enhanced operator perception through three-dimensional (3D) vision and haptic feedback raises similar issues. In this case, the question was whether a robot arm could be manipulated more effectively if the operator had better 3D input and haptic feedback. The answer from the solidly executed study was that these enhancements were helpful. Again, a program of research might yield stronger insights on this topic. In this case, it seems likely that best progress would be made in collaboration with robotics partners beyond HRED, whose role is on the evaluation side of the equation. Other ARL partners would be a possibility, as would industry or academic partnerships arranged through the Army Research Office or a funding vehicle like a CTA. This particular study is highlighted only as an example; the ARLTAB is not advocating for a massive project on robotic arms. HRED leadership should view ODR projects as seeds of larger research projects.
Not all ODR projects need to grow into research programs. An example is the work on evaluating training using the Second Life video game platform to augment mindfulness training for soldiers. Although the topic has value, this project did not include plans for follow-on research and did not fall obviously under the HRED umbrella. If it were to be pursued, then the Simulation & Training Technology Center would be a more obvious home for more extensive study.
Overall Quality of Research
The ODR work that was presented during the reviews was of generally high quality. This work nicely illustrates the ability of HRED to serve as a problem-solving resource for a wide range of Army problems.
SIMULATION AND TRAINING TECHNOLOGY
Changes Since the Previous Review
The previous review occurred as the Simulation and Training Technology Center (STTC) was being integrated into HRED. Therefore, this is the ARLTAB’s first review of the STTC.
Accomplishments and Advancements
The STTC is a large group of researchers and developers based in Orlando, Florida. The group became part of HRED within the past few years. The STTC’s efforts are directed at advancing the Army’s simulation-based capabilities in training, experimentation, analysis, and operational Army needs. Simulation is seen as a cost-effective response to these needs. The pace and diversity of U.S. Army missions require a rapid, responsive training capability. Moreover, in a variety of situations (e.g., adaptive tutoring), training by the use of artificially intelligent agents has been shown to produce better results than has more conventional training.
The STTC designs specific simulations (e.g., simulations of battlefield medical situations), as well as general tools for simulation (e.g., the Generalized Intelligent Framework for Tutoring framework) to make it possible for others to rapidly create new training modules for new content areas.
The STTC has the program management for a University Affiliated Research Center, the Institute for Creative Technology (ICT) at the University of Southern California. The ICT receives 100 percent of the STTC basic research (6.1) funding. The ICT is reviewed by a separate Army assessment board. The ARLTAB was not asked to review the ICT, which limits the scope of the evaluation of STTC presented in this report.
The STTC includes five programs:
1. The purpose of the Adaptive and Intelligent Training Technologies Program is to design, develop, apply, and assess artificially intelligent agent technologies (e.g., adaptive tutoring and virtual human tools and methods) to enhance training effectiveness and reduce costs. An important emphasis is on developing tools that allow others (i.e., researchers, instructional designers, training developers, and trainers) to quickly author new training modules so that artificial tutors can be created for different training needs as they arise. The conscious effort to develop a broadly applicable framework has produced the STTC’s generalized intelligent framework of tutoring (GIFT), which is a sensible and useful tool for the development and evaluation of intelligent training systems.
2. The Synthetic Environments Program develops improvements in synthetic environment modeling, with a particular emphasis on dynamic effects such as changes in weather and lighting.
3. The Immersive Learning Program investigates the issues raised when these synthetic environments are used as learning environments. In addition to studying what makes for a realistic, compelling environment, the program is attempting to address whether or not immersive environments actually promote learning. To the extent that the immersive situation provides an effective version of time-on-task training, it would be expected to lead to better learning outcomes (e.g., skills acquisition, retention, and transfer to the field).
4. Obviously, the above work is related to the Training Applications Program, which creates and tests specific examples of training applications software programs. The training applications program focuses on domain-specific research such as medical training, dismounted soldier training, or ground platform training.
5. The Advanced Distributed Simulation Program focuses on conducting research and developing technology to facilitate local and geographically distributed interactions between models and simulations.
These programs constitute a generally impressive program of work, although comparatively few program details were presented. The panel has not seen the facilities nor met many of the staff.
STTC staff has made impressive efforts to link STTC work to broader work in the field.
Opportunities and Challenges
The main challenge lies in the integration of the STTC into HRED in a manner that is useful to both. Any cognitive psychologists associated with the STTC in Orlando could help to facilitate an increased focus on human factors in training.
Some of the work will likely benefit from a closer link with HRED. The work on real-time monitoring of ECG and GSR signals during computer-based training is a natural point of collaboration with the translational neuroscience group. The current use of galvanic skin response and EEG to measure student arousal seems intrusive and not particularly practical for complex, dynamic training situations. The relationship between EEG and GSR signals and engagement was not obvious, and the generalizability of findings across different contexts and tasks was not clear.
Overall Quality of Research
The overall quality of the work reviewed is good.
