3
Human Research and Engineering Directorate

INTRODUCTION

This chapter is based on the visits by the Soldier Systems Panel to the Human Research and Engineering Directorate (HRED) at Aberdeen Proving Ground, Maryland, on June 9-11, 2009, and July 13-15, 2010, and on examination of written materials provided in association with those visits. During those visits the panel received briefings on substantial portions of nonclassified HRED work, mostly in the 6.1 (basic research) to 6.2 (applied research) categories. This chapter provides an evaluation of that work, recognizing that it represents only a portion of HRED’s portfolio.

Five Broad Areas of Research

HRED research falls into five broad areas:

  1. Human Robot Interaction (HRI): This program deals with the integration of robotic devices into the military and, to some extent, into broader civilian life. These researchers seek to understand how a soldier or group of soldiers can work with autonomous or semiautonomous machines at scales from hand-carried devices to vehicles.

  2. Human System Integration (HSI): This program models the personnel requirements of Army systems, including workload calculations, skill levels demanded, interface designs, and automation. Much of this work is carried out in the context of the Manpower and Personnel Integration (MANPRINT) modeling system.

  3. Neuroscience: This work focuses on the use of neuroscientific methods to monitor and enhance solider performance.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 31
3 Human Research and Engineering Directorate INTRODUCTION This chapter is based on the visits by the Soldier Systems Panel to the Human Research and Engi - neering Directorate (HRED) at Aberdeen Proving Ground, Maryland, on June 9-11, 2009, and July 13-15, 2010, and on examination of written materials provided in association with those visits. During those visits the panel received briefings on substantial portions of nonclassified HRED work, mostly in the 6.1 (basic research) to 6.2 (applied research) categories. This chapter provides an evaluation of that work, recognizing that it represents only a portion of HRED’s portfolio. Five Broad Areas of Research HRED research falls into five broad areas: 1. Human Robot Interaction (HRI): This program deals with the integration of robotic devices into the military and, to some extent, into broader civilian life. These researchers seek to understand how a soldier or group of soldiers can work with autonomous or semiautonomous machines at scales from hand-carried devices to vehicles. 2. Human System Integration (HSI): This program models the personnel requirements of Army systems, including workload calculations, skill levels demanded, interface designs, and automa - tion. Much of this work is carried out in the context of the Manpower and Personnel Integration (MANPRINT) modeling system. 3. Neuroscience: This work focuses on the use of neuroscientific methods to monitor and enhance solider performance. 31

OCR for page 31
32 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY 4. Social and Cognitive Network Science: This research investigates the human behavioral aspects of networked operations. Distributed communication and decision making within and between groups constitute a particular area of interest. 5. Soldier Performance: This area is concerned with the human factors that impinge on the design of military systems, examining how those systems can be best configured for a human operator. Table A.1 in Appendix A characterizes the staffing profile for HRED. Directorate-Wide Themes HRED has a vast and very important mandate: to understand the functions of the human-in-the-loop in a wide range of Army systems. The most sophisticated sensors, weapons, and information systems will not deliver their full potential if they are not well matched to the capabilities of the humans using them, in the conditions under which they are intended to be used. This human-in-the-loop domain includes a wide range of research topics, from basic to applied. Perhaps the fundamental challenge for an organization like HRED is how best to select some subset of the nearly infinite number of possible topics for study. The directorate’s response to this challenge can be understood by considering the nature of HRED. Two visions, not mutually exclusive, compete to shape HRED’s activities: Is HRED intended to make significant contributions to the peer-reviewed scientific enterprise in the manner of a university labo - ratory? Is it supposed to be applying the basic science of others to specific issues raised by its Army customers? Presumably, the answer is “yes” to both questions, but those goals live in necessary tension with each other. Customer-based, applied work is of obvious importance, but the results will often be too specific to be interesting to the broader scientific community. Basic science projects must proceed with the understanding that they might fail or, perhaps more typically, that they will produce valid, sta - tistically significant results of no apparent near-term use to the Army. The accumulation of such results is the required cost for those much rarer basic science breakthroughs that transform practical systems and operations. In a perfect world, resources and personnel would be available to prosecute all of the interesting and valuable projects from the basic and applied realms. In the real world, HRED must continually wrestle with the balance of its allocations of time and money. The issues surrounding the selection of research topics seem to be handled more successfully in some branches of HRED than in others. There are selection mechanisms in place. For example, at the large scale these include Army Technology Objectives (ATOs)—project proposals must describe how the project contributes to formally defined ATOs. At a finer grain, individual research proposals may be reviewed by Army Research Laboratory (ARL) fellows, experts in their scientific domains, whose appraisals of proposed projects are provided to the ARL Director, who decides whether to allocate to proposed projects discretionary funds for the initiation of research. The effectiveness of this rather complex and bureaucratic system varies across HRED. Leaders with clear vision can use the system to shape effective research programs. In other cases, the system may overwhelm the vision. One means of alleviating, though not eliminating, the problems of project selection is to increase resources. The Collaborative Technology Alliances (CTAs) in Cognition and Neuroergonomics, in Net - work Science with the Computational and Information Sciences Directorate (CISD), and in Robotics with the Vehicle Technology Directorate (VTD), are important mechanisms for increasing the capability to contribute to fundamental science in these areas.

OCR for page 31
33 HUMAN RESEARCH AND ENGINEERING DIRECTORATE CHANGES SINCE THE PREVIOUS REVIEW Since the previous ARLTAB review,1 HRED leadership has stimulated the activity of the HRED community. Staff has grown, and there has been significant turnover in personnel, with a resulting influx of early-career talent and energy, to the benefit of the directorate as a whole. There is an increased empha- sis on the impact of HRED work on the broader scientific community. This can be seen in increased publication in peer-reviewed journals, attendance at basic science conferences, and collaborative work with members of the university research community. Significant new HRED facilities have begun to demonstrate their planned functions, including the Cognitive Assessment, Simulation, and Engineering Laboratory (CASEL), the Environment for Auditory Research (EAR), and the Open EAR. Others, such as the Soldier Performance and Equipment Advanced Research (SPEAR) facility, are in development. At the time of this writing, the Army’s Simulation and Training Technology Center in Orlando, Florida, is in the process of becoming a part of HRED. This move brings more than 35 technical staff personnel and their research programs under the HRED umbrella. Significant opportunities for syner- gistic new research co-exist with the challenges of integrating the group into HRED. ACCOMPLISHMENTS AND ADVANCEMENTS Human Robot Interaction Robots continue to be important tools for soldiers in a variety of contexts. Consequently, research in this area is highly relevant to both near-term and long-term Army mission goals. In general, many aspects of the research have improved since the previous review. HRED has successfully identified a number of key problems in this domain. The HRI group clearly realizes that it is not working on traditional robotics problems and that it must pay close attention to soldier-robot interaction, which is important for long-term applicability of the research. Progress was demonstrated on multiple fronts in this research area, particularly in the work incorporating real robots and addressing important field-motivated ques - tions, and in increased consideration of scenarios involving soldiers controlling more than one robotic platform at a time. Similarly, research on alternative modalities for soldier-robot communication and adaptive automation are well motivated by concerns faced by actual soldiers in the field. Another area of substantial improvement is the increased use in research of the actual robotic platforms currently deployed in the field. The HRI Robotics Collaboration Army Technology Objective Capstone Experi - ments represent a good example of meaningful, warfighter-motivated research evaluated in a realistic field setting. This is the kind of research that HRED is uniquely positioned to do, making use of real Army operators testing field-ready technology. Human System Integration The HRED Human System Integration group develops models, tools, and methods to support the assessment and evaluation of warfighter systems. The group is continuing to improve the usability of its models and tools (e.g., the Improved Performance Research Integration Tool [IMPRINT]), and its use 1NationalResearch Council. 2009. 2007-2008 Assessment of the Army Research Laboratory. Washington, D.C.: The National Academies Press.

OCR for page 31
34 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY of these tools to support Army applications is commendable. Overall, HRED has the opportunity to be at the forefront of research in this area. The IMPRINT model development effort continues to reach out to meet the needs of a wide range of users. Users can now develop their own plug-ins and post these on the IMPRINT Pro Online User Community Web site, a SharePoint site. This is very useful for researchers, industry, and Army users. IMPRINT also incorporates existing modules (e.g., the Sleep, Activity, Fatigue and Task Effectiveness model—SAFTE) that have been validated with other Department of Defense (DoD) models or tools. The mission-based testing and evaluation group has demonstrated a clear track record of successful application of IMPRINT in conjunction with field evaluation methods. An outstanding example was provided by the project’s applying domain and mission analysis techniques, IMPRINT modeling, and human-in-the-loop evaluation in close coupling to inform the design of the Joint Light Tactical Vehicle. The analysis and modeling methods developed by this group provide an effective and useful roadmap for how computational models can be used to inform design. In a rather different application, an activity performing the testing and evaluation of a new financial system applied analytical tools normally used in evaluating weapon systems to a large, office-management financial system. The work appears to be well grounded and well tailored for the analysis of complex systems (e.g., a usability matrix that has been established for reliability). Recent initiatives by HRED to incorporate plug-in models into IMPRINT are commendable. Differ- ent subsets of IMPRINT users (both within and outside DoD) have expressed requirements for specific functionality in using this important soldier-workload-assessment tool. This flexibility is especially important when using the IMPRINT tool to assess proposed major changes in equipment design, such as determining the appropriate number of crew members required to operate a new vehicle in a vari - ety of operational scenarios. HRED’s initiative to incorporate the well-established SAFTE model into IMPRINT will provide a basis for subsequent trials of incorporating other plug-in models, such as the Navy’s SeaState, the U.S. Air Force Human Systems Integration module, and the Multimodal Informa - tion Design Support module. The HSI team actively and admirably serves as a bridge between the laboratory and the field, as demonstrated by its effective participation in the ARL-Field Assistance in Science and Technology (FAST) Program, in which HRED representatives interact with soldiers in the field (e.g., in Iraq) and communicate to ARL equipment needs expressed by the soldiers. The FAST Program contributes sig - nificantly to what soldiers, commanders, and military units deployed in the field really need—a two-way conversation with the Army’s research and development (R&D) laboratories to enact quick fixes in the field and to help identify what new systems and technologies will be important for future battles. The project to apply human factors engineering in Iraq compellingly illustrates the value of the deployment of HRED personnel to theater where they can obtain a first-hand understanding of the problems faced by ARL end users (i.e., soldiers). Visiting soldiers downrange in the field is an excellent way to gain the goodwill and respect of those users. It may also be the best place to obtain feedback on ARL prototypes (e.g., chaps, or leggings designed to protect soldiers from contact with blood). Participation by the human factors group in the FAST Program should continue to be supported.

OCR for page 31
35 HUMAN RESEARCH AND ENGINEERING DIRECTORATE Neuroscience In 2009, the National Research Council (NRC) completed a 2-year study designed to advise the Army on opportunities for neuroscience research.2 That study identified 17 key suggestions for Army research in this area. Of necessity, those suggestions spanned the Army’s many laboratories, but at least 4, and perhaps as many as 8, were relevant to the U.S. Army Materiel Command and to ARL in particular. The current neuroscience group at ARL, although still quite new, has clearly begun to make head - way on the three recommendations most relevant to ARL. At a technological level, the group has begun to develop core competencies in electroencephalogram (EEG) measures of soldier performance, in functional magnetic resonance imaging (fMRI), and in the interaction of fMRI and virtual reality. The group also has focused on improving soldier-system interfaces by taking into account neurobiologi - cal constraints on information processing, an area coming to be called neuroergonomics. The group has begun to make some headway in characterizing individual variability at the neural and behavioral levels—another key recommendation of the 2009 NRC study.3 This constitutes solid progress toward the long-term integration of neuroscience research into the Army’s portfolio. It is worth noting that this aspect of neuroscience research is distinct from the medical neuroscience work performed in other DoD laboratories. The emerging neuroscience group at ARL has proceeded as a model of program development. Clearly set goals and innovative experimental programs mark out a young program that is on track to produce highly significant and Army-relevant 6.1 and 6.2 research. HRED has recruited and motivated an effective neuroscience group of early-career scientists whose members are well trained, having come from strong graduate and postdoctoral programs. Overall, the HRED neuroscience group has reached a level of quality that would allow it to fit in well as a solid, small neuroscience group at a reputable research university. The neuroscience group is commendable for its focus on significant aspects of neuroscience that can be translated into the real world of the battlefield. There is a strong focus on Army-relevant psychophysics and neuroscience and on multimodal integration, and there is a recognition that interindividual differ- ences will play an important role in sensory processing, integration, and cognition. The neuroscience group has a healthy respect for real-world complexity conjoined with its policy of keeping translational goals squarely in view. The single most exciting accomplishment by the ARL neuroscience group during this assessment period was its development of single-trial-based, Army-relevant paradigms for brain-behavior analysis. The demonstration that an independent component analysis (ICA)-based EEG can be used at the single- trial level to explain “Shoot/Don’t shoot” behavior was a tour de force. The group demonstrated that classical psychophysics could be extended both to EEG and to Army-relevant tasks. The neuroscience group should be encouraged to explore additional tasks like these that relate single-trial behavior to brain activity in stylized but relevant tasks. The publication rate of the group in peer-reviewed journals is good, and continued publication in Tier 1 journals should be encouraged. It is likely that the group will be highly productive in the years to come. There is no reason that the HRED neuroscience group should achieve any less than the publica - tion rate of a Tier 1 group at a major research university. 2National Research Council. 2009. Opportunities in Neuroscience for Future Army Applications. Washington, D.C.: The National Academies Press. 3Ibid.

OCR for page 31
36 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY Social and Cognitive Network Science As the previous ARLTAB assessment report4 indicated, at ARL generally and in HRED in particular, there are major opportunities within the network science area to address the cross-disciplinary problems that bear on individual, team, and large-group performance in distributed, networked environments. These efforts would be built on HRED’s existing and unique capabilities and resources. The report also noted that the program seemed to be defined to encompass the entire range of physical, information- based, cognitive, and social phenomena emerging from the introduction of network-centric operations as they might influence support for warfighters. Since that review, the program has produced a more circumscribed account of its domain of research. Specifically, its major purpose is to improve distributed collaboration and decision making in warfighters’ complex networked environments by using cognitive science, computer science, and social network innovations. There have been some accomplishments over the 2 years since the previous assessment. These include a number of important steps that the ARLTAB report had recommended. First, it had been noted that the breadth and complexity of the program’s purpose demanded a sufficient research staff. Two Ph.D. researchers were hired in the past year. Both have experience studying teams and networks. These addi - tions should be helpful not only in boosting the productivity of the program, but also in further refining its focus while integrating it into the greater community of network research activity both within and outside ARL. Thus, a closely related second accomplishment is the undertaking of more collaborative work. Given the rapid growth and progress in network sciences on the one hand and the pace of techno - logical change in social media on the other, it is important for researchers in HRED’s program to work collegially with other scientists in this field. Besides involvement with a Multidisciplinary University Research Initiative (MURI) and a large network science collaborative program directed by ARL’s Com - putational and Information Sciences Directorate, the establishment of the Network Science CTA with Pennsylvania State University takes a positive step in this direction. A third promising indicator has to do with improved financial support for the program. This is evi - denced not only by the newly awarded CTA but also by seed money in the form of a Director’s Strategic Initiative (DSI) to generate new research efforts aimed at improving the understanding of the human dimension of network science. However, the DSI project at HRED has been less successful than desired by HRED at bringing in outside funding for the research. Several research projects in the agenda of the social and cognitive network science program have been brought to completion, fulfilling another recommendation from the previous ARLTAB report. These include, for instance, a laboratory study of distributed dyadic interaction and the effects of network delays on pairs’ communication outcomes, as well as a qualitative linguistic study of misunderstandings that arise in communications among members of cross-cultural teams. Soldier Performance Key Facilities Major effort in the area of soldier performance has gone into the creation of two facilities whose establishment represents significant recent accomplishment: 4NationalResearch Council. 2009. 2007-2008 Assessment of the Army Research Laboratory. Washington, D.C.: The National Academies Press.

OCR for page 31
37 HUMAN RESEARCH AND ENGINEERING DIRECTORATE 1. Tactical Environment Simulation Facility: This facility includes two major research features: The first is an immersive environment simulator that integrates visual and auditory displays with an omnidirectional treadmill (ODT) mobility platform that enables test participants to have natural human locomotion through a virtual environment. The second is a hostile environment simulator, an acoustic chamber containing a high-intensity audio system capable of producing sound-pressure levels up to 155 dBP (e.g., weapons-firing noise). Some validation studies with the ODT show its potential to be used in well-controlled experiments with dismounted soldiers progressing over nonlinear paths—for example, studies of load carriage, exertion, and neurocog - nitive variables. 2. Environment for Auditory Research (EAR): Since the previous ARLTAB report, HRED’s auditory research staff has been engaged in finishing the construction of the four acoustic rooms (distance hall, listening laboratory, dome room, and sphere room) that constitute the EAR facility. The research staff has conducted measurements in two separate experiments that helped to verify the properties of those rooms and to establish their baseline. As noted in the previous ARLTAB report, the EAR facility presents remarkable potential for both in-house and joint research with collaborative agencies and other researchers. The leadership of the EAR facility recognizes the desirability of making the EAR available to outside researchers and has taken some steps in that direction. Research in Soldier Performance HRED conducts a wide variety of specific research in the area of soldier performance. Examples include the following: • Auditory Hazard Assessment Algorithm for Humans (AHAAH): A notable accomplishment in sen- sory research is the decades-long, high-quality basic and applied auditory research done by some but not many HRED scientists. The AHAAH is a superbly developed and validated mathematical model of the human auditory system that predicts hazards from any free-field pressure; it can display visually the damage process as it is occurring. ARL should lead the way in promoting HRED’s contributions to what is likely to become an international standardization of auditory damage risk criteria— especially that threatening soldier hearing. • Sensory Research: The soldier performance program has a variety of projects in sensory psycho - physical research. HRED has engaged in basic and applied research on the use of tactile displays to augment overloaded visual and auditory channels. Recent work on the sensitivity of the head to tactile stimuli should be published as a contribution to the basic science on the topic. Some progress was made in characterizing the information that could be conveyed through publications at a later date, since this work is not yet ready for publication. • HRED and Army Medical Training: The HRED field office located at the Army Medical Depart- ment Center and School at Fort Sam Houston, Texas, has been assisting the center in identify - ing soldier performance issues that affect rates of retention (graduation) and academic attrition (failure and dropping out) for the Health Care Specialist (MOS 68W) training course. The work is impressive. The HRED team at Fort Sam Houston is conducting a comprehensive sequence of research, applying qualitative data (focus-group questionnaires and interviews), modeling pre- selection test scores (e.g., Armed Services Vocational Aptitude Battery), and collecting empirical data on class performance. Additionally, the HRED team at Fort Sam Houston began feasibility evaluations of two academic feedback tools for students and faculty. The team evaluated a program

OCR for page 31
38 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY for Personal Academic Strategies for Success (PASS) as a student’s self-evaluation tool, and an Academic Class Composite Tool (AC2T) to inform instructor personnel as to where improvements are needed in the courses that they offer. Results of the PASS and AC2T projects are preliminary, and so it is too early to judge their effects. • Traumatic Brain Injury (TBI) and Post-Traumatic Stress Disorder (PTSD): At Fort Sam Houston an ongoing research protocol to assist in the early identification of the presence of mild forms of traumatic brain injury and perhaps accompanying post-traumatic stress disorder, described for the panel in 2009, will have to move beyond the one-time, opportunistic study to a more program - matic, multistudy approach if it is to become a viable research project. However, because TBI and PTSD are the subject of massive research programs elsewhere, it is not clear that this is an area in which HRED can have a major impact without an appropriate, clearly defined focus. • Brain Training for Resilience: Another HRED medical field office research protocol involves plans to evaluate use of audio-photonic stimulation as a form of neurocognitive brain-training technology to enhance student soldiers’ cognitive resiliency, especially in terms of resisting any - thing that would divert attention from performance during sustained operations. The HRED plan is to determine the effects of audio-photonic stimulation on cognitive performance (automated neuropsychological assessment metric, or ANAM) and academic performance (course grades and pass/fail status), self-reported sleep, mood (e.g., by means of the Profile of Mood States rat - ing scale), and self-reported stress. The goal of the project is to help soldiers perform extremely well during sustained operations. Such a research project could offer significant findings to the Army Medical Department Center and School for its field medical personnel, and it could make contributions to the overall Army program entitled Comprehensive Soldier Fitness, which stresses instilling resilience in individual soldiers and small units. OPPORTUNITIES AND CHALLENGES Human Robot Interaction The Army Research Laboratory could be positioned to take a leadership role in the human robot interaction domain. The Army has extensive numbers of robots and of soldiers training with robotic platforms. HRED has begun to make use of these resources, and more extensive capitalization on this opportunity is warranted. It should be possible to collect fairly extensive data on both the learning and the performance of human-robot teams using currently deployed technology. Collecting such baseline data would provide researchers at HRED and elsewhere with benchmarks for measuring improvements in future robotic systems. It would also provide additional information about the exact nature of the needs and challenges faced by soldiers in the field. There are several points of concern. First, HRED researchers need to improve their contact with contemporary research in this area. Researchers seem somewhat isolated from closely related research being done at universities and at other DoD facilities. Contact with this work is important for several reasons. First, there are technical advances that can be of use to HRED. More broadly, contact helps shape the research questions that actually serve to advance the field. Second, as this research moves into areas that involve traditional human-computer interaction studies, it is important that HRED personnel learn techniques and methodologies from that area rather than trying to apply methods that they already know are inappropriate for the research problems that they face. Third, it is not at all clear that HRED is positioned appropriately to conduct research in some of the areas in which it has engaged (e.g., cognitive robotics). Although engaging with these problems reveals vision and foresight, it is not clear that the full

OCR for page 31
39 HUMAN RESEARCH AND ENGINEERING DIRECTORATE range of required technical knowledge is present in the HRED staff or that the limited resources applied are adequate to make meaningful progress. Fourth, it would be useful to have a centralized mechanism to facilitate information sharing, research review, and discussion among the widely distributed HRED robotics-related researchers. Individuals with limited robotics experience would benefit from interaction with others who have more extensive, ecologically valid robotics evaluation experience. The HRED HRI team should seek to publish and network in relevant venues, such as the IEEE (Institute of Electrical and Electronics Engineers) International Conference on Robotics and Automation and the Association for Computing Machinery/IEEE International Conference on Human-Robot Inter- action. Some members of HRED do so, and other projects would benefit from feedback from relevant peer reviews outside the behavioral sciences. As noted in the previous ARLTAB review, ARL also can bring to bear significant talent from HRED to address human-system integration, along with robotics-related work in the Sensors and Electron Devices Directorate (SEDD) and the Computational and Information Sciences Directorate. There was little evidence that such cross-directorate work is taking place. Many of the research questions being asked by HRED researchers would be well served by more extensive cross-directorate collaboration, because human-robot interaction is intrinsically interdisciplinary in nature. Human System Integration The HSI group has a unique opportunity to contribute to the broader science of human factors. The human-factors issues that the group is addressing and the methodologies that it is developing within the context of particular customer applications are of general interest to the human-factors community (researchers, educators, and practitioners). One mark of the significance of HRED’s work in this area is the impact of IMPRINT on the broader community, which is not appreciated in the community as much as it should be. HRED should continue to collect concrete information about this impact. The information could be obtained in a number of ways: for example, (1) searching for IMPRINT in data - bases such as Google Scholar, Compendex, or PubMed; (2) surveying users through easily available tools like Survey Monkey; and/or (3) compiling data on the number of plug-ins uploaded by users since capability was made available. Some of the challenges faced by HSI are related to the resources available to this group. Within the Army materiel development community, there appears to be great demand for the MANPRINT services of HRED. However, the in-house staff is too small to support numerous service requests effectively, especially if the same HRED members are expected simultaneously to advance the state of the art in HSI models and methods. Although effort has been expended on improving the usability and application range of the HSI tools, there also needs to be work on the validity and reliability of the tools and, most particularly, on the extensions and plug-ins for IMPRINT. The current portfolio of projects within the HSI area may be too service-oriented if broader scientific impact is an important goal. This is one of a number of areas in which ARL leadership must consider the balance between fundamental research and work for DoD consumers. The impact of HRED and the HSI program on new system design is enhanced when the HSI group is invited into the system acquisition process at an early stage, when requirements are first being defined. In many cases this happens. The consequences of a late invitation were clearly shown in the example of the testing and evaluation of the financial system described earlier. With HRED expertise brought into the design cycle late, there was limited opportunity to act on the useful insights gained from the evaluation, which itself was well conducted.

OCR for page 31
40 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY HRED should continue to provide IMPRINT training (beginning and advanced) and demonstra - tions at conferences of the Human Factors and Ergonomics Society and other organizations that focus on human performance modeling (e.g., Annual Conference on Behavior Representation in Modeling Simulation). HRED can also make available more advanced IMPRINT training. It would be worthwhile to develop courses that focus on how to use the IMPRINT model as part of a larger experimental design approach. As HRED researchers pointed out, modeling is an analysis methodology, and IMPRINT is a tool that supports the analysis. Therefore, greater emphasis should be placed on training users on the appropriate domain and mission analysis techniques as well as the mechanics of developing IMPRINT models. The current ratio of in-house to customer-funded HSI work—most of the latter is for direct applica - tions work to systems development projects—could be rebalanced if more in-house funds were available to permit more HSI research to improve the state of the art and HSI’s tools, such as IMPRINT. The HSI group would also benefit from more personnel, given the number of materiel system evaluations that are requested. This would allow the group to pursue basic scientific work to improve the validity of its tools and to enable the group to increase support for and expansion of IMPRINT tools. Work of this sort is hard to accomplish under customer funding. More in-house resources would allow the HSI effort to advance the behavioral science needed to inform HSI issues (e.g., the impact of automation design on situational awareness and workload, the impact of crew size and structure on situational awareness and workload). This would expand the impact of HSI work within and beyond immediate Army customers and would allow the group to publish more in Tier 1 peer-reviewed proceedings and journals. Neuroscience Institutional Challenges The greatest institutional challenge facing the ARL neuroscience program is that of carving out its unique domain within the larger Army neuroscience research establishment. The Army G-1 Office (Personnel) and the Training and Doctrine Command are responsible for training-related neuroscience, and the Army Medical Department manages the Army’s more direct biomedical aspects of neurosci - ence research. Insights and research areas from neuroscience are unlikely to respect these organizational mission boundaries. The neuroscience leadership at ARL has begun to address this issue by engaging in a number of cross-command collaborations that further the overall goals of Army neuroscience while respecting command boundaries. Collaborations of this kind should be strongly encouraged in the years to come as Army neuroscience develops. Another collaborative issue of this kind involves the relationship between the Army Research Office (ARO) and neuroscience at ARL. ARO has recently developed a clear interest in funding Army-relevant neuroscience. The recent fruitful interactions between ARO and ARL neuroscience projects should be continued and strongly encouraged. The Cognition and Neuroergonomics CTA, awarded by HRED in 2010, is likely to immensely strengthen the overall neuroscience investment. The HRED group is to be congratulated on the way that it has formulated the CTA. The CTA leverages the areas of expertise at ARL against areas in which expertise is required. Overall, the CTA shows both vision and direction. As the CTA matures it will be essential that the neuroscience group at ARL remain firmly in control of the CTA’s many elements and that the CTA-wide group retain the flexibility to address Army goals. The leadership of ARL should encourage the HRED neuroscience group to adjust the elements of the CTA flexibly as the neurosci - ence group at HRED grows and develops. It is important that the CTA continue to serve the needs of

OCR for page 31
41 HUMAN RESEARCH AND ENGINEERING DIRECTORATE ARL and not the reverse. Empowering the neuroscience group at HRED to adjust the CTA elements as necessary should yield powerful results. The neuroscience program shows depth, vision, and organization. It is a model for early-stage devel - opment. HRED appears to have an excellent staff on hand, although the continued growth of this staff will be critical for the group. The current staff is, of course, in the early-career stage. This is a strength because it encourages innovation, but maturing the staff and adding senior scientists will be critical as the group matures. A significant challenge faced by the neuroscience group centers on the issue of growth. The addition of a computational neuroscientist as a senior technologist was imminent at the time of this writing. This will be a very important step for the group and points to the fact that growth in staffing will be critical for maintaining excellence in neuroscience. The group is carefully selecting neuroscientific tools for future use. The group is also developing core competencies in a number of areas (EEG, ICA, fMRI) that will be required in the years ahead. If this small and promising group is to transition to a large, excellent, and high-impact group, it will be essential that ARL leadership continue to support the team with additional hires as well as more space and facilities. Space is already tight for the group, and the current facilities cannot support a group of the projected size. ARL leadership will eventually have to address that point. Specific Opportunities in Neuroscience Following are specific opportunities in neuroscience: • Increased focus on multimodal integration: The HRED neuroscience group should consider the interaction of multimodal information processing with stress, fatigue, and strong emotion. It is obvious that these latter will occur on the battlefield and can have unpredictable effects, leading in some cases to greater focus and in other cases to the disorganization of information processing and cognition. In order for basic discoveries to be translated into useful technologies, such factors must be taken into account. The neuroscience group grasps this problem, at least in theory, and is already beginning to think about useful projects to address stress, fatigue, and strong emotion. • Exploring the regulation of brain processes: The most useful purpose in HRED for beginning a neuroscience program, in contrast to extending an existing HRED psychology program, is ulti - mately to gain the ability to regulate brain processes involved in sensory integration and cogni - tive processing and to mitigate confounding factors such as stress, fatigue, and strong emotion. The most important tools for regulating the synaptic mechanisms underlying these functions are pharmacologic. ARL is not currently authorized or equipped to engage in pharmacologic stud - ies; however, a point will come when significant translatable progress will require such inquiry. ARL should begin to consider appropriate collaborations so that organizational boundaries do not inhibit collaborations and thereby limit progress. • Exploring neuroplasticity: The neuroscience group does not currently have a focus on neuro- plasticity, but this will be critical in order to illuminate how training experience and battlefield experience alter complex information-processing tasks carried out by soldiers. The limitation of experiments to a small number of trials in the laboratory does not give a full picture of how soldiers will interact with technologies over time.

OCR for page 31
42 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY Social and Cognitive Network Science The major laboratory program in social and cognitive network science continues to have significant opportunities to contribute to theory, methodology, and application in this rapidly advancing field. For instance, the CASEL facility provides the capability to conduct laboratory research on distributed mul - tiperson team interactions, making it possible to go well beyond the study of dyads and to address not only communication content, but also the varied properties of the technology that supports such teams. As another example, HRED has the ability to capture networked interaction data from large-scale field exercises involving a significant number of participants over several days. These are merely illustrative examples of the opportunities that the social and cognitive network science program has to contribute to research related to how individual and team activities interact with properties of new technologies (including not only e-mail, but other social networking media) to affect collaborative decision making and performance in measurable ways. However, the scientific output of program research, as reflected by publications in peer-reviewed journals, remains less than desired over time and in comparison with other programs within HRED. The program faces significant challenges. First, it must define its approach to the social and cogni - tive network science domain. Two approaches can be seen in the work of other HRED programs. The neuroscience program has made successful use of a top-down approach by first defining key concep - tual areas in which the program could gain ground and make significant advances and then describing specific research “thrusts” based on these concepts. A more bottom-up approach is exemplified by the human systems integration program, in which programmatic thrusts and specific projects are heavily driven by client demand. As noted earlier, this is not without its own difficulties, but in both the HSI and neuroscience cases, it is easier to see the shape of the program than it is with the network science work. There is a noteworthy absence of advancement in articulating the content and structure of the social and cognitive network science domain through an organized and well-defined program of research; there appears to be no strategic plan and logical narrative guiding the program. For instance, the program comprises four major thrusts: multicultural communications, human-team-network interaction, social interaction and simulation, and computational representation. No rationale is provided for the choice of these thrusts: Do they capitalize on existing expertise at ARL? Do they address identified high-priority problem areas? Further, it is not clear how they might be connected into a programmatic research agenda. Indeed, they appear to represent quite different levels of granularity. For example, why is “multicultural communications” not subsumed under “human-team-network interaction”? Similar problems are also evident with subtopics under major thrusts (it appears at times as if an extant project is used to define a thrust, rather than vice versa). Three of the four major thrusts seem to have just a single project each. One would expect a thrust to describe a program of several projects. The existing program seems rather scattershot in nature. Some current projects could be combined and/or restructured around a smaller set of clearly articulated thrust areas. This might result in the phasing out of some projects in order to allow the program’s limited resources to be more coherently focused. There is an urgent need to set out the main conceptual or theoretical building blocks for the program. The social and cognitive network science group needs to develop a strategic plan directing the choice of research thrusts and projects that enable the program to contribute to basic science and to warfighter applications. A second challenge is to address the program’s distinctiveness. How can HRED’s contributions be differentiated from those made by other programs in this domain, such as those supported by the National Science Foundation, the Office of Naval Research, and the Army Research Office? A third challenge will be to increase the program’s productivity in terms of scientific output and outside support. With a

OCR for page 31
43 HUMAN RESEARCH AND ENGINEERING DIRECTORATE sharpened focus on its aims, improved conceptual leadership, and the active involvement of newly hired researchers, such productivity gains would be expected. Soldier Performance As noted above, the Environment for Auditory Research is a world-class auditory facility. However, it remains unclear if there is a utilization plan that will produce world-class results. The panel’s concerns remain strikingly similar to those that can be quoted from the previous ARLTAB report: Their challenge will be to develop a formalized and coherent set of studies that take full advantage of this outstanding facility, while meeting the unique needs of the Army both to protect soldiers against noise-induced hearing loss and to improve auditory communication and performance. There is a risk that both investigator and laboratory time could be absorbed by short-term practical questions. If time is not set aside for more exploratory basic studies, the scientists may not remain at the cutting edge of the research and it will be difficult to attract the best scientists to the laboratory, thus losing the advantage now provided by such a well-conceived physical facility.5 The recruitment of one or more currently productive senior scientists would facilitate the develop - ment of a successful program. Even if a senior hire is not possible, a committee of auditory scientists might be recruited to be involved with the EAR in a project-specific manner. These could be off-site collaborators, but the project should have sufficient funding to commit them to efforts beyond those of occasional consultants. Steps of this sort are needed if this excellent facility is to live up to the invest - ment made in its creation. In the visual domain, there has been a long-standing line of work on the fusion of visible and infra - red signals, among other projects. Although there has been some progress over the years, the visual and tactile projects do not seem to have generated the level of output (e.g., in peer-reviewed publications) that might be expected in these areas. In the case of the sensor fusion work in vision, it is not clear whether the project is in adequate contact with the very substantial work on this topic that occurs elsewhere in DoD and other domains (e.g., medical imaging). A well-thought-out research plan aimed at taking maximum advantage of the unique capabilities of the omnidirectional treadmill facility should be developed. As of this writing, such plans were not in evidence, and they should be developed and presented for review as soon as possible. Additional planning challenges will be to demonstrate how research work planned for the new ODT facility will be complemented by the newly upgraded mobility and portability test course (at the Known Distance Range at Aberdeen Proving Ground) in an integrated research and applications program. The HRED field unit at Fort Sam Houston has been engaged in studies of early identification of the presence of mild forms of traumatic brain injury and perhaps accompanying post-traumatic stress disor- der. This is a topical and important problem. However, as noted earlier, in order to be a viable research project, this would have to become a more substantial research program. ARL needs to determine if there is a viable role for the Fort Sam Houston field office in the study of such medical research issues or whether it needs to expend its energies elsewhere. The group was also asked to evaluate a neurocog - nitive training intervention technology (interactive metronome and videogame playing) on pass/failure rates in the course. The basis for selection of the interactive metronome appears weak (there was little evidence of its effectiveness). If this project were to progress, the metronome should, at a minimum, be compared with some alternative. 5Ibid., p. 34.

OCR for page 31
44 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY OVERALL TECHNICAL QUALITY OF THE WORK To summarize the overall technical quality of work in HRED, it is useful to consider four topics: broad scientific vision, technical resources, quality of work at the individual project level, and scientific output. Vision Clear and appropriate vision is central to the future success of HRED’s scientific mission. There is no doubt that important topics are under study in each of the five broad areas reviewed above: human robot interaction, human system integration, neuroscience, social and cognitive network science, and soldier performance. The neuroscience group articulates a clear vision: It wants to use modern neuro - science to enhance soldier-system performance. The group recognizes this as a very broad mandate, and so it has focused on specific projects in which it can make a unique contribution. Therefore, it is putting considerable effort into taking sensitive methods from the laboratory and getting them to work under less-forgiving conditions—that is, conditions experienced by the Army in the field. This leads to an interest in muscle artifacts in EEG, wearable devices, and single-trial methods, for example. The social and cognitive network science area is also both broad and important. Here, however, it is less clear that there is a vision of HRED’s role. There are multiple projects on many interesting topics, but it was harder to discern the thrust of the overall effort—a strategic plan for an organized, focused program is needed. In the area of human systems integration, the vision is more mature and customer-driven than in an emerging area like neuroscience. This is an observation, not a criticism. The group has a primary focus on modeling how complex systems will function once deployed. The HSI group might consider a long-term strategic planning exercise to identify where it might go next, but for the present, it has a clear role and is setting about fulfilling it to the extent that resources permit. The human robot interaction group would be aided by a clearer vision of where it sits in the broader field both inside and outside the Army. Within the Army, there may be more opportunities for collabo - ration and a sharing of resources. More contact with the broader academic community might help to identify the areas in which the HRI group could make its unique mark. The most creative energies of the soldier performance group seem to have been focused on the devel- opment of new facilities such as the EAR. Like HSI, now that those facilities are in place, the group would benefit from a planning exercise to envision what distinctive HRED work can be done with these tools. Resources The development of the EAR, the omnidirectional treadmill, and other facilities shows HRED’s ability to muster the funding and the skills to create world-class facilities. Ongoing efforts of a similar variety will be needed in other areas. The neuroscience group in particular is likely to need substantial investments in equipment and facilities if it is to live up to its potential. This is simply a recognition that neuroscience is a growing program. In other cases, the resources that could be developed are human resources. Many parts of HRED could readily absorb new hires. The HSI IMPRINT effort is one example.

OCR for page 31
45 HUMAN RESEARCH AND ENGINEERING DIRECTORATE Research Quality As a broad generalization, the HRED work is being done at a high level of competence. As noted above, there might be questions about why a specific study was worth doing, but as a general rule the mechanical details of research are well done. Questions have been raised in the area of statistical analysis and interpretation. In some cases, the analysis did not seem to be the correct analysis. In other cases, the distinction between statistical significance and scientific or practical significance was lost. These issues tend to arise in presentations by junior investigators and point to the ongoing need for mentorship, “brown-bag presentations” of research to internal groups, and the vetting of research by the processes of peer review provided by participation in national and international meetings and the submission of work to rigorous, peer-reviewed journals. Scientific Output In terms of its scientific output, HRED has increased its emphasis on publishing work in the open, peer-reviewed literature, but for a group the size of HRED, that output remains modest. The bulk of publications are book chapters, conference proceedings, and technical reports. Although useful, this output is not the same as publication in journals that are routinely indexed in databases (e.g., PubMed) and whose citations would be tracked by ISI Web of Knowledge or other citation indexes. Publication in these outlets is, in a sense, the currency of academic science and the most direct way for work in one laboratory to influence work in other laboratories. Promotion cases in university departments are heav - ily influenced by the quality and quantity of such publications. The current output at HRED would not support academic promotion in many cases. The Human Research and Engineering Directorate should give stronger consideration to the publica - tion of path-breaking HRED research in high-impact scientific journals.