Summary
The statement of task that guided the work of the Army Research Laboratory Technical Assessment Board (ARLTAB) is as follows:
An ad hoc committee to be named the Army Research Laboratory Technical Assessment Board (ARLTAB), to be overseen by the Laboratory Assessments Board, will be appointed to continue the function of providing biennial assessments of the scientific and technical quality of the Army Research Laboratory (ARL). These assessments will include findings and recommendations related to the quality of ARL’s research, development, and analysis programs. While the primary role of the ARLTAB is to provide peer assessment, it may offer advice on related matters when requested by the ARL Director. The ARLTAB will provide an interim assessment report at the end of Year 1 of each 2-year assessment cycle and a final assessment report biennially. The ARLTAB will be assisted by up to seven separately appointed panels that will focus on particular portions of the ARL program. Each year, up to three additional panels may be appointed to assess special topics, at the request of the ARL Director.
During the 2019-2020 assessment, the ARLTAB is being assisted by five panels, each of which focuses on a portion of the program of the Combat Capabilities Development Command (CCDC) Army Research Laboratory (ARL) of the U.S. Army Futures Command’s (AFC) CCDC. ARL’s research core competencies include network and information sciences, computational sciences, human sciences, materials and manufacturing sciences, propulsion sciences, ballistic sciences, and protection sciences. For the extramural basic research programs at the Army Research Office (ARO), a subdivision of the ARL, each year the ARLTAB will be supported by an additional panel that will focus on selected ARO programs, and the ARLTAB will annually provide a separate assessment report.
This interim report summarizes the findings and recommendations for the first year of this biennial assessment; the current report addresses network and information sciences, computational sciences, and human sciences research core competencies; the remaining competencies will be assessed in 2020 and 2021. It needs to be noted that because a full spectrum of projects and programs within each of the ARL core competencies and the interrelated mapping across all projects and programs of the core competencies was not provided to the ARLTAB, this report represents the assessment on only the projects and programs presented.
During the first year, the following elements within the ARL research core competencies were examined: (1) networks cyber and information sciences; (2) computational sciences and atmospheric sciences; and (3) human sciences—human-autonomy team interactions and humans understanding autonomy; autonomy understanding humans and estimating human-autonomy team outcomes; human interest detection; cyber science and kinesiology; and neuroscience, training effectiveness, and strengthening teamwork for robust operations with novel groups (STRONG). A second, final report will subsume the findings of this interim report and add the findings from the second year of the review, during which the board will examine additional science and technology (S&T) core competency clusters within the ARL.
The mission of ARL is to discover, innovate, and transition S&T to ensure dominant strategic land power. ARL has maintained its organizational structure, consisting of five directorates—Computational and Information Sciences Directorate (CISD), Human Research and Engineering Directorate (HRED), Sensors and Electron Devices Directorate (SEDD), Vehicle Technology Directorate (VTD), and Weapons
and Materials Research Directorate (WMRD)—and the Army Research Office (ARO). The research portfolio has been organized into research core competencies, each of which describes related work supported by staff from multiple directorates. Appendix Table A.1 shows the directorates that supported each of the research core competencies during the 2019 review. ARL’s vision is compelling and raises expectations for an innovative program of research designed to be responsive to the needs of the capabilities for the Army of 2030 and beyond. The reorganization of the portfolio into research core competencies is promising, but it may take some time to transform and mature the program of work to consistently align with the needs and capabilities for the Army of 2030 and beyond.
Four major changes to the Army and the Army Research Laboratory (ARL) clearly present challenges to all research core competencies. These are the recent Army doctrinal changes to Multi-Domain Operations (MDO), the reorganization that put ARL under the newly formed Army Futures Command (AFC), the divestiture of 6.3 work to other organizations,1 and emphasis on “disruptive” technologies. Specifically, with the recent creation of the AFC, ARL has been charged to focus on foundational research; targeting—conducting research to drive change within, across, and between disciplines; creation of knowledge products for warfighting concept development of the future; and interacting with universities via the ARL Open Campus and the ARO.
In general, the quality of the research presented, the capabilities of the leadership, the knowledge and abilities of the investigators,2 and proposed future directions continue to improve. Significant gains were evident in publication rates,3 numbers of postdoctoral and visiting researchers,4 and collaborations with relevant peers outside ARL.5 The research work environments were impressive in terms of their unique and advanced technology capabilities to support research. Overall, these are outstanding accomplishments and mark an advance over prior years.
NETWORK AND INFORMATION SCIENCES
The research areas reviewed were information sciences and networks and cyber. The research projects under the banner of information sciences (IS) had a significant emphasis on artificial intelligence (AI) and machine learning (ML) as applied to diverse areas of Army relevance. Application areas included image understanding, automated language processing, augmented and virtual reality (AR/VR),
___________________
1 Parsing RDT&E funding by the character of the work, the Department of Defense (DoD) has established seven categories identified by a budget activity code (numbers 6.1-6.7) and a description. Budget activity code 6.1 is for basic research; 6.2 is for applied research; 6.3 is for advanced technology development; 6.4 is for advanced component development and prototypes; 6.5 is for systems development and demonstration; 6.6 is for research, development, test, and evaluation (RDT&E) management support; and 6.7 is for operational system development.
2 ARL has increased its civilian scientists and engineers (S&Es) with a Ph.D. degree from 434 to 665 from fiscal year (FY) 2006 to FY 2019. In FY 2019, 59 percent of its S&Es had a Ph.D. degree.
3 For example, in CISD (where network cyber and information sciences and computational sciences and atmospheric sciences core competencies reside) for FY 2019, there were 56 journal publications and 168 conference publications. And in HRED (where human sciences core competency resides) in the 2-year period from March 2017 to March 2019, there were 137 journal publications and 205 conference publications. ARL had 1,145 scientists and engineers as of May 24, 2019.
4 For example, in CISD for FY 2019 there were 27 visiting researchers.
5 ARL had over 515 active Cooperative Research and Development Agreements (CRADAs) with industry and academia as of June 18, 2019. Moreover, ARL’s Open Campus Initiative was started in FY 2014 to link ARL with the global research community. The partners and ARL S&Es work side by side in research facilities. The collaborations are focused on Army-specific challenges of mutual importance to all partners. Partners from Army, industry, and academia engage in research with shared access to people, infrastructure, and resources. More information on ARL’s Open Campus can be found at https://www.arl.army.mil/opencampus/, accessed March 2, 2020.
and learning for control. There was a focus on Multi-Domain Operations (MDO), and research projects were both foundational and disruptive, while maintaining Army relevance. There is a broad embrace of the transformative potential of AI and ML, especially in the context of how groups of people and autonomous systems can seamlessly collaborate, how technology can further aid with time-sensitive decision making in the presence of massive and diverse sources of information, and how virtual and augmented reality can be integrated in Army operations.
Research in the area of networks and cyber includes projects that fall into the general area of human-robot/machine interactions, with the two principal threads being scene narrative generation for humans by robots and robot learning from human demonstrations. This body of work is ambitious and has the potential to disrupt the way human-robot interactions are considered for future battlefields. Another important research methodology, especially for security issues, is to study systems that have both defense and offense techniques.
The research work in IS was assessed to be generally of high scientific quality, but not uniformly so. The research portfolio represents an appropriate balance of theoretical and experimental work, and many of the more mature programs—but not all—show a transition into practice or use by other areas. In most cases, the research projects reflected a good understanding of the problems being considered, an appropriate statement of the problem being pursued, a good knowledge of the appropriate methodologies to address the problem, and acquaintance with the state of the art and the relevant research pursued elsewhere. In many cases, the researchers are able to articulate Army relevance and identify research challenges that are unique to the Army’s operational needs. The researchers are well-qualified to carry out the research problems that they are pursuing and follow rigorous research methodologies and practices. Many of the projects have already resulted in publications in highly visible journals and selective conferences. The computation facilities and instrumentation required are adequate to the needs of the researchers.
The overall scientific quality of the research in the networks and cyber was high, and comparable to research conducted at top research universities and government and industry laboratories. Researchers were very familiar with the underlying science and relevant leading research published and performed elsewhere. In many cases, there are active communications or collaborations with researchers outside ARL, many at the forefront of their respective fields. In all cases, the researchers were aware of the potential challenges, risks, and risk mitigations associated with their projects. In most cases, the researchers were able to incorporate these challenges, risks, and risk mitigations into their research. There is an appropriate balance of theory, computational work, and physical experimentation to inform and investigate multiple areas of research. The facilities and supporting infrastructure are well-suited for collaborative work. There is a good mix of well-trained research personnel who also collaborate with researchers in a broad range of academic and industrial partners in addition to working with the ARL South and ARL West regional sites. The research staff in this area have received outstanding recognition and awards for their research and technical contributions.
COMPUTATIONAL AND ATMOSPHERIC SCIENCES
Projects reviewed related to Battlefield Environment Division (BED) atmospheric observations and modeling and computational sciences research, considering both selected, in-depth research presentations and informal discussions surrounding research posters. Projects presented in depth spanned turbulence modeling in complex environments, using a lattice-Boltzmann computational model, understanding the influence of forest canopies on atmospheric dynamics on complex terrain, uncertainty quantification modeling for atomistic-scale modeling, and AI/ML at the edge implemented in field programmable gate arrays (FPGAs). In addition, posters spanned atmospheric model prediction via radar data assimilation, meteorological sensor arrays, aircraft vortex and rotor wake characterization, and ML characterization of particle shapes.
While daunting, the repurposing of the ARL mission also presents great opportunities to restate, revise, or create a new vision, which would allow scientists to reevaluate their individual programs in terms of how they fit into the new “big picture.” For some projects, the new MDO viewpoint was an immediate and natural fit, while for others, the connections between project objectives and MDO were still being developed. Broadly, the work presented was of high quality, comparable to that conducted at major research universities or leading-edge federal research and development (R&D) organizations.
Examples of this high-quality research include the hierarchical multiscale (HMS) project, which continues to break new ground in mission-relevant research, by maintaining and now extending its quality and utility from earlier reviews. Similarly, the AI project, using FPGAs, seeks to build capabilities into Army weaponry (beginning with gunsights), via a framework that involves software, customizable hardware, and reduced convolutional neural nets (CNNs) to meet space, weight, power, and time-to-solution constraints. Similarly, the work on ML to characterize particle shapes using scattered light images has the potential for wide applicability throughout aerosol science and in the broad area of chemical or biological agent characterization.
There are a few areas of opportunity. As an enabling, broad-based capability, the computational sciences and BED need to maintain a critical mass of expertise, both for targeted projects and for collaborative engagement with other projects. Similarly, the BED atmospheric modeling research is foundational for many MDO activities, but it would benefit from stronger connections to specific projects.
In turn, computational science is becoming a prominent element in training AI/ML systems, such as work using physics-based simulation engines from video games as the source of data to train autonomous vehicles. Further, many of the technical challenges in modern ML involve problems that have been well investigated in computational science work, and emerging research in academia and industry that involves substantive collaborations between computational science and AI/ML is increasingly seen. CISD is well-poised to catalyze important new work that involves collaborations between computational and information sciences, an area that does not currently appear to be pursued.
In summary, there is continued research progress, particularly engagement with academic stakeholders via ARL’s distributed sites and collaborative academic projects. There is hiring and intellectual development of new postdoctoral associates and staff. ARL continues to foster collaboration across its internal organizational structure. Last, all projects would benefit from clearer metrics for research project success and associated project exit strategies, including transitions that maximize Army benefits.
HUMAN SCIENCES
The human sciences (HS) project areas reviewed were human-autonomy team interactions and humans understanding autonomy; autonomy understanding humans and estimating human-autonomy team outcomes; human interest detection; cyber science and kinesiology; neuroscience, training effectiveness, and strengthening teamwork for robust operations with novel groups (STRONG).
ARL’s HS core competency is focused on identifying, creating, and transitioning scientific discoveries and technological innovations underlying three research areas: cognitive dominance, readiness for technological complexity, and teaming overmatch. These areas are critical to the U.S. Army’s future technological superiority. This core competency area concentrates on high-risk and high-payoff transformational basic research with potential for having revolutionary impacts on the Army’s warfighting capabilities. The ultimate goal is to contribute to the creation of disruptive and game-changing soldier-centric technologies for the Army, while also anticipating technological surprises from potential adversaries.
ARL needs to continue to focus on its long-term vision for advancing basic science research in human sciences. Its leadership needs to engage in more dialogue regarding its research strategies with its front-line researchers as well as the greater scientific community. An approach that incorporates both bottom--
up and top-down approaches to advancing basic science research would strengthen the program and allow the ARL to advance its position to the forefront of basic research for mission-critical utility.
ARL continues to collaborate with universities through its state-of-art data collection facilities and through the STRONG program, with the goal of developing the foundation for enhanced teamwork within heterogeneous human-intelligent agent teams. Such continuing collaborations would develop the top talent needed among the next generation of researchers to realize ARL’s long-term research vision for advancing U.S. Army capabilities and to broaden the pool of trainees familiar with the unique challenges that the Army faces. As with the ARL Open Campus initiative, the richness of data that could be collected through ARL facilities is an inexpensive way to collaborate, and if strategically planned, could support several different laboratories asking the same questions from different angles to more robustly inform the research in these multifactorial environments with emergent outcomes.
Human-autonomy teaming needs to include many mission and team phases, including mutual training, planning, execution, and after-action reviews. “Training” is not used in the sense of procedures that are formally specified in another part of ARL, but rather highlights the need for most teams to develop shared understanding and expectations through shared experiences. Algorithms designed through ML based on a preconceived notion of team behavior may not lead to resilient and adaptive human-autonomy teams, so future research needs to consider how mutual adaptation of human to autonomy and vice versa leads to resilient and adaptive human-autonomy teams.
For the presentations of ARL’s current human interest detection (HID) research efforts, context is needed. ARL needs to provide past work in the literature, including previous work in attention focusing and ARL-supported work (especially in the Cognition and Ergonomics Collaborative Technology Alliance [CTA]). A framework is needed for binary target detection using the human interest measurements (probability of detection versus probability of false alarm and receiver operating characteristic [ROC]); this framework is likely to require adaptive decision fusion. A clear roadmap is needed, including potential dependency on related research from other programs, metrics for success, milestones, and timelines. Milestones could include three to five realistic test scenarios specifying how many subjects would be using the HID system in a field test, what tasks and targets will be looked at, what form factor and other wearability constraints would need to be met by the HID system, and what ROC or probability of detection/probability of false alarm performance points on the ROC curve would be achieved.
The cyber science group has made good progress in 2 years, including identifying areas where the Army has unique challenges. The cyber science group brings a unique skill set to the Army that could be left intact or allowed to grow to advance its science. The cyber science group possesses a skill set that could be attractive to other groups—for example, training or computer science. If that skill set were siphoned away into other groups within the Army, the human science that the group is advancing will suffer.
The kinesiology group is working in an important area of research. The research strives to ensure that physical agents such as exoskeletons are designed based on how humans need to move so that they can team well with the warfighter. For the future work that the group envisions, studying smart exoskeletons is an interesting path to pursue. However, this research effort needs to better understand the musculoskeletal “cost” of the technology in order to ensure mission readiness. Supporting the physical performance of tasks that will continue to be allocated to human soldiers is an important contribution of this group. The group can play an important role in identifying those tasks at which human soldiers (with or without augmentation) will excel or be able to keep pace with robotic soldiers, and vice versa, as robotic technology advances.
Changes made in the personnel, organization, and structure of the HS core competency area have provided the tools and opportunities needed to make it a national laboratory in the area of cognitive science and robotics for individual human performance as well as for human teams and mixed teams of humans and robots. The funding directed to this group by the Army has enabled the researchers to build the next generation of high-technology research tools with which to study human performance in combat simulations both with and without robotic teammates.
CROSSCUTTING RECOMMENDATIONS
Based on the 2019 reviews whose assessment is summarized in this interim report, the ARLTAB offers the following recommendations.
With finite resources and the broad array of research topics being pursued by ARL investigators, the probability of final project success will be enhanced through cognizance of outside efforts and, where appropriate, establishing formal collaborations. Establishing contacts will require, at a minimum, attendance at professional meetings and conferences and possible travel to and from leading institutions.
Crosscutting Recommendation 1: The Army Research Laboratory (ARL) should further encourage and facilitate all members of research projects, particularly junior members, to make the scientific contacts necessary to adequately assess their research in the context of the entire field.
There is significant attention given to AI and ML in research programs across different portfolios at ARL. At present, the scope of these research efforts is rather narrowly focused on mission-critical tasks. Fundamental research issues related to innovative ML techniques, AI implementation in resource-constrained environments, and trust and security of AI systems must still be addressed on a broader scale. It is also important to recognize that it is suboptimal to seek algorithmic advances in these areas without due consideration of hardware developments that are taking place in parallel. Given the potential of a disruptive impact of these technologies on Army operations, it is important to develop a comprehensive and integrative research plan in this emerging area. These technologies can have a transformational impact on key elements of Army operations.
Crosscutting Recommendation 2: The Army Research Laboratory (ARL) should emphasize the identification of a set of fundamental research questions underlying the current research portfolio that can provide a long-term focus in areas of artificial intelligence and machine learning.
Software development has advanced at a tremendous pace over the past few years. Much of the reason for this rapid development is the increasingly common practice of employing open source software to build software platforms. Because of this rapid pace, it will be difficult for ARL to remain competitive within the software development space. ARL needs to be a member of GitHub (a major open source group—see https://github.com/) if it is not already. Of course, classified data needs to be handled in other ways.
Crosscutting Recommendation 3: The Army Research Laboratory (ARL) should develop a mechanism for collaboration between ARL and industry on software development. Specifically, ARL should use and develop software platforms in collaboration with open source software libraries that will enable ARL to keep up to date and to rapidly develop software.
