At the request of the U.S. Army, on August 7-9, 2018, the National Academies of Sciences, Engineering, and Medicine’s Panel on Review of Extramural Basic Research at the Army Research Laboratory (ARL) met to review the programs of the Information Sciences Directorate (ISD) of the Army Research Office (ARO), which is an organizational unit within the ARL. The meeting was held at the ARO headquarters in Durham, North Carolina.
The panel’s review was guided by the following statement of task provided by the National Academies of Sciences, Engineering, and Medicine:
An ad hoc committee to be named the Panel on Review of Extramural Basic Research at the Army Research Laboratory, to be overseen by the Laboratory Assessments Board (LAB) of the Division on Engineering and Physical Sciences, will be appointed to provide annual assessments of the Army Research Office (ARO) programs. Each year one of the ARO’s three divisions (Information Sciences, Physical Sciences, and Engineering Sciences) will be assessed by a separately appointed panel. These assessments will address criteria to be defined by the ARO. Each year the panel will provide a report summarizing its findings, conclusions, and recommendations. The panel’s report will be made available to the public on the National Academies Press website and will be disseminated in accordance with National Academies policies.
The current report summarizes the 2018 findings of the Panel on Review of Extramural Basic Research at the ARL, which reviewed the programs at the ARO’s ISD. Over the 2019-2020 period the National Academies is scheduled to conduct reviews of the ARO’s Physical Sciences Directorate’s programs in physics, chemistry, and life sciences and its Engineering Sciences Directorate’s programs in mechanical sciences, electronics, materials science, and Earth sciences.
PROGRAMS WITHIN THE INFORMATION SCIENCES DIRECTORATE
The Army Research Laboratory’s ARO describes its mission as
To serve as the Army’s principal extramural basic research agency in the engineering, physical, and information sciences; developing and exploiting innovative advances to insure the Nation’s technological superiority. Basic research proposals from educational institutions, nonprofit organizations, and private industry are competitively selected and funded. ARO’s research mission represents the most long-range Army view for changes in its technology. ARO priorities fully integrate Army-wide, long-range planning for research, development, and acquisition. ARO executes its mission through conduct of an aggressive basic science research program on behalf of the Army so that cutting-edge scientific discoveries and the general store of scientific knowledge will be optimally used to develop and improve weapons systems that establish land force dominance. The ARO research program consists principally of extramural
academic research efforts consisting of single investigator efforts, university-affiliated research centers, and specially tailored outreach programs.”1
Research programs in the ISD are focused on discovering, understanding, and exploiting the mathematical, computational, and algorithmic foundations that are expected to create revolutionary capabilities for the future Army. Discoveries in this area are expected to lead to capabilities in materials, the information domain, and soldier performance augmentation, well beyond the limits facing today’s Army.2 In addition, such discoveries are intended to help prevent technological surprises. The ISD’s programs are organized in three divisions: Computing Science, Network Science, and Mathematical Sciences.
There are key methodological differences between ARO and research funding entities such as the National Science Foundation (NSF) and the Defense Advanced Research Projects Agency (DARPA). ARO program managers energetically try to create communities with workshops, visits to universities, and talks. ARO program managers engage in an interactive process with proposers to mature ideas into projects through discussion, white papers, and the Short-Term Innovative Research program. ARO program managers are hands-on managers of their projects; this may be a function of the process whereby program managers work with potential proposers (which NSF does not generally do)—the ARO process seems to resemble DARPA’s management style. ARO program managers view as a transition a follow-on effort at another government research funding agency that stems from one of their projects. This has two important benefits: (1) ARO program managers seek opportunities, in particular with other Department of Defense (DoD) components such as DARPA, to keep the basic research results alive in the research and development ecosystem; and (2) ARO program managers are encouraged to trace progress of an idea from its origins in basic research through development, deployment, and use.
Computing Science Division
The vision for the Computing Science Division is to conceive of and develop transformational research programs in the computing sciences for the U.S. Army, exploit new computing paradigms and novel information processing techniques, and provide the scientific foundation to create revolutionary capabilities for the future warfighter. The division has selected its areas of focus to complement work supported by other agencies and does coordinate extensively but informally with them. The intent is to conduct longer term research in areas of Army-specific need that is not addressed by commercial and other government entities. This is a challenge in computing, because the rate of change is so rapid, particularly since the end of Dennard scaling and the rise of data-centric computing. Needs are assessed annually; examples of currently targeted Army needs are modeling of adversaries’ learning, behavior, and social/cultural factors; interaction of soldiers with autonomous systems; and modeling of soldier situational awareness and decision making.
Overall, the scientific strategy and selection of projects were of high quality. The principal investigators (PIs) engaged for the selected projects were highly qualified, and the resulting science was of high caliber. The scientific objectives were generally focused on nearer term opportunities; longer term opportunities could be considered, and higher risk, potentially higher payoff topics could be included in the portfolio.
The programs generally performed very well in terms of funding leverage, relevance to Army needs, number and quality of publications, students supported, and transitions. The mapping of project accomplishments to programs’ strategic plans was not always clear, and consistent, meaningful metrics
for assessing progress were generally lacking. The appendix of this report lists a broad set of metrics that ARO could consider for assessment of its programs.
The division’s programs showed impressive examples of transitions to other Army and wider DoD research and development elements and in some cases to commercial organizations.
Network Science Division
The vision for the Network Sciences Division is to characterize, logically and quantitatively, the emerging macro properties in multigenre networks made up of autonomous agents, human networks, online social networks, and communication networks, leading to design of robust networks with predictable properties.
The division has a unifying scientific vision defining the program area. Scientific objectives were given for fulfilling this vision. Thrust areas were defined to achieve these program objectives. Across the division, the overall scientific quality is high, although some specific programs and investment areas are stronger than others. New areas identified for investment are unique and promising, with strong possibilities for contributing to the Army science and technology (S&T) mission.
The division’s program managers evinced a high level of engagement in community building and discipline building, in venues such as disciplinary meetings and academic institutions. They showed a strong sense of stewardship for these communities, particularly where the division pursued a distinct strategy, as in the social and cognitive networking area. Across the division, program managers are actively seeking emerging developments in relevant scientific fields that can help move their programs forward in useful ways, as well as working to capitalize on recent major scientific advances. It is important that the program managers continue to seek ripe opportunities at the gaps between major fields (e.g., self-organizing biological network structures for applications in medicine) and continue to collaborate across disciplines.
Programs across the division evidenced forethought and focus. As a result, all programs showed strong evidence of recent and ongoing transitions to applied research programs within the Army, as well as several to the broader defense science community.
In the area of quantum networking, considerable benefit could be gained from allying with other research thrusts elsewhere in ARO, the ARL, or elsewhere in DoD to obtain a combination of basic science (e.g., theoretical analyses or mathematical models) and the best experimental science.
Mathematical Sciences Division
The vision for the Mathematical Sciences Division is to develop mathematical understanding and methods that enable fundamental investigations and disciplinary progress in a variety of physical, biological, engineering, and informational areas of study, providing the scientific foundation for revolutionary capabilities for the future warfighter.
The investigators are producing high-quality research, and the subject areas seem to be appropriate and of use to the Army. Examples include projects in quantum annealing, contributions to health monitoring, and contributions to helicopter engine monitoring. The program managers are well qualified and maintain a close and continuous contact with PIs. This way of operating promotes two-way contact and can help to ensure that information is shared. By and large, the program managers followed a deliberate and reasoned process behind the choice of project areas.
The PIs and research outcomes are of generally high quality. The quality of the research outcomes has been tangibly demonstrated through the transition of a number of projects to Army applied research and development activities. The research addresses problems of importance to the Army and DoD. Programs and projects tend to address problems or use approaches that are not considered by other funding agencies such as NSF or the National Institutes of Health.
The division’s implementation of ISD strategic initiatives within the research portfolios of its program managers reflects a view of research approaches that is very much tied to the interests and expertise of the program managers. The division would benefit from enhanced consideration of alternative approaches to address research problems. The division would also benefit from a systematic addition of midperformance period review of all projects for which this is not performed. The feedback from these reviews would provide a basis for the refinement of future funding allocations.
There are several examples of short-term funding (e.g., 6-9 month grants) being used to explore new lines of research. The division’s program might be strengthened if such short-term funding were used as part of an entrepreneurial model with the goal of developing a diverse set of competing technical approaches to high-opportunity topics.
Evidence of a close connection between the division and higher Army echelons was not apparent. Such connections might exist, but they were not elucidated. However, there have been a number of successful transitions of research to DoD and elsewhere. In some cases, program managers have structured projects creatively to enhance potential for transition to practice. For example, researchers who are strong in fundamental work have been paired with researchers with a demonstrated record in translation. Based on the success of the ad hoc pairings to date, the division might benefit from making this a regular practice.
CONCLUSIONS AND RECOMMENDATIONS
Overall, the ISD is producing work of high scientific quality. In general, the ISD program managers are well qualified, and the PIs that were selected for the funded projects are of high caliber. In general, the funded research is relevant to the Army’s S&T mission; there were many examples of transitions of the research to the Army and to the DoD community more broadly.
While the Information Sciences Directorate is producing high-quality research overall, the ISD programs did not evince a clear and consistent set of metrics by which to evaluate program impact and effectiveness. It is necessary that metrics be counted by consistent and transparent methods (e.g., what is meant by a publication, how to count graduate students who are supported by multiple programs, how to assess caliber of awards and recognition) to facilitate measurement of progress. Metrics should reflect accomplishments actually attributable to the ARO projects. The appendix of this report lists a broad set of metrics that ARO could consider for assessment of its programs.
Recommendation 1: The Information Sciences Directorate should develop and apply a set of clear and consistent metrics by which to evaluate program impact and effectiveness.
The directorate follows a system of establishing personal connections between the program managers and their PIs; this almost amounts to collaboration. This system is effective, but it runs the risk that research foci might not change on appropriate time scales and that promising alternative approaches to problems might be missed if they fall outside the knowledge and experience base of the program managers. The system would benefit from deliberate efforts to inject more competition among different research approaches. This would include more rapid turnover in the PI base.
Recommendation 2: The Information Sciences Directorate should consider ways to expand the knowledge base beyond that possessed by the program managers when formulating approaches to selecting programs for funding.
Recommendation 3: To the extent that program managers in the Information Sciences Directorate demonstrate management of successful programs and expanded knowledge of their discipline and of relevant opportunities to support research with potential application to Army needs, they should be encouraged to exercise their vision for basic science to meet
Army needs and be encouraged to maintain their entrepreneurial style in program management.
Recommendation 4: The Information Sciences Directorate should consider shorter time scales and more rapid turnover of the principal investigator base for projects that are not jointly funded or targeted for long-term funding by collaborating Information Sciences Directorate divisions, Army Research Office (ARO) directorates, or other funding agencies. Consideration should include potential impacts on graduate students supporting funded projects, should ARO deem graduate student support a project goal.
The ARO supports research by PIs and by centers of multiple researchers. In contrast with single investigator programs, the Multidisciplinary University Research Initiative (MURI) programs at the ARO support centers whose efforts intersect more than one traditional research specialty, typically at $1.25 million per year for 5 years. Research topics increasingly benefit from such multidisciplinary participation, even in pairs or small sets of investigators and over shorter time periods. Including in such collaborations researchers with knowledge of transitions would be useful.
Recommendation 5: The Information Sciences Directorate should consider funding mechanisms to encourage pairs or small sets of researchers from divergent perspectives to work on the same problem.
The program managers in the Information Sciences Directorate evinced varying levels of engagement with other DoD research, development, and funding agencies such as the Office of Naval Research, the Air Force Office of Scientific Research, DARPA, and other elements within the Army, such as the Research, Development, and Engineering Centers. Such engagement is important for the maintenance of shared situational awareness and is a key enabler for ARO to continue to “outpunch its weight.”
Recommendation 6: Program managers within the Information Sciences Directorate should maintain and seek to expand their engagement with other Department of Defense funding agencies such as the Office of Naval Research, the Air Force Office of Scientific Research, the Defense Advanced Research Projects Agency, and other elements within the Army.
Diversity of gender, age, and geographic location was acknowledged across the ARO as requiring attention.
Recommendation 7: The Information Sciences Directorate should continue encouraging the participation of females and minorities in research funded by the Army Research Office and should collect statistics to track diversity in the broad sense, including gender, age, and geographic location.
The material provided for the review described a limited set of projects funded by the programs. This review could have been augmented by providing copies of the broad agency announcements for all of the currently funded projects, a more complete and organized description of all of the projects, and abstracts of the projects.
There was a wide variety in the content and relevance of the presentations about individual projects. A consistent template for the description of projects could address in sufficient detail such questions as the following:
- How does the project map to the long-term vision for the program?
- What are major consequences for the science if the project succeeds?
- Why is the problem difficult and what are the major technical risks?
- What is the network of contacts involved in the project?
- How, specifically, does the project address one or more critical challenges that the Army of the future will face?