10

Crosscutting Recommendations

Almost all of the In-House Laboratory Independent Research (ILIR) projects involve basic research related to other work conducted at various U.S. Army Research, Development, and Engineering Centers (RDECs) and the Army Research Laboratory (ARL). Enhanced interactions with researchers in these organizations would help accelerate progress on ILIR projects and provide important sources of relevant knowledge, equipment, and software to Army ILIR researchers.

Recommendation 1: The RDECs should explore ways to create greater interactions with one another and with the Army Research Laboratory to enhance their projects.

Conducting basic research activities in an otherwise applied research environment can be very challenging because of a different philosophy and methodology with which such activities are executed. Attendance at national scientific meetings is essential for researchers to experience and assess the state of the art in their disciplines and expand scientific relationships. Building relations beyond the local universities to engage researchers across the United States can bring in additional expertise, potential experimental validation, and theoretical modeling as well as cross-pollination of ideas. For example, many of the projects increasingly employ advanced computational science, including multiscale materials simulations, deep learning algorithms, and multi-agent optimization, for which external expertise would be impactful and when expertise developed via ILIR projects could help others. It appeared that many of the best ILIR projects created synergies with and benefitted from external scientific input, while some of the projects that were less successful appeared to lack such helpful inputs and consequently made less progress.

Recommendation 2: The RDECs should facilitate evaluation and dissemination of ILIR through enhanced engagement with the broader scientific community, including increased presentation at relevant conferences and publication in peer-reviewed journals.

It is important for the Army to have high-caliber researchers who can bring critical and up-to-date thinking to technical challenges that are unique to the Army. One characteristic of such researchers is a demonstrated understanding of the current state of the art, especially when starting a new project, evaluating the novelty of a research result, or documenting a project for publication. State of the art in this sense includes what information is available from the open literature, what relevant tools are available for use in research, and what referable data sets are available for comparative purposes. Many of the high-quality ILIR projects demonstrated such an understanding. Projects that have encountered difficulties often suffered from missing some crucial understanding that has been documented in the literature.

Recommendation 3: The RDECs should ensure that the principal investigators of ILIR projects demonstrate an understanding of the state of the art in the project’s field at the project’s inception and at appropriate project reviews.

Some metrics facilitate internal evaluation of Army-specific research projects, and others represent scientific progress in general; the latter are external, usually quantitative, measures of success acknowledged by the broader research community. Metrics can be qualitative or quantitative. Metrics may include such questions as the following:

• What are the objectives achieved by a project as compared with a measure of feasibility, which requires consideration of available resources and strategic or mission goals?
• Do the reported results address the stated hypotheses?
• Did the research uncover significant or high-impact findings?
• Are the findings unexpected?
• Are the findings beyond the scope of the original hypotheses?
• Are the results breakthroughs or innovations?
• Are the results broad or niche-specific?
• Is the quality of the technical work consistent with the project’s stated objectives and research standards of practice?
• Is the number of refereed publications, conference presentations, and patents appropriate?
• Were unforeseen obstacles overcome?
• Did the project meet schedule and level-of-effort targets?

Such metrics influence management decisions necessary to facilitate efficient conduct of overall ILIR activity at the RDECs. Metrics reflect to a significant degree issues of efficient research management, including setting strategic scope, planning and securing future funding, training staff, and evaluating staff progress in reaching goals by rational critical paths within consistent evaluation criteria. There is sometimes a competition for limited resources that drives the directions of research to meet niche Army needs without sacrificing broader scientific justification.

Recommendation 4: The RDECs should consider developing and implementing technical metrics to gauge progress in their research that are consistent across the entire ILIR program. Given that ILIR projects are restricted to basic research, these metrics should include the boldness and quality of scientific research as well as the possible transition to practical applications.

The ILIR program is a beneficial activity for the Army that can contribute to workforce development by allowing Army research personnel to think creatively and to aim for transformative enhancements in the Army’s capabilities. Because basic research focuses on fundamentals, there is a need to engage the expertise and insight of other investigators in the field. Many presentations did not acknowledge the

broader scientific picture or make efforts to develop concepts that could be generalized. Many RDECs do not avail themselves of external reviewers for project selection and objective evaluation of progress, and the current narrow focus may be a result of reviews and selection done only within the RDECs.

Recommendation 5: The RDECs should enhance the quality of ILIR by using a broader external peer-review process for selection, review, evaluation, and termination of their projects.

The committee judged the amount of ILIR funding to be rather small. It varied between $1 million and $2.2 million across the RDECs that reported ILIR funding (AMRDEC, CERDEC, and SMDC did not report ILIR funding levels).

Within the ILIR budgets at each RDEC, the amount of resources allocated to some projects is limited, and there needs to be a focus on getting the most impact from these available resources. To establish a balanced portfolio, the RDECs may consider conducting fewer projects at higher levels of support for the chosen projects. It is noted that if the level of ILIR funding is maintained and the number of projects is reduced to achieve improvement in research quality, this may reduce the opportunity to recruit and retain scientific and engineering personnel, which is a goal of the ILIR program. It was not apparent that sufficient oversight was provided to ensure effective utilization and leverage of these limited resources to maximize the overall impact of each project.

In addition, although future plans were discussed, it was not clear how to assure successful execution to completion given changes in the investigators’ team membership or availability of research infrastructure. Although the Science, Mathematics and Research for Transformation (SMART) Scholarship program is available for recruiting talent, it does not seem to be effectively leveraged and broadly communicated to the workforce. The RDECs are not fully utilizing this opportunity.

The committee acknowledges that it is challenging to develop project portfolios that reflect consideration of the appropriate number and nature of projects, the goal to recruit and retain talents, resource availability and allocation, and the organization’s mission. Addressing this challenge will require careful examination of the values and priorities that the Army places on these factors, as well as analysis of the pragmatic constraints imposed by limited resources.

Recommendation 6: The RDECs should ensure that ILIR projects are provided with adequate resources, expertise, and researchers’ time to successfully accomplish the objectives of the projects.

In some cases, it was not made clear whether there is a process for researchers to receive feedback and oversight on their work and their thinking processes throughout their projects. Periodic (e.g., quarterly) status reviews by RDEC managers—in addition to feedback at the end of a project—would be beneficial in increasing the chance of project success.

It was also not made clear whether junior researchers receive systematic mentoring by senior personnel throughout the course of a project. Junior personnel would benefit greatly from mentorship with respect to the selection, planning, and execution of the ILIR projects, as well as the presentation of the project results and the planning for future research directions.

As an example in modeling and simulation, it is critical that the developed models adequately represent the real world. Senior researchers can help facilitate an understanding of the implementation of best practices in verification, validation, and uncertainty quantification. It is also worthwhile to explicitly endorse mentorship activities as a positive contribution to the senior researchers’ annual performance evaluation.

Recommendation 7: The RDECs should establish a process for and devote resources to providing effective project oversight and mentorship for junior ILIR researchers.