The overview briefings brought the panels up to date on ARL’s long-range planning. This context-building step is needed because the panels are purposely composed mostly of people who—while experts in the technical fields covered by the directorates(s) that they review—are not engaged in work focused on ARL matters. Technical briefings for the panels focused on the R&D goals, strategies, methodologies, and results of selected projects at the laboratory. Briefings were targeted toward coverage of a representative sample of each directorate’s work over the 2-year assessment cycle. Briefings included poster sessions that allowed direct interaction among the panelists and staff of projects that either were not covered in the briefings or had been covered in prior years.
Ample time during both overview and technical briefings was devoted to discussion, which enabled panel members to pose questions and ARL staff to provide additional technical and contextual information to clarify panel members’ understanding. The panels also devoted sufficient time to closed-session deliberations, during which they developed findings and identified important questions or gaps in panel understanding. Those questions or gaps were discussed during follow-up sessions with ARL staff so that the panel was confident of the accuracy and completeness of its assessments. Panel members continued to refine their findings, conclusions, and recommendations during written exchanges and teleconferences among themselves after the meetings.
In addition to the insights that they gained from the panel meetings, ARLTAB members received exposure to ARL and its staff at ARLTAB meetings each winter. The 2011 ARLTAB meeting focused on the ARL crosscutting research areas, and the 2012 ARLTAB meeting focused on refining elements of the assessment process, including read-ahead materials, review agendas, and expertise required within the panels.
During the assessment, the ARLTAB and its panels considered the following questions posed by the ARL Director:
1. Is the scientific quality of the research of technical quality comparable to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally?
2. Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?
3. Does the research employ the appropriate laboratory equipment and/or numerical models?
4. Are the qualifications of the research team compatible with the research challenge?
5. Are the facilities and laboratory equipment state of the art?
6. Does the research reflect an understanding of the Army’s requirement for the research or the analysis?
7. Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?
8. Is the work appropriate to the ARL niche?
9. Are there especially promising projects that, with application of adequate resources, could produce outstanding results that could be transitioned ultimately to the field?
Within the general framework described above, the ARLTAB also developed and the panels selectively applied detailed assessment criteria organized in the following six categories (Appendix C presents the complete set of assessment criteria):