ASSESSMENT AT THE ARMY RESEARCH LABORATORY
For nearly two decades, the Army Research Laboratory (ARL) has sponsored assessments by the National Research Council of the technical quality of the work at the laboratory’s six directorates, which perform research and development (R&D) in the areas of computational and information sciences, human research and engineering, sensors and electron devices, vehicle technology, weapons and materials research, and survivability/lethality analysis. Accordingly, the NRC appoints six panels of experts that provide annual assessments of the scientific and technical quality of the work at the ARL. These panels provide biennial reports that summarize their findings and recommendations related to the quality and appropriateness of R&D for each of ARL’s technical business areas. The reports are delivered to the Army sponsor and to Army and Department of Defense (DOD) stakeholders and are made available to the public.
The panels are charged to address the following items:
• Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally?
• Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?
• Does the research employ the appropriate laboratory equipment and/or numerical models?
• Are the qualifications of the research team compatible with the research challenge?
• Are the facilities and laboratory equipment state of the art?
• Does the research reflect an understanding of the Army’s requirement for the research or the analysis?
• Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?
In addition to the panel’s addressing of the items listed above, the NRC selects from the panel membership individuals who form ad hoc groups to respond to specific issues identified by the Director of the ARL. The assessments are conducted by annual panel visits to the ARL facilities and review of supporting documentation describing technical projects and programs, equipment and facilities, and staff backgrounds and characteristics. The interactions between the panels and the ARL staff include dialogue during presentations, tours, demonstrations, poster sessions, and other means of presenting the ARL technical work, during which panel members seek clarification of facts and additional contextual information relevant to the assessments. Dialogue also occurs at the end of each review meeting, and panel chairs often engage in follow-up discussions with ARL staff. The purpose of these dialogues is to provide the opportunity for panel members to gather (and for ARL staff to provide) accurate, substantive, relevant information and to clarify contextual factors that may relate to the impressions under formulation by the panels; the dialogues are important so that subsequent findings, conclusions, and recommendations in the biennial report are based on accurate facts and adequate understanding.
As can be discerned from the examples of NIST and ARL assessments, there are various forms of peer review, each tailored to the specific type of organization. Considerable advantage to the process can be gained from the continuity of peer committee membership over time to