Board (AFSAB), the Naval Studies Board, and the Naval Research Advisory Committee. These groups differ in their operations. Some are involved in detailed technical studies; others restrict themselves to policy studies. All of them come under the FACA (with modified FACA rules applied to the Naval Studies Board). On occasion, ad hoc advisory committees may be set up, often by direction of the U.S. Congress. Congressional committees, the Government Accountability Office, and even the Library of Congress have also done policy studies of the laboratories.
Interviews with senior officials at a number of federal laboratories have revealed several different models for peer review. Some are formal review panels that come under the FACA; others are formal but are managed internally by the service or the laboratory being assessed.
An example of one managed by a service is the review by the Air Force Science Advisory Board. The AFSAB, a formal Federal Advisory Committee, reports to the Air Force Chief of Staff and the Secretary of the Air Force. Among other duties, the AFSAB reviews in depth the scientific work of the Air Force Research Laboratory (AFRL) on a 2-year cycle. The review covers the 6.1 through 6.3 work and also work sponsored by downstream users, such as program managers or executive officers. One panel has been created for each AFRL directorate. Panel members are selected by the AFSAB; all are external to the Air Force. Most are members of the AFSAB, but when necessary the AFSAB brings in consultants. At least one member of each panel must be a member of the National Academies. The AFSAB conducts reviews of each directorate, devoting a full week to each. The panels provide an exit briefing and a formal report. The reports are sometimes made available to the public, but some may be restricted (for official use only). The panels look at four factors for the programs: technical work, relevance for the near term (5 years), future impacts, and resources. When evaluating the 6.1 programs, the AFSAB does not look at near-term impacts. For the technical work, it evaluates technical innovation, technical rigor, productivity, and collaboration.
Two external groups have been established to look at various technical topics for the Navy: the Naval Research Advisory Committee and the National Research Council’s Naval Studies Board created by the Chief of Naval Operations. Both groups consider the impact of technical developments on the future of naval forces. For a detailed technical review of its technical base research programs, the Naval Research Laboratory establishes and manages its own peer review. The NRL has seven focus areas: materials and chemistry, electronics, battlespace environment, undersea warfare, electromagnetic warfare, space research/space technology, and information technology. The technical review of the technical base covers about one-third of the total research program each year. The NRL selects members for the external review panels for each focus area. A panel typically has four to six members, drawn from academia and elsewhere, along with at least one member of the National Academies. The NRL asserts that these members are unbiased. The panels meet for from 2 to 4 days, with time for immersion in the laboratories and discussion with the staff. The panels give exit briefings to the NRL management. Subsequently, they submit a formal written report of about 10 to 15 pages, and the NRL responds to this report in writing. The panels evaluate the programs for scientific merit: they examine the research approach, the credentials of the staff, a project’s relevance, equipment, and costs. Details of these differ for 6.1 and 6.2. programs. For a 6.1 program, the panel looks for work that seeks to expand the frontiers of known science; for a 6.2 program, the panel looks for whether the NRL is investigating and developing recent advances in science and technology. The panels have seven categories of evaluation containing metrics. For the customer-funded work, the criteria used are those of the customers.