Definition of Peer Review
Although one can argue legitimately that "peer review"4 is the name given to any judgment of technical5 merit by other experts working in or close to the field in question, the scientific and engineering communities generally use the term "peer review" in a narrower sense. In this report, we adopt these communities' sense of peer review, as articulated in the definition developed by the USNRC:
A peer review is a documented, critical review performed by peers [defined in the USNRC report as "a person having technical expertise in the subject matter to be reviewed (or a subset of the subject matter to be reviewed) to a degree at least equivalent to that needed for the original work"] who are independent of the work being reviewed. The peer's independence from the work being reviewed means that the peer, a) was not involved as a participant, supervisor, technical reviewer, or advisor in the work being reviewed, and b) to the extent practical, has sufficient freedom from funding considerations to assure the work is impartially reviewed.
A peer review is an in-depth critique of assumptions, calculations, extrapolations, alternate interpretations, methodology, and acceptance criteria employed, and of conclusions drawn in the original work. Peer reviews confirm the adequacy of the work. In contrast to peer review, the term "technical review" . . . refers to a review to verify compliance to predetermined requirements; industry standards; or common scientific, engineering, and industry practice. (USNRC, 1988, p. 2)
In this definition, the term peer review has the following characteristics:
- expert (including national/international perspectives on the issue),
- independent,
- external, and
- technical.
Most importantly, peer reviews must be carded out by independent reviewers who are experts in the technical issues relevant to the projects under review. Such reviewers must be highly qualified6 and independent in order to evaluate credibly the scientific and engineering merit of
the project with respect to current technologies, both domestic and international. In the report Allocating Federal Funds for Science and Technology (NRC, 1995a, p. 69), peers are defined as "established working scientists or engineers from diverse research institutions who are deeply knowledgeable about the field of study and who provide disinterested technical judgments as to the competence of the researchers, the scientific significance of the proposed work, the soundness of the research plan, and the likelihood of success." Note that such reviewers need not be expert in or familiar with the agency program or relevant contextual factors. Those are the proper province of agency management.
The USNRC's definition of "peer review" provides some guidance on the issue of potential conflict of interest by explicitly excluding potential reviewers who have been involved with the specific project being reviewed or who have financial interests in the outcome of the reviews. As Chubin and Hackett (1990) have pointed out, however, it is often difficult to identify true "experts" on a subject or technology who do not have some biases that could be perceived as a potential conflict of interest (e.g., competing interests, personal relationships). The committee will provide more specific recommendations on the issue of conflict of interest in its final report.
In the past, OST has used the term "peer review" generally to refer to internal reviews of OST projects by qualified EM technical staff who were not involved directly in the specific project under review. This use of terminology has caused confusion and misunderstanding within both OST and external review groups (e.g., GAO, NRC) who have continued to criticize OST for a lack of a credible peer review program. Although internal reviews of this type are necessary and should continue, they should not be confused with peer review because the term "internal peer review" is not consistent with the usual meaning of peer review (Bozeman, 1993). The committee believes that at least part of the criticism leveled at the OST project review process has resulted from inconsistent and inaccurate descriptions of the processes involved (e.g., internal peer review, technical peer review). To avoid misunderstanding, OST should restrict the term "peer review" to only those technical reviews conducted by independent, external experts. OST should adopt alternative terms, such as "technical review," for its internal reviews of scientific merit and pertinency. Careful attention to nomenclature will eliminate much of the confusion about the nature of OST's review process and could increase appreciation and respect for OST's new peer review program.