Benefits of Peer Review
The OST technology-development program involves the expenditure of hundreds of millions of increasingly scarce federal discretionary dollars each year. It is particularly important that decisions about investment of those funds are based on sound criteria (both technical and nontechnical), and that the decision-making process is respected by all parties in the technology development program. Rigorous peer review can be an important tool for OST in meeting these objectives, as well as improving those projects that are funded.
Effective peer review is first and foremost a valuable way to increase the technical quality of projects in a program. The 1995 CEMT report recommended development and implementation of such a peer review program for the OST technology-development program for just this reason (NRC, 1995b). When peer review results are used to improve the quality of a decision process (e.g., selection of proposals, prioritization of projects for funding), it also enhances the credibility of the decisions. Use of peer review therefore provides observers some confidence that decisions are consistent with the best available scientific and technical information.
When the results of peer reviews are used as important input into funding decisions early in the technology-development process, they also can dramatically increase the productive efficiency of a program. Projects that lack technical merit or have a low likelihood of success can be discontinued early in the research and development cycle (before large investments of funds are made) and the savings can be reallocated to other projects that show greater promise of success. In addition, a greater number of alternative projects can be supported in the early stages of the development cycle. In studies of corporate product development programs, Cooper (1993) has shown that such early decisions can result in a greater than 50 percent overall increase in productive efficiency.
Although "internal reviews" play a valuable role (e.g., in matching projects to program needs), for a number of reasons, peer review, if conducted effectively, can enhance both program quality and the credibility of decision making. These include the following:
- Independent experts who are newly exposed to a project often can recognize technical strengths, weaknesses, and ways to improve the project that may have been overlooked by those close to the project (Bozeman, 1993).
- Drawing from a large pool of independent, external experts can provide more breadth and depth of expertise to the analysis than that available within the internal reviewer pool, resulting in a more effective and meaningful review (NRC, 1995a). For example, such experts may be more effective than internal reviewers in evaluating a project in the context of other comparable or alternative technologies available in the private sector, other government agencies, or other countries.
- Independent experts often can be more open, frank, and challenging to the status quo in their comments than internal reviewers, who may feel constrained by organizational concerns.
- External review can enhance the credibility of the review process by avoiding both the reality and the appearance of conflict of interest (Kostoff, 1997a).
- Peer reviews conducted publicly (i.e., using known reviewers following an established process) that provide immediate feedback in the reviewers' own words can enhance credibility by increasing confidence in the process (Royal Society, 1995; NRC, 1997).
Of course, peer reviews in and of themselves cannot assure the success of a project or program. Effective peer review can increase the probability of project and program success, however. The realization of these benefits requires that the process of peer review be effective and credible and that peer review results are used as an important input in making decisions regarding future support for the reviewed project (Chubin and Hackett, 1990).
The following section of this report includes an initial assessment of OST's new peer review program and offers some general recommendations on how OST could improve both the effectiveness of its program and the usefulness of peer review products (i.e., reports) in helping to make programmatic decisions.
