Peer Review in the Department of Energy-Office of Science and Technology: Interim Report (1997)

The Office of Science and Technology (OST) of the U.S. Department of Energy's (DOE's) Office of Environmental Management (EM) promotes the development of new and improved technologies to lower cleanup costs and risks and to improve cleanup capabilities throughout the nation's nuclear weapons complex. The annual budget for technology development activities within OST in fiscal year 1997 is approximately $300 million, which supports nearly 300 research and development projects at universities, national laboratories, and private-sector companies. These projects are chosen for new and continued funding through a complex technology-selection process, which uses the results from various types of reviews (see Appendix C), including programmatic reviews, technical assessment reviews, and peer reviews.¹

Several recent National Research Council (NRC) reports evaluated DOE-OST's technology-selection process and recommended that OST develop and apply an independent, external review process to all of its technology-development programs (NRC, 1995b,c, 1996). These findings were echoed in a subsequent General Accounting Office (GAO) report, which concluded that ''although the lead sites used significantly different systems to select projects, none of them used disinterested reviewers to determine the technical merit of the proposed work'' (GAO, 1996, p. 7). In response to these NRC and GAO reports, the OST recently has instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development.

OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. This is the first of two reports to be prepared by this committee on OST's new peer review program. OST requested this interim report to provide a preliminary assessment of OST's new peer review program. In the final report, the committee will provide a more detailed assessment of OST's peer review program after its first complete annual cycle.

The committee finds that OST has made progress in its implementation of the peer review program, especially in an environment that has, heretofore, not fully recognized the value of independent assessment of technologies. This new program apparently marks the first time that OST has applied peer review (i.e., technical review by independent, external experts)² to evaluate the technical merit of technology projects as part of its technology-selection process. The program has developed a process for selecting reviewers, developing technology-specific review criteria, and executing peer reviews. And although only a small percentage of OST's technology projects have been reviewed to date, the OST's peer review program appears to have the potential to be fair and credible. To fully achieve the objectives of this program, however, OST must continue to address a number of key issues that hinder the program's successful implementation.

One such problem has been the OST's use of the term "peer review" to refer to technical reviews conducted by EM technical staff or other reviewers not independent of the project under review. Such use of the term "peer review" has caused confusion and misunderstanding within both OST and external review groups (e.g., NRC, GAO), which have continued to criticize OST for a lack of a credible peer review program. To avoid misunderstanding, OST should restrict the term "peer review" to only those technical reviews conducted by independent, external experts. OST should adopt alternative terms such as "technical review" for its internal reviews of scientific merit and pertinency.

The committee has noted a lack of timeliness in a number of the reviews conducted under the new system. For example, some peer reviews have been conducted after OST apparently had already committed to fund the project's next stage of development, or even after a project nearly had been completed. In both such instances the main benefits of peer review were foregone because of the untimely review. The committee also has observed instances where peer reviews of specific projects were canceled shortly before the scheduled peer reviews. To address these issues, the committee recommends that OST develop a targeted plan for the peer review program. In developing this plan, OST will need to consider factors such as how many of its technology projects can be peer reviewed, realistic schedules for the reviews, and the peer review program budget. To be effective, this plan also must assure that peer reviews are conducted early enough in the budget cycle to allow peer review results to be used as an input to meaningful funding decisions.

Because it is a new program, OST's peer review program has been able to review only a small percentage of the technologies currently under development within OST. As a result, OST currently funds a large number of technologies that never have been peer reviewed, and many of these projects are in the later stages of development. To address these issues, the committee recommends that OST develop a rigorous process for selecting projects to be peer reviewed. To be fair and credible, this process should employ well-defined project selection criteria, and OST peer review program staff should be directly involved in making decisions regarding which projects will be reviewed. Due to the large number of projects in the later stages of development, the committee encourages OST to focus much of the short-term peer review efforts on high-budget, late-stage projects that never have been peer reviewed, but that still face upcoming decisions with major programmatic and/or funding implications. In the long term, however, the committee encourages OST to focus on proposals for new projects entering OST's development process to help ensure that only projects with sufficient technical merit are supported by OST.

OST's peer review program has a list of nine general review criteria that includes broad issues such as cost effectiveness, reduced risk, regulatory acceptability, and public acceptance that address many nontechnical considerations which may not be appropriate for peer reviews. The committee recommends that OST revise these general nontechnical criteria to focus on technical aspects of these issues, or remove them from the list of review criteria. In practice, the nine general review criteria are often modified to technology-specific review criteria intended to focus the review on issues of particular interest to OST. The committee encourages OST to continue its practice of developing a minimum number of technology-specific review criteria because it allows OST peer review program staff to focus the reviews on particularly important issues, provided these technology-specific criteria address technical issues and do not obscure or preempt the principal goal of determining technical merit. The committee recommends that OST develop a well-defined general set of technical criteria for peer reviews, to be augmented by technology-specific criteria as needed for particular reviews.

Another important factor in successful peer review programs is that the objectives of the reviews and the review criteria are defined clearly and are understood by all those involved in the peer review. OST policy requires OST program managers to include a statement of desired "scope" and "purpose" when they request a peer review. The committee encourages OST to use these statements to communicate the objectives of the review clearly to all involved in the peer review (i.e., peer reviewers, principal investigators [PIs], and observers).

The selection of peer reviewers is a critically important step in the peer review process. The committee finds that the criteria used to select reviewers are adequate to ensure the technical credibility of the peer review panel. To ensure the independence of the peer reviewers, however, the committee recommends that OST also include a criterion that explicitly excludes EM staff and contractors with real or potential conflicts of interests, including all OST staff and contractors, from consideration as reviewers. OST also should consider modifying the criteria to emphasize expertise relevant to the review.

In reviewer selection, it is also necessary that all relevant areas of expertise required to address the review criteria be represented on the peer review panel. This is particularly important for the very diverse and complex technologies being developed for environmental cleanup of the DOE complex. The size of the review panel should depend on a number of factors, including the complexity and number of projects being reviewed, the stage of development of the project, and the specific review criteria for the review. One approach that could be used to help identify reviewers with relevant expertise would be to develop and use a large data base of potential reviewers.

For a peer review program to be effective, the results should be used as important factors in making decisions regarding future support for the project and/or as input to improve the technical merit of the project. In order to achieve this objective, peer review results should be summarized in clearly written reports that provide the rationale for their conclusions and

recommendations. OST has made progress in improving the technical quality of peer review reports by requiring more complete explanations of the reasoning for the panel's conclusions and recommendations, by adding a section that summarizes the review criteria, and by including short biographical sketches of the peer reviewers. The reports could be improved further, however, by also including a statement of the objectives of the review and a list of references used in the analysis.

OST also should strive to implement its written procedures for feedback of peer review results into program management and development decisions more fully. In particular, the committee encourages OST to enforce its 30-day deadline for written responses to all peer review reports, and to require more detailed written responses from PIs and program managers that fully describe how the recommendations of the peer review panel were seriously considered. OST also should begin to develop and collect data for a set of activity and performance metrics that can measure the overall effectiveness of the peer review program.

To summarize, OST successfully has begun the process of implementing its new peer review program, but considerable effort is needed before peer review becomes an effective part of the standard operating procedures within the organization. OST increasingly has acknowledged that effective peer review can improve the credibility of its programs. The committee hopes that OST staff will recognize that such advice is valuable at many stages in the technology-development process. The committee does not suggest that the OST program staff is not qualified—and in fact believes quite the opposite. Technical reviews by the program staff are extremely valuable, but these programmatic reviews are not substitutes for peer review. Peer review of the program is a form of independent validation and a "reality check" on technical development. The peer review program will be meaningful only if OST acknowledges the benefits of effective peer review and makes peer review a vital part of the decision and management process throughout the organization. Although attaining these benefits will require a sustained effort from management, the entire organization will be rewarded through enhancement of the technology-development program.