Implementation of the Peer Review Program

The committee is encouraged that OST continues to actualize its new goal of implementing peer review of its activities. OST presentations to this NRC committee at its April 1997 meeting showed progress since the February 1997 meeting, and some aspects of the peer review program (e.g., quality of peer review reports) have improved since the program's inception. OST now has a documented peer review process that is being applied on a small scale to evaluate the technical merit of individual projects.

Despite this progress, however, OST must address a number of issues, discussed in the previous sections, that are hindering the effectiveness of its new peer review program (e.g., selection of projects, definition of peer review objectives and criteria, criteria for reviewer selection, and improvement of peer review results). The committee also has perceived a lack of acceptance of peer review as a standard operating procedure within parts of the OST organization. In the following section, the committee discusses this issue and the leadership necessary to implement this peer review program successfully.

OST's "Corporate" Culture

One feature characteristic of organizations that effectively use peer review as an input into management of their research and development portfolios is a peer review process that is ingrained in their organizational cultures; in other words, for these organizations, peer review is "standard operating procedure." Even after deciding on a peer review process that seems adequate on paper, OST still should change its organizational culture so that it embraces peer review as an essential part of its decision-making process. The peer review culture is not yet ingrained within the EM program (NRC, 1995b,c, 1996; GAO, 1996). This may derive from a time when DOE laboratories were working in fields such as weapons development, where the national expertise was predominantly within the DOE organization. Today, technology programs like those of OST are not peculiar to DOE but are common in other organizations, and a broad range of expertise is available outside of the DOE "family." OST is just beginning to turn to the outside world for technical advice, however. Indeed, during the committee's review, it became clear that some OST staff and contractors continue to view peer review as an externally imposed requirement to be complied with, rather than as an opportunity to benefit from independent expert advice. The committee is encouraged, however, that the EM Science Program (see Appendix C), which funds basic environmental research of relevance to EM, has embraced peer review for assessing the scientific merit of proposals. The committee hopes OST staff will recognize that such advice is valuable at many stages in the technology-development process.

An advantage of independent advice throughout the process, as mentioned in the section entitled Benefits of Peer Review, is that external experts do not have the same set of constraints on their thinking and are therefore more likely to recommend termination of projects that have little probability of future utility, because they have no "identity" with the project. Likewise,



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 27
--> Implementation of the Peer Review Program The committee is encouraged that OST continues to actualize its new goal of implementing peer review of its activities. OST presentations to this NRC committee at its April 1997 meeting showed progress since the February 1997 meeting, and some aspects of the peer review program (e.g., quality of peer review reports) have improved since the program's inception. OST now has a documented peer review process that is being applied on a small scale to evaluate the technical merit of individual projects. Despite this progress, however, OST must address a number of issues, discussed in the previous sections, that are hindering the effectiveness of its new peer review program (e.g., selection of projects, definition of peer review objectives and criteria, criteria for reviewer selection, and improvement of peer review results). The committee also has perceived a lack of acceptance of peer review as a standard operating procedure within parts of the OST organization. In the following section, the committee discusses this issue and the leadership necessary to implement this peer review program successfully. OST's "Corporate" Culture One feature characteristic of organizations that effectively use peer review as an input into management of their research and development portfolios is a peer review process that is ingrained in their organizational cultures; in other words, for these organizations, peer review is "standard operating procedure." Even after deciding on a peer review process that seems adequate on paper, OST still should change its organizational culture so that it embraces peer review as an essential part of its decision-making process. The peer review culture is not yet ingrained within the EM program (NRC, 1995b,c, 1996; GAO, 1996). This may derive from a time when DOE laboratories were working in fields such as weapons development, where the national expertise was predominantly within the DOE organization. Today, technology programs like those of OST are not peculiar to DOE but are common in other organizations, and a broad range of expertise is available outside of the DOE "family." OST is just beginning to turn to the outside world for technical advice, however. Indeed, during the committee's review, it became clear that some OST staff and contractors continue to view peer review as an externally imposed requirement to be complied with, rather than as an opportunity to benefit from independent expert advice. The committee is encouraged, however, that the EM Science Program (see Appendix C), which funds basic environmental research of relevance to EM, has embraced peer review for assessing the scientific merit of proposals. The committee hopes OST staff will recognize that such advice is valuable at many stages in the technology-development process. An advantage of independent advice throughout the process, as mentioned in the section entitled Benefits of Peer Review, is that external experts do not have the same set of constraints on their thinking and are therefore more likely to recommend termination of projects that have little probability of future utility, because they have no "identity" with the project. Likewise,

OCR for page 27
--> external reviewers are able to recommend adding additional resources to a promising project without being subject to the charge of programmatic favoritism. The committee does not suggest that the OST program staff is not qualified—and in fact, believes quite the opposite. Technical review by the program staff is extremely valuable and essential, and program staff review is a major component of proper management of the public trust in these high-impact programs. These programmatic reviews are not substitutes for peer review, however. Peer review of the programs is a form of independent validation and a "reality check" on technical development. Role of Leadership Kostoff (1997a,b) recently argued that one of the most important factors for high-quality peer review programs is the commitment of the organization's senior management to high-quality reviews: The primary requirements of excellent peer review are the dedication of an organization's senior management to the highest quality objective review and the motivation of the review manager to conduct a technically credible review. (Kostoff, 1997a, p. 652) Implementing the necessary changes in OST to improve the technology-review process will be difficult. OST leaders will have to commit much attention and energy to integrating peer review into the organization's standard operating procedures. Such a change would lead to an improvement in the planning and execution of the peer review program and, more importantly, its effectiveness (i.e., the use of peer review results in better management decisions and in program improvement). Organizational "culture" can be defined as the norm of what members value. Therefore, to affect a lasting change in an organization's culture, the normative values of its members must change. This change must start at the top, but to be successful it also must pervade all levels of the organization. Members of the organization will come to value peer review when they see benefits to their programs (e.g., through case histories), when management provides logical and consistent messages on the value of peer review, and/or when members are given incentives to use it (Kostoff, 1997b). Support for peer review must be consistent across programs for a sustained period of time to have an impact on the corporate culture. Based on its observations to date, the committee has not seen this level of commitment to peer review in OST. Lack of consistent policy support for peer review is illustrated by two examples: (1) although OST policy requires a peer review before a technology passes through Gate 4 of the TIDM, decisions to conduct such reviews are reserved to the program managers, and (2) a new description of the OST Technical Peer Review Program (DOE, 1997b) describes the actual peer review process as it is currently being implemented, rather than presenting a clear description of how to implement an effective peer review program. As a result, this document implicitly endorses the status quo of an "immature" peer review process. Only when obtaining outside advice through a fair and credible process and incorporating such advice into the routine decision-making process become standard procedure will the goal for peer review be achieved. As a means to that goal, OST needs to recognize the benefits of an

OCR for page 27
--> effective peer review program and become champions of this paradigm. Peer review should be the expected way of doing business. When this is achieved, OST's work will be soundly credible not because it has been given a quick remedial polish but because it is fundamentally good technical work—because it meets high standards of quality.

OCR for page 27
This page in the original is blank.