National Academies Press: OpenBook

Peer Review in the Department of Energy-Office of Science and Technology: Interim Report (1997)

Chapter: Implementation of the Peer Review Program

« Previous: Peer Review Process
Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

Implementation of the Peer Review Program

The committee is encouraged that OST continues to actualize its new goal of implementing peer review of its activities. OST presentations to this NRC committee at its April 1997 meeting showed progress since the February 1997 meeting, and some aspects of the peer review program (e.g., quality of peer review reports) have improved since the program's inception. OST now has a documented peer review process that is being applied on a small scale to evaluate the technical merit of individual projects.

Despite this progress, however, OST must address a number of issues, discussed in the previous sections, that are hindering the effectiveness of its new peer review program (e.g., selection of projects, definition of peer review objectives and criteria, criteria for reviewer selection, and improvement of peer review results). The committee also has perceived a lack of acceptance of peer review as a standard operating procedure within parts of the OST organization. In the following section, the committee discusses this issue and the leadership necessary to implement this peer review program successfully.

OST's "Corporate" Culture

One feature characteristic of organizations that effectively use peer review as an input into management of their research and development portfolios is a peer review process that is ingrained in their organizational cultures; in other words, for these organizations, peer review is "standard operating procedure." Even after deciding on a peer review process that seems adequate on paper, OST still should change its organizational culture so that it embraces peer review as an essential part of its decision-making process. The peer review culture is not yet ingrained within the EM program (NRC, 1995b,c, 1996; GAO, 1996). This may derive from a time when DOE laboratories were working in fields such as weapons development, where the national expertise was predominantly within the DOE organization. Today, technology programs like those of OST are not peculiar to DOE but are common in other organizations, and a broad range of expertise is available outside of the DOE "family." OST is just beginning to turn to the outside world for technical advice, however. Indeed, during the committee's review, it became clear that some OST staff and contractors continue to view peer review as an externally imposed requirement to be complied with, rather than as an opportunity to benefit from independent expert advice. The committee is encouraged, however, that the EM Science Program (see Appendix C), which funds basic environmental research of relevance to EM, has embraced peer review for assessing the scientific merit of proposals. The committee hopes OST staff will recognize that such advice is valuable at many stages in the technology-development process.

An advantage of independent advice throughout the process, as mentioned in the section entitled Benefits of Peer Review, is that external experts do not have the same set of constraints on their thinking and are therefore more likely to recommend termination of projects that have little probability of future utility, because they have no "identity" with the project. Likewise,

Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

external reviewers are able to recommend adding additional resources to a promising project without being subject to the charge of programmatic favoritism.

The committee does not suggest that the OST program staff is not qualified—and in fact, believes quite the opposite. Technical review by the program staff is extremely valuable and essential, and program staff review is a major component of proper management of the public trust in these high-impact programs. These programmatic reviews are not substitutes for peer review, however. Peer review of the programs is a form of independent validation and a "reality check" on technical development.

Role of Leadership

Kostoff (1997a,b) recently argued that one of the most important factors for high-quality peer review programs is the commitment of the organization's senior management to high-quality reviews:

The primary requirements of excellent peer review are the dedication of an organization's senior management to the highest quality objective review and the motivation of the review manager to conduct a technically credible review. (Kostoff, 1997a, p. 652)

Implementing the necessary changes in OST to improve the technology-review process will be difficult. OST leaders will have to commit much attention and energy to integrating peer review into the organization's standard operating procedures. Such a change would lead to an improvement in the planning and execution of the peer review program and, more importantly, its effectiveness (i.e., the use of peer review results in better management decisions and in program improvement).

Organizational "culture" can be defined as the norm of what members value. Therefore, to affect a lasting change in an organization's culture, the normative values of its members must change. This change must start at the top, but to be successful it also must pervade all levels of the organization. Members of the organization will come to value peer review when they see benefits to their programs (e.g., through case histories), when management provides logical and consistent messages on the value of peer review, and/or when members are given incentives to use it (Kostoff, 1997b). Support for peer review must be consistent across programs for a sustained period of time to have an impact on the corporate culture.

Based on its observations to date, the committee has not seen this level of commitment to peer review in OST. Lack of consistent policy support for peer review is illustrated by two examples: (1) although OST policy requires a peer review before a technology passes through Gate 4 of the TIDM, decisions to conduct such reviews are reserved to the program managers, and (2) a new description of the OST Technical Peer Review Program (DOE, 1997b) describes the actual peer review process as it is currently being implemented, rather than presenting a clear description of how to implement an effective peer review program. As a result, this document implicitly endorses the status quo of an "immature" peer review process.

Only when obtaining outside advice through a fair and credible process and incorporating such advice into the routine decision-making process become standard procedure will the goal for peer review be achieved. As a means to that goal, OST needs to recognize the benefits of an

Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

effective peer review program and become champions of this paradigm. Peer review should be the expected way of doing business. When this is achieved, OST's work will be soundly credible not because it has been given a quick remedial polish but because it is fundamentally good technical work—because it meets high standards of quality.

Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
This page in the original is blank.
Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 27
Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 28
Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 29
Suggested Citation:"Implementation of the Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 30
Next: References »
Peer Review in the Department of Energy-Office of Science and Technology: Interim Report Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Office of Science and Technology (OST) of the U.S. Department of Energy's (DOE's) Office of Environmental Management (EM) recently has instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development.

OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. This is the first of two reports to be prepared by this committee on OST's new peer review program. OST requested this interim report to provide a preliminary assessment of OST's new peer review program. In the final report, the committee will provide a more detailed assessment of OST's peer review program after its first complete annual cycle.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!