National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

Introduction

The U.S. Department of Energy's (DOE's) Office of Environmental Management (EM) was established in 1989 to address the health, safety, and environmental challenges associated with cleanup of the nation's nuclear weapons complex. Within EM, the Office of Science and Technology (OST) was created to promote the development of new and improved technologies to lower cleanup costs and risks (to workers, the public, and the environment) and to improve cleanup capabilities. OST supports the entire range of technology development activities—beginning with basic research through the new EM Science Program (NRC, 1997) and extending through development, demonstration, and (with the assistance of industrial partners) deployment into the cleanup program.

In fiscal year 1997 the annual budget for EM is about $6 billion, of which about $300 million is devoted to technology-development activities within OST. The importance of technology development to EM's mission has been recognized in Accelerating Cleanup: Focus on 2006 (DOE, 1997a), a draft planning document for cleanup of the weapons complex. This document discusses the importance of technology development to reduce the "mortgage" at the complex—the long-term costs of maintaining contaminated buildings, equipment, and sites until they can be remediated.

OST sponsors close to 300 research and development (R&D) projects at universities, national laboratories, and private-sector companies on topics ranging from remote detection of contaminants in the subsurface using geophysical techniques to development of melters for waste vitrification. OST uses various types of reviews (e.g., programmatic reviews, technical assessment reviews, and peer reviews)1 in its technology-selection process (see Appendix C for a description of the different types of reviews used within OST). These reviews are used to assess the merit of individual projects as well as the merit of entire technology development thrusts within the office.

Several recent NRC reports evaluated DOE-OST's technology-selection process and recommended that OST should develop and apply an independent, external review process to all of its technology-development programs. The report Improving the Environment recommended that "technology selection should incorporate a knowledgeable independent review group that has no vested interests in the outcome and that includes people from outside the Department who work in the commercial use of technologies" (NRC, 1995c, p. 104). The NRC's Committee on Environmental Management Technologies (CEMT) also evaluated this technology-selection process in its 1994 and 1995 annual reports (NRC, 1995b, 1996). In particular, these reports recommended that OST should develop and apply a standardized, rigorous, and independent external peer review process to all of its technology-development programs. These findings were echoed in a subsequent General Accounting Office (GAO) report, which concluded that "although the lead sites used significantly different systems to select projects, none of them used disinterested reviewers to determine the technical merit of the proposed work" (GAO, 1996, p. 7).

1  

These peer reviews are termed "technical peer reviews" by OST.

Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

In response to these NRC and GAO reports, the OST recently has instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development. According to the OST, the objective of this new program is to serve as a "management tool for assuring that the technology is of high quality and effective, that critical needs have not been overlooked, and that the technology has the best chance possible for implementation."2

OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. In particular, the committee was asked to review the following:

  • internal procedures used by OST to identify the need for timely peer review of projects and programs;
  • structures, protocols, and procedures for obtaining peer reviews of OST projects and programs, including who decides what will be peer reviewed, what criteria for peer review are used, and when in the R&D process peer review is requested; and
  • feedback of peer review results into program management and development decisions.

In performing this assessment, the committee was asked to compare OST's practices to generally accepted norms for scientific and technical peer review, including practices for selection of peer reviewers and screening for bias and conflict of interest.

The responsibility for establishing and executing an effective peer review program lies entirely with EM and specifically with OST. OST has chosen to establish a relationship with two independent organizations, the ASME and RSI, to carry out some aspects of its peer review program. When the committee first began this review and in subsequent meetings, officials from ASME and RSI expressed their concerns about the scope of the study and, in particular, whether the committee would be evaluating the performance of either of these organizations. In undertaking this review, the committee has focused its attention exclusively on the design and effectiveness of the peer review program; the committee has not addressed the performance of ASME or RSI, nor has it evaluated or endorsed the organizational arrangements among ASME, RSI, and OST. The committee's findings and recommendations are directed at OST which, as noted above, has ultimate responsibility for the peer review program.

Another NRC Committee, the Committee on Prioritization and Decision Making in the Department of Energy-Office of Science and Technology, is currently conducting a parallel review of the decision-making process in OST's technology-development program. One aspect of its work will be to examine the role and importance of peer reviews in OST's decision-making process. Our committee therefore has focused its work on OST's peer review program itself (including the peer review results used as an input to decision making), but has not addressed OST's decision-making process explicitly.

This study is being carried out over a 15-month period (from January 1997 through March 1998), during which time the committee will produce both an interim (this report) and a final report. The committee has been briefed on the newly instituted peer review program by DOE staff at three committee meetings, and members and staff of the committee observed peer

2  

Presentation to committee by Anibal Taboas, DOE, February 24, 1997, Washington, D.C.

Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

reviews conducted on the In Sire Redox Manipulation project in Richland, Washington, the Decontamination and Decommissioning (D&D) Large Scale Demonstration projects and three D&D Technology Development projects in Morgantown, West Virginia, the MAG*SEP technique3 in Atlanta, Georgia, the Small In-Tank Processing Modules and Small Modular In-Can Vitrification projects in Columbia Maryland, and several High Temperature Melter and Characterization projects in Idaho Falls, Idaho. The committee also has reviewed all peer review reports that have been produced under the new program from its initiation in October 1996 through June 1997. In addition, during its meetings, the committee was briefed on current peer review practices at the Environmental Protection Agency (EPA), EPA's Science Advisory Board, the U.S. Nuclear Regulatory Commission (USNRC), the DOE-Office of Basic Energy Sciences (BES), Information Dynamics, Inc. (which conducts reviews for NASA's Office of Life and Microgravity Sciences and Applications), and the American Institute of Biological Sciences (AIBS).

This is the first of two reports to be prepared by this committee on OST's new peer review program. OST requested this interim report to provide a preliminary assessment of its new peer review program. In particular, OST asked the committee to consider whether it is moving in the fight direction toward the implementation of a credible, effective, and defensible peer review program. In this report, the committee describes the essential components of a credible peer review program and provides a preliminary assessment of OST's new peer review program and the status of its implementation. Recognizing that this is a new program in its early stages of implementation, the committee has focused on broad issues and has tried to offer constructive recommendations to assist OST in successfully implementing this program. In the final report, the committee will provide a more detailed assessment of OST's peer review program after its first complete annual cycle. This final assessment will develop a general framework for evaluating the level of development (or "maturity level") of a peer review program, focusing on specific components of the peer review process, and will include more detailed recommendations on how to improve OST's peer review program (including issues raised in this report), if appropriate.

3  

The MAG*SEP technique is a means of recovering selected radionuclides and heavy metals from water and other liquids through sorption onto specially coated particles.

Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
This page in the original is blank.
Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 5
Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 6
Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 7
Suggested Citation:"Introduction." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 8
Next: Definition of Peer Review »
Peer Review in the Department of Energy-Office of Science and Technology: Interim Report Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Office of Science and Technology (OST) of the U.S. Department of Energy's (DOE's) Office of Environmental Management (EM) recently has instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development.

OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. This is the first of two reports to be prepared by this committee on OST's new peer review program. OST requested this interim report to provide a preliminary assessment of OST's new peer review program. In the final report, the committee will provide a more detailed assessment of OST's peer review program after its first complete annual cycle.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!