1
Introduction
The U.S. Department of Energy's (DOE's) Environmental Management (EM) Program was established in 1989 to address the health, safety, environmental, and regulatory challenges associated with cleanup of the nation's nuclear weapons complex. Within EM, the Office of Science and Technology (OST) was created to promote the development of improved technologies to lower cleanup costs and risks (to workers, the public, and the environment) and to improve cleanup capabilities. OST supports the entire range of technology development activities—beginning with basic research through the EM Science Program (NRC, 1997a) and extending through development, demonstration, and (with the assistance of industrial partners) deployment into the cleanup program.
In fiscal year 1998 the annual budget for EM is about $5.6 billion, of which about $220 million is devoted to technology development activities within OST. The importance of technology development to EM's mission has been recognized in Accelerating Cleanup: Paths to Closure (DOE, 1998a), which describes EM's plan for cleanup of the weapons complex. This plan discusses the importance of technology development to reduce the "mortgage" at the complex—the long-term costs of maintaining contaminated buildings, equipment, and sites until they can be remediated.
OST sponsors 226 active research and development projects at universities, national laboratories, and private-sector companies on topics ranging from the remote detection of contaminants in the subsurface using geophysical techniques to the development of melters for waste vitrification. The major research and development program units within OST are listed in Table 1.1. OST uses various types of reviews (e.g., programmatic reviews, technical assessment reviews, peer reviews1) in its technology selection process
(see Appendix B for a description of the different types of reviews). These reviews are used to assess the merit of individual projects as well as the merit of all technology development thrusts within the office.
TABLE 1.1: DOE-OST R&D Program Units
Office of Technology Systemsa |
Office of Science and Risk Policy |
Decontamination and decommissioning focus area |
Environmental management science program (EMSP) |
Mixed waste characterization focus area |
Risk policy program |
Radioactive tank waste focus area |
|
Subsurface contamination focus area |
|
Characterization, monitoring, and sensors cross cut |
|
Efficient separations program cross cut |
|
Robotics cross cut |
|
Industry and university programs |
|
Technology integration |
|
a OST's focus areas, cross cuts, and supporting technology areas are administrative units used by OST to manage and coordinate its technology development activities, and are based on DOE-EM's major problems. |
OST's Peer Review Program
Several recent National Research Council (NRC) reports evaluated OST's technology selection process and recommended that OST develop an independent, external review process and apply it to all technology development programs. The report Improving the Environment recommended that "technology selection should incorporate a Knowledgeable independent review group that has no vested interests in the outcome and that includes people from outside the Department who work in the commercial use of technologies" (NRC, 1995c, p. 104). The NRC's Committee on Environmental Management Technologies (CEMT) also evaluated this technology selection process in its 1994 and 1995 annual reports (NRC, 1995b, 1996). In particular, these reports
recommended that OST develop a standardized, rigorous, and independent external peer review process and apply it to all technology development programs. These findings were echoed in a subsequent General Accounting Office (GAO) report, which concluded that "although the lead sites used significantly different systems to select projects, none of them used disinterested reviewers to determine the technical merit of the proposed work" (GAO, 1996, p. 7).
In response to these NRC and GAO reports, OST recently instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development. According to the OST, the objective of this new program is to serve as a "management tool for assuring that the technology is of high quality and effective, that critical needs have not been overlooked, and that the technology has the best chance possible for implementation."2
Charge to the Committee
OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. In particular, the committee was asked to review the following:
- internal procedures used by OST to identify the need for timely peer review of projects and programs;
- structures, protocols, and procedures for obtaining peer reviews of OST projects and programs, including who decides what will be peer reviewed, what criteria for peer review are used, and when in the R&D process peer review is requested; and
- feedback of peer review results into program management and development decisions.
In performing this assessment the committee was asked to compare OST's practices to generally accepted norms for scientific and technical peer review, including practices for selection of peer reviewers and screening for bias and conflict of interest. The assessment has been made more challenging by the
fact that OST has continued to improve its peer review process throughout this study, based on input from the committee's interim report advice from the ASME Peer Review Committee, and OST's peer review program staff.
OST has chosen to establish a relationship with two outside organizations, ASME and RSI, to carry out parts of its peer review process. In performing this assessment, the committee has focused on the structures, protocols, and procedures of OST's peer review program, including those parts carried out by ASME and RSI, but has avoided an explicit evaluation of the performance of these organizations because they are acting simply as agents of OST. The responsibility for establishing and executing an effective peer review program lies entirely with OST. The committee also has not evaluated, or endorsed, the organizational arrangements among ASME, RSI, and OST. The committee's findings and recommendations are directed at OST, which as noted above, has ultimate responsibility for all aspects of the peer review program.
Parallel Efforts
Another NRC Committee, the Committee on Prioritization and Decision Making in the DOE-OST, is currently conducting a parallel evaluation of the decision-making processes throughout OST's technology development program. One aspect of its work will be to examine the role and importance of peer reviews (and other types of reviews) in OST's decision-making processes. Our committee therefore has focused its work on OST's peer review program itself, including an evaluation of how peer reviews, if conducted, could be made more useful as an input to OST's decision-making processes, but has not evaluated OST's decision-making processes explicitly.
Study Process
The committee was briefed on the newly instituted peer review program by DOE staff at six committee meetings and has reviewed the recently developed Implementation Guidance for the Office of Science and Technology Technical Peer Review Process (DOE, 1998b), which documents OST's revised peer review procedures. Members and staff of the committee also observed peer reviews conducted on the In Situ Redox Manipulation project in Richland, Washington; the Decontamination and Decommissioning (D&D) Large Scale Demonstration projects and three D&D Technology Development projects in
Morgantown, West Virginia; the MAG*SEP technique3 in Atlanta, Georgia; the Small In-Tank Processing Modules and Small Modular In-Can Vitrification projects in Columbia, Maryland; several High Temperature Melter and Characterization projects in Idaho Falls, Idaho; and the Vortec Combustion Melter, Catalytic Chemical Oxidation, and Steam Reforming of Low-Level Mixed Waste projects in Columbia, Maryland. Committee members and staff also attended the Annual Meeting of the ASME Peer Review Committee in November 1997 and the ASME Peer Review Committee meeting in January 1998. In addition, the committee reviewed all peer review reports produced under the program from its initiation in October 1996 through April 1998.
In its work, the committee reviewed the literature on peer review and placed an emphasis on comparison with peer review in other organizations—an informal type of benchmarking. To aid in the comparison, during its meetings the committee was briefed on current peer review practices at the Department of Defense (DOD) Strategic Environmental Research and Development Program (SERDP), the National Institutes of Health (NIH), the National Science Foundation's (NSF's) Small Business Innovation Research (SBIR) program, the National Institute of Standards and Technology (NIST), the Environmental Protection Agency (EPA), EPA's Science Advisory Board, the U.S. Nuclear Regulatory Commission (USNRC), DOE's Office of Basic Energy Sciences (BES), Information Dynamics, Inc. (which conducts reviews for the National Aeronautics and Space Administration's [NASA's] Office of Life and Microgravity Sciences and Applications [OLMSA]), and the American Institute of Biological Sciences (AIBS). The committee also drew on its members' personal knowledge of these and other peer review programs.
Reports
This is the second of two reports prepared by the committee. In October 1997, the committee completed its interim report, which provided OST with a preliminary assessment of its new peer review program (NRC, 1997b). The interim report described the essential components of a credible peer review process and provided a preliminary assessment of OST's peer review program and the status of its implementation. In the interim report, the committee examined broad issues and tried to offer constructive recommendations to assist OST in successfully implementing this program, focusing on how OST could most effectively establish the basic elements of a sound peer review process.
OST has moved expeditiously to implement many of the recommendations from the interim report.
In this final report, the committee first reviews the current status of OST's peer review program (as of April 1998) by reevaluating the specific policy changes that have been implemented by OST in response to issues raised in the committee's interim report. The committee also addresses three specific issues that OST has identified as particularly important at this time:
- How should OST deal with the large number of technology projects at late stages of development that have never been peer reviewed? (See Chapter 6.)
- Can the fundamental principles of peer review outlined in the committee's interim report be applied to other types of OST reviews? (See Chapters 3 and 7.)
- How can OST measure the effectiveness of its peer review program? (See Chapter 7.)
In addition, in Chapter 7 the committee discusses a number of approaches that OST could use to continue to improve the peer review program in the future.