National Academies Press: OpenBook

Peer Review in the Department of Energy-Office of Science and Technology: Interim Report (1997)

Chapter: Appendix A: Description of OST's Peer Review Program

« Previous: References
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

Appendixes

Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
This page in the original is blank.
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

Appendix A
Description of OST's Peer Review Program1

The Office of Science and Technology's (OST's) peer review program was established in 1996 to address the concerns of external review groups such as the General Accounting Office and the National Research Council's Committee on Environmental Management Technologies (CEMT), who have criticized OST for its lack of a standardized, independent, external peer review system. This new, office-wide program was designed to provide credible peer reviews of OST's technologies at the appropriate stages in the technology development process.

Peer reviews are used by OST to provide an independent evaluation of the technical merits of a technology. These evaluations are intended to assist decision makers in making "go/no-go" decisions regarding further support for technology projects as they proceed through OST's Technology Decision Process (see Appendix C). In part, because the Technology Decision Process evaluates projects against both technical and nontechnical criteria, OST's peer review program does not directly make decisions regarding the funding of OST technologies. Rather, peer review results are used as one input into the decisions made at critical decision points (or "gates;" Figure C.1) between each stage of the research and development process.

Roles and Responsibilities

The main decision makers regarding the funding of technology projects within OST are the individual Focus Area/Crosscutting (FA/CC) program managers. As such, FA/CC program managers are the main "customer" of the OST peer review program. The peer review program is managed within OST by the Peer Review Coordinator, who represents the Deputy Assistant Secretary for Science and Technology. The peer reviews are conducted by the American Society of Mechanical Engineers (ASME) with administrative and technical support provided by the Institute for Regulatory Science (RSI). Funding for the review program is provided directly to RSI through a $1.2 million annual grant from the OST. This section describes the primary roles and responsibilities of the various entities involved in the OST peer review program.

FA/CC Program Managers

The FA/CC program managers initiate the peer review process by making written requests for peer reviews to the OST Peer Review Coordinator. The FA/CC program managers are responsible for developing a prioritized list of technologies to be reviewed, providing documentation to reviewers, preparing responses to review panels' Consensus Reports, and for covering the cost of FA/CC program personnel and material needed for the peer reviews.2 The

1  

The material in this appendix is based on OST's description of its program, not on the committee's evaluation.

2  

At present, the direct cost of the peer review itself is covered by the grant to RSI rather than by the budget of the FA/CC program manager requesting the review; however the cost for preparing and presenting to the review panel is covered by the FA/CC program.

Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

FA/CC program managers are responsible for incorporating the results of technical peer reviews into the decision-making process.

Principal Investigators

Each project's principal investigators (PIs) are responsible for providing relevant project information for the peer reviewers. Depending on the specific stage of development of project, the OST Revised Guidance (DOE, 1997b) describes the documentation generally required for a peer review. Such information includes written background documentation relevant to the nine general criteria for peer review (see below) and oral presentations to the review panels, if needed. At Gate 4 of the Technology Development Process Procedure, for example, PIs are required to provide the following:

  • literature references,
  • progress report (topical),
  • needs document,
  • test plan,
  • quality assurance plan,
  • proof of design,
  • life cycle cost analysis,
  • risk analysis,
  • regulatory issues and review, and
  • public acceptance issues and plans.
Peer Review Coordinator

The Peer Review Coordinator is the principal federal official responsible for managing the day-to-day activities of the program. Specific responsibilities include receiving, processing, and scheduling peer review requests from FA/CC program managers; coordinating peer review activities among FA/CC program managers, the ASME Peer Review Committee (see below), and the review panels; ensuring reviews are executed in a timely manner; ensuring FA/CC program responses to review recommendations are included in Final Reports; and managing the budget and records for OST.

ASME Peer Review Committee and Its Executive Panel

The Peer Review Committee (PRC) is a standing body of the ASME whose sole purpose is to oversee the OST's peer review program and enforce relevant ASME policies, including compliance with professional and ethical requirements. The PRC consists of 13 to 15 members chosen for their competencies and diversity of views. Membership in ASME is not required for appointment to the PRC. The PRC is responsible for appointing members nominated by the Executive Panel (see below) to individual peer review panels and assessing Interim Reports for conformity to ASME standards before being issued as ASME-sanctioned public documents.3

3  

As part of its assessment, the PRC may add comments to the Interim Reports before issuing them in final form.

Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

The PRC also reviews and approves the Annual Report and presents it, along with a comprehensive evaluation of the technical quality of the OST program, to the Deputy Assistant Secretary for OST at an annual meeting.

The Executive Panel of the Peer Review Committee (EP) consists of three to five ASME members who have served in leadership positions within ASME (e.g., president, vice-president, or division chair). The EP is responsible for overseeing the day-to-day operation of the peer review program and acts on behalf of the PRC between its semi-annual meetings. The EP meets approximately four times per year.

Peer Review Panels

Peer review panels consist of three or more technical experts chosen for their knowledge of the specific technology to be reviewed. Although ASME requires the peer review panel chair to be a member of ASME, other panel members need not be ASME members. The complexity of the technology(ies) being reviewed and the type of review dictate the number of experts on a panel. The peer review panel conducts the review, prepares a Consensus Report detailing its recommendations and observations, and transmits the written report directly to the sponsoring FA/CC program manager. Peer review panels are terminated upon completion of their specific review task.

American Society of Mechanical Engineers

The ASME participates in the OST peer review program by ensuring peer reviews follow ASME procedures. The ASME establishes and sanctions the Peer Review Committee and its Executive Panel through formal approval by the Center for Research and Technology Development of ASME's Council of Engineering. The ASME also serves as a resource for identifying potential peer review panel members.

Institute for Regulatory Science

RSI provides administrative and technical support for peer review activities. RSI's responsibilities include meeting planning, compiling and distributing background materials for members of review panels, facilitating peer reviews, and providing an executive secretary to assist in the preparation of Consensus Reports of peer review panels.

Selection of Technologies to Be Reviewed

OST policy requires that a peer review be conducted on each technology/system before it passes through Gate 4 of OST's Technology Investment Decision Model (TIDM) process (see Appendix B for a description of the TIDM). Specifically, OST policy states that

It is OST policy that a technology peer review will be conducted on each technology/system. However, FA/CC Program Managers will give priority attention to those technologies that are preparing to pass through Gate 4 of the OST Technology Decision Process Procedure (Version 7.0) [DOE, 1997c]. Specifically, this procedure states that technical peer review reports are one of the

Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

documents required to pass through Gate 4. In addition, peer reviews may be appropriate at other phases of the technology maturation stages. (DOE, 1997b, p. 3)

The actual decisions to conduct peer reviews of specific technologies are made by FA/CC program managers in response to five main ''drivers:'' (1) funding (i.e., to justify proposed increases in spending or to justify previous spending on a given technology), (2) changes in status (i.e., when a project approaches a gate in the TIDM process), (3) regulatory issues (i.e., in order to ensure that a technology meets regulatory requirements, (4) stakeholder concerns (i.e., in order to reassure the public or other stakeholders of the technical credibility of a technology, and (5) any technical or other issues that arise concerning a given technology.

Types of Reviews

The size and scope of each review panel depends on the specific technology(ies) being reviewed and the areas of expertise required to address the review criteria. In general, there are three levels of technical reviews:

  1. multi-technology review of a complex nature involving site visits (five or more reviewers);
  2. in-depth review of a technology involving site visits (three or more reviewers); and
  3. review of a document requiting no site visit (i.e., a mail review).

Selection of Reviewers

Reviewers are selected and approved by the Peer Review Committee based on three generally recognized criteria: (1) education and relevant experience, (2) peer recognition, and (3) contributions to the profession. A minimum of a B.S. in an engineering or scientific field and sufficient experience in the technical area being reviewed are required for participation in an OST-related review. Peer recognition is assessed by measures such as activity in a professional society. Contributions to the profession include publications in peer reviewed journals, patents, and meeting presentations.

Conflict of Interest

Individuals with any real or perceived conflicts of interest with respect to the subject of the review may not serve on as a member of a review panel. According to ASME, "an individual who has a personal stake in the outcome of the review may not act as a reviewer" (ASME, 1997, p. 8). Potential conflicts of interest among members of the review panels, the ASME Peer Review Committee, and RSI staff are handled by requiring everyone who participates in the review program to sign a statement indicating a lack of personal or financial interest in the outcome of the review. The ASME conflict of interest policy further requests that reviewers and members of the ASME Peer Review Committee and its Executive Panel recuse themselves from deliberations on any matter in which there may be an appearance of a conflict of interest.

Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

Peer Review Criteria

The following general criteria for peer reviews have been established for OST's peer review program in the ASME Manual for Peer Review (ASME, 1997):

  • 1)  

    Relevancy: All projects supported by the OST must meet the needs of various parts of the Office of Environmental Management, including EM-30, EM 40, EM 60, and EM 70.4 At a minimum, a DOE-wide need must be clearly and unambiguously identifiable.

  • 2)  

    Scientific and technical validity: Every technology supported by the DOE must be consistent with established scientific and engineering principles and standards. Furthermore, the likelihood of its success must be reasonable.

  • 3)  

    State-of-the-art: A prerequisite for desirability of a technology is that its design and engineering tools are state-of-the-art.

  • 4)  

    Filling an existing void: A technology is considered desirable if it fills unmet needs.

  • 5)  

    Non-duplicative/superior to alternatives: A technology can also be desirable if it does not duplicate an existing technology; or if there is duplication of an existing technology, it is superior to it as demonstrated by one or more criteria described under 6-9 below.

  • 6)  

    Cost effectiveness: The life cycle cost of a technology is an important criterion for its desirability.

  • 7)  

    Reduced risk: Reduction of risk to workers or the general public is an important parameter in decisions to implement a technology.

  • 8)  

    Regulatory acceptability: It is unlikely that a technology will be implemented unless it meets applicable regulatory requirements. The regulatory acceptability must be evaluated carefully before a technology starts the permitting process.

  • 9)  

    Public acceptance: Public acceptance of a technology does not necessarily follow peer acceptance of other parameters identified in this guidance. Furthermore, it is not always predictable. However, there are cases in which acceptance or lack of acceptance are clearly demonstrable.

OST has provided no guidance on the relative weighting of these general criteria. Although these nine criteria form the core of the criteria to be used during the peer review, the actual criteria used during each review are determined by the ASME Peer Review Committee in consultation with the FA/CC program manager. For each review, the FA/CC program manager develops a preliminary list of specific review criteria, which are then used by the ASME Peer Review Committee to develop the formal review criteria for the review panel. These criteria are used by project staff and other presenters to organize written materials and oral presentations for the peer review. In addition, even though these review criteria form the basis of the review panel's evaluation, the review panel does have the authority to pursue other issues that arise during the review.

4  

EM-30 is the Office of Waste Management, EM-40 the Office of Environmental Restoration, EM-60 the Office of Facility Transition and Nuclear Materials Stabilization, and EM-70 the Office of Site Operations. OST is EM-50.

Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×

Review Reports

Each member of the review panel prepares a report containing the outcomes of the review. The review panel chair combines these individual reports into a Consensus Report, which typically contains four parts: (1) a brief description of the subject that was reviewed, (2) the consensus of the panel on shortcomings and meritorious aspects of the project, (3) recommendations of the panel, and (4) an appendix containing the comments by each reviewer for which no consensus could be reached, or which were considered by the panel to be beneficial to the investigators and managers but were not important enough to be included in the main body of the Consensus Report.

The Consensus Report is provided to DOE investigators and managers who are requested to respond to the conclusions and recommendations within 30 days. A summary of the responses containing the salient features of the response are combined with the substantive sections of the Consensus Reports as the Interim Report. After DOE's response, the ASME PRC reviews the Interim Report, during which it may add to it substantive, explanatory, clarifying, and supplementary comments and recommendations. The Interim Report as reviewed and approved by the PRC is issued as a Final Report.

Copies of the Consensus Report, the responses from the investigators and managers, and the Interim Reports are made available to members of the PRC for review and comments during the entire review process. At the end of the year (or after a specific period), an Annual Report is prepared by the PRC. The report contains Final Reports of all projects, recommendations of the PRC, and other information that, in the judgment of the PRC, would be beneficial to OST. This report will be printed and widely distributed.

Feedback of Peer Review Results into Program Management and Development Decisions

One to two days after the completion of the peer review, RSI submits the peer review panel's Consensus Report to the FA/CC program manager who requested the review. The program manager is asked to prepare, within 30 days, a written response that describes how the PI and/or the FA/CC manager intend to respond to the findings and recommendations of the Consensus Report. This formal response is then combined with the substantive sections of the Consensus Report as an ASME Interim Report, which is used by DOE headquarters' staff to evaluate the management of the technology program. The Interim Report is then reviewed and approved by the ASME Peer Review Committee and released as an ASME Final Report.

OST's Technology Decision Process requires that peer review reports be used in all funding decisions made at Gate 4 or any subsequent gate in the technology development process. In these gate reviews, these reports constitute one input to be considered in determining whether to continue funding for a project.

Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 33
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 34
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 35
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 36
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 37
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 38
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 39
Suggested Citation:"Appendix A: Description of OST's Peer Review Program." National Research Council. 1997. Peer Review in the Department of Energy-Office of Science and Technology: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/5939.
×
Page 40
Next: Appendix B: Description of OST's Technology Investment Decision Model »
Peer Review in the Department of Energy-Office of Science and Technology: Interim Report Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Office of Science and Technology (OST) of the U.S. Department of Energy's (DOE's) Office of Environmental Management (EM) recently has instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development.

OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. This is the first of two reports to be prepared by this committee on OST's new peer review program. OST requested this interim report to provide a preliminary assessment of OST's new peer review program. In the final report, the committee will provide a more detailed assessment of OST's peer review program after its first complete annual cycle.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!