4
OST's Peer Review Program

In this chapter, the committee presents a description of OST's peer review program as of April 1998. Much of the information was based on Implementation Guidance for the OST Technical Peer Review (DOE, 1998b), ASME's Manual for Peer Review (ASME, 1998), RSI's Handbook of Peer Review (RSI, 1998), as well as presentations by OST, ASME, and RSI staff. The committee's analysis of the current state of the peer review program is presented in Chapter 5.

OST's peer review program was designed to provide program managers with credible, independent evaluations of the scientific and engineering merits of its technology projects. The peer review program evaluates technology projects at various stages of development (from basic research to late-stage demonstrations and implementation of the technologies)—not solely at the proposal stage, as practiced by many other government agencies. OST established the peer review program to provide one important input to its focus area/crosscutting area (FA/CC) program managers as they make "go or no-go" decisions on technology projects supported by OST.

Technology Investment Decision Model

OST's Technology Investment Decision Model is a procedure OST developed to provide a common basis on which to assess and manage the performance, expectations, and transition of technologies through the development process (Paladino and Longsworth, 1995). It is a user-oriented decision-making process for managing technology development and for linking technology development activities with cleanup operations (see Appendix A for a more complete description of the TIDM). It should be noted that the TIDM



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 53
--> 4 OST's Peer Review Program In this chapter, the committee presents a description of OST's peer review program as of April 1998. Much of the information was based on Implementation Guidance for the OST Technical Peer Review (DOE, 1998b), ASME's Manual for Peer Review (ASME, 1998), RSI's Handbook of Peer Review (RSI, 1998), as well as presentations by OST, ASME, and RSI staff. The committee's analysis of the current state of the peer review program is presented in Chapter 5. OST's peer review program was designed to provide program managers with credible, independent evaluations of the scientific and engineering merits of its technology projects. The peer review program evaluates technology projects at various stages of development (from basic research to late-stage demonstrations and implementation of the technologies)—not solely at the proposal stage, as practiced by many other government agencies. OST established the peer review program to provide one important input to its focus area/crosscutting area (FA/CC) program managers as they make "go or no-go" decisions on technology projects supported by OST. Technology Investment Decision Model OST's Technology Investment Decision Model is a procedure OST developed to provide a common basis on which to assess and manage the performance, expectations, and transition of technologies through the development process (Paladino and Longsworth, 1995). It is a user-oriented decision-making process for managing technology development and for linking technology development activities with cleanup operations (see Appendix A for a more complete description of the TIDM). It should be noted that the TIDM

OCR for page 53
--> procedure has been documented by OST, and OST does use the overall TIDM framework to track its projects, but OST does not yet use the TIDM approach in its decision making consistently across the organization. The TIDM identifies seven R&D stages from basic research through implementation of a technology (see Figure 4.1). At each stage, specific criteria, requirements, and deliverables form a common basis for technology assessment. In the model, stages are separated by "gates"—decision points at which projects are evaluated for funding of the next stage. The stage-gate process is meant to provide for evaluation of projects at all stages of development against technical and nontechnical criteria selected to ensure that the technologies developed will provide superior performance, will meet the acceptance requirements of the intended customers, and can be moved into the marketplace. At each gate, OST's FA/CC program managers are responsible for evaluating a technology's documentation in accordance with the appropriate criteria. If the FA/CC program manager determines that the technology warrants passing through a gate, the technology maturation process continues. If the program manager determines that the technology does not warrant further consideration, then funding is discontinued. See Appendix A for a description of individual stages and gates of the TIDM. Peer reviews are intended to be used by OST in the TIDM to provide an independent, external evaluation of the technical merits of a technology. These evaluations are intended to assist decision makers in making decisions regarding further support for technology projects as they proceed through OST's TIDM. In part, because the TIDM evaluates projects against both technical and nontechnical criteria, OST's peer review program does not directly make decisions regarding the funding of OST technologies. Rather, peer review results are intended to be one input into the decisions made at the critical decision points (gates) of the research and development process. Roles and Responsibilities As noted above, the main decision makers in terms of funding technology projects within OST are the individual FA/CC program managers. As such, FA/CC program managers are the main "customers" of the OST peer review program. The peer review program is managed within OST by the Peer Review Coordinator, who represents the Deputy Assistant Secretary for Science and Technology. The peer review sessions themselves are conducted by the American Society of Mechanical Engineers, with administrative and technical

OCR for page 53
--> Figure 4.1 Diagram of OST's Technology Investment Decision Model (DOE, 1998b).

OCR for page 53
--> support provided by the Institute for Regulatory Science. Funding for the review program is provided directly to RSI through a $1.2 million annual grant from OST. According to OST, ASME was chosen to conduct parts of OST's peer review process because of its long history as a technical professional society, its work in developing and categorizing technical and engineering codes and standards, its experience conducting peer review for other organizations, and its independence from DOE. This section describes the primary roles and responsibilities of the various entities involved in the OST peer review program. Peer Review Coordinator The Peer Review Coordinator (under DOE's Chicago Operations Office) is the principal federal official responsible for managing the peer review program's day-to-day activities. Specific responsibilities include developing a targeted plan for the program including a prioritized list of projects to be reviewed; receiving, processing, and scheduling peer review requests from FA/CC program managers; coordinating peer review activities among FA/CC program managers, the ASME Peer Review Committee (see below), and the review panels; ensuring that reviews are executed in a timely manner; ensuring that FA/CC program responses to review recommendations are included in Final Reports; managing the budget and records for OST; and tracking activity metrics for the program (DOE, 1998b). Headquarters Peer Review Program Manager The headquarters Peer Review Program Manager is responsible for monitoring all peer review activities within OST. The program manager also works with the Peer Review Coordinator to provide direction on policy, program planning, and budgetary issues. FA/CC Program Managers The FA/CC program managers work with the Peer Review Coordinator to develop a prioritized list of technologies to be reviewed and initiate the peer review process by making written requests for peer reviews to the OST Peer Review Coordinator. FA/CC program managers also provide documentation to reviewers, prepare responses to the Report of the Review Panel, coordinate the responses with the director of the Office of Technology Systems, and cover the

OCR for page 53
--> cost of FA/CC program personnel and material needed for the peer reviews.1 The FA/CC program managers are responsible for incorporating the results of peer reviews into the decision-making process. Principal Investigators The principal investigators of a project are the scientists and engineers who perform the technology development work. Each project's PIs are responsible for providing relevant technical background information for the peer reviewers 30 days prior to the review (See Table 4.1). For reviews of Types I and II (see below), the PIs are also responsible for presenting and responding to the review panel. American Society of Mechanical Engineers The ASME establishes and sanctions the Peer Review Committee and its Executive Panel through formal approval by the Center for Research and Technology Development of ASME's Council of Engineering. The ASME ensures that peer reviews follow ASME procedures and serves as a resource for identifying potential peer review panel members. As well as being able to provide names from its own membership, the ASME will attempt to reach collegiality agreements with other relevant professional societies, as necessary, to identify additional potential nominees. ASME Peer Review Committee and its Executive Panel The Peer Review Committee (PRC) is a standing body of the ASME whose sole purpose is to oversee OST's peer review program and enforce relevant ASME policies, including compliance with professional and ethical requirements. The PRC is a consensus body that meets several times per year and consists of members chosen for their competences and diversity of views. Thirteen members currently serve on the PRC (see Box 4.1). Membership in ASME is not required for appointment to the PRC (except for members of its Executive Panel—see below). The PRC is responsible for appointing persons 1   At present, the direct cost of the peer review itself is covered by the grant to RSI, rather than by the budget of the FA/CC program manager requesting the review; however, the costs of preparing for the reviews and presenting to the review panel are covered by the FA/CC program.

OCR for page 53
--> nominated by the Executive Panel to individual peer review panels and approving Interim Reports (see section ''Review Reports'' below) before they are issued as ASME-sanctioned public documents.2 The PRC also reviews and approves the Annual Report and presents it, along with a comprehensive evaluation of the technical quality of the OST program, to the Deputy Assistant Secretary for OST at an annual meeting. The Executive Panel (EP) of the PRC consists of three to five ASME members who typically have served in leadership positions within ASME (i.e., president, vice-president, division chair). The EP is responsible for overseeing the day-to-day operation of the peer review program and acts on behalf of the PRC between its meetings. The EP meets approximately four times per year but conducts most of its business by correspondence. The first annual meeting of the ASME Peer Review Committee was held in Washington, D.C., on November 24, 1997, at which time the first Annual Report (ASME, 1997) was released. The report includes a description of the program, recommendations addressed to OST regarding the conduct of reviews, and a compilation of all the peer review reports produced in Fiscal Year 1997. Peer Review Panels Peer review panels consist of three or more technical experts chosen for their knowledge of the specific technology to be reviewed. The complexity of the technology being reviewed and the type of review are used by the PRC to choose the number of experts needed on a particular panel. The peer review panel conducts the review, prepares a Report of the Review Panel (formerly termed Consensus Report) detailing its recommendations and observations, and transmits the written report to the Peer Review Coordinator, who forwards it to the FA/CC program managers for response. In most cases, peer review panels produce a single report that summarizes the consensus views of all members. In cases where a consensus among all members is not reached, however, the Report of the Review Panel can include a minority opinion as an appendix (see discussion of Review Reports below). Peer review panels are terminated upon completion of their specific review task. 2   As part of its assessment, the PRC may add comments to the Interim Reports before issuing them in final form.

OCR for page 53
--> BOX 4.1 ASME Peer Review Committee Charles O. Velzy (chair), consultant1, 2 Gary A Benda, NUKEM Nuclear Technologies, Corp.2 Erich W. Bretthauer, Bryce Meadows Development Corporation Ernest L. Daman, Foster Wheeler Development Corporation1, 2 Robert A. Fjeld, Clemson University John T. Greeves, U.S. Nuclear Regulatory Commission William T. Gregory III, Foster Wheeler Environmental Corporationfr2] Nathan H. Hurt, IDM Environmental Corp.1, 2 Peter B. Lederman, New Jersey Institute of Technology Jeffrey A. Marqusee, Office of the Deputy Under Secretary of Defense for Environmental Security A. Alan Moghissi, Institute for Regulatory Science1, 2 Goetz K. Oertel, Association of Universities for Research in Astronomy, Inc. Staff Howard E. Clark, American Society of Mechanical Engineers 1   Member of Executive Panel 2   Member of ASME Institute for Regulatory Science OST chose RSI to provide administrative and technical support for peer review activities. RSI's responsibilities include meeting planning, compiling and distributing background materials for members of review panels, facilitating peer reviews, assisting in the identification of potential review panel nominees, and providing a Technical Secretary who participates in all meetings of the review panels and assists in the preparation of their technical reports. The RSI President manages the review program, sits on the ASME EP, and supervises RSI's activities related to the review program.3 3   Because the committee was asked not to review the DOE-ASME-RSI relationship, it could not arrive at a complete understanding of this relationship and therefore makes no comment on it.

OCR for page 53
--> Selection of Technologies to be Reviewed OST policy requires that a technology peer review be conducted on each technology or system. In response to this committee's interim report (NRC, 1997b), OST recently adopted a new policy requiring that candidate technologies for peer review be identified as those that fulfill at least one of the following three project selection criteria (not in priority order): Gate 4: technologies or systems passing into the engineering development stage; or More than three years of funding: projects that have been funded for more than three years and have not been peer reviewed; or New start: projects that were initiated in FY97 or FY98 and proposed new starts. In planning for peer reviews to be conducted in FY98, OST used its Technology Management System Database (consisting of more than 800 technologies or systems that have been funded by OST from FY89 to FY98) to identify more than 200 technologies that meet at least one of these three criteria. This list was sorted by the amount of funding each project had received to date. The list was then provided to all FA/CC program managers so that they could further prioritize their list of projects by the dates at which peer review results would be most valuable for making programmatic and/or funding decisions (DOE, 1998b, p. 11). Ultimately, a plan to review 38 specific technologies during FY98 was prepared. Documentation Required for Review OST's Implementation Guidance (DOE, 1998b) lists the written documentation generally required for a peer review at each stage of development of a project (Table 4.1).

OCR for page 53
--> TABLE 4.1 Documentation Required by OST for Peer Reviews Conducted at Different Gates of its Technology Investment Decision Model.   Gates Documents 1 2 3 4 5 6 Proof of principle __ __ __ __ __ __ Literature references __ __ __ __ __ __ Progress report (topical) __ __ __ __ __ __ Needs document   __ __ __ __ __ Test plan at the appropriate scale   __ __ __ __ __ Data quality assurance plan     __ __ __ __ Proof of design       __ __ __ Construction plan         __ __ Implementation plan           __   Source: DOE, 1998b, p. 18. Types of Reviews The size and scope of each review panel depends on the specific technology(ies) being reviewed and the areas of expertise required to address the review criteria. In general, there are three types of peer reviews: Type I: multitechnology reviews of a complex nature (five or more reviewers); Type II: in-depth reviews of a single technology (three or more reviewers); and Type III: reviews of documents requiring no meeting of the review panel (i.e., mail review; three or more reviewers). To reduce costs and increase the number of projects that can be reviewed, OST now reserves Type II reviews for emergency situations where a program manager requires a short-term evaluation of the technical merits of a specific project or proposal, and encourages peer reviews of similar technologies to be grouped together in Type I reviews whenever possible (DOE, 1998b).

OCR for page 53
--> Selection of Reviewers Selection Criteria Peer reviewers are selected by three generally recognized criteria: (1) education and relevant experience, (2) peer recognition, and (3) contributions to the profession. A minimum of a B.S. in a relevant engineering or scientific field and significant experience in the technical area being reviewed is required for participation on a review panel. Peer recognition is assessed by measures such as activity in professional societies and scholarly organizations. Contributions to the profession include publications in peer reviewed journals, patents, and meeting presentations. Panel Formation According to presentations by RSI staff at the ASME Peer Review Committee meeting in November 1997, ASME uses three electronic databases and the ASME membership list (approximately 120,000 names) to identify reviewers meeting the above criteria. In addition to a database of those who have previously served as reviewers (approximately 130 names), ASME and RSI have a working database of nominated reviewers that contains approximately 1,000 entries and a list of potential reviewers with approximately 5,000 entries. RSI uses these databases to identify potential reviewers who then must be approved by the ASME PRC. Panel members are reimbursed for travel expenses and generally receive an honorarium of several hundred dollars per review. Conflict of Interest Individuals with any real or perceived conflicts of interest with respect to the subject of the review may not serve as members of a review panel. According to ASME, "an individual who has a personal stake in the outcome of the review may not act as a reviewer" (ASME, 1997, p. 8). The ASME PRC has interpreted this provision as excluding all employees of DOE and DOE-owned facilities operated by contractors, including national laboratories. Potential conflicts of interest among members of the review panels, the PRC, and RSI staff are handled by requiring all who participate in the review program to sign a statement certifying that they do not have personal and financial interest in the outcome of the review. The ASME conflict-of-interest policy further requests that reviewers and members of the PRC and its Executive Panel recuse

OCR for page 53
--> themselves from deliberations on any matter in which there may be an appearance of a conflict of interest. Review Criteria OST established the following general review criteria (DOE, 1998b): Relevancy: Is the technology applicable to the specific contaminants or conditions as claimed? Scientific and technical validity: Is the technology consistent with established scientific, engineering, and industry principles and standards? Is the likelihood of its success reasonable? Are the data reliable? Are the stated scientific deductions meritorious? Are the project personnel technically qualified? Are the best available R&D practices used? Nonduplicative or superior to alternatives: Is the technology duplicative or inferior to existing technology? Data validity: Is the data collection process complete and valid, and can results be applied to regulatory, cost-benefit, stakeholder, and risk evaluations? OST has provided no guidance on the relative weighting of these general criteria. However, OST policy states that these general review criteria must be augmented by technology-specific criteria. For each review, the FA/CC program manager submits a preliminary list of technology-specific criteria to the Peer Review Coordinator, who forwards them to RSI in the written request that initiates the review process. The proposed criteria are discussed among the Technical Secretary, Peer Review Coordinator, and OST managers, as necessary. Final criteria are approved by the EP of the ASME Peer Review Committee. These criteria are used by PIs and other presenters to organize written materials and oral presentations for the peer review. In addition, the review panel does have the authority to pursue other issues that arise during the review.

OCR for page 53
--> Review Session The peer review session for Type I and II reviews consists of five parts: Introduction. A representative of the PRC describes the review program and discusses the review criteria, and the Peer Review Coordinator describes the OST process. Description of the project. PIs give presentations describing the details of the technology under review. Stakeholders may present technical aspects of their concerns. Audience members and the review panel may ask questions of the presenters. Executive session. The review panel meets in closed session to identify unresolved issues, unanswered questions, and additional needs for information. Discussion (questions and answers). The review panel may ask questions of the presenters and other proponents of the project to resolve the questions raised in closed session. Executive session. The review panel writes its report in closed session with the assistance of the Technical Secretary. This session commonly lasts half a day. Throughout the open sessions, presenters are given the opportunity to respond to questions and clarify information provided to the panel. In many cases, presenters and other proponents of the project have been allowed to provide additional information or background materials to the review panel, on the condition that it is presented before the panel completes its deliberations. All of the peer review session is open to the public, with the exception of the executive sessions. The chair of the review panel presides over the entire session. For Type III reviews, the format of the review is similar, but occurs by mail and phone. The introduction of the review is conducted via conference call. The review panel is generally given one to two weeks to read the written material being reviewed and to prepare comments. The Technical Secretary consolidates the comments and prepares the Report of the Review Panel, which is provided to the panel members for their review and comment. An executive session is held via conference call to discuss unresolved issues before the report is submitted to the Peer Review Coordinator.

OCR for page 53
--> Review Reports The outcome of each review is contained in a technical peer review report, which is prepared in three phases: Report of the Review Panel, Interim Report, and Final Report. The Report of the Review Panel is prepared at the review session with the participation of all panel members and the assistance of the Technical Secretary. According to OST policy, it should typically contain the following items (DOE, 1998b): introduction; project summary; objectives of the review; evaluation criteria; findings of the panel; recommendations of the panel; appendix including any minority reports or extra information that may seem unnecessary for the report but be of potential interest to readers; biographical summaries of the members of the review panel; and a list of references used in the analysis. The Report of the Review Panel is provided to OST investigators and managers who are requested to notify the Technical Secretary of any errors within 5 days and to respond in writing to the recommendations within 30 days. The response is required to be clear and detailed and to include a schedule of implementation, if possible. A summary of the responses containing their salient features is added to the Report of the Review Panel to form the Interim Report. After DOE's response, the ASME PRC reviews the Interim Report, and may add to it substantive, explanatory, clarifying, and supplementary comments and recommendations. It is this committee's understanding that the PRC may not alter the sense of the review panel's report. The Interim Report as reviewed and approved by the PRC is issued as the Final Report. Copies of the Reports of the Review Panels, the responses from the investigators and managers, and the Interim Reports are made available to members of the PRC for review and comment during the entire review process. At the end of the fiscal year (or after a specific period), an Annual Report is prepared by the PRC. This contains Final Reports of all projects, recommendations of the PRC, formal letters to OST,4 other information that, 4   At its January 26, 1998, meeting the PRC voted to send several formal letters to OST, in order to provide more timely recommendations prior to the next Annual Report.

OCR for page 53
--> in the judgment of the PRC, would be beneficial to OST. This report is printed and widely distributed to FA/CC managers, PIs, and others. Feedback of Peer Review Results into Program Management and Development Decisions OST's TIDM requires that peer review reports be used in all funding decisions made at Gate 4 and any subsequent gate in the technology development process. In the gate reviews, these reports are intended to constitute one input to be considered in determining whether to continue funding for a project. As discussed in Chapter 5, OST has not yet demonstrated that the results from this new peer review program are used consistently in its decision-making process.