This page in the original is blank. |
Appendix A
Description of OST's Peer Review Program1
The Office of Science and Technology's (OST's) peer review program was established in 1996 to address the concerns of external review groups such as the General Accounting Office and the National Research Council's Committee on Environmental Management Technologies (CEMT), who have criticized OST for its lack of a standardized, independent, external peer review system. This new, office-wide program was designed to provide credible peer reviews of OST's technologies at the appropriate stages in the technology development process.
Peer reviews are used by OST to provide an independent evaluation of the technical merits of a technology. These evaluations are intended to assist decision makers in making "go/no-go" decisions regarding further support for technology projects as they proceed through OST's Technology Decision Process (see Appendix C). In part, because the Technology Decision Process evaluates projects against both technical and nontechnical criteria, OST's peer review program does not directly make decisions regarding the funding of OST technologies. Rather, peer review results are used as one input into the decisions made at critical decision points (or "gates;" Figure C.1) between each stage of the research and development process.
Roles and Responsibilities
The main decision makers regarding the funding of technology projects within OST are the individual Focus Area/Crosscutting (FA/CC) program managers. As such, FA/CC program managers are the main "customer" of the OST peer review program. The peer review program is managed within OST by the Peer Review Coordinator, who represents the Deputy Assistant Secretary for Science and Technology. The peer reviews are conducted by the American Society of Mechanical Engineers (ASME) with administrative and technical support provided by the Institute for Regulatory Science (RSI). Funding for the review program is provided directly to RSI through a $1.2 million annual grant from the OST. This section describes the primary roles and responsibilities of the various entities involved in the OST peer review program.
FA/CC Program Managers
The FA/CC program managers initiate the peer review process by making written requests for peer reviews to the OST Peer Review Coordinator. The FA/CC program managers are responsible for developing a prioritized list of technologies to be reviewed, providing documentation to reviewers, preparing responses to review panels' Consensus Reports, and for covering the cost of FA/CC program personnel and material needed for the peer reviews.2 The
FA/CC program managers are responsible for incorporating the results of technical peer reviews into the decision-making process.
Principal Investigators
Each project's principal investigators (PIs) are responsible for providing relevant project information for the peer reviewers. Depending on the specific stage of development of project, the OST Revised Guidance (DOE, 1997b) describes the documentation generally required for a peer review. Such information includes written background documentation relevant to the nine general criteria for peer review (see below) and oral presentations to the review panels, if needed. At Gate 4 of the Technology Development Process Procedure, for example, PIs are required to provide the following:
- literature references,
- progress report (topical),
- needs document,
- test plan,
- quality assurance plan,
- proof of design,
- life cycle cost analysis,
- risk analysis,
- regulatory issues and review, and
- public acceptance issues and plans.
Peer Review Coordinator
The Peer Review Coordinator is the principal federal official responsible for managing the day-to-day activities of the program. Specific responsibilities include receiving, processing, and scheduling peer review requests from FA/CC program managers; coordinating peer review activities among FA/CC program managers, the ASME Peer Review Committee (see below), and the review panels; ensuring reviews are executed in a timely manner; ensuring FA/CC program responses to review recommendations are included in Final Reports; and managing the budget and records for OST.
ASME Peer Review Committee and Its Executive Panel
The Peer Review Committee (PRC) is a standing body of the ASME whose sole purpose is to oversee the OST's peer review program and enforce relevant ASME policies, including compliance with professional and ethical requirements. The PRC consists of 13 to 15 members chosen for their competencies and diversity of views. Membership in ASME is not required for appointment to the PRC. The PRC is responsible for appointing members nominated by the Executive Panel (see below) to individual peer review panels and assessing Interim Reports for conformity to ASME standards before being issued as ASME-sanctioned public documents.3
The PRC also reviews and approves the Annual Report and presents it, along with a comprehensive evaluation of the technical quality of the OST program, to the Deputy Assistant Secretary for OST at an annual meeting.
The Executive Panel of the Peer Review Committee (EP) consists of three to five ASME members who have served in leadership positions within ASME (e.g., president, vice-president, or division chair). The EP is responsible for overseeing the day-to-day operation of the peer review program and acts on behalf of the PRC between its semi-annual meetings. The EP meets approximately four times per year.
Peer Review Panels
Peer review panels consist of three or more technical experts chosen for their knowledge of the specific technology to be reviewed. Although ASME requires the peer review panel chair to be a member of ASME, other panel members need not be ASME members. The complexity of the technology(ies) being reviewed and the type of review dictate the number of experts on a panel. The peer review panel conducts the review, prepares a Consensus Report detailing its recommendations and observations, and transmits the written report directly to the sponsoring FA/CC program manager. Peer review panels are terminated upon completion of their specific review task.
American Society of Mechanical Engineers
The ASME participates in the OST peer review program by ensuring peer reviews follow ASME procedures. The ASME establishes and sanctions the Peer Review Committee and its Executive Panel through formal approval by the Center for Research and Technology Development of ASME's Council of Engineering. The ASME also serves as a resource for identifying potential peer review panel members.
Institute for Regulatory Science
RSI provides administrative and technical support for peer review activities. RSI's responsibilities include meeting planning, compiling and distributing background materials for members of review panels, facilitating peer reviews, and providing an executive secretary to assist in the preparation of Consensus Reports of peer review panels.
Selection of Technologies to Be Reviewed
OST policy requires that a peer review be conducted on each technology/system before it passes through Gate 4 of OST's Technology Investment Decision Model (TIDM) process (see Appendix B for a description of the TIDM). Specifically, OST policy states that
It is OST policy that a technology peer review will be conducted on each technology/system. However, FA/CC Program Managers will give priority attention to those technologies that are preparing to pass through Gate 4 of the OST Technology Decision Process Procedure (Version 7.0) [DOE, 1997c]. Specifically, this procedure states that technical peer review reports are one of the
documents required to pass through Gate 4. In addition, peer reviews may be appropriate at other phases of the technology maturation stages. (DOE, 1997b, p. 3)
The actual decisions to conduct peer reviews of specific technologies are made by FA/CC program managers in response to five main ''drivers:'' (1) funding (i.e., to justify proposed increases in spending or to justify previous spending on a given technology), (2) changes in status (i.e., when a project approaches a gate in the TIDM process), (3) regulatory issues (i.e., in order to ensure that a technology meets regulatory requirements, (4) stakeholder concerns (i.e., in order to reassure the public or other stakeholders of the technical credibility of a technology, and (5) any technical or other issues that arise concerning a given technology.
Types of Reviews
The size and scope of each review panel depends on the specific technology(ies) being reviewed and the areas of expertise required to address the review criteria. In general, there are three levels of technical reviews:
- multi-technology review of a complex nature involving site visits (five or more reviewers);
- in-depth review of a technology involving site visits (three or more reviewers); and
- review of a document requiting no site visit (i.e., a mail review).
Selection of Reviewers
Reviewers are selected and approved by the Peer Review Committee based on three generally recognized criteria: (1) education and relevant experience, (2) peer recognition, and (3) contributions to the profession. A minimum of a B.S. in an engineering or scientific field and sufficient experience in the technical area being reviewed are required for participation in an OST-related review. Peer recognition is assessed by measures such as activity in a professional society. Contributions to the profession include publications in peer reviewed journals, patents, and meeting presentations.
Conflict of Interest
Individuals with any real or perceived conflicts of interest with respect to the subject of the review may not serve on as a member of a review panel. According to ASME, "an individual who has a personal stake in the outcome of the review may not act as a reviewer" (ASME, 1997, p. 8). Potential conflicts of interest among members of the review panels, the ASME Peer Review Committee, and RSI staff are handled by requiring everyone who participates in the review program to sign a statement indicating a lack of personal or financial interest in the outcome of the review. The ASME conflict of interest policy further requests that reviewers and members of the ASME Peer Review Committee and its Executive Panel recuse themselves from deliberations on any matter in which there may be an appearance of a conflict of interest.
Peer Review Criteria
The following general criteria for peer reviews have been established for OST's peer review program in the ASME Manual for Peer Review (ASME, 1997):
-
1)
Relevancy: All projects supported by the OST must meet the needs of various parts of the Office of Environmental Management, including EM-30, EM 40, EM 60, and EM 70.4 At a minimum, a DOE-wide need must be clearly and unambiguously identifiable.
-
2)
Scientific and technical validity: Every technology supported by the DOE must be consistent with established scientific and engineering principles and standards. Furthermore, the likelihood of its success must be reasonable.
-
3)
State-of-the-art: A prerequisite for desirability of a technology is that its design and engineering tools are state-of-the-art.
-
4)
Filling an existing void: A technology is considered desirable if it fills unmet needs.
-
5)
Non-duplicative/superior to alternatives: A technology can also be desirable if it does not duplicate an existing technology; or if there is duplication of an existing technology, it is superior to it as demonstrated by one or more criteria described under 6-9 below.
-
6)
Cost effectiveness: The life cycle cost of a technology is an important criterion for its desirability.
-
7)
Reduced risk: Reduction of risk to workers or the general public is an important parameter in decisions to implement a technology.
-
8)
Regulatory acceptability: It is unlikely that a technology will be implemented unless it meets applicable regulatory requirements. The regulatory acceptability must be evaluated carefully before a technology starts the permitting process.
-
9)
Public acceptance: Public acceptance of a technology does not necessarily follow peer acceptance of other parameters identified in this guidance. Furthermore, it is not always predictable. However, there are cases in which acceptance or lack of acceptance are clearly demonstrable.
OST has provided no guidance on the relative weighting of these general criteria. Although these nine criteria form the core of the criteria to be used during the peer review, the actual criteria used during each review are determined by the ASME Peer Review Committee in consultation with the FA/CC program manager. For each review, the FA/CC program manager develops a preliminary list of specific review criteria, which are then used by the ASME Peer Review Committee to develop the formal review criteria for the review panel. These criteria are used by project staff and other presenters to organize written materials and oral presentations for the peer review. In addition, even though these review criteria form the basis of the review panel's evaluation, the review panel does have the authority to pursue other issues that arise during the review.
Review Reports
Each member of the review panel prepares a report containing the outcomes of the review. The review panel chair combines these individual reports into a Consensus Report, which typically contains four parts: (1) a brief description of the subject that was reviewed, (2) the consensus of the panel on shortcomings and meritorious aspects of the project, (3) recommendations of the panel, and (4) an appendix containing the comments by each reviewer for which no consensus could be reached, or which were considered by the panel to be beneficial to the investigators and managers but were not important enough to be included in the main body of the Consensus Report.
The Consensus Report is provided to DOE investigators and managers who are requested to respond to the conclusions and recommendations within 30 days. A summary of the responses containing the salient features of the response are combined with the substantive sections of the Consensus Reports as the Interim Report. After DOE's response, the ASME PRC reviews the Interim Report, during which it may add to it substantive, explanatory, clarifying, and supplementary comments and recommendations. The Interim Report as reviewed and approved by the PRC is issued as a Final Report.
Copies of the Consensus Report, the responses from the investigators and managers, and the Interim Reports are made available to members of the PRC for review and comments during the entire review process. At the end of the year (or after a specific period), an Annual Report is prepared by the PRC. The report contains Final Reports of all projects, recommendations of the PRC, and other information that, in the judgment of the PRC, would be beneficial to OST. This report will be printed and widely distributed.
Feedback of Peer Review Results into Program Management and Development Decisions
One to two days after the completion of the peer review, RSI submits the peer review panel's Consensus Report to the FA/CC program manager who requested the review. The program manager is asked to prepare, within 30 days, a written response that describes how the PI and/or the FA/CC manager intend to respond to the findings and recommendations of the Consensus Report. This formal response is then combined with the substantive sections of the Consensus Report as an ASME Interim Report, which is used by DOE headquarters' staff to evaluate the management of the technology program. The Interim Report is then reviewed and approved by the ASME Peer Review Committee and released as an ASME Final Report.
OST's Technology Decision Process requires that peer review reports be used in all funding decisions made at Gate 4 or any subsequent gate in the technology development process. In these gate reviews, these reports constitute one input to be considered in determining whether to continue funding for a project.