National Academies Press: OpenBook

Peer Review in Environmental Technology Development Programs (1998)

Chapter: 6 “Triage” Approach for Reducing Project Backlogs

« Previous: 5 Analysis of OST's Peer Review Program
Suggested Citation:"6 “Triage” Approach for Reducing Project Backlogs." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

6
"Triage" Approach for Reducing Project Backlogs

OST has participated in more than 800 environmental restoration technology development projects since 1989 (DOE, 1998b), 226 of which are currently funded technologies and most of which have not been peer reviewed. Peer review of these technologies would be invaluable in helping determine which are ready for deployment,1 which merit further development, and which should be canceled. The number of technologies is too large to be reviewed rapidly by the current OST peer review process, however, because only 28 peer reviews were completed in 1997 and only 38 reviews have been scheduled for FY98 (Table 4.1).2 The relatively small percentage of active projects that have been peer reviewed also raises questions about the effectiveness of the peer review process, because it does not allow a consistent application of peer review results in OST's decision-making process. For example, projects that have been peer reviewed may be more likely, or less likely, to be funded solely because they were reviewed, rather than because of their technical merit.

In Chapter 4, the committee recommends two procedural changes to improve both the rigor of the selection process and the efficiency of the review process. Although these changes would help deal with OST's backlog of projects, eliminating the problem would require more fundamental changes in its peer review process. In this chapter, the committee focuses specifically on how OST could reduce this large backlog of OST-supported projects that have never been peer reviewed.

OST's current practice in which nearly all peer reviews include formal presentations by the project team, followed by deliberations by the panel, and further question-and-answer sessions over the course of two to three days, places

1  

In this context, deployment refers to large-scale application of a process.

2  

As of April 15, 1998.

Suggested Citation:"6 “Triage” Approach for Reducing Project Backlogs." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

significant limits on the number of projects that can be peer reviewed by a single panel.3 Even if the number of projects that were peer reviewed during a single Type I review could be increased by improved efficiency, OST's backlog of technologies that have never been peer reviewed still would take years to be eliminated through its current process. If OST is to fulfill its policy that "all projects are to be peer reviewed" in the short-term (i.e., the next year), it will have to make significant changes in how peer reviews are conducted.

The committee recommends that OST consider adopting a "triage" approach that would allow far greater numbers of technologies to be peer reviewed. This approach would involve a formal prescreening of projects by peer reviewers based exclusively on the written documentation on the project—in effect, a "mail review" of projects, to be followed by a formal meeting of the panel to discuss and rank the projects. During this prescreening review, panel members would be asked to rank all related technologies in a given area that are being considered for additional development or deployment.4 Rankings from the panel as a whole could then be used by OST program managers to determine those highly ranked, low-budget projects that should be considered for funding without additional peer review; those highly-ranked projects that should receive a more detailed evaluation (including presentations by the project team and question and answer sessions); and those technically weak projects that should not be considered for funding. This approach would provide OST program managers the basis for discontinuing funding for technically weak projects and might provide them with sufficient technical input (to be supplemented by input on nontechnical factors) to make a decision to fund a low-budget project. The prescreening evaluations should not be used as the sole means for providing technical input into decisions to fund high-budget, environmental remediation projects, however. Peer reviews involving presentations by the project team and question-and-answer sessions should be carried out for all projects involving significant capital investment by OST.

Because prescreening evaluations require only written documentation on the projects to be reviewed, the triage approach could be used to evaluate a large number of projects at a single review. For example, a single review could evaluate all projects developed to address a specific type of environmental management problem or an entire OST FA/CC program. The approach also could include related projects from OST's inventory of nearly 600 projects that

3  

According to RSI staff, the maximum number of projects that can be peer reviewed effectively during a Type I review using the current OST process is approximately five.

4  

If the prescreening review involves technologies at very different stages of development, it might be necessary to develop somewhat different review criteria for each general stage of development. The same reviewers could conduct all of the reviews, however.

Suggested Citation:"6 “Triage” Approach for Reducing Project Backlogs." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

are not currently funded. This might allow OST to identify especially promising technologies within its inventory that should be funded for demonstration or deployment.

Because of the breadth of technical issues that might be evaluated during a review of a large number of technologies, the number of reviewers required for the prescreening evaluation could have to be significantly larger than the five to six members typically employed for an OST Type I review. If the prescreening review process results in a greatly reduced number of technologies requiring a formal, panel review (as intended), fewer reviewers might be needed for the panel review itself, however.

Increasing the efficiency of OST's review process through the triage approach outlined above could result in some trade-offs in terms of the quality of the peer reviews, however. The same level of detail and thoroughness that is possible through panel reviews involving lengthy interaction and questioning of the project team would not be possible in the prescreening evaluations. The difference in quality of the reviews could be minimized by ensuring that the written documentation provided to the panel is complete and of high quality. A detailed proposal or statement of work for every project, as recommended earlier in this report, would be especially important for these prescreening reviews. The committee believes that the potential decrease in quality of some peer reviews, however, would be more than offset by the increased effectiveness of a peer review system in which all projects funded by OST have been through a credible peer review.

Suggested Citation:"6 “Triage” Approach for Reducing Project Backlogs." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 82
Suggested Citation:"6 “Triage” Approach for Reducing Project Backlogs." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 83
Suggested Citation:"6 “Triage” Approach for Reducing Project Backlogs." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 84
Next: 7 Improving the Effectiveness of OST's Peer Review Program »
Peer Review in Environmental Technology Development Programs Get This Book
×
Buy Paperback | $44.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!