Building an Effective Environmental Management Science Program: Final Assessment (1997)

In response to your letter of August 9, 1996, the National Research Council's Committee on Building an Environmental Management Science Program offers this letter report on the fiscal year (FY) 1997 program announcement for the U.S. Department of Energy's (DOE's) Environmental Management Science Program (EMSP).

The committee has been charged to advise the Department on ways to improve the effectiveness of the EMSP The statement of task for the committee's work is given in Attachment A. The committee members were selected to provide a balance of expertise and perspectives, including knowledge of and experience with the weapons complex and its clean-up challenges, and the proposal solicitation process related to basic research. A list of committee members is given in Attachment B.

The committee held its first meeting on May 11-12, 1996, and published the first of three reports, Building an Effective Environmental Management Science Program: Initial Assessment, in July.¹ The Initial Assessment report presents the committee's preliminary evaluation of the EMSP, findings regarding the FY 1996 proposal competition, and recommendations for the FY 1997 program announcement. The committee specifically recommended that the Department postpone the release of the FY 1997 program announcement to allow time to identify and incorporate lessons learned from the FY 1996 program competition and to determine how the program should be structured and managed. The committee also noted that the FY 1997 competition likely will have a major role in shaping the program and ensuring its future success.

Reasons for Writing this Letter Report

Your letter of August 9, 1996 (Attachment C), requests that the committee provide additional advice to the Department regarding the content of the FY 1997 program announcement, and, in particular, advice on research needs. This letter is meant to address this request. This letter reflects a consensus of the committee and has been reviewed in accordance with the procedures of the National Research Council. This letter does not take the place of the committee's final report, which will be completed by the end of the year, but rather is intended to

provide the Department with more timely advice to avoid an unnecessary delay in the release of the FY 1997 program announcement

As a first step in its deliberations that led to this letter, the committee reviewed the results of the FY 1996 proposal competition to assess the effectiveness of the FY 1996 program announcement. To this end, the committee asked for—and received—from the Department the following data on the successful proposals for the FY 1996 proposal competition

• proposal titles and names of principal investigators (Pls),

• their institutional affiliations;

• award amounts; and

• scientific field of proposal (e.g., geoscience) and area of potential impact of the proposed research (e.g., contaminant plume treatment).

Because the information provided to the committee lists only the principal investigator of each project, evidence of collaborations between individuals or institutions is lacking Thus, while collaborations may exist, the committee was not able to determine them from the information provided. The committee is seeking additional information on collaborations for its final report

The information listed above was provided to the committee the day before its fourth meeting, which was held on August 21-22, 1996. The committee concluded that it did not have enough time or information to assess conclusively the overall success of the FY 1996 program and the effectiveness of the FY 1996 program announcement. However, the committee did have first-hand information on the makeup and operation of one of the review panels and was able to confirm the overall quality of the proposals, the review process, and the review panelists. The committee intends to provide additional comments on the success of the FY 1996 competition in its final report.

The committee also requested and was provided with a list of the titles of unsuccessful projects. This information allowed the committee to inform itself generally on the nature of proposed projects, but the titles themselves did not provide the members with enough information to make an effective assessment of the quality of the research or proposers. The committee does note, however, that the titles indicate that the Department received proposals in a wide range of research areas listed in the FY 1996 program announcement

The committee also received copies of the guidelines that were given to the merit and relevance review panelists by the offices of Environmental Management (EM) and Energy

Research (ER)² The committee requested—but did not receive—the names of the merit and relevance review panelists, and the final ratings of the proposals This information is considered confidential by the Department

The committee expects to receive additional information on the FY 1996 proposal competition at a later date. This information includes abstracts of the successful projects. biographical sketches of investigators who received funding, and, for successful investigators, a list of recent, current, and pending research support. The committee is especially interested in the number of successful investigators with recent, current, or pending Department support relative to the total number of investigators. The committee believes that this information will help it to assess whether the EMSP was successful in attracting high-quality researchers and innovative proposals to the program

Recommendations for the FY 1997 Program Announcement

The FY 1996 program announcement provided a fairly complete description of the EMSP. The announcement included a statement of purpose, a list of research needs, a brief description of the criteria used for proposal review and selection, a schedule for proposal submission and review, and a financial plan. The committee believes that such a self-contained announcement is helpful to the research community because it provides most of the information needed to prepare a competitive proposal in a single, readily accessible package. The committee recommends that the Department use the same approach in developing the FY 1997 program announcement, and it offers suggestions below on the following elements of the announcement:

• criteria for proposal review and selection;

• research areas;

• proposal format;

• program schedule;

• review process; and

• financial plan.

1. Criteria for proposal review and selection. The FY 1997 program announcement should be explicit about what criteria will be used to select proposals for funding. The committee recommends that the Department utilize the following criteria:

• Focus on basic research. The purpose of the EMSP is to foster basic research that will contribute to long-term clean up of the weapons complex. The focus on basic research should be articulated clearly in the program announcement

• Scientific merit. As noted in the committee's Initial Assessment report, merit should be the primary criterion for proposal selection. Using scientific merit as a first screen will help ensure that only high-quality proposals are supported by the program, and it will help keep the focus of the program on basic research.

• Relevance to mission. Also as noted in the Initial Assessment report, research should be broadly relevant to EM's clean up mission. That is, the basic research supported in the EMSP should address the phenomena and processes that underpin EM's clean up problems The proposal need not demonstrate knowledge of problems at weapons complex sites to be useful to the clean up mission in the long-term, and such knowledge should not be required in the proposal.

• A demonstrated record of research accomplishment. As noted in the committee's Initial Assessment report, the EMSP should aim to attract outstanding researchers to work on EM's problems.

• The project must he able to demonstrate progress (but not necessarily completion) in the 3-year time period. As mandated by Congress, one purpose of the program is to "stimulate the required basic research." which may require longer-term commitments beyond 3 years. The committee recognizes, of course, that even long-term projects may yield important "deliverables" over much shorter time frames.

• Training opportunities. In its Initial Assessment report, the committee commented on the need to build a "committed cadre" of researchers for the EMSP. The committee believes that graduate student training is an effective mechanism for building a community of researchers knowledgeable of EM's problems and responsive to EM's research needs The program announcement should encourage (but not require) graduate student involvement in research proposals submitted to the program.

2. Research areas. In the committee's original statement of task (Attachment A) and your letter requesting this report (Attachment C), the committee was asked to identify additional areas of research that should be included in the EMSP. In its deliberations on this issue, the committee has concluded that the EMSP is more likely to attract innovative proposals from creative researchers if the focus of the program announcement is shifted away from a statement of suggested solutions (i.e., research areas), as was provided in the FY 1996 program announcement, to a statement of EM's problems that require basic research As an aid to researchers, the Department also may wish to include in its program announcement examples of

innovative proposals that were submitted by researchers in the FY 1996 proposal competition The committee believes that this approach would encourage researchers who are not knowledgeable of EM's clean up problems to apply their expertise and suggest solutions that may not have occurred to the authors of the program announcement.

It is beyond the experience and the expertise of the committee to provide a list of EM problems that should be included in the FY 1997 program announcement In its Initial Assessment report, the committee recommended that "concise technical summaries" of clean-up problems be prepared by the Department. The committee reaffirms the importance of these summaries and recommends that they be prepared forthwith. Such summaries should include examples of the types of problems that exist at specific sites as well as more generic problems that apply across sites, such as ground water contamination.

In formulating this problem list, the committee encourages the Department to broaden the solicitation to include problems related to risk, quantitative methodologies, and health assessment As noted in two recent National Research Council reports,³ relevant research on risk would be especially valuable for prioritizing clean up efforts and allocating limited resources Indeed, a risk-based approach is currently being used by the Department to help identify and rank the important problems and prioritize clean up (DOE, 1995)⁴ Currently, there is much scientific uncertainty about the very existence of risk to human health at the low levels projected for the end stages of the clean-up effort. To establish standards and measures of progress, substantial improvement in the scientific state-of-art is needed. The EMSP could contribute further to the understanding of risk and risk-based approaches to priority setting. Accordingly, the committee recommends that the program announcement be expanded to include risk as it relates to the clean-up program, both now and in the future.

3. Proposal format. To emphasize important information that is required in a proposal, the committee recommends that the Department specify a format for proposals that incorporates the elements shown in Appendix D. A standard format would be a major aid to reviewers in assessing and comparing proposals.

4. Program schedule. The accelerated FY 1996 competition schedule presented a significant challenge to researchers, review panelists, and EM and ER program managers.⁵ The compressed schedule in the first round gave researchers little time to educate themselves on EM clean-up problems, to develop proposals, or to establish new collaborations. For program managers and review panelists, the tight schedule placed severe pressures on the preproposal selection process and final proposal reviews. The committee believes that more time should be allowed in the FY 1997 program competition to alleviate these pressures. To this end, the committee recommends that the Department provide researchers with at least one month to prepare preproposals and two months to prepare full proposals.

The committee believes that the FY 1997 review process also would be improved by giving review panelists more time to examine proposals In the FY 1996 competition, most merit panelists received proposals to review only two weeks prior to the panel meetings, and most relevance panelists did not receive proposals in advance of their meetings. The committee believes that the panelists will do a better job of evaluating these proposals if they are given more time to review them prior to their panel meetings.

5. Review process. The committee recommends that the FY 1997 program announcement provide a clear description of the process that will be used to review proposals and select awards. The committee offers the following comments and recommendations for the Department's consideration

The collaborative management efforts between ER and EM have been very successful to date, and the committee urges continued interactions and open communications between staff in these offices in the FY 1997 program competition. The committee reaffirms its endorsement (from the Initial Assessment report) of the two-phase review process used in the FY 1996 competition that first evaluates the scientific and technical merit of the proposals and then examines more closely the relevance of the proposed work to the clean-up mission. The committee believes that this two-phase review should continue in FY 1997 and that it should continue to be managed as a partnership between ER and EM.

The committee further recommends that the Department retain, to the extent possible, continuity in merit and relevance review panels to take advantage of the experience gained in the FY 1996 competition. Additionally, the committee recommends that the Department's preproposal screening process involve, to the extent possible, members of the merit and relevance panels. The involvement of the research and clean-up communities in preproposal review will

strengthen the preselection process, which in the FY 1996 competition eliminated roughly two-thirds of the preproposals.

In the FY 1996 EMSP competition, the merit review panels convened by ER⁶ were constituted as non-FACA⁷ committees. In this capacity, the panelists were allowed to discuss the proposals and to provide ER program managers with individual scores on each proposal The panels were not allowed to reach consensus, nor were they allowed to provide ER program managers with a rank ordering of the proposals considered by each panel Further, the names of the panelists were kept confidential to the proposers and the research community at large, including this committee.⁸

As noted in its Initial Assessment report, building credibility in the research community is a singular challenge for the EMSP. The committee believes that such credibility is less likely to be achieved when the review process has the appearance of a "black box" into which proposals are fed and out of which funding decisions emerge To achieve more transparency in the process—and to provide for a higher quality of merit review by allowing panelists to reach consensus on proposal scoring and ranking—the committee strongly recommends that ER follow established practices of other federal agencies with basic research programs, such as the National Institutes of Health and the National Science Foundation, by constituting the FY 1997 review panels as FACA committees. The committee further recommends that ER announce its intention to follow the FACA process for merit review in the FY 1997 program announcement

The committee recognizes that this recommendation may be difficult to implement given DOE's history with the FACA process and ER's current practice with respect to FACA review panels.⁹ Nevertheless, the committee offers this recommendation because it believes that merit review panels constituted under FACA will improve significantly the quality and credibility of the review process.

6. Financial plan. In its Initial Assessment report, the committee recommended that "successful proposals should be funded fully 'up front' to help ensure the stability and continuity of the research projects and to establish a solid foundation on which a stable, long-term program can be built." The committee believes that the full-funding of proposals is essential for

establishing credibility for the program in the research community and, therefore, is an important factor in attracting high-quality researchers and proposals.

In FY 1996, the Department was able to provide full funding for proposals submitted by non-DOE performers (i.e., proposals from universities, industry, and non-profit research performers), but DOE was not able to provide full funding for proposals from national laboratory performers. The Department committed about $112 million in the FY 1996 competition A total of $43 million was provided out of FY 1996 funds to provide full funding for the 3-year non-DOE projects, and $4 million was provided to national laboratory projects. The remainder, about $65 million, will be provided to national laboratory projects out of future-year program funds (i.e., FY 1997, FY 1998, and FY 1999 funds) The committee believes that this ''mortgage" represents a significant challenge to the future viability of the program. In FY 1997, for example, $23 of the $50 million allocated to this program¹⁰ already has been committed to funding FY 1996 projects at national laboratories.

The committee has reviewed the future-year commitments from the FY 1996 awards to the national laboratories and has concluded that, on the current path, considerably fewer new or competitive renewal awards will be made in future-years unless significantly more funding is made available. Attachment E provides two simple scenarios for future funding that illustrate the committee's concerns regarding the "mortgage" and balance of funding for universities and national laboratories. Table E. shows that if the current pattern of funding is continued, approximately $112 million in program funds will be required annually by FY 2000 to maintain current levels of funding for new or renewal projects. If funding is constrained to approximately $50 million per year—the amount of funding available to the program in FY 1996—then funds for new or competitive renewal projects will decrease by approximately 75 percent.

Recognizing that a serious funding problem may be developing, the committee strongly encourages the Department to explore mechanisms to provide full funding for successful national laboratory proposals for the FY 1997 proposal competition. This issue should be resolved, if possible, before the FY 1997 program announcement is released, because it will govern the amount of funding available to the program next year, and hence the number of new starts.¹¹ Additionally, the committee recommends that funding guidelines, but not dollar limits, be provided in the FY 1997 program announcement. Specific dollar limits may restrict potentially outstanding research proposals from being submitted, which in turn, could limit the development of effective academic-laboratory-industry partnerships

Announcements/Publication

The Department published the FY 1996 program announcement in the Federal Register and on its home page, and it sent notices by mail to approximately 200 universities. The committee encourages the Department to utilize these dissemination mechanisms to publicize the FY 1997 proposal competition, and it recommends that the Department explore additional mechanisms to make the research community more broadly aware of the FY 1997 proposal competition. To this end, the committee recommends the use of paid advertisements in professional journals such as Science, Chemical and Engineering News, and EOS to publicize this program in FY 1997.

Summary

The FY 1997 program announcement will have a major impact on the future direction and viability of the EMSP. Although the committee had previously recommended postponing the release of the FY 1997 program notice in its Initial Assessment report, the committee recognizes the urgency of the Department's request for advice on the content of the notice. With the suggestions for modifications to the FY 1997 program announcement provided in this letter, the committee now urges the Department to move forward expeditiously to release the program announcement as soon as possible.

ATTACHMENT A

Statement Of Task

The committee will produce two reports that address the science and management needs of the Department of Energy's (DOE's) Environmental Management (EM) Science Program These reports will be produced in two separate activities as noted below.

ACTIVITY #1 FY97 RESEARCH PROGRAM

The committee will draw on the expertise of its members and other outside experts, the results of the 1996 DOE workshops on research needs, and previous NRC and federal government reports in order to address the following questions

1. How can basic research be used to help DOE EM "to complete its mission successfully in the next few decades"?

2. How can a basic research program help add value to DOE EM's cleanup efforts?

3. What kinds of technical challenges would likely benefit from a program in basic research?

4. How can the research program take advantage of the unique capabilities of U S. universities and federal labs?

5. How can the research program take advantage of research efforts and capabilities in other DOE programs and other federal agencies?

6. What, if any, additional areas of research should be included in the fiscal year (FY) 1997 program announcement as the DOE EM Science Program evolves?

The committee will not attempt to be comprehensive in addressing these questions, but, rather, its focus will be on providing guidance to DOE-EM for use in the FY97 program solicitation

ACTIVITY #2: SCIENCE AND MANAGEMENT NEEDS

The committee will produce a final report that provides a more detailed assessment of the science and management needs of the EM Science Program. This report will address the following questions:

Science Needs

1. How can science needs most effectively feed into the development of the EM research agenda?

2. How can the research program be structured to take advantage of research efforts and capabilities in other DOE programs and other federal agencies? (The committee would revisit the issue from the first activity.)

3. How can the research program be structured to broaden the community of researchers that can be called upon to address environmental problems?

4. What areas of basic research are likely to provide the best payoffs for FM cleanup efforts over the next few decades?

5. What additional areas of research should be included in future program announcements as the DOE EM Science Program evolves? (The committee would revisit the issue from the first activity.)

Management Needs

1. How can the DOE evaluate the quality of the basic research it supports and the impact of this research on its cleanup mission?

2. How can DOE identify changing needs for basic research as the program evolves?

3. How should the program be structured and operated in order to assist the DOE in overall reduction of cleanup costs, risks, waste generation, and time requirements?

4. How can the program be structured take advantage of the unique capabilities of U.S. universities and federal labs? (The committee would revisit the issue from the first activity)

Sponsor(s): Department of Energy

Date of Statement: 10/8/96

Date of Previous Statement: 7/15/96

ATTACHMENT B

Steering Committee on Building an Environmental Management Science Program

John F Ahearne. CHAIR

Sigma Xi. The Scientific Research Society & Duke University

Edward M Amett

Duke University (emeritus)

Stanley I Auerbach

Oak Ridge National Laboratory (retired)

Edward J. Bouwer

The Johns Hopkins University

John I Brauman

Stanford University

Naomi H Harley

New York University Medical Center

Harold Lewis

University of California, Santa Barbara (emeritus)

Derek R. Lovley

University of Massachusetts, Amherst

Alexander MacLachlan

DuPont (retired)

Gene G Mannella

Gas Research Institute (retired)

Norine E Noonan

Florida Institute of Technology

Jerome Sacks

National Institute of Statistical Sciences

Alfred P Sattelberger

Los Alamos National Laboratory

Leon T Silver

California Institute of Technology

COMMITTEE CONSULTANTS

Gregory R. Choppin

Florida State University

Donald J DePaolo

University of California, Berkeley

George M. Hornberger

The University of Virginia

ATTACHMENT C

ATTACHMENT D

Example Proposal Format

Project Abstract

Project Narrative

Goals

Scientific Significance of Project

Relevance of Project to the EM Cleanup Mission

Background

Research Plan

Preliminary Studies (if applicable)

Literature Cited

Research Design and Methodologies

Collaborative Arrangements (if applicable)

Appendices

Biographical Sketches

Description of Facilities and Resources

Budget

Budget Explanation

Current and Pending Support

ATTACHMENT E

Future Funding Scenarios for the Environmental Management Science Program

The purpose of this attachment is to illustrate two scenarios for funding of the Environmental Management Science Program (EMSP) through FY 2002 by extrapolating, under two sets of assumptions, from the FY 1996 award results The objective of these scenarios is to illustrate some consequences of the "mortgage" problem created by the "outyear" (i.e., post FY 1996) funding commitments made in the FY 1996 proposal competition¹

The unconstrained funding scenario, which is shown in Table E. l, was generated using the following set of assumptions:

• Funding of new awards for non-DOE performers (i.e., university, industry, and nonprofit performers) is continued at the FY 1996 level of $43 million for 3-year grants, and these awards are funded fully in the first year, as was the case for the FY 1996 proposal competition.

• The ratio of dollars committed each year to awards to non-DOE performers to the dollars committed each year to new awards to national lab performers remains constant at FY 1996 levels.²

• Awards to national lab performers are paid in equal installments over 3 years

• Total annual funding for the EMSP is allowed to increase as necessary to satisfy the foregoing assumptions.

As shown in Table E.1, in order to maintain funding at FY 1996 levels, the total annual funding for the program would almost triple, to $131 million in FY 1999, before declining to a steady-state value of $1 12 million in FY 2000. This amount is roughly 225 percent of the current annual budget for the program.

The constrained funding scenario, which is shown in Table E.2, was generated using the following set of assumptions:

• Total annual program funding is constrained to FY 1996 levels of $50 million³

• As in the unconstrained funding scenario, the ratio of dollars committed each year to awards to non-DOE performers to the dollars committed to new awards to national laboratory performers remains constant at FY 1996 levels.

• As in the unconstrained funding scenario, awards to national laboratory performers are paid in equal installments over 3 years. The first installment is paid during the fiscal year in which the awards were made. The two remaining installments are paid in the two succeeding fiscal years As shown by the scenario in Table E. 1, for example, the $69 million awarded to national laboratories in FY 1997 would be paid in three equal installments of $23 million in FY 1997, $23 million in FY 1998, and $23 million in FY 1999.

This scenario illustrates the full effects of the mortgage when national laboratory performers receive funding one year at a time and non-DOE performers receive all of their funding up front. As shown in Table E.2, the mortgage from the FY 1996 award cycle creates a significant drain on program funds through FY 1999. Indeed, by FY 1999 only $10 million in new funds are available to non-DOE performers and $6 million in new funds are available to national laboratory performers, about a quarter of the funding available in FY 1996.⁴

The committee believes that the following conclusions can be inferred reasonably from the scenarios shown above: (1) Funding for the program will have to increase significantly in future-years (e.g., as shown in Table E. ) in order to maintain current levels of program funding and a reasonable distribution of funding between non-DOE and national lab performers; or (2) both non-DOE and national lab performers will see a significant drop in funding for new or competitive renewal projects (e.g., Table E.2) if total annual funding for the program remains constant or decreases.