Building an Effective Environmental Management Science Program: Final Assessment (1997)

The Department of Energy's (DOE's) Environmental Management Science Program (EMSP) was created by the 104^th Congress to stimulate basic research and technology development for cleanup of the nation's nuclear weapons complex. The EMSP is a mission-directed basic research program and is designed to support a much larger technology development program within the Office of Environmental Management (EM). The program is managed jointly by EM and the Office of Energy Research (ER). Unlike other federal programs that address environmental problems, the EMSP is explicitly focused on EM's problems and has the specific objective to improve the effectiveness of the cleanup effort over the long-term.

This is the third of three reports written by this committee at the request of Thomas P. Grumbly, Under Secretary of Energy, to provide advice to the Department on the structure and management of the EMSP.¹ Summaries of the committee's principal conclusions and recommendations are provided in the following sections. More detailed explanations and supporting discussions can be found in the text of the report.

Many of EM's cleanup problems cannot be solved or even managed efficiently and safely with current technologies, in part owing to their tremendous size and scope. However, cleanup would benefit greatly from the involvement of basic researchers, as noted in recent NRC and DOE reports (see Chapter 2). The committee believes that a basic research program focused on EM's most difficult cleanup problems may have a significant long-term impact on the EM mission. Basic research can provide new knowledge to allow the Department to attack cleanup

problems that are currently intractable or exorbitantly expensive using current technologies; it can lead to the development of better technologies to allow current cleanup to be accomplished at lower costs or with fewer hazards to workers and the public; it can improve understanding of risks and how to discuss them with local stakeholders; and it can lead to the development of new or improved technologies that will allow cleanup to a higher state than is presently possible, thereby making sites available for less restrictive uses. Simply put, new technologies are required to deal with EM's most difficult problems, and new technologies demand new science.

The EMSP is different in several respects from other federal basic research programs, including other DOE programs, that support fundamental research related to the environment. Although several federal programs support basic research in fields broadly relevant to environment science, none are focused explicitly on EM's problems, and none have an explicit link to the problem holders at the sites. In addition, the EMSP will promote the development of partnerships among universities, national laboratories, other federal agencies, and the private sector. These partnerships can bring together highly creative and innovative researchers, provide access to unique national research facilities, and provide a multidisciplinary focus on EM's most difficult problems.

Funding for the EMSP should be viewed as an investment that may, in the long-term, lead to more effective cleanup. The EMSP alone will not solve all of EM's cleanup problems—but given the sheer magnitude of the cleanup mission and its estimated cost, coupled with the technological challenges, the committee views the investment in EMSP as both prudent and timely.

If the EMSP is to have a significant impact on the cleanup mission, the Department must incorporate this program into its strategic plans. Indeed, as the deadline for the Government Performance and Results Act's reporting requirements draws near, it is essential to the survival of the EMSP that a plan for applying basic research in the cleanup program—a science plan—be explicitly and officially articulated by the Department. The committee recommends that the Department

develop a science plan for the EMSP. This science plan should provide a comprehensive list of significant cleanup problems in the nation's nuclear weapons complex that can be addressed through basic research and a strategy for addressing them. This science plan should serve as the primary guiding document for the Department's research investment in the cleanup mission.

The committee recommends both a near-term and a long-term process for developing a science plan for the EMSP. For the near term (i.e., the fiscal year 1997 [FY97] competition), the committee recommends that the Department develop a science plan from existing Department documents. Examples of documents that could be used for this purpose are provided in Chapter 3. For the longer term (i.e., the FY98 competition), the committee recommends that the Department consult with its ''problem holders"—the technical staff, managers, and stakeholder advisory groups at the sites who have some understanding of cleanup issues—to obtain guidance on cleanup problems that cannot be addressed practically or efficiently with current knowledge or technologies. The committee recognizes, of course, that the technical expertise and knowledge for assessing cleanup problems among these groups is uneven and, consequently, suggestions from these groups will have to be considered against that knowledge. Given the large number of DOE sites, these consultations will have to be structured carefully to be manageable by and useful to EMSP staff.

The committee's Letter Report encouraged the Department to broaden its research solicitations and to include problems related to risk, health assessment, and quantitative methodologies (i.e., statistical methods, numerical [simulation] methods and the combination of the two sets of techniques), mainly because the committee believes that research in these areas could have a direct impact on the cleanup mission. In addition, the committee believes that ER should ensure that the pertinent merit review panelists are knowledgeable in the risk research field.

The science plan is likely to be very broad in scope—both in terms of the range of problems and the disciplinary coverage—and will likely require an investment in basic research that is larger than the

current $50 million annual investment in the EMSP. To implement the science plan, Department staff should find ways to utilize relevant research being sponsored in other federal programs and to focus the EMSP on those problems that are unique to the weapons complex.

Given the relatively small size of the EMSP and its staff, the committee does not deem it prudent to recommend formal coordination mechanisms between the EMSP and other research programs. The committee does, however, offer several examples of the kinds of coordinating activities that could be of value to the program in Chapter 3.

Department staff should strive to broaden the community of investigators involved in the EMSP and to expand the core or "committed cadre" of investigators who are knowledgeable about EM's problems. The Department can broaden the community of investigators concerned with its cleanup problems by encouraging (but not requiring) appropriate collaborations among university, industry, and national laboratory researchers. These collaborations are not an end in themselves but rather a route for stimulating new research, introducing new investigators to the Department's problems, and assuring relevance of the projects. By additional encouragement of graduate and postdoctoral training in areas of interest, the Department can further broaden the community of investigators over the longer term.

The committee recommends that collaborations be Encouraged where appropriate—but they should not be a requirement for the program. The committee also reaffirms the recommendation from its Letter Report (p. 4) that the program "should encourage (but not require) graduate student involvement in research proposals submitted to the program." The committee would add to this recommendation that appropriate postdoctoral training opportunities, including training opportunities within current DOE programs, also should be encouraged to sustain the interest of talented young scientists.

Based on its review of the data received from the Department, the committee reached the following conclusions about the proposals selected for funding in the FY96 competition: (1) meritorious projects appear to have been selected; (2) collaborative efforts were well represented among the list of successful projects; (3) the program appears to have been successful in attracting some "new" (to DOE) researchers to the program; and (4) in the one case where firsthand information was available, the committee was able to confirm the overall quality of the merit review panel.

The committee has two concerns about the transparency and technical credibility of the merit review process used in the FY96 competition. First, the merit review process was "opaque" to those who submitted proposals to the program and the broader research community. Second, the merit review panels were not allowed to reach consensus on individual proposals or to provide ER program managers with a ranking of proposals because the panels were not constituted under the Federal Advisory Committee Act (FACA). The committee recommends that the Department examine the entire review process for the EMSP with the goal of increasing its transparency and technical credibility. To this end, the committee recommends that the Department carry through on its stated intention (in its response to a 1991 General Accounting Office report) to seek a change in its legislation to allow FACA proposal review panels—and to convene the EMSP merit review panels under FACA once this change is made.

The committee also is concerned about the lack of timely feedback to proposers in the FY96 proposal competition. In at least some instances, panelist reviews were not sent to principal investigators (P.I.s) unless requested, and these reviews did not always reflect the discussions in the panel meetings. The committee recommends that in future competitions the proposal reviews be modified to reflect the discussions at the panel meetings and, further, that applicants receive feedback on the content and result of the reviews in a timely fashion.

The committee remains concerned about the developing "mortgage" on future-year budgets in the program from commitments made in the FY96 proposal competition. Based on its analysis of future-year funding (Chapter 4), the committee reached the following conclusions about the budget for the program: (1) the annual budget for the EMSP will have to increase significantly to maintain a reasonable number of new starts with an equitable distribution of funding between DOE and non-DOE performers or (2) if the budget remains at current levels, both non-DOE and DOE performers could see about a 75 percent drop in funding for new and competitive renewal projects. The committee believes that, without some assurance that funding will be available to support a reasonable number of new awards annually, EMSP will simply not be viewed as "worth the effort" by potential proposers.

The committee appreciates the difficult budget environment that DOE now finds itself in and recognizes that any increases in the budget for the EMSP may be at the expense of other Department programs. In the committee's view, however, this funding should not come from existing ER programs, which are vital to the Department's long-term mission and are an important part of the nation's basic research portfolio. Nevertheless, the EMSP cannot live up to its potential without careful consideration by DOE of both the total funding levels and the funding patterns (i.e., the balance between new and continuing awards). The committee urges DOE to find a solution to the problem of not being able to "forward fund" projects at national laboratories and reiterates its recommendation from the previous reports to fully fund all awards in the first year.

The committee does not believe that stakeholders should be involved in the day-to-day management of the program and, in particular, the proposal review and selection process. To be effective and credible, the review and selection process should be carried out by technical experts and should remain free of local concerns and special-interest pressures. Stakeholders should be consulted for guidance on site

problems for the EMSP science plan. The committee suggests a process in Chapter 3 for obtaining this guidance.

The committee also believes that participation of EMSP investigators in the proposal selection process would be very helpful in future-years. These individuals can bring an important perspective that helps link EMSP more closely to the broad research community, which will benefit the process of shaping the longer-term character of the program.

DOE should also improve and enhance the ways in which it informs the potential users of EMSP results (e.g., technology managers at the various sites) about the process and the outcome of EMSP proposal selection. The hoped-for result of such improved information flow is that these "problem holders" will become more attuned to the long-term benefits of EMSP to their efforts.

The committee believes that simplification of program management and a clearer delineation of responsibilities among all management participants is needed to ensure the long-term effectiveness of the EMSP. To this end, the committee recommends that management of the EMSP be vested in a single individual—an EMSP Program Director—who should have authority, responsibility, and accountability for meeting the program's objectives.

This Program Director must be involved in the planning activities of both EM and ER and must have the support of the Director of Energy Research and the Assistant Secretary for Environmental Management to utilize the considerable resources from both organizations for the benefit of the EMSP. At the same time, the Program Director must be able to balance the interests of ER and EM and must have the independence to resolve conflicts when these interests come into competition. To allow for such independence, the committee recommends that the EMSP Program Director report to the Under Secretary for Energy.

The committee recognizes that this recommendation might be seen by some in the Department as unrealistic when the small size of this program is considered against the other responsibilities of the Under Secretary. Nevertheless, the committee makes this recommendation because it believes that, although the program is small, the success of the

EMSP can be vital to the Department's ability to resolve the contamination legacy and to utilize effectively the several hundred billion dollars estimated to be spent on the cleanup effort.

To maintain the quality of the EMSP, the committee recommends that the Department convene an independent review panel at appropriate intervals to review the performance and effectiveness of the following aspects of the program:

• merit and relevance review processes,

• quality of funded proposals,

• effectiveness of the application of research results to technology development and cleanup,

• effectiveness of the program in attracting outstanding researchers and innovative research ideas, and

• overall management efficiency and effectiveness.

The Department must provide information about performance of the EMSP to meet the requirements of the Government Performance and Results Act of 1993. The committee believes that the best way to assess the performance of the EMSP is through independent peer review. Such review will assess the overall scientific quality of the program and the extent to which the research it supports has led to technical or intellectual "breakthroughs" of value to the scientific community and technology development efforts.

The committee recommends that the independent review panel be charged with the responsibility of assessing the quality of EMSP science and its impacts. The committee recommends that the Program Director assume the responsibility for developing a "portfolio" of information that would support both shorter-term and long-term assessment of EMSP by the independent review panel.

The movement of new knowledge and insights from investigators to full-scale application is a slow and diffuse process. As a way of facilitating this information flow and stimulating new research ideas, the EMSP Program Director should convene annual workshops, seminars, and symposia that bring together EMSP investigators, program managers from EM and ER (including those in the EM focus areas), site contractors and other problem holders, and, when appropriate, other stakeholders, regulators, and P.I.s and managers from other research programs. The Program Director should assume responsibility for determining how to best structure such activities so that they serve the interests of investigators and EM's needs for information transfer. It will be important in any effort that is undertaken to improve communication and information flow to involve the problem holders at the sites. These individuals will not only have the greatest knowledge about the sites but will also be able to assist in integrating the results of EMSP into the long-term EM effort.

The responsibility for disseminating results from EMSP is not EMSP's alone. Other offices in EM, especially other parts of the Office of Science and Technology, must take an active role in ensuring that the Department and the nation reap the full benefits from EMSP-supported research. Without an active effort to move research into technology development and application, the EMSP may become a high-quality research program but have little limited impact on the EM's cleanup mission.