Building an Effective Environmental Management Science Program: Final Assessment (1997)

This chapter addresses the Statement of Task questions related to management needs for the Environmental Management Science Program (EMSP) (Appendix A). The Statement of Task directs the committee to provide advice on evaluation of the basic research supported by the EMSP and its impact on the cleanup mission, as well as the overall structure and management of the program. The committee summarizes the conclusions from its previous reports and provides additional comments in this chapter on the following issues:

• long-term management strategies,

• maintaining program quality,

• assessing outcomes, and

• applying the results of basic research.

The conference report that created the EMSP directed that the program be managed by the Office of Energy Research (ER). The Secretary of Energy subsequently decided to establish the program as a joint effort between the Office of Environmental Management (EM) and ER to ensure a continuing focus on both research merit and program relevance. The committee endorsed this joint management approach in its earlier reports and most members of the committee remain convinced that such an approach is necessary for the continued success of the program.

During the short time the program has been in operation, EM and ER staff have worked within a management structure that provides similar levels of responsibility for both offices. Most of the management processes were put into place during the first proposal competition, and many of these processes have yet to be tested through a full project cycle. The fact that EM and ER staff were able to establish this "hybrid" management structure in the middle of a proposal competition attests both to their dedication and energy. It also is a testimony to the efforts

made by ER and EM management to devote some of their best people to this program.

During the course of this study, the committee received considerable information from the Department describing the joint management structure for the EMSP.¹ The committee summarizes its understanding of this structure below.

• EM and ER are described as ''partnering" at headquarters to set policy for the EMSP and to carry out key tasks such as (1) assuring the quality of ongoing research, (2) determining future research needs, and (3) strengthening the linkage between research and the cleanup activities. Each EMSP project has both a designated ER program manager and an EM program manager.

• Actual administration of EMSP projects is done through the Idaho National Engineering Laboratory (INEL) field office. This office is described as the lead organization to administer, manage, and coordinate the award of research grants. The committee was told that INEL also will be used to pull together information from the focus areas and develop a list of problem needs.²

• In the situation where a national laboratory receives a grant, the DOE Operations Office that has oversight for that laboratory also is responsible for administering funding for the award. In addition, the Operations Office coordinates with the headquarters program manager(s) responsible for the award(s) in their laboratory. Some Operations Offices also have the responsibility to identify site-specific needs, to ensure research results are applied, to coordinate interactions with the Site Technology Coordination Groups,³ to set up various kinds of site-specific workshops, and to do other things that may help with use of the research.

• The national laboratories also have a role in the program. They are charged with managing EMSP funding for work in their facility; they must put in place mechanisms to promote interactions among inside and outside resources; and they are responsible for organizing and running the topical workshops.

These management activities seem reasonable to the committee when considered individually. In the aggregate, however, these activities and the structure that supports them seem unnecessarily complicated. Indeed, when considered against the small size of the EMSP and its focus on basic research, the management structure seems overloaded with administrators and coordinators.

As the program settles into a "steady state" over the next several years, the committee believes that simplification of program management and a clearer delineation of responsibilities among all management participants is needed to ensure its continued effectiveness. The committee believes that in the future program management staff will take on new, important, and potentially conflicting management responsibilities, for example:

• Maintaining internal and external advocacy for the program.

• Developing and maintaining performance measures for accountability to Congress and stakeholders.

• Developing outreach initiatives to ensure the continuing quality of grant recipients.

• Ensuring the continuing cooperation and coordination between EM and ER.

• Ensuring that the results of the research are utilized at the earliest possible time.

• Enhancing the productivity of the program.

In the committee's opinion, EMSP staff will have great difficulty in executing these tasks effectively under the current management structure where no single individual is "in charge" of the program. Therefore, the committee recommends that management of the EMSP be vested in a

single individual—an EMSP Program Director—who should have authority, responsibility, and accountability for meeting the program's objectives.

The EMSP Program Director should be an individual with a research background who is respected within the research community and understands the mission and responsibilities of EM. The Program Director must have access to and be included in the strategic planning activities within EM and must be utilized by EM management as an important scientific voice in the planning of the EM research and technology development, agenda. Involvement in the latter activity is particularly important to achieve the earliest deployment of EMSP research results into technology development and, ultimately, cleanup activities. Similarly, the Program Director must be included in the planning activities within ER to ensure the proper coordination of the EMSP with other ER research programs. The Program Director also should be responsible for ensuring outreach and coordination activities among performers and stakeholders. He or she must have the responsibility to set policy for grant administration and reporting requirements and provide direction to the program managers who oversee the proposals and grants on a day-to-day basis.

The Program Director must have the support of both the Director of the Office of Energy Research and the Assistant Secretary for Environmental Management to utilize the considerable resources of both organizations for the benefit of the EMSP. At the same time, the Program Director must be able to balance the interests of ER (to support high-quality basic research) and EM (to support research that is relevant to the cleanup mission) and must have the authority to resolve conflicts when these interests come into competition. In the committee's view, the Program Director can be effective in achieving and sustaining this balance only if she or he is functionally independent of both EM and ER. To allow for such independence, the committee recommends that the EMSP Program Director report to the Under Secretary for Energy.

The committee spent a great deal of time discussing alternative management strategies for the EMSP before making the recommendations that appear above. In fact, the committee considered the following five alternatives: (1) status quo, that is, joint management by EM and ER with no Program Director, (2) management of the program by ER with management responsibility vested in an ER Program

Director, (3) management of the program by EM with management responsibility vested in an EM Program Director, (4) joint management by EM and ER with responsibility vested in a Program Director reporting to both EM and ER, and (5) joint management by EM and ER with responsibility vested in a Program Director reporting to another office in DOE.

Initially, the status quo alternative had considerable appeal for the committee. The hybrid organization comprised of EM and ER staff is already in place and has worked well to date. As noted at the beginning of this section, however, the committee concluded that the current arrangement structure would not be workable for addressing the longer range needs of the EMSP or for balancing the near-term and long-term pressures on the program.

One could interpret the congressional language that established the EMSP (Chapter 1) as supporting ER management with an ER Program Director (alternative 2). This alternative does have appeal—the EMSP is a basic research program, and ER is highly skilled at managing basic research. In fact, ER is now managing the merit review process in the EMSP precisely because this is something it does well.

The major disadvantage of this alternative is that the EMSP would likely lose its strong linkages to the users and their problems, which is not what the Congress intended to happen: "This funding is to be used to stimulate the required basic research, development and demonstration efforts to seek new and innovative cleanup methods to replace current conventional approaches . . ." (H.R. 1905; see Chapter 1). A majority of the committee was of the strong view that alternative 2 would not accomplish this linkage. However, one member of the committee believes that the only way the EMSP program can hope to be successful is if it is managed by ER alone (see Appendix D).

Alternative 3 (EM management with an EM Program Director) has some appeal as well. Under EM management, the EMSP would maintain a strong focus on the cleanup mission and would be more responsive to immediate and site-specific technology needs. However, management by EM would likely result in a shift of emphasis toward projects with more immediate payoffs at the expense of longer-term, higher-risk, or more innovative projects. In addition, EM has little experience in managing basic research programs and peer review of basic research, and it has relatively little contact with the basic research

community in universities, national laboratories, or industry. Thus, EM would be on a steep and rocky "learning curve" were it given the management responsibility for this program.

The committee has an additional concern with alternatives 2 and 3: if the EMSP Program Director reported either to EM or to ER, he or she would be driven by the interests of those offices and would find it difficult to operate independently of those interests. The committee believes that it is essential to keep both EM and ER involved in the EMSP because each plays different, largely complementary, and equally important roles in the program.

Joint management with the Program Director reporting to both EM and ER (alternative 4) also was judged to be unworkable by the committee, largely for the same reasons that the current management arrangement was deemed to be unworkable over the long-term—namely, the Program Director would likely find it difficult to please two masters having fundamentally different missions.

Thus, the committee settled on alternative 5 (joint management with the Program Director reporting to another office within DOE) largely by a process of elimination. This alternative maintains the productive collaborations that are occurring currently between EM and ER, it gives both offices some "ownership" of the program, and it provides leadership to deal with the longer-term issues identified at the beginning of this section. Further, it puts a single individual in charge of and accountable for the program and allows this individual to balance the competing interests of EM and ER.

The committee recommended that the Program Director report to the Under Secretary because both EM and ER report directly to the Under Secretary's office. The committee recognizes that the recommendation on reporting responsibilities for the Program Director could be viewed as unrealistic when the small size of this program is considered against the other responsibilities of the Under Secretary. Nevertheless, the committee makes this recommendation because it believes that, although small, the EMSP can contribute significantly to the Department's ability to resolve the contamination legacy and to utilize effectively the several hundred billion dollars that has been estimated will be spent on the cleanup effort.

The committee notes that agencies such as the National Science Foundation (NSF), the National Institutes of Health (NIH), and the

National Aeronautics and Space Administration (NASA) have small offices and programs that report to high levels (frequently to the agency heads) within their organizations, particularly when the activities of the offices cross internal organizational lines. These arrangements are frequently transient and are intended to bring visibility, emphasis, coordination or management attention to specific initiatives. Examples of such arrangements include NSF's Office of Science and Technology Infrastructure, NASA's Office of Safety and Mission Assurance, and NIH's Office of AIDS Research and Office of Behavioral and Social Science Research.

The committee understands and appreciates the difficult task the Department of Energy (DOE) faces in creating a basic research program that will serve the needs of a highly goal-oriented organization such as EM. In some respects the committee is troubled by the prospect of a program enmeshed in an irreconcilable conflict between the character of its basic research and the need for this research to be ultimately useful to EM and the cleanup effort, particularly given the small size of the program in relation to the total EM effort. Indeed, there were some committee members who believed that basic research was fundamentally incompatible with the strongly needs-driven mission of the EMSP. The committee discussed various ways that this conflict might be addressed (e.g., setting aside money for "blue sky" projects with no particular relevance in order to reinforce the basic character of the program). But at this early stage and given the program's small size, it is not clear to the committee what the near-term versus long-term pressures will be, so the committee thought it inappropriate to be overly prescriptive because it did not want to drive the program in unproductive directions.

The committee notes, however, that there are other agencies, for example, the NIH, the Department of Agriculture, and the National Oceanic and Atmospheric Administration, where mission-oriented basic research is performed, supported, and managed reasonably well and where the long-term outcome has been both high-quality research and significant advances in achieving those agencies' missions. The issue of how to manage such an effort within DOE was one with which the committee struggled mightily, perhaps in large part because there is still a significant lack of clarity about what EM's mission really is.

Many federal agencies have found that over time their research programs are strengthened and their credibility reaffirmed through periodic, rigorous, independent peer review of all aspects of the programs. For example, the National Institute of Standards and Technology has for many years used "visiting committees" to review each of its major divisions. These committees are usually comprised of eminent scientists and engineers from industry and academia and often include senior industry managers. NSF also uses such visiting committees in many of its research programs. These committees review the operations of the program or division (i.e., effectiveness of peer review, processing time for grants), and also the program's or division's strategic directions and scientific focus. Many universities also use visiting committees to review the quality of their academic programs—in some cases, members of the committee include representative sponsors of research on campus and can effectively articulate the viewpoint of a "customer."

The committee believes that the EMSP would similarly benefit from periodic, independent peer reviews. These reviews should address all aspects of EMSP program management, including

• the merit and relevance review processes,

• quality of funded proposals,

• effectiveness of the application of research results to technology development and cleanup,

• effectiveness of the program in attracting outstanding researchers and innovative research ideas, and

• overall management efficiency and effectiveness.

The committee recommends that the Department convene an independent review panel at appropriate intervals to review the performance and effectiveness of the EMSP.⁴

The committee recognizes that the long-term success of the EMSP depends on the quantity and quality of the "outcomes"—namely, the impacts on fundamental scientific understanding and, ultimately, on cleanup. However, the time scale for basic research may be quite long. The committee also recognizes that the measurement of outcomes from basic research is currently receiving thorough and careful consideration by many federal research agencies.⁵

At present, no criteria have been established to measure outcomes from the EMSP, although EM staff have proposed two performance criteria to provide such measurements: (1) the number of research projects tied to science needs as identified by Site Technology Coordination Groups, site-specific science research agendas, and program offices and (2) the number of research projects with documented peer-reviewed research results.⁶

In view of the wide breadth of disciplines supported within EMSP and the well-recognized problems of assessing performance of basic research,⁷ the committee advises the Department against attempting the development of a general, formal quantitative structure for assessing the performance of the work of its investigators. Nevertheless, the committee believes that it will be essential to review and assess the quality of EMSP on a periodic basis. In the committee's view, the most

important component of an evaluation of research performance is through a review of the work of investigators supported by the program by an independent review panel of leaders in the field. Such review will assess the overall scientific quality of the program and the extent to which the research it supports has led to technical or intellectual "breakthroughs" of value to the scientific community and technology development efforts.

Despite the acknowledged limitations of review by peers, no better means has been found to evaluate and assure research quality over the long-term. As noted in a recent report by the Office of Science and Technology Policy,⁸ "for evaluating current programs in individual agencies, merit review based on peer evaluation will continue to be the primary vehicle for assessing the excellence and conduct of science at the cutting edge." Ultimately, of course, it will be the quality of the panel members carrying out such reviews that will determine the quality of the EMSP-supported research.

The committee recommends that the independent review panel be charged with the responsibility of assessing the quality of EMSP science and its impacts.

To accomplish this task, the panel should be provided with information about EMSP by the Program Director that includes but is not limited to the following:

• a comprehensive listing of publications by EMSP grantees;

• a listing of graduate and postdoctoral students trained by EMSP investigators; the degrees, if any, awarded; and current positions of these students;

• a compilation of the most significant scientific results of EMSP with a discussion of how these were selected;

• a compilation of the linkages to the larger EM effort developed with EMSP-supported research; and

• where possible, retrospective studies of the long-term impacts of EMSP results on technology development and cleanup.

The committee recognizes that it could take several years for the compilation of this information to be meaningful even for an initial assessment of the quality of science and its impact. This is inherent in the

nature of a basic research program. But the committee also recognizes that there are shorter-term "drivers" for program assessment, notably the Government Performance and Results Act of 1993. The Department must provide information to the Congress on an annual basis about its performance in response to the mandate of GPRA. Indeed, there may be some advantages to the Department and the EMSP in considering performance measures that have somewhat more immediacy than those listed above but that recognize that the "payback time" for EMSP as a basic research program will be long. Such shorter-term measures might help to sustain the interest in and commitment to EMSP of managers of technology development and cleanup activities. For example, such assessments might focus on processes for evaluating the quality of research proposals and for applying results to cleanup. This might then help to reinforce the linkages to the larger EM effort, the impact of which could only be fully assessed years later.

The committee, therefore, recommends that the Program Director assume the responsibility for developing a "portfolio" of information that would support both short-term and long-term assessment of EMSP by the independent review panel. The Program Director might be well served in this regard by exploring what strategies are being used by other federal agencies that support basic research.

However, the committee believes that attempts at short-term assessments of basic research programs such as EMSP will have very limited value at best. Information of the kind noted above, namely long-term data on outcomes and impacts, is, indeed, the most effective way to assess the value of EMSP and presents the most complete picture of both the quality of the EMSP research activities and their ultimate impact on the cleanup mission.

The EMSP is designed to support high-quality basic research that has the potential to have significant positive impacts on the broader cleanup effort. It is not possible to predict when and where such impacts will occur. What can be predicted is that by supporting high-quality basic research, new knowledge and insights will be gained and, over time, the

benefits from such knowledge and insights will pay off in the broader cleanup mission.

The movement of new knowledge and insights from investigators to full-scale application is a slow and diffuse process—a process without clear pathways in most cases. As a way of facilitating this information flow and stimulating new research ideas, the EMSP Program Director should convene annual workshops, seminars, and symposia that bring together EMSP investigators, program managers from EM and ER (including those in the EM focus areas), site contractors and other ''problem holders," and, when appropriate, other stakeholders, regulators, and principal investigators (P.I.s) and managers from other research programs. The Program Director should assume responsibility for determining how to best structure such activities so that they serve the interests of investigators and EM's needs for information transfer. Of course, such gatherings should not take the place of papers and reports, which, particularly when peer reviewed, form the basis for wide communication among scientists. However, the committee cautions that whatever mechanisms are developed, they must add value to the EMSP and should not be simply a check mark on a "to do" list.

It will be important in any effort that is undertaken to improve communication and information flow to involve the problem holders at the sites. These individuals will not only have the greatest knowledge about the sites but will also be able to assist in integrating the results of EMSP into the long-term EM effort. The ultimate success of EMSP may depend in no small part on the support and participation of these problem holders.

The responsibility for disseminating results from EMSP is not EMSP's alone. Other offices in EM, especially the other parts of the Office of Science and Technology, must take an active role in ensuring that the Department and the nation reap the full benefits from EMSP-supported research. It is beyond the committee's charge to advise the Office of Science and Technology on how to move the research from the EMSP and other federal research programs into application—the committee simply notes that, without an active effort to move research into technology development and application, the EMSP will become a high-quality research program with a limited impact on EM's cleanup mission.