BACKGROUND AND CONTEXT
From the time the Department of Energy (DOE) was formed in 1977, successive administrations in Washington have looked to technological innovation as a critical tool for ensuring that the nation has affordable, clean, and reliable energy. To help stimulate this innovation, DOE, since its inception, has invested about $29 billion in energy efficiency and fossil energy research and development (R&D).1 Looking ahead, innovation continues to be a centerpiece of energy policy. Indeed, visions of a hydrogen economy and zero-emissions coal power plants have increased the importance of energy R&D among national priorities.
Recognizing the importance of technological innovation, DOE, the Office of Management and Budget (OMB), and congressional appropriations committees have paid more attention to understanding the effectiveness of federal funding for applied energy R&D. The conference report of the Consolidated Appropriations Act2 for fiscal year (FY) 2000 requested the National Academy of Sciences to assess the benefits and costs of DOE’s R&D programs in fossil energy and energy efficiency3—a task agreed on by the House and Senate subcommittees with jurisdiction over funding of these programs. The retrospective study (NRC, 2001) reported that, in the aggregate, the benefits of federal applied energy R&D exceeded the costs but observed that the DOE portfolio included both striking successes and expensive failures. As important, the study noted that the methodologies by which DOE had calculated the benefits of its programs varied considerably, making comparisons of program benefits difficult.
The retrospective study also confirmed that evaluating government investment in these programs is not a trivial matter. For example, the analysis of costs and benefits must reflect the full range of public benefits—environmental and national security impacts as well as economic impacts. In addition, the value of government funding depends on what might happen if the government did not support a project. It is possible, though not certain, that a private entity would undertake the project or an equivalent activity that would produce some or even all of the benefits of government involvement. To deal with these and other issues, the committee that wrote the retrospective study developed and applied its own uniform methodology for assessing the benefits of applied energy R&D programs (NRC, 2001).
PROSPECTIVE BENEFITS STUDY
The Congress continued to express its interest in R&D benefits assessment by providing funds for the NRC to build on the retrospective methodology to develop a methodology for assessing prospective benefits.4 Funds were made available for this purpose in both FY 2003 and FY 2004. In response, the NRC began a project for the development of a prospective benefits methodology in two overlapping phases and formed the Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs (see biographies of committee members in Appendix A). The current study, which is the first of the two phases, began in December 2003. The task assigned to the committee for Phase One was to
adapt the results of the previous committee with an aim to develop a methodology and matrix for evaluating prospective benefits of DOE’s energy efficiency and fossil energy programs. In addition, the committee will apply its newly developed methodology to evaluate energy efficiency and fossil energy programs,
as expressed in the pertinent part of the statement of task. (The complete statement of task for Phase One can be found in Appendix B.5) Phase Two will begin in early 2005. Funds were appropriated in FY 2004 for Phase Two and FY 2005 for Phase Three. Activities undertaken by the committee in connection with Phase One are listed in Appendix C.
Three considerations were particularly important in formulating this project. One was that the committee must build on the work of the retrospective study and, by extension, the work of DOE and OMB that has taken place since the retrospective study was completed. DOE has made important strides in developing tools for assessing the likely benefits of its R&D programs. Of special note is that elements of DOE—particularly the fossil energy and energy efficiency programs—have worked together toward common methodologies and approaches. The scale of DOE’s effort has been impressive, involving substantial staff commitments and the extensive use of sophisticated economic models. The efforts of DOE to improve and standardize its own estimates of benefits using a common methodology and approach contributed to the committee’s ability to fully understand the programs and their anticipated impacts.
OMB has also made considerable progress in creating tools for assessing the value of future energy R&D investments and has promulgated specific investment guidelines for applied R&D, as well as for more basic research programs.6 The OMB began the evolutionary development of the Program Assessment and Rating Tool (PART), which ranks federal programs on the basis of both their likely benefits and the quality of program design and management.7 One of the first applications of PART was to DOE’s applied energy R&D programs. OMB now makes public its summary assessments of major DOE programs.8
As the committee studied these developments in methodology, it became increasingly clear that this project would be most useful if it helped to move the entire enterprise toward a common methodology that assists decision makers at every level to consider “what programs should be continued, expanded, scaled-back, or eliminated.”9
The second consideration was that the committee must develop not only a methodology that is rigorous in its calculation of benefits and assessment of risks but also a practical process for applying that methodology in a consistent manner across a variety of DOE programs. The process needs the participation of outside experts who are familiar with the specific technologies and markets and can provide an independent assessment of the program. Familiarity with the programs, technology, and industry is critical to determining what needs to be considered in the analysis, assessing the likelihoods of achieving the technical goals, and calculating the benefits if they are successful. The process of assembling and applying expert panels efficiently within the constraints of a consistent analytic methodology is thus central to the committee’s work.
The third consideration was that the methodology being developed must be transparent and easy to use. The methodology should not require extensive resources to implement nor should the results from its application be difficult for key stakeholders in the program to understand. Further, to the extent possible, the analysis should be consistent with and extend the PART analysis.
To carry out its assigned task, the committee organized its work into the following subtasks:
Review the methodologies for assessing R&D benefits developed by DOE, OMB, and other agencies.10
Propose a conceptual framework that captures the key features of prospective benefits evaluation. The benefits matrix developed for the retrospective study was taken as a useful starting point for developing the prospective framework.
Appoint expert panels to apply the framework to the three DOE programs that had been selected by the committee. A committee member chaired each panel, but the panel members were mainly persons with expertise in the technology and markets pertaining to the programs.11 The panels
were provided with the proposed benefits framework but were given considerable flexibility in applying the methodology for completing the matrix. It was the judgment of the committee that this flexible approach would stimulate critical thinking about the methodological needs and practical problems of providing the relevant information. The committee believes that the panels met this objective in exemplary fashion.
Receive a report from each panel and use it as a basis for recommendations.
Evaluate the experience reported by the panels and modify the methodology accordingly.
Present conclusions in the Phase One report, which will also contain recommendations for specific steps to be taken in Phase Two of the ongoing effort.
The committee prepared a letter report, dated November 15, 2004, responding to a request from the staff of the House Appropriations Subcommittee on the Interior and Related Agencies to “… provide an overview of the study committee’s methodological approach; how that approach differs from the retrospective study delineated in the Academies’ report (also initiated by the Subcommittee through the appropriations process) Energy Research at DOE, Was It Worth It? (NRC, 2001); a synopsis of the current committee’s activities; and what is expected to be contained in the committee’s final report.” The letter report is included as Appendix D.
It is important to emphasize that Phase One was focused tightly on the basic problem of adapting the retrospective methodology to a prospective context. As a result, some important issues had to be deferred to Phase Two. For example, the issue of how to characterize national security benefits has not been explored. Security threats to the energy system have multiplied in number and kind since the retrospective study, but the committee intentionally deferred this and other issues to Phase Two of the project. In addition, the committee cautions against using the three expert panel case studies to assess the benefits of the three programs because the panels’ findings on the programs were developed for the express purpose of helping the committee to refine its methodology.
The balance of this report is organized as follows: Chapter 2 describes the evolution of the conceptual framework from retrospective to prospective evaluation and recalls the chief methodological and application problems encountered by the expert panels. Chapter 3 presents the committee’s recommended methodology. Chapter 4 presents the committee’s recommendations for applying the methodology in a consistent but practical way. Chapter 5 summarizes the work of the panels. Chapter 6 presents the committee’s overall conclusions about the Phase One project and its recommendations for Phase Two.