National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"1 Introduction." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

1
Introduction

BACKGROUND AND CONTEXT

From the time the Department of Energy (DOE) was formed in 1977, successive administrations in Washington have looked to technological innovation as a critical tool for ensuring that the nation has affordable, clean, and reliable energy. To help stimulate this innovation, DOE, since its inception, has invested about $29 billion in energy efficiency and fossil energy research and development (R&D).1 Looking ahead, innovation continues to be a centerpiece of energy policy. Indeed, visions of a hydrogen economy and zero-emissions coal power plants have increased the importance of energy R&D among national priorities.

Recognizing the importance of technological innovation, DOE, the Office of Management and Budget (OMB), and congressional appropriations committees have paid more attention to understanding the effectiveness of federal funding for applied energy R&D. The conference report of the Consolidated Appropriations Act2 for fiscal year (FY) 2000 requested the National Academy of Sciences to assess the benefits and costs of DOE’s R&D programs in fossil energy and energy efficiency3—a task agreed on by the House and Senate subcommittees with jurisdiction over funding of these programs. The retrospective study (NRC, 2001) reported that, in the aggregate, the benefits of federal applied energy R&D exceeded the costs but observed that the DOE portfolio included both striking successes and expensive failures. As important, the study noted that the methodologies by which DOE had calculated the benefits of its programs varied considerably, making comparisons of program benefits difficult.

The retrospective study also confirmed that evaluating government investment in these programs is not a trivial matter. For example, the analysis of costs and benefits must reflect the full range of public benefits—environmental and national security impacts as well as economic impacts. In addition, the value of government funding depends on what might happen if the government did not support a project. It is possible, though not certain, that a private entity would undertake the project or an equivalent activity that would produce some or even all of the benefits of government involvement. To deal with these and other issues, the committee that wrote the retrospective study developed and applied its own uniform methodology for assessing the benefits of applied energy R&D programs (NRC, 2001).

PROSPECTIVE BENEFITS STUDY

The Congress continued to express its interest in R&D benefits assessment by providing funds for the NRC to build on the retrospective methodology to develop a methodology for assessing prospective benefits.4 Funds were made available for this purpose in both FY 2003 and FY 2004. In response, the NRC began a project for the development of a prospective benefits methodology in two overlapping phases and formed the Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs (see biographies of committee members in Appendix A). The current study, which is the first of the two phases, began in December 2003. The task assigned to the committee for Phase One was to

1  

This amount includes only those funds under the jurisdiction of the U.S. House Appropriations Subcommittee on the Interior and Related Agencies.

2  

House Report 106-479, p. 493. November 18, 1999. U.S. Government Printing Office: Washington D.C.

3  

The fossil energy R&D programs are administered by the Office of Fossil Energy and the energy efficiency programs by the Office of Energy Efficiency and Renewable Energy.

4  

Specifically, the Congress provided funds for “a continuing annual review by the [National] Academy [of Sciences] of programs, using the Academy’s matrix, to measure the relative benefits expected to be achieved and to inform decision making on what programs should be continued, expanded, scaled-back, or eliminated.” (House Report 107-564, p. 125. July 11, 2002. U.S. Government Printing Office: Washington, D.C.)

Suggested Citation:"1 Introduction." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

adapt the results of the previous committee with an aim to develop a methodology and matrix for evaluating prospective benefits of DOE’s energy efficiency and fossil energy programs. In addition, the committee will apply its newly developed methodology to evaluate energy efficiency and fossil energy programs,

as expressed in the pertinent part of the statement of task. (The complete statement of task for Phase One can be found in Appendix B.5) Phase Two will begin in early 2005. Funds were appropriated in FY 2004 for Phase Two and FY 2005 for Phase Three. Activities undertaken by the committee in connection with Phase One are listed in Appendix C.

Three considerations were particularly important in formulating this project. One was that the committee must build on the work of the retrospective study and, by extension, the work of DOE and OMB that has taken place since the retrospective study was completed. DOE has made important strides in developing tools for assessing the likely benefits of its R&D programs. Of special note is that elements of DOE—particularly the fossil energy and energy efficiency programs—have worked together toward common methodologies and approaches. The scale of DOE’s effort has been impressive, involving substantial staff commitments and the extensive use of sophisticated economic models. The efforts of DOE to improve and standardize its own estimates of benefits using a common methodology and approach contributed to the committee’s ability to fully understand the programs and their anticipated impacts.

OMB has also made considerable progress in creating tools for assessing the value of future energy R&D investments and has promulgated specific investment guidelines for applied R&D, as well as for more basic research programs.6 The OMB began the evolutionary development of the Program Assessment and Rating Tool (PART), which ranks federal programs on the basis of both their likely benefits and the quality of program design and management.7 One of the first applications of PART was to DOE’s applied energy R&D programs. OMB now makes public its summary assessments of major DOE programs.8

As the committee studied these developments in methodology, it became increasingly clear that this project would be most useful if it helped to move the entire enterprise toward a common methodology that assists decision makers at every level to consider “what programs should be continued, expanded, scaled-back, or eliminated.”9

The second consideration was that the committee must develop not only a methodology that is rigorous in its calculation of benefits and assessment of risks but also a practical process for applying that methodology in a consistent manner across a variety of DOE programs. The process needs the participation of outside experts who are familiar with the specific technologies and markets and can provide an independent assessment of the program. Familiarity with the programs, technology, and industry is critical to determining what needs to be considered in the analysis, assessing the likelihoods of achieving the technical goals, and calculating the benefits if they are successful. The process of assembling and applying expert panels efficiently within the constraints of a consistent analytic methodology is thus central to the committee’s work.

The third consideration was that the methodology being developed must be transparent and easy to use. The methodology should not require extensive resources to implement nor should the results from its application be difficult for key stakeholders in the program to understand. Further, to the extent possible, the analysis should be consistent with and extend the PART analysis.

To carry out its assigned task, the committee organized its work into the following subtasks:

  • Review the methodologies for assessing R&D benefits developed by DOE, OMB, and other agencies.10

  • Propose a conceptual framework that captures the key features of prospective benefits evaluation. The benefits matrix developed for the retrospective study was taken as a useful starting point for developing the prospective framework.

  • Appoint expert panels to apply the framework to the three DOE programs that had been selected by the committee. A committee member chaired each panel, but the panel members were mainly persons with expertise in the technology and markets pertaining to the programs.11 The panels

5  

The statement of task estimated that the committee’s methodology might be applied to perhaps 5 projects in fossil energy and 10 projects in energy efficiency. In choosing three rather large programs to which the panels were to apply the committee’s prospective benefits methodology, the committee ensured that many projects within each office were considered as part of the committee and panel efforts.

6  

John H. Marburger and Mitchell E. Daniels Jr. “Memorandum for the Heads of Executive Departments and Agencies: FY2005 Interagency Research and Development Priorities.” Executive Office of the President, June 5, 2003.

7  

Office of Management and Budget. President’s Management Agenda, Fiscal Year 2002.

8  

Office of Management and Budget. Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments.

9  

House Report 107-564, p. 125. July 11, 2002. U.S. Government Printing Office: Washington, D.C.

10  

Including, for example, calculations of program benefits performed pursuant to the Government Performance Results Act (GPRA) of 1993.

11  

The committee decided that three studies would be needed to test the methodology and selected the advanced lighting, fuel cells, and carbon sequestration programs. The advanced lighting program is managed by the Office of Energy Efficiency and Renewable Energy and the sequestration program by the Office of Fossil Energy; both organizations have fuel cell programs. These programs also involve the full range of benefits and likely future scenarios (i.e., future states of the world) relevant to the study. Although the committee believes these programs are sufficient for the purposes of the Phase One study, it recognizes that the methodology will be further refined and tested in Phase Two.

Suggested Citation:"1 Introduction." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

were provided with the proposed benefits framework but were given considerable flexibility in applying the methodology for completing the matrix. It was the judgment of the committee that this flexible approach would stimulate critical thinking about the methodological needs and practical problems of providing the relevant information. The committee believes that the panels met this objective in exemplary fashion.

  • Receive a report from each panel and use it as a basis for recommendations.

  • Evaluate the experience reported by the panels and modify the methodology accordingly.

  • Present conclusions in the Phase One report, which will also contain recommendations for specific steps to be taken in Phase Two of the ongoing effort.

The committee prepared a letter report, dated November 15, 2004, responding to a request from the staff of the House Appropriations Subcommittee on the Interior and Related Agencies to “… provide an overview of the study committee’s methodological approach; how that approach differs from the retrospective study delineated in the Academies’ report (also initiated by the Subcommittee through the appropriations process) Energy Research at DOE, Was It Worth It? (NRC, 2001); a synopsis of the current committee’s activities; and what is expected to be contained in the committee’s final report.” The letter report is included as Appendix D.

It is important to emphasize that Phase One was focused tightly on the basic problem of adapting the retrospective methodology to a prospective context. As a result, some important issues had to be deferred to Phase Two. For example, the issue of how to characterize national security benefits has not been explored. Security threats to the energy system have multiplied in number and kind since the retrospective study, but the committee intentionally deferred this and other issues to Phase Two of the project. In addition, the committee cautions against using the three expert panel case studies to assess the benefits of the three programs because the panels’ findings on the programs were developed for the express purpose of helping the committee to refine its methodology.

The balance of this report is organized as follows: Chapter 2 describes the evolution of the conceptual framework from retrospective to prospective evaluation and recalls the chief methodological and application problems encountered by the expert panels. Chapter 3 presents the committee’s recommended methodology. Chapter 4 presents the committee’s recommendations for applying the methodology in a consistent but practical way. Chapter 5 summarizes the work of the panels. Chapter 6 presents the committee’s overall conclusions about the Phase One project and its recommendations for Phase Two.

Suggested Citation:"1 Introduction." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 6
Suggested Citation:"1 Introduction." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 7
Suggested Citation:"1 Introduction." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 8
Next: 2 From Retrospective to Prospective Evaluation »
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward Get This Book
×
Buy Paperback | $56.00 Buy Ebook | $44.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 2001, the National Research Council (NRC) completed a congressionally mandated assessment of the benefits and costs of DOE's fossil energy and energy efficiency R&D programs, Energy Research at DOE: Was It Worth It? The Congress followed this retrospective study by directing DOE to request the NRC to develop a methodology for assessing prospective benefits. The first phase of this project—development of the methodology—began in December 2003. Phase two will make the methodology more robust and explore related issues, and subsequent phases will apply the methodology to review the prospective benefits of different DOE fossil energy and energy efficiency R&D programs. In developing this project, three considerations were particularly important. First, the study should adapt the work of the retrospective study. Second, the project should develop a methodology that provides a rigorous calculation of benefits and risks, and a practical and consistent process for its application. Third, the methodology should be transparent, should not require extensive resources for implementation, and should produce easily understood results. This report presents the results of phase one. It focuses on adaptation of the retrospective methodology to a prospective context.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!