Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) (2007)

Since its inception in 1977 from an amalgam of federal authorities, the U.S. Department of Energy (DOE) has administered numerous programs aimed at developing applied energy technologies. The better portion of the annual expenditures dedicated to such technologies is spent on research and development (R&D) and is administered by three DOE offices—the Office of Energy Efficiency and Renewable Energy (EERE), the Office of Fossil Energy (FE), and the Office of Nuclear Energy Science and Technology (NE).¹ These three offices received approximately $2.6 billion total for fiscal year (FY) 2006, approximately $1.5 billion of which is being devoted to R&D. The R&D programs are complemented by policy measures, such as tax incentives to encourage early adoption of advanced technologies by consumers, efficiency standards for household appliances, and production tax credits for a variety of energy production technologies, including certain renewable energy sources (EIA, 2006).

In recent years, federal oversight of public expenditures has emphasized the integration of performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 “in response to questions about the value and effectiveness of federal programs” (GAO, 1997, p. 11). GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which has requested of the National Research Council (NRC) a series of reports using quantitative indicators to evaluate the effectiveness of applied energy R&D. The first such report² took a retrospective view of the first 23 years of DOE R&D programs on fossil energy and energy efficiency.³ The report found that DOE-sponsored research had netted large commercial successes, such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization (NRC, 2001). However, some programs were judged to be costly failures in which large R&D expenditures did not result in a commercial energy technology (NRC, 2001). A follow-up NRC committee was assigned the task of adapting the methodology to the assessment of the future payoff of continuing programs (NRC, 2005a).

Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic effects. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government did not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement?

The present report continues to investigate the development and use of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. It provides further definition for the development of indicators for environmental and security benefits and refines the evaluation process based on its experience with the six DOE R&D case studies.

The DOE divides its mission according to four strategic goals—one each dealing with defense, science, environment, and energy⁴ (DOE, 2005a)—and seven general goals. The energy strategic goal is supported by the general goal of energy security.⁵ The general goal underscores the role of technology development:

The three applied energy offices implement this goal using R&D programs, intergovernmental grants, reserves of fossil fuels, and appliance efficiency standards. Within EERE, R&D programs can be considered in two more or less distinct groups. The first consists of R&D directed at biomass and biorefinery systems R&D, geothermal technology, hydrogen technology, hydropower, solar energy, and wind energy technologies. Program activities range from basic research at universities and national laboratories to cost-shared applied research, development, and field validation in partnership with the private sector (OMB, 2005). The second group within EERE R&D consists of energy conservation R&D relating to the efficient end use of fuels and electricity in vehicles, in industrial processes and manufacturing, and in building envelopes. In general, this second group of energy conservation EERE R&D programs has the objective of achieving an output of a good or service with less energy input⁶—that is, achieving greater energy efficiency. The FY06 appropriation for EERE was $1,173 million,^7,8 the R&D component of which totaled $770 million, including $151 million for congressionally mandated R&D activities.⁹

Research in FE has traditionally been divided between the Office of Coal and Power Systems (CPS) and the Office of Oil and Natural Gas. CPS administers a suite of clean coal R&D, which aims at ensuring the generation of clean, reliable, affordable electricity from coal. One large planned demonstration project is FutureGen, which has as its aim the development of a zero-atmospheric-emissions power plant. Additional programs focus on all the key technologies needed for FutureGen—carbon sequestration, membrane technologies for oxygen and hydrogen separation, advanced turbines, fuel cells, technologies related to gasifiers for coal-to-hydrogen conversion, and others. The Clean Coal Power Initiative emphasizes the development of pollution controls and efficiency improvements for existing plants (DOE, 2006a). The Office of Oil and Natural Gas supports research and policy options to ensure clean, reliable, and affordable supplies of oil and natural gas for American consumers.¹⁰ The FY06 appropriation for FE was $841 million, with $592 million of it allotted for fossil energy R&D.¹¹

NE supports innovative applications of nuclear technology and administers four principal R&D programs. The Generation IV Nuclear Energy Systems Initiative and the Nuclear Hydrogen Initiative, which according to the information DOE provided on its budget for FY07 (DOE, 2006b) will develop a new generation of nuclear technologies to produce cost-effective electricity and commercial quantities of economic hydrogen to support the development of a transportation infrastructure. Nuclear Power 2010 is a demonstration program applicable to untested licensing and other regulatory processes. The Advanced Fuel Cycle Initiative aims to improve the efficiency and proliferation resistance of the nuclear fuel cycle.

The scope of programs subject to the NRC studies in this series has been limited to fossil energy R&D and the energy conservation portion of EERE’s R&D. Prior to FY06, funding for this group of programs fell under the jurisdiction of the Interior and Related Agencies appropriation subcommittees owing to the programs’ origins in the Department of Interior; the balance of EERE’s R&D funding, on energy efficiency, was included on a separate appropriations bill. However, in FY06, the House Committee on Appropriations restructured its subcommittees’ jurisdictions and reduced them in number from 13 to 10. Funding for all FE and EERE R&D programs was consolidated into one account subject to the jurisdiction of the House Appropriations Subcommittee on Energy and Water Development and Related Agencies

(CRS, 2005).¹² Four of the program funding line items that made up the pre-FY06 accounts were merged into two. In another instance—that of distributed energy—a program was moved out of EERE and into the appropriation account corresponding to the DOE Office of Electric Transmission and Distribution. (See Table 1-1 for a listing of programs and legislated funding sources in FY05 and earlier years and in FY06 and beyond.)

GPRA requires agencies to annually submit or update a strategic plan, an annual performance plan, and an annual performance report (GAO, 2005). Program costs and benefits are analyzed and submitted, pursuant to GPRA, in an annual report to Congress with the President’s budget. GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. Additional analyses are required of agencies by the Office of Management and Budget (OMB), the executive agency that formulates and administers the federal budget and evaluates program effectiveness. The President’s Management Agenda (PMA) (OMB, 2001) sets forth nine agency-specific reforms and five government-wide goals (OMB, 2005); a set of R&D investment criteria was spelled out in 2003, implementing one of the agency-specific goals of the PMA (Marburger and Daniels, 2003). Also in 2003, OMB inaugurated the Program Assessment Rating Tool (PART) (OMB, 2003) to assess and improve program performance. The Government Accountability Office—a congressional agency—regularly audits, examines, and evaluates government programs (GAO, 2005). A few of the aforementioned assessments, and their relationship to one another, are discussed below.

GPRA and PART require DOE to report on the performance of its R&D programs. The OMB guidance document for PART (OMB, 2005) states that “PART is a vehicle for achieving the goals of GPRA.” To encourage compatibility between the two systems, the PART guidance states that

In principle, therefore, PART and GPRA should use the same benefits estimates. However, it is difficult to tell whether this is being done because the DOE Performance and Accountability Report (DOE, 2005a) only presents a general statement of objectives and an assessment of whether a series of annual program targets was reached. Moreover, the PART Assessment Report¹³ does not consistently state specific long-term goals for each program. Table 1-2 contains the goal statements in the 2005 GPRA and PART reports in

the five technologies that the NRC evaluated in the current study.¹⁴

GPRA requires a statement of expected program results. EERE develops this statement through its benefits estimation process. That statement describes the relation between GPRA and benefits assessment as follows:

This benefits assessment process is the same one that EERE discussed with the NRC Phase Two committee. Note that the GPRA benefits are not discounted for risk.

PART does not report benefits estimates per se but rather asks a series of questions about the program as a basis for assessing its performance and management. The specific questions are listed in Appendix A. The relationship between PART and the present study depends on whether the NRC committee’s methodology helps to provide answers for the questions in PART. In particular, a credible benefits estimate is persuasive in answering PART questions on program purpose and design.

DOE has made important strides in developing tools for assessing the likely benefits of its R&D programs. Of special note is that elements of DOE—particularly the fossil energy and energy efficiency programs—have worked together toward common methodologies and approaches. In this regard, the efforts of DOE to improve and standardize its own estimates of benefits contributed to the committee’s ability to fully understand the programs and their anticipated impacts. At the undersecretary level, DOE maintains the R&D Council to handle the PMA, including R&D investment criteria, portfolio analysis, and budgeting. Reporting to the R&D Council are the Science and Technology (S&T) Integration Working Group, which facilitates compliance with administration and DOE requirements through the use of common methods and procedures and provides portfoliolevel decision support briefings, and the S&T Laboratory Working Group, which enhances the alignment of the R&D agenda with national energy goals.¹⁵

The conference report of the Consolidated Appropriations Act¹⁶ for FY00 requested the NRC to assess the benefits and costs of DOE’s R&D programs in fossil energy and energy efficiency, a task agreed on by the House and Senate subcommittees then having jurisdiction over funding of these programs. The retrospective study—Energy Research at DOE: Was It Worth It? (NRC, 2001)—reported that, in the aggregate, the benefits of federal applied energy R&D exceeded the costs but observed that the DOE portfolio included both striking successes and expensive failures. As important, the study noted that the methodologies by which DOE had calculated the benefits of its programs varied considerably, making comparisons of program benefits difficult.

Congress subsequently provided funds for “a continuing annual review by the [National] Academy [of Sciences] of programs … to measure the relative benefits expected to be achieved and to inform decision making on what programs should be continued, expanded, scaled-back, or eliminated.”¹⁷ The methodology suggested by the Phase One committee in its report, Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward (NRC, 2005a), used expert panels to review the DOE R&D program. The panels estimated the expected benefits¹⁸ in three categories—economic, environmental, and energy security—and in three global economic scenarios and summarized the results in a matrix. The panels constructed simple decision trees to describe the key technical and market uncertainties associated with the program and to evaluate the impact DOE support would have on the probability of various technical and market outcomes. To estimate benefits, simple spreadsheet models were deployed in conjunction with more sophisticated models (such as the Energy Information Administration’s (EIA’s) National Energy Modeling System (NEMS).¹⁹ The overall benefit of the DOE R&D program was given as the difference between the expected benefits with DOE support and the expected ben-

efits without DOE support. The process ensured consistency through the use of common scenarios and assumptions by each panel and the guidance on process and methodology provided by the parent committee, which also provided oversight.

The current NRC study continues the work of the first prospective benefits study, which evaluated the stream of benefits resulting from applied energy programs. It has the objective of testing, refining, and extending the proposed methodology. To conduct this study, the NRC formed the Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy Programs (Phase Two) (see biographies of committee members in Appendix B). The task assigned to the committee for Phase Two was as follows:

(The complete statement of task for Phase Two can be found in Appendix C.) Funds were appropriated in FY04 for Phase Two²⁰ and FY05 for Phase Three.²¹

To obtain feedback from stakeholders on its proposed methodology and its then-pending selection of DOE programs for further case study, the committee held a workshop on July 14, 2005, in Washington, D.C. Topics of discussion included the methodology developed in Phase One, the suggested revisions thereto, and the choice of case studies. Soon after the workshop, the committee selected six programs for case study and informed DOE of its choices. The programs selected comprised three in EE, including the Industrial Technologies Program (ITP)–Chemicals, the Distributed En-ergy Resources (DER) program, and the Light-Duty Vehicles Hybrid Technologies program, and three in FE, including the IGCC program, the Carbon Sequestration program, and the Natural Gas Exploration and Production program.

The NRC formed six panels of experts to conduct the case studies. Each panel held two meetings between September and December 2005. The panels carried out their evaluations following the methodology of Phase One with some modifications suggested by the Phase Two committee. The panels wrote short reports summarizing their findings and calculations and submitted them to the committee.

In a letter report issued December 14, 2005 (see Appendix D), the committee discussed the principal comments made during the workshop, the case studies it intended to perform in Phase Two, and the changes to the process and methodology that had occurred since Phase One. The committee maintained its dialogue with stakeholders throughout the study, including interfacing with the DOE R&D Council structure (see above) and ensuring frequent briefings by DOE to the committee and panels.

In addition, the committee refined its calculation of environmental and security benefits and added enhancements to the methodology based on its experience with the six panels. Activities undertaken by the committee during Phase Two are listed in Appendix E.

The balance of the current report is organized as follows: Chapter 2 presents the results of the six case studies. Chapter 3 discusses the methodology—the analytical framework for calculating quantitative program outcome indicators—in particular the way the framework has changed since the completion of the Phase One report (NRC, 2005a). Chapter 4 discusses the expert panel process—the sequence of tasks and events needed to apply the methodology to a specific program—and the changes to the process that have occurred since the Phase One study. Chapter 5 presents the conclusions and recommendations of the study. Appendix F contains material from Chapter 3 of the committee’s Phase One report to provide guidance to the panels on the methodology. Appendix G describes the kinds of information that DOE should provide and suggests forms it can fill out to submit that information. Appendixes H to M are the reports prepared by the six panels.