1
Introduction

BACKGROUND

Since its inception in 1977 from an amalgam of federal authorities, the U.S. Department of Energy (DOE) has administered numerous programs aimed at developing applied energy technologies. The better portion of the annual expenditures dedicated to such technologies is spent on research and development (R&D) and is administered by three DOE offices—the Office of Energy Efficiency and Renewable Energy (EERE), the Office of Fossil Energy (FE), and the Office of Nuclear Energy Science and Technology (NE).1 These three offices received approximately $2.6 billion total for fiscal year (FY) 2006, approximately $1.5 billion of which is being devoted to R&D. The R&D programs are complemented by policy measures, such as tax incentives to encourage early adoption of advanced technologies by consumers, efficiency standards for household appliances, and production tax credits for a variety of energy production technologies, including certain renewable energy sources (EIA, 2006).

In recent years, federal oversight of public expenditures has emphasized the integration of performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 “in response to questions about the value and effectiveness of federal programs” (GAO, 1997, p. 11). GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which has requested of the National Research Council (NRC) a series of reports using quantitative indicators to evaluate the effectiveness of applied energy R&D. The first such report2 took a retrospective view of the first 23 years of DOE R&D programs on fossil energy and energy efficiency.3 The report found that DOE-sponsored research had netted large commercial successes, such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization (NRC, 2001). However, some programs were judged to be costly failures in which large R&D expenditures did not result in a commercial energy technology (NRC, 2001). A follow-up NRC committee was assigned the task of adapting the methodology to the assessment of the future payoff of continuing programs (NRC, 2005a).

Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic effects. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government did not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement?

The present report continues to investigate the development and use of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. It provides further definition for the development of indicators for environmental and security benefits and refines the evaluation process based on its experience with the six DOE R&D case studies.

1

The Office of Electricity Delivery and Energy Reliability, created in 2002, could be considered a fourth applied energy program.

2

Requested by Congress in the conference report of the Consolidated Appropriations Act for FY00 (House Report 106-479, p. 493. November 18, 1999).

3

These programs include only those that were at the time under the jurisdiction of the U.S. House Appropriations Subcommittee on the Interior and Related Agencies.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 8
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) 1 Introduction BACKGROUND Since its inception in 1977 from an amalgam of federal authorities, the U.S. Department of Energy (DOE) has administered numerous programs aimed at developing applied energy technologies. The better portion of the annual expenditures dedicated to such technologies is spent on research and development (R&D) and is administered by three DOE offices—the Office of Energy Efficiency and Renewable Energy (EERE), the Office of Fossil Energy (FE), and the Office of Nuclear Energy Science and Technology (NE).1 These three offices received approximately $2.6 billion total for fiscal year (FY) 2006, approximately $1.5 billion of which is being devoted to R&D. The R&D programs are complemented by policy measures, such as tax incentives to encourage early adoption of advanced technologies by consumers, efficiency standards for household appliances, and production tax credits for a variety of energy production technologies, including certain renewable energy sources (EIA, 2006). In recent years, federal oversight of public expenditures has emphasized the integration of performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 “in response to questions about the value and effectiveness of federal programs” (GAO, 1997, p. 11). GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which has requested of the National Research Council (NRC) a series of reports using quantitative indicators to evaluate the effectiveness of applied energy R&D. The first such report2 took a retrospective view of the first 23 years of DOE R&D programs on fossil energy and energy efficiency.3 The report found that DOE-sponsored research had netted large commercial successes, such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization (NRC, 2001). However, some programs were judged to be costly failures in which large R&D expenditures did not result in a commercial energy technology (NRC, 2001). A follow-up NRC committee was assigned the task of adapting the methodology to the assessment of the future payoff of continuing programs (NRC, 2005a). Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic effects. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government did not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement? The present report continues to investigate the development and use of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. It provides further definition for the development of indicators for environmental and security benefits and refines the evaluation process based on its experience with the six DOE R&D case studies. 1 The Office of Electricity Delivery and Energy Reliability, created in 2002, could be considered a fourth applied energy program. 2 Requested by Congress in the conference report of the Consolidated Appropriations Act for FY00 (House Report 106-479, p. 493. November 18, 1999). 3 These programs include only those that were at the time under the jurisdiction of the U.S. House Appropriations Subcommittee on the Interior and Related Agencies.

OCR for page 8
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) APPLIED ENERGY RESEARCH AND DEVELOPMENT AT DOE The DOE divides its mission according to four strategic goals—one each dealing with defense, science, environment, and energy4 (DOE, 2005a)—and seven general goals. The energy strategic goal is supported by the general goal of energy security.5 The general goal underscores the role of technology development: Improve energy security by developing technologies that foster a diverse supply of reliable, affordable and environmentally sound energy by providing for reliable delivery of energy, guarding against energy emergencies, exploring advanced technologies that make fundamental improvement in our mix of energy options, and improving energy efficiency. (DOE, 2005a, p. 20) The three applied energy offices implement this goal using R&D programs, intergovernmental grants, reserves of fossil fuels, and appliance efficiency standards. Within EERE, R&D programs can be considered in two more or less distinct groups. The first consists of R&D directed at biomass and biorefinery systems R&D, geothermal technology, hydrogen technology, hydropower, solar energy, and wind energy technologies. Program activities range from basic research at universities and national laboratories to cost-shared applied research, development, and field validation in partnership with the private sector (OMB, 2005). The second group within EERE R&D consists of energy conservation R&D relating to the efficient end use of fuels and electricity in vehicles, in industrial processes and manufacturing, and in building envelopes. In general, this second group of energy conservation EERE R&D programs has the objective of achieving an output of a good or service with less energy input6—that is, achieving greater energy efficiency. The FY06 appropriation for EERE was $1,173 million,7,8 the R&D component of which totaled $770 million, including $151 million for congressionally mandated R&D activities.9 Research in FE has traditionally been divided between the Office of Coal and Power Systems (CPS) and the Office of Oil and Natural Gas. CPS administers a suite of clean coal R&D, which aims at ensuring the generation of clean, reliable, affordable electricity from coal. One large planned demonstration project is FutureGen, which has as its aim the development of a zero-atmospheric-emissions power plant. Additional programs focus on all the key technologies needed for FutureGen—carbon sequestration, membrane technologies for oxygen and hydrogen separation, advanced turbines, fuel cells, technologies related to gasifiers for coal-to-hydrogen conversion, and others. The Clean Coal Power Initiative emphasizes the development of pollution controls and efficiency improvements for existing plants (DOE, 2006a). The Office of Oil and Natural Gas supports research and policy options to ensure clean, reliable, and affordable supplies of oil and natural gas for American consumers.10 The FY06 appropriation for FE was $841 million, with $592 million of it allotted for fossil energy R&D.11 NE supports innovative applications of nuclear technology and administers four principal R&D programs. The Generation IV Nuclear Energy Systems Initiative and the Nuclear Hydrogen Initiative, which according to the information DOE provided on its budget for FY07 (DOE, 2006b) will develop a new generation of nuclear technologies to produce cost-effective electricity and commercial quantities of economic hydrogen to support the development of a transportation infrastructure. Nuclear Power 2010 is a demonstration program applicable to untested licensing and other regulatory processes. The Advanced Fuel Cycle Initiative aims to improve the efficiency and proliferation resistance of the nuclear fuel cycle. The scope of programs subject to the NRC studies in this series has been limited to fossil energy R&D and the energy conservation portion of EERE’s R&D. Prior to FY06, funding for this group of programs fell under the jurisdiction of the Interior and Related Agencies appropriation subcommittees owing to the programs’ origins in the Department of Interior; the balance of EERE’s R&D funding, on energy efficiency, was included on a separate appropriations bill. However, in FY06, the House Committee on Appropriations restructured its subcommittees’ jurisdictions and reduced them in number from 13 to 10. Funding for all FE and EERE R&D programs was consolidated into one account subject to the jurisdiction of the House Appropriations Subcommittee on Energy and Water Development and Related Agencies 4 The energy strategic goal is “[t]o protect our national security by promoting a diverse energy supply and delivery of reliable, affordable, and environmentally sound energy” (DOE, 2005, p. 7) 5 The final level in the goal cascade is the GPRA unit defining a major activity that aligns resources with a unique program goal. Some examples of GPRA units are hydrogen technology, solar energy, and wind energy. 6 Examples of this include, in the automotive industry, vehicles that obtain more miles per gallon (mpg); in the aluminum industry, production of more pounds of aluminum per Btu of energy; in lighting, more lumens per watt (NRC, 2001). 7 Budget of the United States Government, Fiscal Year 2007—Appendix, p. 390. 8 DOE. FY2007 Control Table by Organization, p. 5. Available at <http://www.cfo.doe.gov/budget/07budget/Content/orgcontrol.pdf>. 9 DOE. Department of Energy 2007 Budget Request: Fossil Energy and Other. Available at <http://www.cfo.doe.gov/budget/07budget/Content/Volumes/Vol_7_FE.pdf>. 10 Available at <http://www.fossil.energy.gov/programs/oilgas/index.html>. 11 The balance of budget authority within the Office of Fossil Energy went for the Strategic Petroleum Reserve and Northeast Home Heating Oil Reserve ($164 million), a rescission of uncommitted balances for clean coal technology (–$20 million), and for other purposes.

OCR for page 8
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) TABLE 1-1 Summary of Congressional Appropriations Oversight of EERE Programs       FY06   FY05 and Earlier EWD Subaccount Interior Bill: Energy Conservation Account EWD: Energy Supply Account Energy Supply and Conservation Account Electricity Delivery and Energy Reliability Account Vehicle technologies X   X   Fuel cell technologies X   (1)   Hydrogen technology   X X Distributed energy X   X Building technologies X   X   Industrial technologies X   X   Biomass and biorefinery systems R&D X   (2)   Solar energy   X X   Wind energy   X X   Hydropower   X X   Geothermal technology   X X   Biomass and biorefinery systems R&D   X X   NOTES: (1) Merged with the hydrogen technology subaccount; (2) merged with the identically named biomass and biorefinery systems R&D subaccount on the energy and water development bill. FY, fiscal year; EWD, Energy and Water Development Bill. (CRS, 2005).12 Four of the program funding line items that made up the pre-FY06 accounts were merged into two. In another instance—that of distributed energy—a program was moved out of EERE and into the appropriation account corresponding to the DOE Office of Electric Transmission and Distribution. (See Table 1-1 for a listing of programs and legislated funding sources in FY05 and earlier years and in FY06 and beyond.) EVALUATING THE FEDERAL INVESTMENT IN ENERGY RESEARCH AND DEVELOPMENT Federal Oversight GPRA requires agencies to annually submit or update a strategic plan, an annual performance plan, and an annual performance report (GAO, 2005). Program costs and benefits are analyzed and submitted, pursuant to GPRA, in an annual report to Congress with the President’s budget. GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. Additional analyses are required of agencies by the Office of Management and Budget (OMB), the executive agency that formulates and administers the federal budget and evaluates program effectiveness. The President’s Management Agenda (PMA) (OMB, 2001) sets forth nine agency-specific reforms and five government-wide goals (OMB, 2005); a set of R&D investment criteria was spelled out in 2003, implementing one of the agency-specific goals of the PMA (Marburger and Daniels, 2003). Also in 2003, OMB inaugurated the Program Assessment Rating Tool (PART) (OMB, 2003) to assess and improve program performance. The Government Accountability Office—a congressional agency—regularly audits, examines, and evaluates government programs (GAO, 2005). A few of the aforementioned assessments, and their relationship to one another, are discussed below. Reporting Requirements and Management Systems GPRA and PART require DOE to report on the performance of its R&D programs. The OMB guidance document for PART (OMB, 2005) states that “PART is a vehicle for achieving the goals of GPRA.” To encourage compatibility between the two systems, the PART guidance states that The PART … strengthens and reinforces performance measurement under GPRA by encouraging careful development of performance measures according to the outcome-oriented standards of the law and by requiring that agency goals be appropriately ambitious. Therefore, performance measures included in GPRA plans and reports and those developed or revised through the PART process must be consistent. The PART should help develop and identify meaningful performance measures to support GPRA reporting…. [T]he measures used for GPRA should be the same as those included in the PART. In all cases, performance measures included in GPRA plans and reports should meet the standards of the 12 The name of the combined account is Energy Supply and Conservation. Page 97 of House Report 109-275 explains it thus: “Energy Conservation programs previously funded by the Interior and Related Agencies Appropriations Act are now funded by the Energy Supply and Conservation appropriation, and are combined with energy efficiency activities in the Energy Efficiency and Renewable Energy account.” See also OMB (2005, p. 391).

OCR for page 8
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) TABLE 1-2 Stated Goals of GPRA and PART Program GPRA Objective PART Objective Sequestration/ integrated gasiification combined cycle (IGCC) Create public/private partnerships to develop technology to ensure continued electricity generation and hydrogen production from the extensive U.S. fossil fuel resource, including control technologies to permit reasonable cost compliance with emerging regulations and, ultimately, by 2015, near-zero atmospheric emission plants (including carbon) that are fuel-flexible and capable of multiproduct output and energy efficiencies over 60 percent with coal. Long-term goals are 50% efficient coal-based power generation (IGCC) in 2010; CO2 capture at 10% increase in cost of electricity by 2012; 90% reduction in mercury emissions at less than 75% current cost by 2012; $1,000/kW capital cost for IGCC technology in 2010. Natural gas Provide technology and policy options capable of ensuring abundant, reliable, and environmentally sound gas supplies. Long-term goals are (1) by 2015, develop technologies to expand the 2002 domestic gas economically recoverable resource base by 100 Tcf; (2) develop technologies that will by 2025 add 20 Tcf of technically recoverable resources of natural gas from methane hydrates; (3) by 2013, reduce the cost of hydrogen production from natural gas by 25% from current baseline of $5.54/MMBtu (steam reforming of methane at natural gas price of $3.15/MMBtu). Distributed energy Develop and facilitate market adoption of a diverse array of cost-competitive, integrated distributed generation and thermal energy technologies in homes, businesses, industry, communities, and electricity companies, increasing the efficiency of electricity generation, delivery, and use, improving electricity reliability, and reducing environmental impacts. The program has two long-term goals that capture most of the activities supported in each of the two subprograms. One subprogram focuses on the development of next-generation distributed energy technologies (e.g., microturbines, reciprocating engines, industrial gas turbines, thermally activated cooling and humidity control devices, combined heat and power systems) that are cleaner and more reliable, fuel efficient, fuel flexible, and affordable than existing equipment. The second subprogram concentrates on the development of technologies, tools, and techniques to enable prospective users of distributed energy systems—regardless of the type of technology—to evaluate benefits, install, operate, control, and maintain those systems in an optimized manner to meet the needs of their facilities and business operations and those of the electric power and natural gas utilities to which the systems are interconnected. Vehicle technologies Develop technologies that enable cars and trucks to become highly efficient, through improved power technologies and cleaner domestic fuels, and to be cost- and performance-competitive. The program has established three long-term performance measures. These measures are focused on program outputs (not processoriented) that could enable significant oil savings. However, success is dependant on the rate and level of market penetration, which are strongly affected by other policy instruments (such as fuel economy standards) and market conditions (such as fuel prices). The program also tracks progress against additional goals with its industry partners from the FreedomCAR and 21st Century Truck partnerships. Industrial technologies Partner with our most energy-intensive industries in strategic planning and energy-specific research, development, and demonstration to develop the technologies needed to use energy efficiently in their industrial processes and cost-effectively generate much of the energy they consume. The result of these activities will save feedstock and process energy, create domestic supply, improve the environmental performance of industry, and help America’s economic competitiveness. Not available. PART – they must be outcome oriented and have ambitious targets. (OMB, 2005, p. 3, italics added) In principle, therefore, PART and GPRA should use the same benefits estimates. However, it is difficult to tell whether this is being done because the DOE Performance and Accountability Report (DOE, 2005a) only presents a general statement of objectives and an assessment of whether a series of annual program targets was reached. Moreover, the PART Assessment Report13 does not consistently state specific long-term goals for each program. Table 1-2 contains the goal statements in the 2005 GPRA and PART reports in 13 Available at <http://www.whitehouse.gov/omb/budget/fy2006/pma/energy.pdf>.

OCR for page 8
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) the five technologies that the NRC evaluated in the current study.14 GPRA requires a statement of expected program results. EERE develops this statement through its benefits estimation process. That statement describes the relation between GPRA and benefits assessment as follows: EERE develops these benefits projections annually to help meet the requirements of the Government Performance and Results Act (GPRA) of 1993 and the President’s Management Agenda (PMA). GPRA requires Federal Government agencies to develop and report on output and outcome measures for each program. This analysis helps meet GPRA requirements by identifying the potential outcomes and benefits of realizing EERE program goals (outputs). The benefits estimates do not reflect the risk of realizing these goals, which is being addressed separately. (DOE, 2005b, italics added) This benefits assessment process is the same one that EERE discussed with the NRC Phase Two committee. Note that the GPRA benefits are not discounted for risk. PART does not report benefits estimates per se but rather asks a series of questions about the program as a basis for assessing its performance and management. The specific questions are listed in Appendix A. The relationship between PART and the present study depends on whether the NRC committee’s methodology helps to provide answers for the questions in PART. In particular, a credible benefits estimate is persuasive in answering PART questions on program purpose and design. Analysis Within the Department of Energy DOE has made important strides in developing tools for assessing the likely benefits of its R&D programs. Of special note is that elements of DOE—particularly the fossil energy and energy efficiency programs—have worked together toward common methodologies and approaches. In this regard, the efforts of DOE to improve and standardize its own estimates of benefits contributed to the committee’s ability to fully understand the programs and their anticipated impacts. At the undersecretary level, DOE maintains the R&D Council to handle the PMA, including R&D investment criteria, portfolio analysis, and budgeting. Reporting to the R&D Council are the Science and Technology (S&T) Integration Working Group, which facilitates compliance with administration and DOE requirements through the use of common methods and procedures and provides portfoliolevel decision support briefings, and the S&T Laboratory Working Group, which enhances the alignment of the R&D agenda with national energy goals.15 National Research Council Studies The conference report of the Consolidated Appropriations Act16 for FY00 requested the NRC to assess the benefits and costs of DOE’s R&D programs in fossil energy and energy efficiency, a task agreed on by the House and Senate subcommittees then having jurisdiction over funding of these programs. The retrospective study—Energy Research at DOE: Was It Worth It? (NRC, 2001)—reported that, in the aggregate, the benefits of federal applied energy R&D exceeded the costs but observed that the DOE portfolio included both striking successes and expensive failures. As important, the study noted that the methodologies by which DOE had calculated the benefits of its programs varied considerably, making comparisons of program benefits difficult. Congress subsequently provided funds for “a continuing annual review by the [National] Academy [of Sciences] of programs … to measure the relative benefits expected to be achieved and to inform decision making on what programs should be continued, expanded, scaled-back, or eliminated.”17 The methodology suggested by the Phase One committee in its report, Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward (NRC, 2005a), used expert panels to review the DOE R&D program. The panels estimated the expected benefits18 in three categories—economic, environmental, and energy security—and in three global economic scenarios and summarized the results in a matrix. The panels constructed simple decision trees to describe the key technical and market uncertainties associated with the program and to evaluate the impact DOE support would have on the probability of various technical and market outcomes. To estimate benefits, simple spreadsheet models were deployed in conjunction with more sophisticated models (such as the Energy Information Administration’s (EIA’s) National Energy Modeling System (NEMS).19 The overall benefit of the DOE R&D program was given as the difference between the expected benefits with DOE support and the expected ben- 14 Table 1-2 also shows that the programs reported in PART and GPRA are generally larger than the specific technologies that the Phase Two study is evaluating. However, the programs in PART and GPRA appear to be the same. 15 John Sullivan, Deputy Under Secretary for Energy, Science, and the Environment, “Energy, Science, and Environment R&D Council,” Presentation to the committee on October 26, 2005. 16 House Report 106-479, November 18, 1999. 17 House Report 107-564, July 11, 2002. 18 The expected benefit is a single quantity that incorporates information about the various possible outcomes and their respective probabilities and levels of benefits (NRC, 2005a). 19 NEMS is a computer-based, energy-economy system for modeling U.S. energy markets that projects the production, imports, conversion, consumption, and prices of energy, subject to assumptions about macroeconomic and financial factors, world energy markets, resource availability and costs, behavioral and technological choice criteria, cost and performance characteristics of energy technologies, and demographics.

OCR for page 8
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) efits without DOE support. The process ensured consistency through the use of common scenarios and assumptions by each panel and the guidance on process and methodology provided by the parent committee, which also provided oversight. THE CURRENT STUDY The current NRC study continues the work of the first prospective benefits study, which evaluated the stream of benefits resulting from applied energy programs. It has the objective of testing, refining, and extending the proposed methodology. To conduct this study, the NRC formed the Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy Programs (Phase Two) (see biographies of committee members in Appendix B). The task assigned to the committee for Phase Two was as follows: The work of the Phase 2 committee will be supported by several panels that will be separately appointed by the NRC to apply the methodology developed in Phase 1 and evaluate the prospective benefits of individual EE and FE programs/ projects. Since a methodology will have been developed in Phase 1, it is expected that a greater number of panels can be formed in Phase 2 and more time and resources can be devoted to evaluating prospective benefits. It is proposed that approximately 6 panels will be appointed by the NRC. (The complete statement of task for Phase Two can be found in Appendix C.) Funds were appropriated in FY04 for Phase Two20 and FY05 for Phase Three.21 To obtain feedback from stakeholders on its proposed methodology and its then-pending selection of DOE programs for further case study, the committee held a workshop on July 14, 2005, in Washington, D.C. Topics of discussion included the methodology developed in Phase One, the suggested revisions thereto, and the choice of case studies. Soon after the workshop, the committee selected six programs for case study and informed DOE of its choices. The programs selected comprised three in EE, including the Industrial Technologies Program (ITP)–Chemicals, the Distributed En-ergy Resources (DER) program, and the Light-Duty Vehicles Hybrid Technologies program, and three in FE, including the IGCC program, the Carbon Sequestration program, and the Natural Gas Exploration and Production program. The NRC formed six panels of experts to conduct the case studies. Each panel held two meetings between September and December 2005. The panels carried out their evaluations following the methodology of Phase One with some modifications suggested by the Phase Two committee. The panels wrote short reports summarizing their findings and calculations and submitted them to the committee. In a letter report issued December 14, 2005 (see Appendix D), the committee discussed the principal comments made during the workshop, the case studies it intended to perform in Phase Two, and the changes to the process and methodology that had occurred since Phase One. The committee maintained its dialogue with stakeholders throughout the study, including interfacing with the DOE R&D Council structure (see above) and ensuring frequent briefings by DOE to the committee and panels. In addition, the committee refined its calculation of environmental and security benefits and added enhancements to the methodology based on its experience with the six panels. Activities undertaken by the committee during Phase Two are listed in Appendix E. The balance of the current report is organized as follows: Chapter 2 presents the results of the six case studies. Chapter 3 discusses the methodology—the analytical framework for calculating quantitative program outcome indicators—in particular the way the framework has changed since the completion of the Phase One report (NRC, 2005a). Chapter 4 discusses the expert panel process—the sequence of tasks and events needed to apply the methodology to a specific program—and the changes to the process that have occurred since the Phase One study. Chapter 5 presents the conclusions and recommendations of the study. Appendix F contains material from Chapter 3 of the committee’s Phase One report to provide guidance to the panels on the methodology. Appendix G describes the kinds of information that DOE should provide and suggests forms it can fill out to submit that information. Appendixes H to M are the reports prepared by the six panels. 20 House Report 108-330, p. 83 and p. 163. 21 House Report 108-792, p. 1032, and House Report 108-542, p. 125.