2
Results of Applying the Methodology

CASE STUDY SELECTION AND EXECUTION

The statement of task for the study called for case studies of the prospective benefits of approximately six DOE programs on energy efficiency and fossil energy R&D. Early in the study, during the open session of its July 14, 2005, meeting, the committee on prospective benefits held discussions with federal stakeholders on its pending selection of case studies. Based on that input, the committee finalized its choices and on July 22, 2005, sent letters to DOE—one each to the Office of Fossil Energy (FE) and the Office of Energy Efficiency and Renewable Energy (EERE)—informing them of the six selected programs (see Appendix D), which were as follows:

  • FE programs and activities:

    • Integrated Gasification Combined Cycle (IGCC);

    • Carbon Sequestration; and

    • Natural gas technologies (exploration and production).

  • EERE programs and activities:

    • Distributed Energy Resources (DER) program (enduse system integration and interface);

    • Vehicle technologies program (hybrid and electric propulsion, advanced combustion R&D and materials technology—excluding projects related to heavy-duty vehicles); and

    • Industrial Technologies Program–Chemicals.

In a few cases, the scope of the evaluation was limited to a subset of activities within a program. This is indicated in parentheses, where applicable.

The NRC formed six panels to conduct the case studies. The chairs of five of the six panels were members of the committee on prospective benefits. (A list of panel activities is included as Appendix E.) Each panel held two meetings between September and December 2005. A description of the evaluation process and methodology (see Appendix F) was distributed to panelists before the first panel meetings. After discussing this methodology early in the first meeting, the panel heard presentations delivered by DOE staff on their programs’ FY05 budgets, objectives, R&D portfolios, and estimated future benefits; the latter had been calculated by DOE assuming program technical goals would be met. The panels used this and other information available to them, along with their own knowledge of the subjects, to construct a decision tree showing the timing and likelihood of the technical success and market acceptance of the technology or technologies supported by the program. The panels next calculated the benefits—economic, environmental, and security—associated with each level of technical and market success in three global economic scenarios. The overall benefit of the DOE R&D program is given as the difference between the expected benefits with DOE support and the expected benefits without DOE support. To ensure consistency, the panels employed the process recommending common scenarios and assumptions across evaluations, and they were aided by the committee, which served in an oversight capacity. A decision analysis consultant assisted with the preparation of decision trees and benefits calculations. The panels wrote short reports summarizing their findings and calculations and submitted them to the committee.

STATEMENT OF TASK

The statements of task for the six panels were nearly identical. Here is the statement of task for the Panel on Light-Duty Vehicle Hybrid Technology:


The Panel on Light-Duty Vehicle Hybrid Technology will apply the methodology developed by the Committee on Prospective Benefits of Energy Efficiency and Fossil Energy R&D Programs to assess the potential benefits of DOE’s R&D activities that are focused on hybrid electric vehicle technologies using more efficient internal combustion engine



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) 2 Results of Applying the Methodology CASE STUDY SELECTION AND EXECUTION The statement of task for the study called for case studies of the prospective benefits of approximately six DOE programs on energy efficiency and fossil energy R&D. Early in the study, during the open session of its July 14, 2005, meeting, the committee on prospective benefits held discussions with federal stakeholders on its pending selection of case studies. Based on that input, the committee finalized its choices and on July 22, 2005, sent letters to DOE—one each to the Office of Fossil Energy (FE) and the Office of Energy Efficiency and Renewable Energy (EERE)—informing them of the six selected programs (see Appendix D), which were as follows: FE programs and activities: Integrated Gasification Combined Cycle (IGCC); Carbon Sequestration; and Natural gas technologies (exploration and production). EERE programs and activities: Distributed Energy Resources (DER) program (enduse system integration and interface); Vehicle technologies program (hybrid and electric propulsion, advanced combustion R&D and materials technology—excluding projects related to heavy-duty vehicles); and Industrial Technologies Program–Chemicals. In a few cases, the scope of the evaluation was limited to a subset of activities within a program. This is indicated in parentheses, where applicable. The NRC formed six panels to conduct the case studies. The chairs of five of the six panels were members of the committee on prospective benefits. (A list of panel activities is included as Appendix E.) Each panel held two meetings between September and December 2005. A description of the evaluation process and methodology (see Appendix F) was distributed to panelists before the first panel meetings. After discussing this methodology early in the first meeting, the panel heard presentations delivered by DOE staff on their programs’ FY05 budgets, objectives, R&D portfolios, and estimated future benefits; the latter had been calculated by DOE assuming program technical goals would be met. The panels used this and other information available to them, along with their own knowledge of the subjects, to construct a decision tree showing the timing and likelihood of the technical success and market acceptance of the technology or technologies supported by the program. The panels next calculated the benefits—economic, environmental, and security—associated with each level of technical and market success in three global economic scenarios. The overall benefit of the DOE R&D program is given as the difference between the expected benefits with DOE support and the expected benefits without DOE support. To ensure consistency, the panels employed the process recommending common scenarios and assumptions across evaluations, and they were aided by the committee, which served in an oversight capacity. A decision analysis consultant assisted with the preparation of decision trees and benefits calculations. The panels wrote short reports summarizing their findings and calculations and submitted them to the committee. STATEMENT OF TASK The statements of task for the six panels were nearly identical. Here is the statement of task for the Panel on Light-Duty Vehicle Hybrid Technology: The Panel on Light-Duty Vehicle Hybrid Technology will apply the methodology developed by the Committee on Prospective Benefits of Energy Efficiency and Fossil Energy R&D Programs to assess the potential benefits of DOE’s R&D activities that are focused on hybrid electric vehicle technologies using more efficient internal combustion engine

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) power trains for light-duty vehicles. The program has R&D activities in combustion, lightweight materials for vehicle components and structures, electrical and power electronic systems, emission control technologies, and fuels. The panel will be composed of about 8 members who will have experience with hybrid vehicle technologies, combustion engine technology, emission controls, the automotive market and sector, economics, and vehicle engineering and marketing. Panel members may have expertise in more than one area. The panel will apply the committee’s methodology by first assessing the probability of success of meeting the DOE’s time frame and targets for technology development, consider alternative paths of development with and without DOE funding, review DOE estimates of the economic, environmental, and national security benefits of its program on light-duty vehicle technology, and estimate the benefits under the alternative future states of the world (scenarios) that have been specified by the committee, namely, a base case EIA scenario, a high oil/gas price scenario, and a carbon-constrained scenario. Consideration of technical and market risk will be a key element of this assessment. It is expected that the panel will hold two 2-day meetings and will write a short 10-page report on its assessment of the benefits of this program and fill in the benefits matrix and explanation of it as outlined in the committee’s report, Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. ADVICE TO USERS The benefits methodology presented in this report is intended to provide consistent information that is useful to decision makers who are considering what programs should be continued, expanded, scaled back, or eliminated. The committee’s charge did not extend to reaching conclusions or making recommendations about such decisions for the six programs that it reviewed for this Phase Two project. However, the committee has developed some insights that may assist decision makers with interpreting and applying the results of the analysis. These insights are recorded in the next section, “Commonly Encountered Issues.” Although the committee has developed a process whereby all the panels evaluating DOE programs will use a common methodology and has suggested a procedure whereby all evaluations will be performed on a consistent basis, it is important that the decision makers understand the assumptions that the panelists had to use in assessing the programs. This understanding is critical to giving them the confidence to use the results as they make decisions about DOE programs and as they recommend funding. Commonly Encountered Issues The committee’s review of the six panel reports of Phase Two, along with the experience from the two earlier benefits evaluations, noted that several issues of interpretation came up frequently. Decision makers may thus encounter these same issues as they apply the results of the methodology presented here. A clear understanding of the reason for government-funded research is fundamental to the entire evaluation. For this reason, the program summaries produced in Phase Two require an explicit statement of the reason for government action. The government must demonstrate on the one hand the importance of a technology and the compelling need for it, and, on the other hand, that it will not be achieved in a timely fashion without the government’s participation. Readers are encouraged to consider this statement of justification carefully to verify that it accurately expresses the basic reason for the program. Several of the expert panels concluded that the probability of meeting the technical goals of a DOE program on the stated timetable was very small. While it is important for a decision maker to know this, it is also frequently true that a program can produce significant benefits at lower levels of technical achievement or over longer times. The committee recommends that users investigate these trade-offs carefully. The budgets of some programs may be inconsistent with the program goals. In some cases, this inconsistency is the reason that the program’s stated goals are not likely to be met, as noted above. But in other cases the issue is that budgets have been declining for several years; the combined heat and power (CHP) program and the ITP–Chemicals subprogram in Phase Two study exhibited this characteristic. In the latter case, the benefits of the subprogram may be over-ambitious because the analysis assumes that the goals will be accomplished when the reality is that declining budgets, relative to those planned, make that unlikely. In these cases, decision makers should recognize that realizing the estimated benefit may require increasing (or at least stabilizing) the budget. At some point, declining budgets probably reach a threshold below which the program is not viable.1 The benefits of some programs are strongly influenced by nonresearch policy decisions. For example, the benefits of carbon capture and sequestration depend on the size and timing of the carbon tax (or equivalent policy intervention in the market). In this case, a delay in the imposition of the tax, by delaying an incentive to the private sector, would accelerate technology development attributable to government funding, although not on a one-for-one basis. Imposition of appliance efficiency standards might have a similar effect. 1 At some point, however, a small program to maintain a federal presence might be justifiable. For example, the Panel on DOE’s Distributed Energy Resources Program believes that the federal program is helpful in encouraging states to fund similar programs.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) While not directly evaluated in this study, the social welfare impact of such policies could be considerable. Programs having very large benefits might alter the supply and demand characteristics of the entire energy system. For example, a successful deployment of IGCC technology appears to be accompanied by a large role for DER in order to meet peak electricity demand. This, of course, would affect the benefits calculation for DER. NEMS is particularly useful in understanding these changes in the overall energy system.2 Using the Benefits Analysis for Decision Making The expected net benefits calculated according to the methodology provide a useful basis for comparing programs. However, other elements of the analysis will contribute to the decision-making process. During its discussion of the results of the Phase Two work, the committee identified three such considerations that it believes deserve comment: The decision maker will often want to know how best to spend incremental resources (or how to minimize the impact on benefits of a reduction in resources). It is not the case, however, that incremental resources should be spent on programs with the highest benefit-cost ratio. What counts most in making this decision is finding the greatest incremental benefit produced by the additional resources. The methodology does not calculate marginal benefits and so is not directly useful in making these incremental decisions. However, the committee suggests that the decision tree analysis would be of considerable use in identifying actions most likely to increase the expected benefit. A recommended method for calculating marginal benefits appears in Chapter 5. The methodology presents benefits for each of three scenarios that describe future states of the world, but it does not attempt to combine the three sets of benefits into a single set. As a result, the decision maker must weigh the alternative scenarios in arriving at judgments about the benefits of the overall research portfolio. Ideally, a well-designed portfolio would contain a balance of projects that would produce acceptable results across the range of outcomes for different scenarios. The methodology estimates public benefits in three areas—economic, environmental, and energy security. While these three types of benefits reflect DOE’s strategic goals (DOE, 2005a), the committee recognizes that other kinds of benefits may be important in evaluating some projects. For example, markets will demand that automobile manufacturers produce cars that not only meet the fuel economy of importance to DOE but have several additional attributes as well. Another example might be employment impacts. Because including such benefits could affect the comparability among programs, the committee recommends that they be considered apart from the results matrix developed by the methodology or that, if included, they should be part of the text of a panel report. The panel should also discuss noteworthy assumptions it has made in calculating benefits, particularly if the results are sensitive to that assumption or the assumptions itself is novel or interesting. RESULTS OF CASE STUDIES The six expert panels that evaluated DOE applied energy programs estimated prospective benefits—economic, environmental, and security—using quantitative outcome indicators. In some cases, security and environmental benefits had to be characterized qualitatively. These estimates were facilitated by an assessment of the key technical and market risk faced by the technology being developed. Each set of results is reported in a two-page format accompanied by a decision tree. The latter illustrates how the expert panel assessed the possible future outcomes of the R&D program in terms of the timing, probability, and levels of technical performance and market acceptance of the technology. The goals and history of the program are summarized, and the expert panels have offered further observations about the program. The expert panels have contemplated outcomes that might occur with and without federal funding and taken the difference between the outcomes as the net expected value of the program. The findings of the six panels are presented in summary fashion in Figures 2-1 through 2-6: Figure 2-1, integrated gasification combined cycle technology, pages 17-19, Figure 2-2, carbon sequestration, pages 20-22, Figure 2-3, natural gas exploration and production, pages 23-28, Figure 2-4, distributed energy resources, pages 29-31, Figure 2-5, light-duty hybrid vehicles, pages 32-36, and Figure 2-6, Industrial Technologies Program––Chemicals, pages 37-39. The complete panel reports from which this information was derived are found in Appendixes H through M. 2 NEMS determines capacity additions in its electricity market module as follows: “Capacity expansion is determined by the least cost mix of all costs, including capital, O&M, and fuel.” And further: “Fossil-fired steam and nuclear plant retirements are calculated endogenously within the model. Plants are retired if the market price of electricity is not sufficient to support continued operation.” (EIA, 2003, pp.43-46)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) FIGURE 2-1 Findings for DOE’s gasification technologies R&D.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) FIGURE 2-2 Findings for DOE’s carbon sequestration program.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) FIGURE 2-3 Findings for DOE ’s natural gas exploration and production program.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) FIGURE 2-4 Findings for DOE’s distributed energy resources program.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) FIGURE 2-5 Findings for DOE’s light-duty vehicle hybrid technology R&D.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) FIGURE 2-6 Findings for DOE’s chemical industrial technologies program.

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)

OCR for page 14
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)