Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward (2005)

From the time the Department of Energy (DOE) was formed in 1977, successive administrations in Washington have looked to technological innovation as a critical tool for ensuring that the nation has affordable, clean, and reliable energy. Recognizing the importance of technological innovation, DOE, the Office of Management and Budget (OMB), and congressional committees have given increasing attention to understanding the effectiveness of federal funding for applied energy research and development (R&D).¹ The conference report of the Consolidated Appropriations Act² for fiscal year (FY) 2000 requested the National Academy of Sciences to assess the benefits and costs of DOE’s R&D programs in fossil energy and energy efficiency. Completed in 2001, the retrospective study Energy Research at DOE: Was It Worth It? (NRC, 2001) reported that, in the aggregate, the benefits of federal applied energy R&D had exceeded the costs but observed that the DOE portfolio had included both striking successes and expensive failures. Just as important, the study noted that the methodologies by which DOE calculated the benefits of its programs had varied considerably, making comparisons of program benefits difficult.

The Congress continued to express its interest in R&D benefits assessment by providing funds for the National Research Council (NRC) to build on the retrospective methodology to develop a methodology for assessing prospective benefits. Specifically, the Congress provided funds for “a continuing annual review by the [National] Academy [of Sciences] of programs … to measure the relative benefits expected to be achieved and to inform decision making on what programs should be continued, expanded, scaled-back, or eliminated.”³ In response, the NRC began a project for the development of a prospective benefits methodology in two overlapping phases and formed the Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs. The current study, which is Phase One, began in December 2003. Phase Two will begin in early 2005.

Three considerations were particularly important in formulating this project. One was that the committee should adapt the work of the retrospective study and, by extension, the work of DOE and OMB that has taken place since the retrospective study was completed. As the committee studied these developments in methodology, it became increasingly clear that the project would be most useful if it helped to move the entire enterprise toward a common methodology that assists decision makers at every level to consider “what programs should be continued, expanded, scaled-back, or eliminated.”⁴

The second consideration was that the committee must develop not only a methodology that is rigorous in its calculation of benefits and assessment of risks but also a practical and consistent process for applying that methodology across a variety of DOE programs. The third consideration was that the methodology being developed must be transparent and easy to use. Further, the methodology should not require extensive resources to implement nor should the results from its application be difficult for key stakeholders in the program to understand.

To carry out its assigned task, the committee organized the work into subtasks:

• Review the methodologies for assessing R&D benefits developed by DOE, OMB, and other agencies.

• Propose a conceptual framework that captures the key features of prospective benefits evaluation.

• Appoint expert panels to apply the framework to three DOE programs. The committee decided that three studies would be needed to test the methodology and selected the advanced lighting, fuel cells, and carbon sequestration programs for such study. Each panel was chaired by a committee member, but the panel members were mainly persons with expertise in the technology and markets pertaining to the programs. It was the judgment of the committee that this flexible approach would stimulate critical thinking about the methodological needs and practical problems of obtaining the relevant information.

• Receive a report from each panel and use it as a basis for the committee’s recommendations.

• Evaluate the experience reported by the panels and modify the methodology accordingly.

• Present conclusions in the Phase One report, which will also contain recommendations for specific steps to be taken in Phase Two of the ongoing effort.

It is important to emphasize that Phase One has been focused tightly on the basic problem of adapting the retrospective methodology to a prospective context. As a result, some important issues had to be deferred to Phase Two. For example, the issue of how to characterize security benefits has not been explored. In addition, the committee cautions against using the three expert panel case studies to assess the benefits of the three programs, because the panels’ findings on the programs were developed for the express purpose of helping the committee to refine its methodology.

The methodology of the retrospective study rested on two principal concepts. One was the benefits matrix. This matrix proved to be useful to decision makers for assessing R&D benefits because it focused attention on the public good benefits—economic, environmental, and security—that are the objectives of DOE’s applied energy R&D programs. These benefits were captured in the rows of the matrix. Additionally, the matrix identified a number of possible outcomes of the R&D programs, ranging from successful deployment of a technology in private markets to the generation of knowledge that was useful but did not result in a successful technology.

The other key concept in the retrospective study was the “cookbook” (Appendix D in that study), which contained detailed instructions for how to calculate the benefits in each cell of the matrix. These instructions were written for analysts, but they proved to be of value to decision makers as well. The cookbook provided a consistent set of assumptions, concepts, and rules that all analysts should use. This consistency, in turn, allowed decision makers to confidently compare the benefits reported for different technologies.

In contrast to retrospective evaluation, however, prospective evaluation is complicated by uncertainty about how the future will unfold. In the committee’s view, the chief uncertainties are of three kinds:

• Uncertainty about the technological outcome of a program. Research is inherently an uncertain process, and any evaluation of a research program must consider the likelihood that the program’s goals will be met. However, if a program’s goals are not fully met on time and on budget, the program may still produce important technological advances that have benefits that should be reflected in the evaluation.

• Uncertainty about the market acceptance of a technology. It is possible for a research program to meet all of its own technical goals yet produce a technology that is not accepted in the marketplace and therefore has no economic benefit, because another technology has met the same need sooner, better, or more cheaply.

• Uncertainty about future states of the world. The benefit of a new technology will often depend on developments quite unrelated to the technology itself. For example, the benefits of carbon sequestration technology will be greatest if carbon emissions are regulated, giving electricity producers incentives to deploy the technology.

These three categories of uncertainty will apply to all programs, but the relative impact of a particular category will vary from program to program.

The benefits framework that the committee proposed to the expert panels for prospective evaluation included a results matrix that summarizes possible outcomes and attendant investment risk for various categories of benefits.

Building on the experience gained by the expert panels, the committee recommends a methodology for prospective benefits evaluation having six main elements:

• Rigorous definition of benefits, to be used consistently for all programs. One important addition to the techniques used for the retrospective study is discounting. Another is the use of “expected benefits” as the measure of a program’s value. An expected benefit is the expected value of net benefits and should be calculated as the probability-weighted average of benefits for all important outcomes analyzed in each scenario.

• Common scenarios, used for all technologies, that describe future states of the world for which benefits are being estimated. The committee uses the three scenarios that are currently used by DOE’s Office of Fossil Energy in its own benefits analysis (NETL, 2004): (1) the Reference Case,

based on the Energy Information Administration’s (EIA’s) reference case, described in the Annual Energy Outlook, (2) a High Oil and Gas Prices scenario, and (3) a Carbon Constrained scenario wherein carbon emissions are limited by legislation. The committee believes that these three scenarios succinctly account for the key issues that have wide-ranging effects across all the DOE programs.

• A decision tree framework for ensuring that the role of government support and the important technology and market uncertainties are considered in the benefits calculation (see Figure ES-1). The value of DOE funding depends on the technological risks involved, advances in competing technology that could deliver the same or similar benefits as the DOE program, and the extent to which non-DOE resources are being applied to the technology—for example, by the private sector or foreign governments. It is essential

FIGURE ES-1 Decision tree.

FIGURE ES-2 Results matrix for evaluating benefits and costs prospectively.

• to evaluate these key alternative outcomes in addition to evaluating the benefits associated with complete success of the nominal DOE program. In principle, each potential outcome must be considered when evaluating government programs. In practice, however, evaluators need only consider those features that are important in a specific case.

• Procedures for estimating probabilities for the relevant uncertainties of the decision tree. The recommended expected value approach would help capture the range of probabilities perceived by the panel experts. Guidelines for risk assessment are essential to achieving a reasonable degree of uniformity across the program evaluations.

• A results matrix that uniformly summarizes important data and estimated benefits for all technology programs. Such a matrix⁵ is shown in Figure ES-2.

• Simplified models for calculating the benefits associated with the critical pathways in the decision tree. This cal-

culation requires estimates of prices and quantities of the various energy commodities in a scenario-specific market equilibrium for the relevant time horizon and at a level of detail consistent with the new technology being examined. Typically such data could be derived from a large-scale energy model. The committee believes that the EIA’s National Energy Modeling System (NEMS) is, in fact, suitable for calculating equilibrium prices and quantities.⁶ For calculating benefits, however, NEMS raises three problems. One is the lack of transparency—the difficulty of identifying the critical assumptions on which the NEMS calculation is based. Second, it appears that NEMS does not calculate benefits using the rules established in Appendix D of the retrospective report (NRC, 2001) and recommended by the committee for prospective evaluations as well. Finally, running NEMS is cumbersome and very resource-intensive, which greatly limits its utility for exploring alternative scenarios and program outcomes.

As important as the evaluation methodology is the process by which the methodology is applied. The process proposed by the committee reflects the experience of the expert panels during Phase One as they attempted to determine prospective benefits.

The proposed process for program assessment centers on establishing an expert panel to review a particular DOE program. The process calls for a standard form summarizing the information the panel needs from DOE. This expert panel will begin its deliberations by conducting a technical assessment based on a brief description of the program and its component projects provided by DOE. It will assess the conditional benefits of the program, assuming the program meets the stated goals. The panel members’ expertise and the decision tree assessment tool will be used to develop the probabilities for technical and market risk for the program as a whole. The results of the probability analysis will be used to estimate the expected value of program benefits. The panel will be supported by a consultant in decision analysis who has a working knowledge of the benefits assessment methodology being proposed by the committee. The panel will report its results and comment on the program risks in a defined format.

In addition, the committee recommends establishing an overall quality assurance function. The committee believes that such a function is essential for ensuring consistency in the application of the benefits methodology and that it will be a mechanism for the methodology’s continuing refinement.

The committee recognizes that because the proposed methodology relies on expert opinion and analytic approximations, it leaves room for judgments about which reasonable people may disagree. The committee believes that eliminating such disagreements would be undesirable if not impossible. With the proposed methodology, the committee is to provide a structure facilitating key judgments and a process that encourages consistent application of the structure. In addition, the methodology is designed to report the results of the analysis in a way that makes transparent the underlying assumptions and range of judgments.

The methodology is not yet fully tested and has some weaknesses that must be overcome. However, its use by the panels led to the identification of a number of inconsistencies and weaknesses in DOE’s benefits estimates. Based on the work of the panels, the committee believes the consistent application of the methodology will improve the quality and comparability of benefits estimates in ways that should enhance the confidence that decision makers can place in the analysis. Specific areas of DOE’s analyses that could be improved include the following:

• Since the publication of the retrospective report, DOE has significantly advanced its approach for calculating benefits. However, DOE’s definitions of benefits do not always conform to those specified in the recommended methodology, which are essentially unchanged from the retrospective study. The uniform definition of various kinds of benefits would ensure the comparability of benefits estimates across programs.

• DOE calculates benefits assuming that the performance and cost goals of a technology are fully attained. The panels generally found that DOE had set stretch goals that by their nature would be hard to meet. Knowing the benefits of achieving these stretch goals is important, of course. However, since their achievement is uncertain and, in most cases, not very likely, the expected value of the benefit is a more suitable parameter for comparing program benefits.

• Comparability of benefits across programs requires using comparable and reasonable assumptions. The lighting panel found significant incompatibilities in the assumptions used by DOE in calculating benefits.

Further, the analysis provides substantive insights that should be useful for allocating resources. Examples of the methodology’s utility include the following:

• Benefits can vary significantly from one future scenario to another. For example, the sequestration panel estimated large benefits for the Carbon Constrained scenario

and zero benefits for the Reference Case and the High Oil and Gas Prices scenario. Because the relative importance of the different scenarios is a consequence of policy judgments, the difference in benefits of a particular technology under different scenarios can be crucial information for decision makers.

• Both the lighting and the sequestration panels observed an inconsistency between stated program goals, which were predicated on certain funding levels, and the actual funding, which was inadequate. In both cases, however, the potential benefits associated with full funding were large. In allocating resources, it is essential that decision makers know the benefits expected to be produced by a given funding pattern.

• DOE often sets stretch goals for its programs. Some panels found that the probability of meeting those goals at current funding levels is low. Nevertheless, the estimated benefits of high-risk, high-payoff programs might exceed their projected cost by a significant amount. For example, the vehicle fuel cell panel believed there was little probability that DOE’s stretch goals would be achieved but concluded that the expected economic benefit would still be well in excess of program costs. This result is consistent with the rationale for public funding of high-risk, high-payoff research. In addition, the methodology helps to pinpoint the key risks to which benefits are sensitive, serving as a basis for monitoring the progress of such programs in a useful way.

• Even when the stretch goals are unlikely to be met, DOE’s programs might achieve lower levels of performance at which benefits still exceed their costs. For example, the stationary fuel cell subpanel estimated that achieving a $600 per kilowatt cost goal (instead of $400 per kilowatt) would still have positive benefit. Similarly, the lighting panel estimated that the expected benefits of achieving a 100-lumens-per-watt (lpw) solid state device would be similar to those of achieving the stretch goal of 150 lpw and that the benefit of achieving either would substantially exceed the cost of a program aimed at the stretch goal.

In developing its recommended methodology and process, the committee has been sensitive to the demands it places on DOE’s resources. Although the committee believes that a reasonable balance between rigor and practicality has been struck, applying the methodology will require the use of scarce DOE financial, program management, and analytic resources.

Accordingly, the committee recommends that DOE explicitly recognize in its resource allocation processes the need for resources to support use of the methodology. The committee believes that such resources are not likely to be large in relation to the size of programs being evaluated and that the value of prospective benefits analysis justifies making them available.

The goal of Phase One of the prospective benefits project has been to develop a methodology that can be applied at the program level. The committee believes that the experience amassed during Phase One is sufficient to accomplish this goal. The committee expects that experience gained during Phase Two will make the methodology more robust. Further, in concentrating on developmental issues, a number of other important issues were intentionally deferred to Phase Two.

Now that the proposed methodology is in hand, the committee recommends that the following steps be taken in Phase Two to integrate benefits estimation into the program budget process and to maximize the value of such estimation to decision makers:

1. Review the proposed template for a two-page summary panel report with its intended users to ensure that their decision-making needs are met.

2. Prepare guidance for the program reviews to be conducted in Phase Two.

3. Make an initial analysis of the application of the benefits methodology to portfolio design.

4. Improve the process for estimating national security and environmental benefits.

5. Refine and amplify the quality control process.

6. Recommend how to integrate the benefits estimation process into the program-budget cycle.

7. Select anywhere from two to six programs to which the methodology will be applied in Phase Two.