Program Evaluation

The committee found that managers of both the energy efficiency and the fossil energy RD&D programs did not utilize a consistent methodology or framework for estimating and evaluating the benefits of the numerous projects within their programs. In addition to a tendency to assign too much weight to realized economic benefits, especially avoided costs and unshared costs, the inconsistent approach adopted by DOE policy makers to evaluation of their programs often was associated with an overstatement of economic benefits.

The benefits matrix adopted for this study is a robust framework for evaluating program outcomes. Its application imposes a rigor on the evaluation process that clarifies the benefits achieved and the relationship among them.

Recommendation. DOE should adopt an analytic framework similar to that used by this committee as a uniform methodology for assessing the costs and benefits of its R&D programs. DOE should also use an analytic framework of this sort in reporting to Congress on its programs and goals under the terms of the Government Performance and Results Act.

Recommendation. To implement this recommended analytic approach, DOE should consider taking the following steps:

  1. Adopt and improve guidelines for benefits characterization and valuation. Convene a workshop of DOE analysts, decision makers, and committee members to discuss the problems encountered in the application of the committee’s guidelines and to consider how to begin the improvement process.

  2. Adopt consistent assumptions to be used across programs.

  3. Adopt procedures to enhance the transparency of the process.

  4. Provide for external peer review of the application of the analytic framework to help ensure that it is applied consistently for all programs.

  5. Seek to include the views of all stakeholders in public reviews of its R&D programs.

DOE programs may be effective in very diverse ways, and better data on the nature of program results will aid policy makers in assessing the appropriateness of program structures. It is essential to report specifically the concrete results achieved by DOE’s participation in such programs relative to the efforts of other investors. Application of this framework requires data that often are difficult to obtain within DOE. Public costs may be quite modest compared with the benefits if they catalyze private investments in innovation.

Recommendation. DOE should consistently record historical budget and cost-sharing data for all RD&D projects. Industry incurs significant costs to commercialize technology developed in DOE programs, and—especially in the assessment of economic benefits—these costs should be documented where possible.

Portfolio Management

The committee’s review of the fossil energy and energy efficiency programs underscores the significant changes in energy policy during the nearly three decades of the programs’ existence. There have been changes in technological possibilities; expectations about energy supply, prices, and security; DOE programmatic goals; the national and international political environment; and the feasibility and accomplishments of various technological approaches and R&D performers. A balanced R&D portfolio is particularly important since individual R&D projects may well fail to achieve their goals. Rather than viewing the failure of individual R&D projects as symptoms of overall program failure, DOE and congressional policy makers should recognize that project failures generate considerable knowledge and that a well-designed R&D program will inevitably include such failures. An R&D program with no failures in individual research projects is pursuing an overly conservative portfolio.

Recommendation. DOE’s R&D portfolio in energy efficiency and fossil energy should focus first on DOE (national) public good goals, and it should have (1) a mix of exploratory, applied, development, and demonstration research and related activities, (2) different time horizons for the deployment of any resulting technologies, (3) an array of different technologies for any programmatic goals, and (4) a mix of economic, environmental, and security objectives. In addition, it is important to effectively integrate the results of exploratory research projects with applied RD&D activities within individual programs.

Recommendation. DOE should develop clear performance targets and milestones, including the establishment of intermediate performance targets and milestones, at the inception of demonstration and development programs (in cooperation with industry collaborators, where appropriate) and employ these targets and milestones as go/no-go criteria within individual projects and programs.

The committee’s review of DOE RD&D programs suggests that programs seeking to support the development of technologies for rapid deployment are more likely to be successful when the technological goals of these programs are consistent with the economic incentives of users to adopt such technologies. For the programs in which these goals are central, the case studies illustrate a number of instances in

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement