. "2 From Retrospective to Prospective Evaluation." Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press, 2005.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward
proved to be of value to decision makers as well. The cookbook provided a consistent set of assumptions, concepts, and rules that all analysts should use. This consistency, in turn, allowed decision makers to confidently compare the benefits reported for different technologies.
UNCERTAINTIES OF PROSPECTIVE ANALYSIS
The committee’s proposed methodology for the prospective evaluation of benefits builds on the above features to provide an analytical framework adapted to the needs of prospective analysis. It is a more technical, but still consistent, methodology for creating the information required by the matrix. In contrast to retrospective evaluation, however, prospective evaluation is complicated by uncertainty about how the future will unfold. In the committee’s view, the chief uncertainties are these:
Uncertainty about the technological outcome of a program. Research is inherently an uncertain process, and any evaluation of a research program must consider the likelihood that the program’s goals will be met. Even if a program’s goals are not fully met on time and on budget, the program may still produce important technological advances that have considerable benefits that should be reflected in the evaluation.
Uncertainty about the market acceptance of a technology. It is possible for a research program to meet all of its own technical goals yet produce a technology that is not accepted in the marketplace and therefore has no economic benefit because another technology has met the same need sooner, better, or more cheaply. For example, DOE’s research program in solid state lighting may achieve its technical goals only to find that fluorescent lights have improved and dominate the lighting market. Similarly, flex-fueled hybrid-electric automotive technology and cellulosic ethanol production may both improve enough to reduce the anticipated benefits of hydrogen-fueled vehicles. Of course, the success of these competing technologies is itself uncertain.2
Uncertainty about future states of the world. The benefit of a new technology will often depend on developments quite unrelated to the technology itself. For example, the benefit of energy-efficient lights will depend on the cost of electricity, which in turn depends on the cost of fuels like natural gas and coal used to generate the electricity. Similarly, the benefits of carbon sequestration will be greatest if carbon emissions are regulated, giving electricity producers incentives to deploy this technology.
Uncertainty about the attribution of program benefits to DOE’s R&D investments. In some instances, a DOE R&D program investment might be the sole reason for a technological breakthrough envisioned by the program. In other
FIGURE 2-2 Results matrix for evaluating benefits and costs prospectively. The matrix shown in the figure differs from the preliminary matrix provided to the panels in June 2004.
instances, a private sector or foreign government investment might be the primary determining factor for technological success. It is difficult to know this with any certainty prospectively.
These four categories of uncertainties will apply to all programs, but the relative impact of a particular category will vary from program to program. A standard way to take uncertainties into account in cost-benefit analysis is to consider the “expected benefit,” which is the probability-weighted average of the benefits associated with all the possible outcomes of a program.
The benefits framework that the committee proposed for prospective evaluation incorporates these characteristics of the possible outcomes and attendant investment risk. This framework is summarized in matrix form in Figure 2-2. The bottom three rows represent the same three kinds of benefits—economic, environmental, or security benefits—considered in the retrospective analysis.3
The columns of the matrix represent possible future states of the world. The first of these is a reference case—here, the Energy Information Administration’s (EIA’s) Reference
The need to capture the technical and market risks associated with technology development is the principal reason for organizing expert panels.
These three classes of benefits have been chosen to reflect the programmatic goals of the DOE offices covered in Phase One and are not meant to reflect other benefits such as those gained from basic scientific research.