Conclusions and Recommendations
The committee’s task has been to propose a methodology for estimating the prospective benefits of DOE’s applied R&D programs that is of value to decision makers and practical for analysts to use. Even so, the committee recognizes that its proposed methodology relies on expert opinion and analytic approximations and, accordingly, leaves room for judgments about which reasonable people may disagree. The committee believes that eliminating such disagreements is undesirable if not impossible. What the proposed methodology aims to do is to provide a structure facilitating key judgments and a process that encourages consistent application of the structure. In addition, the methodology is designed to report the results of the analysis in a way that makes transparent the underlying assumptions and range of judgments. The committee expects that experience gained during Phase Two will help make the methodology more robust but recognizes that prospective analysis is inherently uncertain.
Although additional experience will refine the methodology and process outlined in the preceding chapters, the committee believes that it has met the objectives set for this phase of the project. In the following sections, the reasons for this conclusion are summarized and recommendations are made for proceeding to Phase Two.
VALUE OF PROPOSED METHODOLOGY FOR DECISION MAKING
In providing funds for the development and use of a methodology for evaluating the prospective benefits of DOE’s energy R&D programs, the Congress intended to “… inform decision making on what programs should be continued, expanded, scaled-back, or eliminated.”1 This report recommends such a methodology. However, it necessarily focuses on developing the methodology rather than on reporting information that could be used for decision-making purposes. Indeed, more than one of the panels that utilized the methodology to assess DOE programs for this report were explicitly concerned that their results not be interpreted as a balanced or even a complete collection of the information needed to make go/no-go decisions. The committee concurs with this concern while at the same time believing that the panel reports form a fully adequate basis for methodology development and for assisting decision makers.
The methodology is not yet fully tested and has some weaknesses that must be overcome. However, its use by the panels led to the identification of a number of inconsistencies and weaknesses in DOE’s benefits estimates. First, based on the work of the panels, the committee believes consistent application of the methodology will improve the quality and comparability of benefits estimates in ways that should enhance the confidence that decision makers can place in the analysis. Second, the analysis provides substantive insights that should be useful for allocating resources. Examples of these values are provided below.
Since the publication of the retrospective report Energy Research at DOE: Was It Worth It? (NRC, 2001), DOE has significantly advanced its approach for calculating benefits. However, DOE’s definitions of benefits did not always conform to those specified in the recommended methodology, which is essentially unchanged from the retrospective study. For example, the sequestration panel noted that because DOE’s definition of economic benefits2 is not consistent with the committee’s definition, which is “total net benefits to the U.S. economy,” it overstates those benefits. The uniform
definition of various kinds of benefits would ensure the comparability of benefits estimates across programs.
DOE calculates benefits assuming that the performance and cost goals of a technology are fully attained. The panels generally found that DOE had set stretch goals that by their nature would be hard to meet. Knowing the benefits of achieving these stretch goals is important, of course. However, since their achievement is uncertain and, in most cases, not very likely, the expected value of the benefit is a more suitable parameter for comparing program benefits. The difference can be substantial: The sequestration panel, for example, calculated an expected benefit of $35 billion, while it estimated that full attainment of DOE’s stated goals, on schedule, would produce a $120 billion benefit.
Comparability of benefits across programs requires using comparable and reasonable assumptions. The vehicle fuel cell subpanel found significant incompatibilities in the assumptions DOE used to calculate benefits. For example, DOE assumed that fuel cell vehicles would, by 2040, have 3.4 times the fuel economy of conventional gasoline vehicles of the same vintage, while gasoline hybrid vehicles would have only 2.0 times the fuel economy of conventional vehicles by 2040. In the view of the fuel cell panel, the assumed gasoline hybrid fuel economy was too conservative and resulted in DOE overestimating the benefits of the vehicle fuel cell program.3
Insights Useful for Decision Making
Benefits can vary significantly from one future scenario to the next. For example, the sequestration panel estimated large benefits for the Carbon Constrained scenario and zero benefits for the Reference Case and the High Oil and Gas Prices scenario. Because the relative importance of the different scenarios is a consequence of policy judgment, the differences in benefits of a particular technology under different scenarios can be crucial information for decision makers.
Both the lighting and the sequestration panels observed an inconsistency between stated program goals, which were predicated on certain funding levels, and the actual funding, which was inadequate. In both cases, however, the potential benefits associated with full funding were large. In allocating resources, it is essential that decision makers know the benefits expected to be produced by a given funding pattern.
As noted above, DOE often sets stretch goals for its programs. Some of the panels/subpanels found that the probability of meeting those goals at current funding levels is low. Nevertheless, the estimated benefits of high-risk, high-payoff programs might exceed their projected cost by a significant amount. For example, the vehicle fuel cell subpanel assigned a very low probability for the achievement of DOE’s stretch goals but concluded that the expected economic benefit was still well in excess of program costs. A conclusion such as this is consistent with the rationale for public funding of high-risk, high-payoff research. In addition, the methodology for prospective evaluation helps to pinpoint the key risks to which benefits are sensitive, serving as a basis for monitoring the progress of such programs in a useful way.
Even when the stretch goals are unlikely to be met, DOE’s programs might achieve lower levels of performance at which benefits would still exceed their costs. For example, the stationary fuel cell subpanel estimated that achieving a $600/kW cost goal (instead of $400/kW) would still have a positive benefit. Similarly, the lighting panel estimated that the expected benefits of achieving a 100 lpw solid state device would be similar to those of achieving the stretch goal of 150 lpw and that the benefits of achieving either would substantially exceed the cost of a program aimed at the stretch goal.
The committee stresses that the above examples are only that—illustrations of the kinds of insight that careful analysis of program benefits can provide.
BALANCE BETWEEN ANALYTIC RIGOR AND PRACTICAL UTILITY
The methodology and process for prospective benefits calculations recommended by the committee are intended to balance the need for analytic rigor and the need for practical utility. Rigor is required to assure that the benefits information offered to decision makers is credible and consistent across DOE’s programs. At the same time, the committee is aware that benefits estimation can become excessively complex, bringing resistance to the application of the methodology.
To characterize how the committee has balanced these conflicting demands, it could be useful to examine how the six main issues raised by the panels have been resolved. (These issues were presented in Chapter 2, “From Retrospective to Prospective Evaluation.”)
Calculating benefits. The recommended methodology calls for a simplified spreadsheet model for calculating benefits instead of relying on NEMS. This approach allows a panel to explore alternative outcomes quickly and efficiently, conforms to the committee’s definition of benefits, and helps make the analytic assumptions more transparent. An initial NEMS run calibrates the spreadsheet model to ensure consistency with the key energy prices and quantities used in benefits calculation. Preplanning of the panel’s activities is suggested to establish agreement on, and the transparency of, the assumptions that go into the NEMS runs.
Isolating critical outcomes. The committee has developed a conceptual decision tree that provides for systematic consideration of the alternative outcomes created by various levels of technological success, market success, and government funding. Although this is a standard decision analysis approach, the committee recognizes that the conceptual tree has to be simplified for practical application by the panels. Accordingly, the committee suggests a qualified expert to guide the decision tree analysis. This expert would be able to interpret the panel’s expertise, place the information in a decision framework, and aid in eliciting risk estimates (probabilities), perhaps by means of a questionnaire. Furthermore, while the full decision tree helps ensure that all relevant pathways are examined, usually only a very few pathways through the tree are of real interest in the benefits calculation.
Assessing probabilities. The Phase One panels all estimated the probabilities associated with technical and market risks, although each used a different approach. The committee has consolidated and built on this experience to recommend a process for aggregating the opinions of the experts into probabilities (or, when appropriate, ranges of probabilities) used in calculating expected benefits. In addition, the panel experience argued strongly against the specification of a single possible outcome and an associated probability of success for any DOE program. The recommended methodology thus employs an expected value approach, which helps to capture the range of outcomes identified as plausible and important by members of the expert panels.
Obtaining program information. The recommended methodology provides for a standard form summarizing the information a panel needs from DOE. A planning meeting of the panel and DOE staff is suggested to help ensure that the necessary information is available to the panel before it begins its deliberations.
Managing the panels. The committee has recommended a structure for panel meetings and for preparations leading up to the panel meetings. This structure is important because it would help the panels to bridge the gap between the substantive expertise of their members and the analytic mechanics of benefits estimation.
Presenting the results. The committee has recommended the use of a summary template to present a panel’s results in a form useful to decision makers. The template presents the panel’s conclusions about the DOE program as presented by DOE but provides opportunity to qualify and interpret the panel’s conclusions. This format is intended both to make the information more useful to decision makers and to avoid misuse of incomplete results.
Phase Two of the prospective benefits project will test whether the committee’s methodology in fact balances rigor and utility in a useful way. However, because the key elements of its recommendations build on the actual experience of one or more of the Phase One panels, the committee believes that its recommended methodology will be practical and will achieve the goal of providing useful decision information.
NEED FOR ADEQUATE RESOURCES AND MANAGEMENT PRIORITY
In developing its recommended methodology and process, the committee has been sensitive to the demands it places on DOE’s resources. Although the committee believes that a reasonable balance between rigor and practicality has been struck, applying the methodology will require the use of scarce DOE financial, program management, and analytic resources. A financial cost is associated with the conduct of the panel evaluations, including such items as travel expenses, staff and consultant costs, and the like. Additional resources may also be required to improve DOE’s analytic tools over time. In particular, while NEMS is used in a limited way in the committee’s methodology, it is still an essential part of the recommended methodology and remains a cumbersome model for benefits estimation. Over time, DOE may wish to improve the capability of NEMS, and doing so would probably require significant resources.
Accordingly, the committee recommends that DOE explicitly recognize in its resource allocation processes the need for resources to support use of the methodology. The committee believes that such resources are not likely to be large in relation to the size of programs being evaluated and that the value of prospective benefits analysis justifies making them available. The visible support of key DOE officials—notably, the under secretary—will be essential to ensure these resources are made available. The committee recognizes that the value of prospective benefits analysis for decision making must justify this support and will continue to work with DOE, OMB, and the Congress to ensure that the recommended methodology realizes its full value.
PRIORITIES IDENTIFIED FOR PHASE TWO OF THIS PROJECT
The goal of Phase One of the prospective benefits project has been to develop a methodology that can be applied at the program level. The committee believes that the experience amassed during Phase One has been sufficient to accomplish this goal. The committee expects that experience gained during Phase Two will make the methodology more robust. Further, in concentrating on the developmental issues, a number of important issues were intentionally deferred to Phase Two.
Now that the proposed methodology is in hand, the committee recommends that the following steps be taken in Phase Two to integrate benefits estimation into the program budget process and to maximize the value of such estimation to decision makers:
Review the proposed two-page summary panel report with its intended users to ensure that decision-making needs are met. Within DOE, these users should include the under secretary, the chief financial officer, relevant assistant secretaries, and program managers. Congress should be consulted, including congressional appropriations staff. OMB analysts should be consulted to meet the needs of budget examiners as well as to establish the specific linkages to the Program Assessment and Rating Tool (PART). The linkages of the panel’s summary reports and full reports to the needs for Government Performance and Results Act (GPRA) reporting should also be examined.
Prepare guidance for the program reviews to be conducted in Phase Two. The guidance would set forth in how-to style the committee’s recommended methodology contained in this Phase One report and would be used by each expert panel to ensure consistency of treatment among programs. To develop this guidance, the Phase Two committee should hold a workshop involving users and analysts to secure their comments on the recommended methodology. The committee will write a letter report that contains comments from the workshop and describes any modifications to the committee’s methodology.
Make an initial analysis of how the benefits methodology can be applied to portfolio design. The committee recommends that this analysis be conducted for major portions of the EE and FE programs. Of course, these programs will not all have been analyzed with the recommended methodology, but the committee believes that a panel that understands how the methodology would work can take a useful first step toward integrating benefits estimation and portfolio design.
Improve the process for estimating national security and environmental benefits. The committee intentionally did not address the issue of security benefits in Phase One. However, the definition of security has changed dramatically since the retrospective study. The committee believes that it will be possible to address this issue in Phase Two. The approach to estimating environmental benefits developed for the retrospective study is less likely to change, but the committee believes that a systematic review of this approach in the prospective context is warranted.
Refine and amplify the quality control process. In Chapter 4 of this report, the committee develops the general outlines of a process to assure the quality of the benefits estimates. Using the experience of applying the recommended methodology to several programs, Phase Two should identify the specific elements of the quality assurance process. For example, Phase Two might develop more specific guidance on minimum quality levels for individual program evaluations and the role and qualifications of the consultants who support the analytic process. In addition, OMB and the Congress should be consulted, since the quality assurance issue is of particular interest to them.
Recommend how to integrate the benefits estimation process into the program-budget cycle. The committee believes that benefits estimation provides important background information for the annual program-budget cycle but that it is not necessary to update the analysis for an individual program every year. Thus, the policy for updating analyses and for linking the available analyses to the annual program-budget cycle should be determined in Phase Two.
Select between two and six programs to which the methodology will be applied in Phase Two. This selection should be carried out in consultation with DOE, OMB, and the Congress and should consider the following:
The need to develop a basis for accomplishing the key objectives of Phase Two, as outlined above. Testing a methodology for estimating security benefits might be important here.
The importance of testing elements of the recommended methodology, such as the availability of proprietary information, an issue that was prominent in the panel report on stationary fuel cells.
The interests of OMB and Congress. For example, OMB may wish to include a program that will help identify the linkage to PART, and the Congress may have a need for information on which to base a particular decision.
The stress the methodology places on DOE resources. While the committee recommends that resources be made available for benefits estimation, it recognizes that DOE may have to redeploy resources to participate appropriately in Phase Two.