Conclusions and Recommendations
The Phase Two study continued the work of the Phase One study by refining the methodology and applying it on a consistent basis to the energy efficiency (EE) and fossil energy (FE) programs. The methodology is meant to provide consistent information that will enable decision makers to better allocate the funds available for R&D and to identify programs where funding should be continued, expanded, scaled back, or eliminated. However, the methodology is only one piece of what is needed for an allocation decision, so it would not be appropriate to force-fit it to yield a particular decision, e.g., to increase or decrease marginal funds. The key to the methodology is the use of a panel of experts with a balance of skills and a wide range of expertise and experience to ensure that all relevant issues are identified, fully discussed, and factored into the assessment. The panel was charged not with performing a traditional program review or evaluation but with understanding programmatic issues to independently establish probabilities and expected benefits using the committee’s methodology.
Applying the methodology to the six case studies,1 the committee and expert panels developed recommendations on obtaining the results, using the results, methodological issues, and the continuity of the evaluation activity.2
PRIORITIES IDENTIFIED FOR PHASE THREE OF THIS PROJECT
The scope of programs subject to the NRC studies in this series has been limited to fossil energy R&D and the energy conservation portion of the R&D efforts by the Office of Energy Efficiency and Renewable Energy (EERE) R&D. Prior to FY06, funding for this group of programs fell under the jurisdiction of the Interior and Related Agencies appropriation subcommittees owing to the programs’ origins in the Department of Interior; the balance of EERE’s R&D funding, on energy efficiency, was included on a separate appropriations bill. However, in FY06, the House Committee on Appropriations restructured its subcommittees’ jurisdictions and reduced them in number from 13 to 10. Funding for all FE and EERE R&D programs was consolidated into one account subject to the jurisdiction of the House Appropriations Subcommittee on Energy and Water Development and Related Agencies (CRS, 2005).3 Four of the program funding line items that made up the pre-FY06 accounts were merged into two. In another instance—that of distributed energy—a program was moved out of EERE and into the appropriation account corresponding to the DOE Office of Electric Transmission and Distribution.
Finding: Phase Two showed that the basic structure of the methodology—scenarios, decision trees, technical and market risk assessments, and economic, environment, and security benefits—could be implemented by six panels of experts on a consistent basis. The panels were able to obtain the quantitative and qualitative information they needed
to assess individual programs in fossil energy and energy efficiency.
Recommendation: The committee should undertake the following activities in Phase Three to further demonstrate the robustness of the methodology and maximize its value for decision makers:
Expand the case studies to include at least one program in renewable energy and one in nuclear energy.
Determine how the benefits methodology can be applied for portfolio analysis and evaluate a portfolio. Portfolios were not evaluated in Phase Two, but the review of the light-duty vehicle hybrid technologies encompasses three separate program elements and provides an opportunity to aggregate several activities. IGCC and sequestration represent two major components of the FutureGen program.
Continue to evaluate and refine the quality control process.
Continue to communicate and have informal conversations with stakeholders throughout the process. These would include discussions with the committee and panel chairs and with some members of the committee about the case studies and process enhancements or modifications.
Make recommendations and provide for transition to full-scale implementation by either DOE or the NRC.
THE PROCESS FOR OBTAINING RESULTS
The components necessary for completing the assessment include the methodology, the panel of experts, input from DOE, and a quality control process.
Panel of Experts
Finding: The commitment and the technology background of the panel members determine the quality of the assessment of the program.
Recommendation 4: Panel composition and level of expertise must be critically considered during the selection process. If a panel concludes that certain skills are not possessed by its members, it should consider expanding its membership or using an outside expert to brief it.
Finding: The leadership role of the panel chair cannot be overemphasized. For the panel to succeed, the chair has to take a lead role in interacting with DOE to ensure that the best possible information is available to the panel before it meets for the first time. Panels where the chair devoted significant time to ensuring that all panel members were fully familiar with the process and methodology produced the best assessments.
Recommendation 5: The panel chair should spend a fair amount of time outside the actual panel meetings working with DOE program managers, DOE management, and the independent consultant(s).
Finding: The main responsibility of a consultant is to maintain consistency across the panels in applying the methodology and to facilitate the analysis. This includes structuring the decision trees, facilitating the assignment of probabilities to technical and market outcomes, and assisting in the modeling of benefits. Phase Two made use of a consultant for all the panels, which worked very well.
Recommendation 6: Depending on the number of programs being evaluated and the panels’ schedules, it might be necessary to have more than one consultant. One consultant should focus on the decision tree development and probability assessment and the other on the modeling of benefits.
Input from DOE
Finding: The level and timeliness of information provided by DOE to the various panels play a critical role in facilitating the deliberations and conclusions on the panel. Completion of panel evaluations is contingent on the panel’s receiving synoptic information and inputs for benefits calculations. The timeliness and quality of this information impacts the quality and utility of the panel evaluation.
Recommendation 7: Since the usefulness of the benefits estimates depends on the quality and timeliness of information available to the panels, DOE management should give its full support for providing the necessary information. DOE at all levels should buy into this process because it is useful for managing and assessing its programs. If this commitment is not clear, the committee should explore all avenues for gaining DOE support.
Finding: Quality control continues to be important in ensuring the consistency, and therefore the utility, of panel evaluations.
Recommendation 8: An oversight committee should apply the quality control process to several elements of the study process, including ensuring appropriate panel membership and composition, orienting the panel chair and consultant, monitoring the panel’s progress, monitoring information
received from DOE for adequacy and consistency, and reviewing and revising the process itself.
USING THE RESULTS
Impact of Policy Measures Unrelated to Research on Realization of Program Benefits
Finding: Policy measures unrelated to research have an effect on when and whether the benefits of some programs will be realized. For example, the benefits of carbon capture and storage depend on the size and timing of a carbon tax (or an equivalent policy intervention in the market). The scenarios, which should include some of these factors, are a valuable tool for characterizing and quantifying the benefits of the DOE R&D program.
Recommendation 9: Decision makers should consider the impact of other policy measures—that is, policies not related to research—in all domains of action (federal, state, and international, say) when considering the results of prospective benefits evaluations. Having a common set of scenarios is useful in general, although additional scenarios may be called for in some cases. While defining the scenarios more completely would be helpful for interpreting the outcomes of the analysis, at the same time it is essential to preserve flexibility by keeping the scenarios as broad as possible.
Guidance for Budget Formulation
Finding: Phase Two showed that the basic structure, using decision trees, worked well and could be implemented with the panels. The panel evaluations permit calculation of a benefit-to-cost ratio, which is not, however, the correct metric to use when allocating resources among the programs in a portfolio.
Recommendation 10: To allocate resources, DOE should know the marginal benefit of a budget increase on a programby-program basis. To calculate the marginal benefit, the decision tree should be examined to identify the outcomes that would be most sensitive to changes in budget levels. In the Phase One study, for example, the lighting program proved to be highly budget-dependent. When such sensitivities exist, the decision tree can be re-estimated for a different budget level, using the committee’s methodology. The marginal benefit associated with the change in budget level is the difference between the net benefits of the two calculations.
Consideration of Alternative Futures
Finding: The methodology presents benefits for each of three scenarios that describe future states of the world, but does not attempt to combine the three sets of benefits into a single set.
Recommendation: DOE should weigh the alternative scenarios in arriving at judgments about the benefits of the overall research portfolio. The portfolio should contain a balance of projects that will produce acceptable results across the range of scenarios.
Benefits Not Captured by the Methodology
Finding: The methodology estimates public benefits in three areas—economic, environmental, and energy security. While these three types of benefits reflect DOE’s strategic goals (DOE, 2005a), the committee recognizes that other kinds of benefits may be important in evaluating some projects. For example, market forces demand that automobile manufacturers produce cars that not only meet the fuel economy standard criterion of importance to DOE but that also have several other attributes. A technology that achieves DOE’s objectives must also provide these additional attributes. As another example, DOE’s research might have employment impacts.
Recommendation 11: If benefits in areas other than economic, environmental, or energy security are found to occur, they should be noted in the text accompanying the results matrix. However, the matrix should stay focused on the three main types of benefits to facilitate comparisons across programs.
ESTIMATING NATIONAL SECURITY AND ENVIRONMENTAL BENEFITS
Finding: Valuation of benefits in monetary units related to reducing air pollution is both the primary environmental benefit identified in the present study and the class of benefits for which valuation methods are most advanced.
Recommendation 1: Panels should apply valuations in monetary units to criteria air pollutant emissions in the results matrix, but not to other types of pollutant emissions. The valuations used should be the allowance price forecasts for the future period.
Energy Security Benefits: Electricity
Finding: While the complex relationship between electricity supply and security is becoming clearer, analysts are a long way off from having methods for valuing reductions in security threats contributed by technologies such as distributed generation.
Recommendation 2: Panels conducting prospective benefits assessments should describe reductions in threats to
energy security related to electricity supply as physical quantities of oil and gas.
Energy Security Benefits: Oil and Gas
Finding: Increases in U.S. oil and gas consumption and imports may impose incremental costs that are not fully reflected in the market price. The cost components of this oil premium have been estimated in various studies. In principle, similar estimates could be made for natural gas, but the committee is unaware of such research having been done.
Recommendation 3: Panels should describe energy security benefits related to reduced oil and natural gas consumption quantitatively in the benefits matrix as physical quantities of oil and gas. The time pattern of the oil consumption impacts should be made explicit, along with an assessment of the probable state of the oil market during those future times.
CONTINUITY: INSTITUTIONALIZING THE EVALUATION PROCESS
Finding: Prospective benefits evaluations would be most useful if DOE would adopt them. This would allow integration with GPRA, the President’s Management Agenda, and other tools and systems related to performance and budgeting.
Recommendation: DOE should create a triennial program evaluation cycle using the methodology of this Phase Two study for all the applied energy programs. If DOE chooses to undertake this internally, it would need to create a set of FACA committees managed by the DOE and reporting to the under secretary or higher. If DOE chooses to use the NRC, the NRC would independently appoint the oversight committee and panels to undertake the prospective benefits evaluations. A third possibility would be for a contractor working in the model of DOD’s JASON4 to perform the assessment.