National Academies Press: OpenBook
« Previous: 4 Expert Panel Process
Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.

Conclusions and Recommendations


The Phase Two study continued the work of the Phase One study by refining the methodology and applying it on a consistent basis to the energy efficiency (EE) and fossil energy (FE) programs. The methodology is meant to provide consistent information that will enable decision makers to better allocate the funds available for R&D and to identify programs where funding should be continued, expanded, scaled back, or eliminated. However, the methodology is only one piece of what is needed for an allocation decision, so it would not be appropriate to force-fit it to yield a particular decision, e.g., to increase or decrease marginal funds. The key to the methodology is the use of a panel of experts with a balance of skills and a wide range of expertise and experience to ensure that all relevant issues are identified, fully discussed, and factored into the assessment. The panel was charged not with performing a traditional program review or evaluation but with understanding programmatic issues to independently establish probabilities and expected benefits using the committee’s methodology.

Applying the methodology to the six case studies,1 the committee and expert panels developed recommendations on obtaining the results, using the results, methodological issues, and the continuity of the evaluation activity.2


The scope of programs subject to the NRC studies in this series has been limited to fossil energy R&D and the energy conservation portion of the R&D efforts by the Office of Energy Efficiency and Renewable Energy (EERE) R&D. Prior to FY06, funding for this group of programs fell under the jurisdiction of the Interior and Related Agencies appropriation subcommittees owing to the programs’ origins in the Department of Interior; the balance of EERE’s R&D funding, on energy efficiency, was included on a separate appropriations bill. However, in FY06, the House Committee on Appropriations restructured its subcommittees’ jurisdictions and reduced them in number from 13 to 10. Funding for all FE and EERE R&D programs was consolidated into one account subject to the jurisdiction of the House Appropriations Subcommittee on Energy and Water Development and Related Agencies (CRS, 2005).3 Four of the program funding line items that made up the pre-FY06 accounts were merged into two. In another instance—that of distributed energy—a program was moved out of EERE and into the appropriation account corresponding to the DOE Office of Electric Transmission and Distribution.

Finding: Phase Two showed that the basic structure of the methodology—scenarios, decision trees, technical and market risk assessments, and economic, environment, and security benefits—could be implemented by six panels of experts on a consistent basis. The panels were able to obtain the quantitative and qualitative information they needed


The activities selected for review included three within EE—the Chemicals subprogram of the Industrial Technologies Program, the Distributed Energy Resources (DER) program, and Light-Duty Vehicle Hybrid Technology activities within the Vehicle Technologies Program—and three within FE—the Integrated Gasification Combined Cycle (IGCC) subprogram, the Carbon Sequestration program, and the Natural Gas Exploration and Production R&D program.


This chapter contains 14 recommendations, 11 of which have been given numbers and carried forward to the summary (the numbering reflects the order there). Three recommendations are not numbered.


The name of the combined account is Energy Supply and Conservation. Page 97 of House Report 109-275 explains it thus: “Energy Conservation programs previously funded by the Interior and Related Agencies Appropriations Act are now funded by the Energy Supply and Conservation appropriation, and are combined with energy efficiency activities in the Energy Efficiency and Renewable Energy account.” See also OMB, 2005, p. 391.

Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.

to assess individual programs in fossil energy and energy efficiency.

Recommendation: The committee should undertake the following activities in Phase Three to further demonstrate the robustness of the methodology and maximize its value for decision makers:

  • Expand the case studies to include at least one program in renewable energy and one in nuclear energy.

  • Determine how the benefits methodology can be applied for portfolio analysis and evaluate a portfolio. Portfolios were not evaluated in Phase Two, but the review of the light-duty vehicle hybrid technologies encompasses three separate program elements and provides an opportunity to aggregate several activities. IGCC and sequestration represent two major components of the FutureGen program.

  • Continue to evaluate and refine the quality control process.

  • Continue to communicate and have informal conversations with stakeholders throughout the process. These would include discussions with the committee and panel chairs and with some members of the committee about the case studies and process enhancements or modifications.

  • Make recommendations and provide for transition to full-scale implementation by either DOE or the NRC.


The components necessary for completing the assessment include the methodology, the panel of experts, input from DOE, and a quality control process.

Panel of Experts

Panel Composition

Finding: The commitment and the technology background of the panel members determine the quality of the assessment of the program.

Recommendation 4: Panel composition and level of expertise must be critically considered during the selection process. If a panel concludes that certain skills are not possessed by its members, it should consider expanding its membership or using an outside expert to brief it.

Panel Chair

Finding: The leadership role of the panel chair cannot be overemphasized. For the panel to succeed, the chair has to take a lead role in interacting with DOE to ensure that the best possible information is available to the panel before it meets for the first time. Panels where the chair devoted significant time to ensuring that all panel members were fully familiar with the process and methodology produced the best assessments.

Recommendation 5: The panel chair should spend a fair amount of time outside the actual panel meetings working with DOE program managers, DOE management, and the independent consultant(s).

Independent Consultant(s)

Finding: The main responsibility of a consultant is to maintain consistency across the panels in applying the methodology and to facilitate the analysis. This includes structuring the decision trees, facilitating the assignment of probabilities to technical and market outcomes, and assisting in the modeling of benefits. Phase Two made use of a consultant for all the panels, which worked very well.

Recommendation 6: Depending on the number of programs being evaluated and the panels’ schedules, it might be necessary to have more than one consultant. One consultant should focus on the decision tree development and probability assessment and the other on the modeling of benefits.

Input from DOE

Finding: The level and timeliness of information provided by DOE to the various panels play a critical role in facilitating the deliberations and conclusions on the panel. Completion of panel evaluations is contingent on the panel’s receiving synoptic information and inputs for benefits calculations. The timeliness and quality of this information impacts the quality and utility of the panel evaluation.

Recommendation 7: Since the usefulness of the benefits estimates depends on the quality and timeliness of information available to the panels, DOE management should give its full support for providing the necessary information. DOE at all levels should buy into this process because it is useful for managing and assessing its programs. If this commitment is not clear, the committee should explore all avenues for gaining DOE support.

Quality Control

Finding: Quality control continues to be important in ensuring the consistency, and therefore the utility, of panel evaluations.

Recommendation 8: An oversight committee should apply the quality control process to several elements of the study process, including ensuring appropriate panel membership and composition, orienting the panel chair and consultant, monitoring the panel’s progress, monitoring information

Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.

received from DOE for adequacy and consistency, and reviewing and revising the process itself.


Impact of Policy Measures Unrelated to Research on Realization of Program Benefits

Finding: Policy measures unrelated to research have an effect on when and whether the benefits of some programs will be realized. For example, the benefits of carbon capture and storage depend on the size and timing of a carbon tax (or an equivalent policy intervention in the market). The scenarios, which should include some of these factors, are a valuable tool for characterizing and quantifying the benefits of the DOE R&D program.

Recommendation 9: Decision makers should consider the impact of other policy measures—that is, policies not related to research—in all domains of action (federal, state, and international, say) when considering the results of prospective benefits evaluations. Having a common set of scenarios is useful in general, although additional scenarios may be called for in some cases. While defining the scenarios more completely would be helpful for interpreting the outcomes of the analysis, at the same time it is essential to preserve flexibility by keeping the scenarios as broad as possible.

Guidance for Budget Formulation

Finding: Phase Two showed that the basic structure, using decision trees, worked well and could be implemented with the panels. The panel evaluations permit calculation of a benefit-to-cost ratio, which is not, however, the correct metric to use when allocating resources among the programs in a portfolio.

Recommendation 10: To allocate resources, DOE should know the marginal benefit of a budget increase on a programby-program basis. To calculate the marginal benefit, the decision tree should be examined to identify the outcomes that would be most sensitive to changes in budget levels. In the Phase One study, for example, the lighting program proved to be highly budget-dependent. When such sensitivities exist, the decision tree can be re-estimated for a different budget level, using the committee’s methodology. The marginal benefit associated with the change in budget level is the difference between the net benefits of the two calculations.

Consideration of Alternative Futures

Finding: The methodology presents benefits for each of three scenarios that describe future states of the world, but does not attempt to combine the three sets of benefits into a single set.

Recommendation: DOE should weigh the alternative scenarios in arriving at judgments about the benefits of the overall research portfolio. The portfolio should contain a balance of projects that will produce acceptable results across the range of scenarios.

Benefits Not Captured by the Methodology

Finding: The methodology estimates public benefits in three areas—economic, environmental, and energy security. While these three types of benefits reflect DOE’s strategic goals (DOE, 2005a), the committee recognizes that other kinds of benefits may be important in evaluating some projects. For example, market forces demand that automobile manufacturers produce cars that not only meet the fuel economy standard criterion of importance to DOE but that also have several other attributes. A technology that achieves DOE’s objectives must also provide these additional attributes. As another example, DOE’s research might have employment impacts.

Recommendation 11: If benefits in areas other than economic, environmental, or energy security are found to occur, they should be noted in the text accompanying the results matrix. However, the matrix should stay focused on the three main types of benefits to facilitate comparisons across programs.


Environmental Benefits

Finding: Valuation of benefits in monetary units related to reducing air pollution is both the primary environmental benefit identified in the present study and the class of benefits for which valuation methods are most advanced.

Recommendation 1: Panels should apply valuations in monetary units to criteria air pollutant emissions in the results matrix, but not to other types of pollutant emissions. The valuations used should be the allowance price forecasts for the future period.

Energy Security Benefits: Electricity

Finding: While the complex relationship between electricity supply and security is becoming clearer, analysts are a long way off from having methods for valuing reductions in security threats contributed by technologies such as distributed generation.

Recommendation 2: Panels conducting prospective benefits assessments should describe reductions in threats to

Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.

energy security related to electricity supply as physical quantities of oil and gas.

Energy Security Benefits: Oil and Gas

Finding: Increases in U.S. oil and gas consumption and imports may impose incremental costs that are not fully reflected in the market price. The cost components of this oil premium have been estimated in various studies. In principle, similar estimates could be made for natural gas, but the committee is unaware of such research having been done.

Recommendation 3: Panels should describe energy security benefits related to reduced oil and natural gas consumption quantitatively in the benefits matrix as physical quantities of oil and gas. The time pattern of the oil consumption impacts should be made explicit, along with an assessment of the probable state of the oil market during those future times.


Finding: Prospective benefits evaluations would be most useful if DOE would adopt them. This would allow integration with GPRA, the President’s Management Agenda, and other tools and systems related to performance and budgeting.

Recommendation: DOE should create a triennial program evaluation cycle using the methodology of this Phase Two study for all the applied energy programs. If DOE chooses to undertake this internally, it would need to create a set of FACA committees managed by the DOE and reporting to the under secretary or higher. If DOE chooses to use the NRC, the NRC would independently appoint the oversight committee and panels to undertake the prospective benefits evaluations. A third possibility would be for a contractor working in the model of DOD’s JASON4 to perform the assessment.


JASON is a third-party review of the DOD weapons program, established in the 1950s, managed by MITRE Corporation and funded by the Department of Defense Research and Engineering (Finkbeiner, 2006).

Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
Page 60
Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
Page 61
Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
Page 62
Suggested Citation:"5 Conclusions and Recommendations." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
Page 63
Next: References »
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) Get This Book
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since its inception in 1977 from an amalgam of federal authorities, the U.S. Department of Energy (DOE) has administered numerous programs aimed at developing applied energy technologies. In recent years, federal oversight of public expenditures has emphasized the integration of performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 in response to questions about the value and effectiveness of federal programs. GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which has requested of the National Research Council (NRC) a series of reports using quantitative indicators to evaluate the effectiveness of applied energy research and development (R&D).

The first such report took a retrospective view of the first 3 years of DOE R&D programs on fossil energy and energy efficiency. The report found that DOE-sponsored research had netted large commercial successes, such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization. However, some programs were judged to be costly failures in which large R&D expenditures did not result in a commercial energy technology. A follow-up NRC committee was assigned the task of adapting the methodology to the assessment of the future payoff of continuing programs.

Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic effects. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government did not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement?

This second report continues to investigate the development and use of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. It provides further definition for the development of indicators for environmental and security benefits and refines the evaluation process based on its experience with the six DOE R&D case studies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook,'s online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!