The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward
adapt the results of the previous committee with an aim to develop a methodology and matrix for evaluating prospective benefits of DOE’s energy efficiency and fossil energy programs. In addition, the committee will apply its newly developed methodology to evaluate energy efficiency and fossil energy programs,
as expressed in the pertinent part of the statement of task. (The complete statement of task for Phase One can be found in Appendix B.5) Phase Two will begin in early 2005. Funds were appropriated in FY 2004 for Phase Two and FY 2005 for Phase Three. Activities undertaken by the committee in connection with Phase One are listed in Appendix C.
Three considerations were particularly important in formulating this project. One was that the committee must build on the work of the retrospective study and, by extension, the work of DOE and OMB that has taken place since the retrospective study was completed. DOE has made important strides in developing tools for assessing the likely benefits of its R&D programs. Of special note is that elements of DOE—particularly the fossil energy and energy efficiency programs—have worked together toward common methodologies and approaches. The scale of DOE’s effort has been impressive, involving substantial staff commitments and the extensive use of sophisticated economic models. The efforts of DOE to improve and standardize its own estimates of benefits using a common methodology and approach contributed to the committee’s ability to fully understand the programs and their anticipated impacts.
OMB has also made considerable progress in creating tools for assessing the value of future energy R&D investments and has promulgated specific investment guidelines for applied R&D, as well as for more basic research programs.6 The OMB began the evolutionary development of the Program Assessment and Rating Tool (PART), which ranks federal programs on the basis of both their likely benefits and the quality of program design and management.7 One of the first applications of PART was to DOE’s applied energy R&D programs. OMB now makes public its summary assessments of major DOE programs.8
As the committee studied these developments in methodology, it became increasingly clear that this project would be most useful if it helped to move the entire enterprise toward a common methodology that assists decision makers at every level to consider “what programs should be continued, expanded, scaled-back, or eliminated.”9
The second consideration was that the committee must develop not only a methodology that is rigorous in its calculation of benefits and assessment of risks but also a practical process for applying that methodology in a consistent manner across a variety of DOE programs. The process needs the participation of outside experts who are familiar with the specific technologies and markets and can provide an independent assessment of the program. Familiarity with the programs, technology, and industry is critical to determining what needs to be considered in the analysis, assessing the likelihoods of achieving the technical goals, and calculating the benefits if they are successful. The process of assembling and applying expert panels efficiently within the constraints of a consistent analytic methodology is thus central to the committee’s work.
The third consideration was that the methodology being developed must be transparent and easy to use. The methodology should not require extensive resources to implement nor should the results from its application be difficult for key stakeholders in the program to understand. Further, to the extent possible, the analysis should be consistent with and extend the PART analysis.
To carry out its assigned task, the committee organized its work into the following subtasks:
Review the methodologies for assessing R&D benefits developed by DOE, OMB, and other agencies.10
Propose a conceptual framework that captures the key features of prospective benefits evaluation. The benefits matrix developed for the retrospective study was taken as a useful starting point for developing the prospective framework.
Appoint expert panels to apply the framework to the three DOE programs that had been selected by the committee. A committee member chaired each panel, but the panel members were mainly persons with expertise in the technology and markets pertaining to the programs.11 The panels
The statement of task estimated that the committee’s methodology might be applied to perhaps 5 projects in fossil energy and 10 projects in energy efficiency. In choosing three rather large programs to which the panels were to apply the committee’s prospective benefits methodology, the committee ensured that many projects within each office were considered as part of the committee and panel efforts.
John H. Marburger and Mitchell E. Daniels Jr. “Memorandum for the Heads of Executive Departments and Agencies: FY2005 Interagency Research and Development Priorities.” Executive Office of the President, June 5, 2003.
Office of Management and Budget. President’s Management Agenda, Fiscal Year 2002.
Office of Management and Budget. Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments.
House Report 107-564, p. 125. July 11, 2002. U.S. Government Printing Office: Washington, D.C.
Including, for example, calculations of program benefits performed pursuant to the Government Performance Results Act (GPRA) of 1993.
The committee decided that three studies would be needed to test the methodology and selected the advanced lighting, fuel cells, and carbon sequestration programs. The advanced lighting program is managed by the Office of Energy Efficiency and Renewable Energy and the sequestration program by the Office of Fossil Energy; both organizations have fuel cell programs. These programs also involve the full range of benefits and likely future scenarios (i.e., future states of the world) relevant to the study. Although the committee believes these programs are sufficient for the purposes of the Phase One study, it recognizes that the methodology will be further refined and tested in Phase Two.