Transparency. Although the projects being evaluated are complex, in order for the results to be understood and used, decision makers must be able to understand the critical assumptions underlying the analysis, including the assumptions about the likelihood of various levels of success and the benefits in each scenario.
Consistency. Although each project has its own technical goals and potentially different markets, the committee requires that the same summarizing matrix be used for each project. The expected economic, environmental, and security benefits in the matrix must be defined in the same way, and each program must be considered under the same three scenarios. Using a consistent set of definitions and assumptions makes it easier to study a portfolio of programs,17 as the investment in learning the terminology associated with the evaluation of one program pays dividends when studying subsequent programs. It also facilitates comparisons between and aggregation across programs, as the expected benefits of different programs can be compared and totaled.
The committee expects to release its Phase 1 final report on methodology for estimating R&D benefits in February 2005. (In Phase 2, the committee will apply the methodology developed to a larger set of DOE programs and resolve any remaining methodological issues.) The Phase 1 final report will contain the following:
A description of the prospective benefits matrix, which the committee expects will be similar to the outline provided in this letter report;
Detailed guidance on the methodology for completing the matrix;
An outline of the process for applying the methodology, including the use of expert panels;
A summary of the reports prepared by the expert panels for the three test programs—advanced solid state lighting, carbon sequestration, and fuel cells—that were selected for evaluation; and
Recommendations for key issues to be resolved in the Phase 2 study, as well as suggestions to DOE on ways to improve its prospective benefits evaluations.
In its work to date, the committee has benefited greatly from the input and support of the DOE staff. It looks forward to their continued assistance in its future endeavors.
Robert W. Fri,
Robert W. Fri (Chair)
Senior Fellow Emeritus
Resources for the Future
Professor, Department of Economics
University of California, Irvine
Energy Alternatives Studies, Inc.
Paul A. DeCotis
Director of Energy Analysis
New York State Energy Research and Development Authority (NYSERDA)
Wesley Harris, NAE18
Head, Department of Aeronautics and Astronautics
Massachusetts Institute of Technology
Martha A. Krebs
Consultant (Science Strategies)
George W. Norton
Professor, Agricultural and Applied Economics
Virginia Polytechnic Institute and State University
Technology Impact Assessment (TIA) Consulting, Inc.
Maxine L. Savitz, NAE
General Manager, Honeywell, Inc. (retired)
President, Technology and Markets Group
Energy Resources International, Inc.
James E. Smith
Professor, Fuqua School of Business
Electricity Innovation Institute
James L. Sweeney
Professor of Management Science and Engineering
John J. Wise, NAE
Vice President (retired)
Mobil Research and Development Corporation