to assess individual programs in fossil energy and energy efficiency.
Recommendation: The committee should undertake the following activities in Phase Three to further demonstrate the robustness of the methodology and maximize its value for decision makers:
Expand the case studies to include at least one program in renewable energy and one in nuclear energy.
Determine how the benefits methodology can be applied for portfolio analysis and evaluate a portfolio. Portfolios were not evaluated in Phase Two, but the review of the light-duty vehicle hybrid technologies encompasses three separate program elements and provides an opportunity to aggregate several activities. IGCC and sequestration represent two major components of the FutureGen program.
Continue to evaluate and refine the quality control process.
Continue to communicate and have informal conversations with stakeholders throughout the process. These would include discussions with the committee and panel chairs and with some members of the committee about the case studies and process enhancements or modifications.
Make recommendations and provide for transition to full-scale implementation by either DOE or the NRC.
The components necessary for completing the assessment include the methodology, the panel of experts, input from DOE, and a quality control process.
Finding: The commitment and the technology background of the panel members determine the quality of the assessment of the program.
Recommendation 4: Panel composition and level of expertise must be critically considered during the selection process. If a panel concludes that certain skills are not possessed by its members, it should consider expanding its membership or using an outside expert to brief it.
Finding: The leadership role of the panel chair cannot be overemphasized. For the panel to succeed, the chair has to take a lead role in interacting with DOE to ensure that the best possible information is available to the panel before it meets for the first time. Panels where the chair devoted significant time to ensuring that all panel members were fully familiar with the process and methodology produced the best assessments.
Recommendation 5: The panel chair should spend a fair amount of time outside the actual panel meetings working with DOE program managers, DOE management, and the independent consultant(s).
Finding: The main responsibility of a consultant is to maintain consistency across the panels in applying the methodology and to facilitate the analysis. This includes structuring the decision trees, facilitating the assignment of probabilities to technical and market outcomes, and assisting in the modeling of benefits. Phase Two made use of a consultant for all the panels, which worked very well.
Recommendation 6: Depending on the number of programs being evaluated and the panels’ schedules, it might be necessary to have more than one consultant. One consultant should focus on the decision tree development and probability assessment and the other on the modeling of benefits.
Finding: The level and timeliness of information provided by DOE to the various panels play a critical role in facilitating the deliberations and conclusions on the panel. Completion of panel evaluations is contingent on the panel’s receiving synoptic information and inputs for benefits calculations. The timeliness and quality of this information impacts the quality and utility of the panel evaluation.
Recommendation 7: Since the usefulness of the benefits estimates depends on the quality and timeliness of information available to the panels, DOE management should give its full support for providing the necessary information. DOE at all levels should buy into this process because it is useful for managing and assessing its programs. If this commitment is not clear, the committee should explore all avenues for gaining DOE support.
Finding: Quality control continues to be important in ensuring the consistency, and therefore the utility, of panel evaluations.
Recommendation 8: An oversight committee should apply the quality control process to several elements of the study process, including ensuring appropriate panel membership and composition, orienting the panel chair and consultant, monitoring the panel’s progress, monitoring information