in managing and facilitating meetings and be familiar with benefit assessment of R&D programs. The panel chair should be recognized as having technical and/or assessment knowledge in the program area that the panel will be evaluating. It would also be helpful if the chair is familiar with how DOE conducts R&D activities and with DOE’s operating procedures. The chair is identified before the panel is nominated.

To keep the size of the panel to six to eight people, some of the members might possess more than one kind of expertise. Members may come from institutions currently engaged in activities with DOE, but they should not be directly involved with DOE in the program being reviewed. The panels will be chosen by an entity independent of the programs being reviewed.

To assure independence, freedom from conflict of interest, and balanced composition, a process similar to that used by the National Research Council (NRC) to form committees is adopted, whether the review is being performed under the auspices of the NRC, of a DOE-appointed Federal Advisory Committee Act (FACA) committee, or of another organization. While the panel is clearly not being charged with performing a traditional program review or evaluation, it must understand the programmatic issues in order to independently establish probabilities and expected benefits using the committee’s methodology.

Panel members are chosen based on their expertise in the specific technology being reviewed, in business development, or in related policy areas. They are not necessarily expert in or even familiar with the methods used by DOE to administer, implement, analyze, or evaluate programs, including the Energy Information Administration’s (EIA’s) NEMS, decision-tree analysis, or the benefits methodology proposed by this committee. Accordingly, the committee recommends that one or two consultants (depending on the number of programs that are being reviewed) provide support to the panels. The consultant(s) should have expertise in (1) decision analysis and methodology and (2) modeling and economics. They work with the panel members and, if it is an NRC review, with NRC staff. The one or two consultants each work with all the panels and assure consistency among the panels in their use of the methodology and calculation of benefits.



The panel chair schedules, organizes, and facilitates meetings of the panel and is responsible for report completion. In addition, the panel chair

  • Recommends potential panel members.

  • Meets with other panel chairs before the first meeting of the panel to coordinate and ensure consistency of activities.

  • Meets with the consultant and DOE program staff to review the panel’s need for data, as illustrated in the program assessment summary (PAS) forms (Appendix G).

  • In preparation for the first meeting, draws up, with the consultant(s), an initial request to DOE for information, consistent with the discussion in the section “Interactions with DOE and Information Request” in this report which calls for the chair and consultant(s) to meet with the DOE program management prior to the first panel meeting.

  • Discusses with the consultant(s) any questions about the use of the methodology as well as any emphasis (or deemphasis) to be applied to portions of the methodology to make it relevant to the needs of the program being reviewed.

  • At the completion of each panel meeting, meets with the oversight committee chair to assess crosscutting and portfolio issues as well as lessons learned.

The panel chair needs to spend a fair amount of time outside the actual panel meetings working with DOE program managers, DOE management—perhaps at the level of assistant secretary—and the independent consultant(s). If the study is conducted by the NRC, the panel chair’s primary point of contact is NRC staff.

At the first panel meeting, the chair ensures that all panelists are familiar with the procedures outlined in this chapter and that they know what is expected of them in the study. With support from the consultants, the chair opens the first meeting with a briefing to panelists on the following topics:

  • History of the prospective benefits study being undertaken by the panel, brief review of past studies, and review of other panel studies under this phase of work.

  • Description of the methodology to be used.

  • Emphasis that this is not a review of a past program but is instead an assessment of the current program.

  • Role of the consultant(s).

In addition, at the outset of the first panel meeting, the chair outlines the schedule for the study and the tasks that will be undertaken each day, the use of the study results, and panel study focus. The discussion of these topics by the chair, who will impart a firm understanding of the approach to be taken, focuses the panel on the task being undertaken.

Independent Consultant(s)

The primary responsibility of the consultant(s) is to maintain consistency across the panels in applying the committee’s methodology. The consultant(s) might also suggest and implement modifications to the process to address the needs of any specific program being evaluated while maintaining consistency with the committee’s approach across other panels. Responsibilities of the independent consultant(s) are

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement