program review, but that the programmatic issues need to be understood to independently establish probabilities and expected benefits. The expert panel chair would be identified when the Phase Two (or subsequent phase) committee is appointed, if possible.

Panel members will be chosen because of their expertise in the specific technology, in business development, or in related policy issues. They will not necessarily be expert in or even familiar with the methods used by DOE to administer, implement, analyze, or evaluate programs, including the Energy Information Administration’s National Energy Modeling System (NEMS) model, decision tree analysis, or the benefits methodology proposed by this committee. Accordingly, the committee recommends that a consultant with the needed methodological expertise work with the panel members, and, if an NRC review, with NRC staff. This person (or persons) would work with all the panels and would assure consistency among the panels in their use of the methodology.



The panel chair will need to spend a fair amount of time outside the actual meetings working with the DOE program managers, the assistant secretary (as necessary), and the independent consultant. If the study is conducted by the NRC, the panel chair’s primary point of contact will be the NRC staff. Responsibilities of the chair will include these:

  • In preparation for the first meeting, draw up, with the consultant, an initial request to DOE for the information consistent with the discussion in the section “DOE Program Information Request.” The chair and consultant will meet with the DOE program management prior to the first meeting.

  • Discuss with the consultant any questions about the use of the cookbook as well as any emphasis (or de-emphasis) to be applied to portions of the cookbook to make it relevant to the needs of the particular program being reviewed.

  • Prepare a brief lessons-learned report on the committee’s methods at the completion of the assessment.

  • Lead the final debriefing for DOE, the Office of Management and Budget, and the Congress, as appropriate. If the report is prepared by an NRC committee, this will occur after the report has been approved for release.

Independent Consultant

The key responsibility of the consultant is to maintain consistency across the panels in applying the committee’s method. The consultant might also suggest modifications of the process to address the needs of specific programs. Responsibilities of the independent consultant will include these:

  • Participate in initial meeting(s) with DOE program management and the panel chairs to review information needs for each expert panel.

  • Review the committee’s methodology and recommend modifications for the panel’s consideration.

  • Attend all expert panel meetings. Work with panel members, individually and collectively, to structure and work through the necessary analyses.

  • Review each report’s output to ensure that analyses and recommendations are consistent with internal panel discussions and modeling and with reports of the other panels.


The panel will convene at least two meetings lasting 2 full days with the possibility of a short third meeting or conference calls.

Premeeting Work

A significant effort is expected on the part of the panel chair and the panel consultant. Together they must determine the specifics of the information to be requested from the Department of Energy following the general guidelines in the section “DOE Program Information Request.” They will then meet with DOE staff to review the request and identify any DOE concerns with the information request or the methodology. The chair and panel consultant will document the methods by which DOE calculates program benefits, including the use, if any, of reduced-forms methods.1 The consultant will review and develop proposed modifications to the methods appropriate to the program under review. The chairman and consultant will review the character of government support for the program consistent with the discussion in the subsection “Expert Evaluation of Probabilities” in Chapter 3.

Panel members will receive a package of information from staff supporting the panel a few weeks before the first meeting. This package will include the statement of task and the methodology cookbook, with proposed adjustments and information on DOE methodology for calculating benefits. The panel members will also receive the program and project summaries described in the section “DOE Program Information Request” and a list of panel members and short biographies.


Reduced-form versions approximate the more detailed modules in translating input variables (e.g., prices for demand or quantities for supply) to output variables (e.g., quantities and investments for demand and prices and investment for supply). Reduced-form versions can be easily substituted for the full modules. They would not be independent models but simple mathematical structures estimated from the original modules to approximate those modules’ full responses (NRC, 1992).

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement