National Academies Press: OpenBook
« Previous: 3 Methodology for Prospective Evaluation of Department of Energy Programs
Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

4
Expert Panel Process

INTRODUCTION

The proposed program assessment activity centers on establishing an expert panel to review selected DOE research and development (R&D) programs or projects. This expert panel will begin by conducting a technical assessment based on a brief description of the specific program and its component projects provided by DOE. It will assess the conditional benefits of the program, assuming the program meets the stated goals. The panel should estimate benefits using simplified models, as described in Chapter 3, but can rely on a general equilibrium model like the National Energy Modeling System (NEMS) to create internally consistent price and quantity schedules. The panel members’ expertise and the decision tree assessment tool described in Chapter 3 will be used to develop the probabilities for technical and market risk for the program as a whole. The results of the probability analysis will be used to estimate the expected value of program benefits. The panel will be supported by a consultant in decision analysis who has a working knowledge of the benefits assessment methodology being proposed by the committee. The panel will report its results and comment on the program risks in the format defined in Appendix J, “Expected Benefits Results and Report Guidance.”

This chapter describes a specific procedure for the work of the expert panels. The basis for the recommended procedure is the accumulated experience of the three panels convened by this committee to refine the methodology for evaluating prospective benefits. This chapter also contains recommendations for establishing an overall quality assurance function. The committee believes that such a function is essential for ensuring consistency in the application of the benefits methodology and that it will also serve as a mechanism for the continuing refinement of the methodology.

EXPERT PANEL COMPOSITION

A panel of about six to eight experts would be appointed to evaluate a given DOE program and apply the benefits methodology. The panel would need a wide range of expertise and experience to ensure that the issues are fully discussed and evaluated effectively. The expertise of the panel would include the following:

  • A manufacturer knowledgeable about both the conventional and the new technology being advanced by the program;

  • An end user of the technology, possibly a builder or utility representative or the like;

  • An academic, who could be helpful in determining the scientific requirements for important breakthroughs in the new or existing technology;

  • A chief technology officer or an R&D manager (or equivalent) from industry familiar with how to take a technology from the laboratory to the market;

  • A public policy analyst or decision maker with expertise in energy;

  • An environmental specialist;

  • An economist;

  • An expert in the technology being reviewed, who could come from industry, academia, or a national laboratory; and

  • A venture capital investor active in the relevant arena.

To keep the size of the panel to six to eight people, some of the members might have more than one kind of expertise. Members could come from institutions currently engaged in activities with DOE, but they should not be funded by the program being reviewed. The panels will be chosen by an entity independent of the programs being reviewed.

To assure independence, reasonable freedom from conflict of interest, and balance of composition, a process similar to that used by the National Research Council (NRC) to form committees should be adopted, whether the review is being performed under the auspices of the NRC, of a DOE-appointed Federal Advisory Committee Act (FACA) committee, or of another organization. It must be made clear that the panel is not being charged with performing a traditional

Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

program review, but that the programmatic issues need to be understood to independently establish probabilities and expected benefits. The expert panel chair would be identified when the Phase Two (or subsequent phase) committee is appointed, if possible.

Panel members will be chosen because of their expertise in the specific technology, in business development, or in related policy issues. They will not necessarily be expert in or even familiar with the methods used by DOE to administer, implement, analyze, or evaluate programs, including the Energy Information Administration’s National Energy Modeling System (NEMS) model, decision tree analysis, or the benefits methodology proposed by this committee. Accordingly, the committee recommends that a consultant with the needed methodological expertise work with the panel members, and, if an NRC review, with NRC staff. This person (or persons) would work with all the panels and would assure consistency among the panels in their use of the methodology.

ROLE OF CHAIR AND CONSULTANT

Chair

The panel chair will need to spend a fair amount of time outside the actual meetings working with the DOE program managers, the assistant secretary (as necessary), and the independent consultant. If the study is conducted by the NRC, the panel chair’s primary point of contact will be the NRC staff. Responsibilities of the chair will include these:

  • In preparation for the first meeting, draw up, with the consultant, an initial request to DOE for the information consistent with the discussion in the section “DOE Program Information Request.” The chair and consultant will meet with the DOE program management prior to the first meeting.

  • Discuss with the consultant any questions about the use of the cookbook as well as any emphasis (or de-emphasis) to be applied to portions of the cookbook to make it relevant to the needs of the particular program being reviewed.

  • Prepare a brief lessons-learned report on the committee’s methods at the completion of the assessment.

  • Lead the final debriefing for DOE, the Office of Management and Budget, and the Congress, as appropriate. If the report is prepared by an NRC committee, this will occur after the report has been approved for release.

Independent Consultant

The key responsibility of the consultant is to maintain consistency across the panels in applying the committee’s method. The consultant might also suggest modifications of the process to address the needs of specific programs. Responsibilities of the independent consultant will include these:

  • Participate in initial meeting(s) with DOE program management and the panel chairs to review information needs for each expert panel.

  • Review the committee’s methodology and recommend modifications for the panel’s consideration.

  • Attend all expert panel meetings. Work with panel members, individually and collectively, to structure and work through the necessary analyses.

  • Review each report’s output to ensure that analyses and recommendations are consistent with internal panel discussions and modeling and with reports of the other panels.

PANEL ACTIVITIES AND PROCESS

The panel will convene at least two meetings lasting 2 full days with the possibility of a short third meeting or conference calls.

Premeeting Work

A significant effort is expected on the part of the panel chair and the panel consultant. Together they must determine the specifics of the information to be requested from the Department of Energy following the general guidelines in the section “DOE Program Information Request.” They will then meet with DOE staff to review the request and identify any DOE concerns with the information request or the methodology. The chair and panel consultant will document the methods by which DOE calculates program benefits, including the use, if any, of reduced-forms methods.1 The consultant will review and develop proposed modifications to the methods appropriate to the program under review. The chairman and consultant will review the character of government support for the program consistent with the discussion in the subsection “Expert Evaluation of Probabilities” in Chapter 3.

Panel members will receive a package of information from staff supporting the panel a few weeks before the first meeting. This package will include the statement of task and the methodology cookbook, with proposed adjustments and information on DOE methodology for calculating benefits. The panel members will also receive the program and project summaries described in the section “DOE Program Information Request” and a list of panel members and short biographies.

1  

Reduced-form versions approximate the more detailed modules in translating input variables (e.g., prices for demand or quantities for supply) to output variables (e.g., quantities and investments for demand and prices and investment for supply). Reduced-form versions can be easily substituted for the full modules. They would not be independent models but simple mathematical structures estimated from the original modules to approximate those modules’ full responses (NRC, 1992).

Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

First Meeting, First Day

In closed session, the panel will be introduced to the methodology and any modifications proposed to respond to specific program needs. It will adopt the proposed changes or recommend them to the committee, including reduced-form techniques in the package received in advance of the meeting. After finalizing the method that will be used for benefit assessment, members will familiarize themselves with the methodology using their current knowledge about the program. The committee believes that it is very important that this initial discussion and introduction to the methodology take place before the panel meets with DOE. This will focus the panel on carrying out the specific form of probability analysis and benefits assessment exactly as the committee recommends. It will be emphasized to DOE that this is not a traditional program review. This activity would probably take up the first morning.

In the afternoon open session, the panel would hear presentations by the DOE program manager, who would elaborate on information provided to the panel in the program and project summaries and answer questions from the panel. The panel would also hear a presentation on the models, scenarios, assumptions, and other techniques the program uses to calculate benefits. Throughout these presentations, in an end-of-day review, and during the second day, the panel would identify additional information that it requires from DOE. The panel might choose to meet in closed session at the end of the day.

First Meeting, Second Day

The second day continues in closed session. The panel will have read the project descriptions before the meeting. They will have heard the DOE presentations and had a chance to raise questions. They are ready to begin discussion of the program goals, timing, budget, and so forth. The panel will decide whether to carry out the review at the program level or the project level.2 Beginning the review at the project level would force the panel to look at the details systematically and would make for a more informed panel. Eventually, however, the probability of a successful outcome would have to be assessed at the program level, and it is unlikely that this assessment could be done by mathematically combining the probabilities assigned to the individual projects. Rather, it would be based on the panel’s judgment using knowledge gleaned from the project assessments.

Once the panel has finished this work, the members will familiarize themselves with the committee’s results matrix (see Chapter 3) and discuss what work they will need to do to provide inputs to the matrix.

With the help of the consultant, a decision tree will be constructed. The panel will identify the variables that need to be looked at in the decision tree. If the panel believes the program goals cannot be completed in the time allotted, it might recommend a longer time frame. Similarly, the panel will make recommendations on other parameters such as budget and goals, as appropriate. A questionnaire soliciting probability estimates from each panel member for each major variable in the decision tree will be developed and refined by the consultant(s) and agreed to by the panel. The questionnaire will be sent to panel members and returned to the consultant before the second meeting; the consultant will aggregate the data for the panel’s review. The questionnaire could be at the project level or at the program level. The panel will also review the panel chair’s draft outline for its report.

Second Meeting, First Day

The panel members’ aggregated response to the questionnaire will be discussed. DOE will be invited back to answer the questions generated at the first meeting and subsequently. Using the decision tree process, each member of the panel will reassess the probabilities for each node and each relevant outcome in the decision tree.

Second Meeting, Second Day

Using the set of probabilities supplied by each panelist for the decision tree discussed in Chapter 3, the expected benefits will be calculated.

The second task for this day is to develop the full matrix for the program, including the explanatory material that accompanies the matrix, consistent with the template provided in Appendix J, “Expected Benefits Results and Report Guidance.” This is the critical deliverable of the panel. The panel may also choose to provide expanded commentary on its approach to the evaluation and its use of the methodology. The complete panel report, including the completed version of the two-page template, should be approximately 10 pages long. In the closing session of the meeting, the panel will review the final report outline and hand out writing assignments. It will also determine if any more information is needed from DOE and decide if a third meeting is called for.

Third Meeting or Teleconference Sessions

The panel might need follow-on discussions to review new DOE information, to write the report, or to review draft material. After the report has been completed and reviewed and is ready for publication, the panel chair and members, as

2  

In this chapter, the term “program” refers to the highest level of organization associated with the collection of R&D activities under review. Thus, the panel should consider such terms as technical objective and mission to be synonymous with “program objective.” Likewise, the term “project” can refer to a single R&D activity such as a grant or contract or to a collection of activities addressing a single technical task, or to a quantitative technical target that must be met at the system level.

Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

appropriate, will prepare a summary for briefing DOE management, the Office of Management and Budget, and Congress.

DOE PROGRAM INFORMATION REQUEST

Information required for the panel’s deliberation is to be provided by DOE before the first meeting, as described in Appendix K. The panel chair, independent consultant, and NRC staff (if it is an NRC study) will meet with DOE to discuss the information and data needs of the panel. The information and data request will be standardized across programs and projects and will be the same for all panels. However, if deemed necessary by the panel chair, a departure from the standardized information and data reporting might be permitted. The panels will need both program- and project-level information. DOE should turn over information on broad program and project goals, funding, and benefits expected to be realized if goals are met. DOE should also explain the interrelationships among projects within the program and complementary DOE programs that are required to achieve the overall program objective. It should summarize relevant R&D being conducted abroad and by industry to the extent this can be characterized.

Reporting Protocols

Information provided by DOE should satisfy the following criteria:

  • Data should be the most current available.

  • Data should be consistently reported across individual projects to support data roll-up to the program level.

  • Monetized data should be reported in nominal as well as real dollars using the same discount rate across projects and programs.

  • NEMS and other model results and key assumptions should be consistently reported over like time periods.

  • Data should be presented in a form consistent with the program assessment summary that follows.

DOE should use NEMS to develop the inputs to calculation of benefits. The underlying assumptions used in NEMS should be clearly set forth, including estimates of market penetration. Moreover, DOE should explain the interdependencies among individual projects within the program and the expected mutual contributions. Every effort should be made to monetize economic benefits on a net basis—that is, to compare them with the benefits that might otherwise have accrued over the same time period from continued advancements in the conventional technology or the next-best technology alternative.

In addition to completing the program assessment summary (PAS), DOE is asked to briefly describe the knowledge anticipated to be created if the program milestones are met. The description should include a brief discussion of how the knowledge is to be used within DOE and how it is expected to affect other programs. DOE should also explain the specific commercialization process and assumptions used in NEMS to support the benefits calculations.

DOE should provide estimates of the economic, environmental, and security benefits, along with a brief discussion of the methods used to derive these benefits, including any caveats or other commentary. The information request and supporting documentation should take the form of a brief PAS. Individual assessment summaries should be prepared for each project in the program portfolio. Finally, DOE should provide the panel with a program roadmap if one exists and an outline of the program’s logic, including an identification of critical program inputs and key activities, and expected benefits of the program assuming that the program’s goals are met. The information needed for each program is listed in Appendix K, along with a generic template for summarizing the program assessment.

DOE should provide the panel with a brief descriptive overview of the NEMS model and its strengths and shortcomings; an overview of the modeling scenarios and methods used by NEMS; and an identification of key input assumptions. DOE will also perform limited additional NEMS model runs and/or spreadsheet calculations at the panel’s request.

DURATION AND FREQUENCY OF THE EXPERT PANEL REVIEWS

These expert panel assessments of the benefits of DOE programs should occur at least once every 3 years. In addition, the expert panels should review programs in which significant changes have occurred. During the years between expert panel reviews, DOE will comment on and update the program status annually. Individual expert panels, once convened, should finish their work within 3 months of the first meeting, because the reviews and recommendations should tie into and be relevant to the congressional budget process.

GENERAL ISSUES

As it assesses a program and reviews DOE activities, a panel will need to give special consideration to several issues. (If the experts determine that additional expertise or information is required, they may ask for it.) Each issue is discussed below:

  • Showstoppers

    • The panel should identify projects whose success is absolutely critical for program success and determine whether they are receiving sufficient attention and resources from DOE.

    • The panel should identify other projects and programs that are enabling and complementary with respect to

Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×

the program under review. If the attainment of the program goal is dependent on parallel DOE programs, the panel will require sufficient information to assess this interdependence.

  • The panel should determine if DOE has a project termination strategy in the event that a project is not successful and it should determine the likely effect of that termination on the program under review.

  • Program disconnects

    • The panel should determine if success of the projects that constitute the program will translate easily into achievement of the program goal. The panel should identify gaps in the program that would keep the program goals from being achieved.

    • The panel should determine if the program goal is realistic in light of current and/or expected future funding levels.

    • The panel should evaluate a project’s funding and determine if it is sufficient to permit the project to proceed to a go/no go decision point. The panel should determine if this decision point is clearly defined by DOE.

  • Assessment of non-DOE activities

    • The panel should assess industry programs—to the extent they are known to DOE or to the panel itself—that may reach the DOE program goal or its equivalent before or at the same time as the DOE-funded activities.

    • The panel should assess international R&D activities that might support or compete with the DOE program and evaluate their impact on expected benefits.

    • The panel should assess industry projects that DOE is supporting or working on jointly with private industry. For this purpose the panel will need information some of which might be proprietary. Under such circumstances, the panel will need to enter into a nondisclosure agreement to review this information.

  • Next-best technology

    • The panel should review the next-best technology that would be competing with the new concepts being developed under the DOE program and should track the evolution of competing concepts throughout the conduct of the DOE program.

QUALITY ASSURANCE

Oversight Committee

An oversight committee, which would be similar to the current Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs, will primarily ensure consistency over time across the various assessments of DOE programs being conducted by the expert panels. Several options for the format of such an oversight committee were examined: a standing committee of the National Research Council, a DOE-appointed FACA committee; a committee of panel chairs, an independent contractor, and an internal DOE committee. It was concluded that an NRC committee or a DOE-appointed FACA committee would be most appropriate because of their independence, their institutionalized mechanisms for preventing bias, and their access to a broad, high-quality pool of potential committee members.

Characterizations of Assurance

To be of value to the decision makers at DOE, OMB, and the Congress, there must be consistency among the panels in the conduct of their activities, the use of the methodology, and the products that are delivered. Examples of panel activities that need a consistent approach and quality assurance are these:

  • Use of the methodology and approach. The panel should obtain equivalent information on programs, projects and modeling from DOE. Simplified models (suggested by the committee) should be based on assumptions consistent with those of NEMS. Meetings and completion of the matrix are conducted according to the section “Panel Activities and Process.” There is flexibility in how technical and market risk may be determined, but any deviation from the committee-defined cookbook needs to be explained in the report.

  • Data. Data for a program or its projects should cover the same time frame as the budget year of the program that is under consideration. Modeling data from DOE should be accompanied by the assumptions used to generate the outcomes in order to ensure that the same assumptions were used for the calculations of all of the programs. The data include energy prices, environmental benefits, energy savings, and time frames.

  • Composition of expert panels. All the panels should have the mix of expertise and experience described in the section “Expert Panel Composition.”

  • Panel reports. The two-page template described in Appendix K should be completed by each panel, as appropriate. Each panel chair should submit a lessons-learned report to the oversight committee on the panel’s experience using the committee’s method.

Function

As the committee performs its oversight role, it should evaluate the methods that are set forth in this report and are being used by the expert panels to perform these assessments and recommend changes if they are needed. It should also review the scenarios, which are part of the method, and suggest modifications to accommodate changes in policy and in the economic, energy, and environmental outlooks.

Key to the assessment process is a decision analysis consultant who will work with each panel to ensure that the committee’s methods are being applied in a consistent manner. This consultant would work with the oversight committee and report periodically to this committee.

Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Operational Characteristics

It is recommended that the oversight committee assign one of its members as a liaison to each expert panel throughout the assessment. The liaison member would keep the oversight committee informed of the panel’s assessment and report any concerns arising from the evaluations.

The draft reports of the expert panels should be given to the oversight committee for its information, but not for formal approval.

Periodically—say, every 6 to 12 months—the entire oversight committee should review for consistency all the panel assessments that were conducted during that time. (It will have received the reports from the panels.) The consultant and the committee member serving as liaison to the panel should prepare briefings for the committee, highlighting any inconsistencies. The chair of the expert panel may need to attend these review meetings to provide additional information.

Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 32
Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 33
Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 34
Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 35
Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 36
Suggested Citation:"4 Expert Panel Process." National Research Council. 2005. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward. Washington, DC: The National Academies Press. doi: 10.17226/11277.
×
Page 37
Next: 5 Results of Applying the Methodology »
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase One): A First Look Forward Get This Book
×
Buy Paperback | $56.00 Buy Ebook | $44.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 2001, the National Research Council (NRC) completed a congressionally mandated assessment of the benefits and costs of DOE's fossil energy and energy efficiency R&D programs, Energy Research at DOE: Was It Worth It? The Congress followed this retrospective study by directing DOE to request the NRC to develop a methodology for assessing prospective benefits. The first phase of this project—development of the methodology—began in December 2003. Phase two will make the methodology more robust and explore related issues, and subsequent phases will apply the methodology to review the prospective benefits of different DOE fossil energy and energy efficiency R&D programs. In developing this project, three considerations were particularly important. First, the study should adapt the work of the retrospective study. Second, the project should develop a methodology that provides a rigorous calculation of benefits and risks, and a practical and consistent process for its application. Third, the methodology should be transparent, should not require extensive resources for implementation, and should produce easily understood results. This report presents the results of phase one. It focuses on adaptation of the retrospective methodology to a prospective context.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!