National Academies Press: OpenBook
« Previous: 3 Methodology for Prospective Evaluation of Deparment of Energy Programs
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

4
Expert Panel Process

INTRODUCTION

The program assessment process centers on establishing an expert panel to review selected DOE research and development (R&D) programs or projects. The expert panel conducts a technical assessment based on a brief description by DOE of the program under review and its component projects. The panel assesses the conditional benefits of the program, assuming the program meets its stated goals. The panel also considers other perhaps more appropriate or more likely to be attained goals of the program. The panel estimates benefits using simplified models whenever possible, following the methodology described in Appendix F, “Guidance on Evaluation of Program Benefits,” but it can also obtain benefit information from a general equilibrium model like the National Energy Modeling System (NEMS) to provide internally consistent information on price and quantity and their impacts. In this instance, benefits estimates are provided by DOE to the panel as illustrated in Appendix G, “Information to be Requested of the Department of Energy.” The panel members’ expertise and the decision trees assessment tool described in Appendix F are used to develop the probabilities for technical and market risk for the program as a whole. The results of the probability analysis are used to estimate the expected program benefits. The panel is supported by two consultants: one, perhaps in decision analysis, who has a working knowledge of the benefits assessment methodology being proposed by the committee and the other in modeling, to calculate the benefits of the program using inputs from DOE and the probabilities arrived at by the panel. The panel reports its results, including its decision trees and its comments on the program risks, in the format defined in Appendix F, Figure F-4. In addition, the panels must consider a number of issues and adhere to the quality assurance guidelines described in the last two sections of the present chapter. The committee supports the panels by overseeing and reviewing their work to ensure consistency in the way the panels apply the methodology.

This chapter describes a specific process for the work of the expert panels. It also contains suggestions for establishing an overall quality assurance function.

EXPERT PANEL COMPOSITION

Six to eight experts are empanelled to evaluate a given DOE program and to apply the process, including the risk and benefits assessment methodologies. The size of the panel is dependent on the breadth of the program to be reviewed. Each panel will require a balance of skills and a wide range of expertise and experience to ensure that all relevant issues are identified, fully discussed, and factored into the assessment. The membership of the panel might include the following:

  • A manufacturer knowledgeable about both the conventional technology and the new technology being advanced by the program, market issues associated with its commercial adoption, and research, development, and commercial application of the technology domestically and internationally;

  • An end user of the technology, possibly a builder or utility representative who can provide a user’s perspective;

  • A chief technology officer or an R&D manager (or equivalent) from industry familiar with how to take a technology from the laboratory to the market;

  • A public policy analyst or decision maker with expertise in energy, environmental, and economic analysis;

  • An expert in the technology being reviewed, who could come from industry, academia, or a national laboratory;

  • An economist who can review and provide insights on the economic information provided by DOE and who will assure that the evaluation methodology is properly used by the panel; and

  • Other related expertise as deemed necessary.

The panel is led by a chair with broad expertise and experience in analyzing energy and environmental issues and technologies. The panel chair should have considerable skills

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

in managing and facilitating meetings and be familiar with benefit assessment of R&D programs. The panel chair should be recognized as having technical and/or assessment knowledge in the program area that the panel will be evaluating. It would also be helpful if the chair is familiar with how DOE conducts R&D activities and with DOE’s operating procedures. The chair is identified before the panel is nominated.

To keep the size of the panel to six to eight people, some of the members might possess more than one kind of expertise. Members may come from institutions currently engaged in activities with DOE, but they should not be directly involved with DOE in the program being reviewed. The panels will be chosen by an entity independent of the programs being reviewed.

To assure independence, freedom from conflict of interest, and balanced composition, a process similar to that used by the National Research Council (NRC) to form committees is adopted, whether the review is being performed under the auspices of the NRC, of a DOE-appointed Federal Advisory Committee Act (FACA) committee, or of another organization. While the panel is clearly not being charged with performing a traditional program review or evaluation, it must understand the programmatic issues in order to independently establish probabilities and expected benefits using the committee’s methodology.

Panel members are chosen based on their expertise in the specific technology being reviewed, in business development, or in related policy areas. They are not necessarily expert in or even familiar with the methods used by DOE to administer, implement, analyze, or evaluate programs, including the Energy Information Administration’s (EIA’s) NEMS, decision-tree analysis, or the benefits methodology proposed by this committee. Accordingly, the committee recommends that one or two consultants (depending on the number of programs that are being reviewed) provide support to the panels. The consultant(s) should have expertise in (1) decision analysis and methodology and (2) modeling and economics. They work with the panel members and, if it is an NRC review, with NRC staff. The one or two consultants each work with all the panels and assure consistency among the panels in their use of the methodology and calculation of benefits.

ROLE OF THE PANEL CHAIR, THE CONSULTANT(S), AND THE OVERSIGHT COMMITTEE

Chair

The panel chair schedules, organizes, and facilitates meetings of the panel and is responsible for report completion. In addition, the panel chair

  • Recommends potential panel members.

  • Meets with other panel chairs before the first meeting of the panel to coordinate and ensure consistency of activities.

  • Meets with the consultant and DOE program staff to review the panel’s need for data, as illustrated in the program assessment summary (PAS) forms (Appendix G).

  • In preparation for the first meeting, draws up, with the consultant(s), an initial request to DOE for information, consistent with the discussion in the section “Interactions with DOE and Information Request” in this report which calls for the chair and consultant(s) to meet with the DOE program management prior to the first panel meeting.

  • Discusses with the consultant(s) any questions about the use of the methodology as well as any emphasis (or deemphasis) to be applied to portions of the methodology to make it relevant to the needs of the program being reviewed.

  • At the completion of each panel meeting, meets with the oversight committee chair to assess crosscutting and portfolio issues as well as lessons learned.

The panel chair needs to spend a fair amount of time outside the actual panel meetings working with DOE program managers, DOE management—perhaps at the level of assistant secretary—and the independent consultant(s). If the study is conducted by the NRC, the panel chair’s primary point of contact is NRC staff.

At the first panel meeting, the chair ensures that all panelists are familiar with the procedures outlined in this chapter and that they know what is expected of them in the study. With support from the consultants, the chair opens the first meeting with a briefing to panelists on the following topics:

  • History of the prospective benefits study being undertaken by the panel, brief review of past studies, and review of other panel studies under this phase of work.

  • Description of the methodology to be used.

  • Emphasis that this is not a review of a past program but is instead an assessment of the current program.

  • Role of the consultant(s).

In addition, at the outset of the first panel meeting, the chair outlines the schedule for the study and the tasks that will be undertaken each day, the use of the study results, and panel study focus. The discussion of these topics by the chair, who will impart a firm understanding of the approach to be taken, focuses the panel on the task being undertaken.

Independent Consultant(s)

The primary responsibility of the consultant(s) is to maintain consistency across the panels in applying the committee’s methodology. The consultant(s) might also suggest and implement modifications to the process to address the needs of any specific program being evaluated while maintaining consistency with the committee’s approach across other panels. Responsibilities of the independent consultant(s) are

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

summarized below, with additional description provided in the next section, “Panel Activities and Process.”

  • Work closely with the panel chair to plan activities and clarify roles, responsibilities, and expectations for the panels’ work.

  • Participate in initial meeting(s) with DOE program management, the panel chairs, and NRC staff to review the information needs of each expert panel.

  • Review the committee’s methodology and recommend modifications for the panel’s consideration, as necessary.

  • Attend all expert panel meetings. Work with panel members, individually and collectively, to structure and work through the necessary analyses. This will include structuring the decision tree or trees, facilitating the assignment of probabilities to technical and market risk outcomes, and guiding and assisting in the modeling of benefits. Review each report’s output to ensure that its analyses and recommendations are consistent with internal panel discussions and modeling and with reports of the other panels, explaining any needed modifications.

Depending on the number of programs being evaluated and the panel’s schedule, it may be necessary to use more than one consultant. In that case, it is recommended that one consultant should focus on the decision-tree development and probability assessments and the second should focus on the benefits modeling. Because the technical and market risk assessment and the benefits calculations are tightly linked, if there are two consultants, they would work closely together. To help ensure panel-to-panel consistency, the consultant(s) work with all panels assessing the programs.

Oversight Committee

An oversight committee, similar in responsibilities to the current Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs (Phase Two), will ensure consistency across the various programs being addressed by the expert panels. There are several options for the form of such an oversight committee: (1) a standing committee of the National Research Council; (2) a DOE-appointed FACA committee; (3) a committee of panel chairs; (4) an independent contractor; and (5) an internal DOE committee. The committee concluded that either an NRC committee or a DOE-appointed FACA committee would be most appropriate because, independently, both organizations have institutionalized mechanisms for preventing bias, and both have access to a broad, high-quality pool of potential committee members.

A key role of the oversight committee is to ensure that the panels are performing their assessments in a consistent manner that allows their results to inform decision making.

Several specific functions of the committee are as follows:

  1. Review the composition of the panels to determine where additions to the panels and/or information presented to the panels could aid in the evaluation.

  2. Evaluate the consultant(s) and ensure that they are knowledgeable about the process/methodology and can work with the panels to develop the decision trees and benefits calculations in a consistent and timely manner.

  3. Meet with the panel chairs prior to the first panel meeting and instruct the chairs on the process and the role the consultant(s) will play in the evaluation.

  4. Evaluate the panels’ progress, working with information from the panel chairs and the consultant(s) between panel meetings. The oversight committee might at this point recommend modification to the panels to ensure consistent assessment. It will evaluate the information being provided to the panels by the DOE program under review. If the DOE information is not sufficient for the panels to perform their assessment, the committee will take action to ensure that DOE is being responsive.

  5. Hold a debriefing meeting with the panel chairs to review the process used and to determine if there were inconsistencies in the evaluations.

  6. Review the panel reports. If it is necessary to modify the panel assessment, the committee will provide comments to the panel chair and offer suggestions for addressing the suggested modifications.

  7. Be responsible for briefing the funding agency and other stakeholders on the results of the benefits assessments.

PANEL ACTIVITIES AND PROCESS

The panel convenes at least two meetings lasting 2 full days each, with the possibility of a third meeting or conference calls, as necessary.

Premeeting Work

The panel chair and the panel consultant(s) can expect to expend a significant effort prior to the first panel meeting. Together they determine the specifics of the information to be requested from the DOE following the general guidelines in the section “DOE Interactions and Information Request.” They then meet with DOE staff to review the request and identify any DOE concerns with the information request or the methodology and review the methods by which DOE calculates program benefits. The consultant reviews and develops proposed modifications to the methods appropriate to the program under review, including any modifications to the decision tree implementation and suggestions for how the economic and other benefits should be calculated.

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

Panel members receive a package of information from the staff supporting the panel at least 2 weeks before the first meeting. This package includes the statement of task and the process/methodology description. The panel members also receive the program and project summaries described in the section “DOE Interactions and Information Request” and a list of panel members and short biographies. Prior to the first panel meeting, panel chairs meet with the consultant(s) to coordinate activities and ensure consistency among panel approaches. The results of this meeting are communicated to the panels by the panel chair on the first day of the panel meeting. It would also be helpful to the work of the panel if a teleconference could be held among panel members before the first meeting to determine whether any additional expertise or members should be added to the panel. This teleconference should be held well in advance of the first meeting to give time for additional appointments to the panel. If this turns out not to be possible, consideration of such matters will be deferred to first meeting.

First Meeting, First Day

In closed session, the panel members are introduced to one another and they assess panel balance and ask for any additional expertise they require. The panel also discusses the process/methodology, schedule, role of the consultant(s), and deliverables. The panel members are advised about their roles and responsibilities for the effort. It is important for this initial discussion and introduction to the process/methodology to take place before the full panel meets with DOE. This focuses the panel on carrying out the specific form of probability analysis and benefits assessment exactly as the committee recommends. This activity is expected to take up the first morning. If time permits, the panel chair develops a list of panel member questions to be shared with DOE prior to DOE’s program presentations.

In the afternoon open session, the panel hears presentations by the DOE program manager, who elaborates on information provided to the panel and answers questions from the panel. The panel also hears a presentation on the models, scenarios, assumptions, and other techniques DOE uses to calculate benefits. It emphasizes to DOE that it is not conducting a traditional program review but is assessing the prospective benefits of the program.

Throughout these presentations, in an end-of-day review and during the second day, the panel identifies additional information that it requires from DOE. The panel might choose to meet in closed session at the end of the day.

First Meeting, Second Day

The second day continues in closed session. The panel will have read the project descriptions before the meeting. It will have heard the DOE presentations and had a chance to raise questions. Members are ready to begin discussion of the program goals, timing, budget, and benefits estimates. They decide whether to carry out the review at the program level or the project level.1 Beginning the review at the project level forces the panel to look at the details systematically and makes for a more informed panel. Eventually, however, the probability of a successful outcome has to be assessed at the program level, and it is unlikely that this assessment can be done by mathematically combining the probabilities assigned to the individual projects. Rather, it would be based on the panel’s judgment using knowledge gleaned from the project assessments.

Once the panel finishes this work, the members familiarize themselves with the committee’s results matrix (see Appendix F) and discuss what work they need to do to provide inputs to the matrix.

With the help of the consultant(s), a decision tree is constructed and a benefits modeling approach specified. The panel identifies the technical and market risks that need to be looked at in the decision tree and, if time permits, begins discussion of the probabilities of reaching the specified levels of technical and market success. If the panel believes the program goals cannot be completed in the time allotted, it uses a different time frame for assigning probabilities. Similarly, the panel identifies other parameters to consider in the decision tree analysis, such as dependencies with other programs, milestones to be met in order to meet program goals, and reasonableness of the program’s budget to meet its goals.

By the end of the second day, the panel agrees to a decision-tree framework, including branches to account for both with and without DOE funding. The panel should also have a good understanding of the manner in which probabilities are to be assigned for each branch of the tree.

Between the First and Second Meetings

The consultant(s) prepares and distributes a questionnaire soliciting probability estimates from each panel member for each major variable in the decision tree. The questionnaire is sent to panel members and, when completed, returned to the consultant(s) before the second meeting. The consultant(s) compiles the data for the panel’s review. The questionnaire can be at the project level or at the program level.

The panel also reviews the chair’s draft outline for the panel report and receives writing assignments for initial report drafting, to be completed prior to the second meeting.

During this time the consultant(s) uses the panel members’ first-cut probability assessments using the decision tree as the basis for an initial estimate of the program benefits. The con-

1

In this chapter, the term “program” refers to the highest level of organization associated with the collection of R&D activities under review. Thus, the panel should consider such terms as technical objective and mission to be synonymous with “program objective.” Likewise, the term “project” can refer to a single R&D activity such as a grant or contract or to a collection of activities addressing a single technical task, or to a quantitative technical target that must be met at the system level.

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

sultant, working with the panel chair and/or individual panel members, develops a prototype benefits model that can be used in conjunction with the decision tree and preliminary probabilities to quantify the estimated benefits of the program.

Second Meeting, First Day

The panel members’ compiled responses to the questionnaire are discussed. The panel reviews the decision tree and the preliminary probabilities assigned by individual panelists, identifies and discusses any areas where significant difference of opinion on technical and market risks exists among panel members, and determines whether additional information is necessary to resolve those disagreements or refine individual inputs. Using the decision-tree process, each member of the panel reassesses the probabilities for each node and each relevant outcome in the decision tree. The panel decides to estimate probabilities individually or to develop consensus estimates. DOE is invited back to answer the questions generated at the first meeting and additional questions from the panelists. Using the decision-tree process, each member of the panel once again reassesses the probabilities for each node and each relevant outcome in the decision tree. The panel discusses more fully the next-best alternative (competing technology) to the DOE program, non-DOE technology funding, and other issues related to the decision tree, assignment of probabilities by panel members, and the consultants’ estimation of benefits.

Second Meeting, Second Day

Several tasks will be completed on this day:

  • Finalize the decision tree by agreeing to the probabilities for each branch of the tree.

  • Complete a review of the benefits calculations conducted by the consultant(s) and provide guidance to the consultant(s) to enable them to finalize the calculations and the decision matrix.

  • Review the report drafts of panel members and make writing assignments for the final report.

If possible, the panel develops the full matrix for the program, including the explanatory material that accompanies the matrix, consistent with the template provided at the end of Appendix F, “Expected Benefits Results and Report Guidance.” This is the critical deliverable of the panel. The panel may also choose to provide expanded commentary on its approach to the evaluation and its use of the methodology. The complete panel report, including the completed version of the two-page template, should be as brief as possible. The panel also determines if it needs any more information from DOE and if a third meeting is necessary. The panel has the option of holding discussions with DOE to clarify any aspects of the program in order to complete these tasks.

Third Meeting or Teleconference Sessions

The panel probably needs follow-up discussions to review new DOE information and to review its draft report. After the report has been completed and reviewed and is ready for publication, the panel chair prepares a summary briefing for DOE management, the Office of Management and Budget, and Congress. These briefings are coordinated and managed by the oversight committee.

Meeting to Ensure Consistency Among Panels

Following the panel meetings, and before finalizing their reports, the panel chairs and consultant(s) meet again with the chair of the oversight committee to discuss panel activities, analyses conducted, issues raised and addressed or left outstanding, and critical assumptions upon which the analysis rests. The group then assesses the consistency of panel approaches and use of the process/methodology, and determines whether any panel report should be modified to ensure consistency with the process/methodology. Any changes to the panel reports are the responsibility of the panel chair. Issues that cannot be resolved by the panels will be brought to the oversight committee for resolution. (See discussion in other sections of Chapter 4 for more details.)

INTERACTIONS WITH DOE AND INFORMATION REQUEST

Information required for the panel’s deliberation is provided by DOE at least two weeks prior to the panel’s first meeting to give panel members sufficient time to get acquainted with the materials. To help ensure that the information provided is most relevant to the panel and least burdensome for DOE, the panel chair and NRC staff meet with the DOE program manager to discuss the panel’s needs and the form in which the information is to be provided. The chair also discusses with DOE the expectations for DOE’s presentation of the information to the panel, including presentation template, program information, and data content, results of modeling analysis, and program and project highlights. The request to DOE includes all information and data that the panel believes it needs to complete its task. To the extent possible, but with exceptions as defined by the chair owing to the unique nature of the DOE program, information and data are standardized across programs and projects.

DOE provides the panel with the program’s goals, budget, and schedule for achieving its goals as well as program plans and roadmaps. It also provides the panel with individual project goals if such projects are a significant part of the larger program and if the goals represent milestones that need to be met if the larger program is to succeed. The conditional relationship is such that the goals of the larger program depend on the individual projects succeeding in their own right. The panel also needs DOE’s estimates of the

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

expected net economic, environmental, and security benefits of the program once goals are met. In addition, the primary assumptions associated with DOE’s benefits analyses should be provided. The net benefits analysis requires that benefits are reported as being over and above those of the next-best alternative to the R&D technology or program under review. Information provided by DOE should comply with the following requirements:

  • Data should be consistent with DOE’s reporting under the Program Assessment Review Tool (PART) and/or the Government Performance Review Act (GPRA) and be the most current available.

  • Data should be reported consistently across individual projects in the program to support project data aggregation at the program level.

  • Net economic benefits data should be reported in nominal as well as real dollars using the same discount rate across projects and programs and should reasonably account for known life-cycle benefits and costs.

  • Net environmental and security benefits should be quantified to the extent possible and qualified as necessary.

  • NEMS and MARKAL modeling results and key assumptions should be consistently reported over a like time period for benefits calculations and simply reported numerically and graphically for ease of understanding. DOE should also explain the specific commercialization process and assumptions used in the benefits calculations.

  • Technology goals must be clearly stated, and the extent of market adoption of the technology once relevant goals are met and the technology is commercially available must be reported along with the underlying assumptions reflected in the arithmetic market adoption function.

  • Information should be provided on external (to DOE) RD&D funding and activities by governments, institutions, and industry to develop and deploy the technologies being evaluated.

The information request and supporting documentation take the form of a brief program assessment summary (PAS), discussed in Appendix G. Individual assessment summaries are prepared for each program under review and each project in the program if projects are also going to be subject to the panel review.

DURATION AND FREQUENCY OF THE EXPERT PANEL REVIEWS

Expert panel assessments of the benefits of each major DOE program occur at least once every 3 years. Programs in which significant changes have taken place are assessed by the expert panels soon after the changes. Between expert panel reviews, DOE comments on and updates the program status annually. Individual expert panels, once convened, aim to finish their work within 3 months of the first meeting, because the reviews and recommendations should tie into and be relevant to the administration and congressional budget processes.

ASSESSMENT OF ACTIVITIES BY NON-DOE ENTITIES

DOE’s expenditure of public funds should be employed to “make the difference” in areas where other public entities, other national governments, and the private sector are not succeeding at spurring innovation and advancing critical technology. Therefore, an assessment of DOE’s R&D investment needs to also examine the effectiveness and potential for success of the non-DOE programs.

To establish the character of the non-DOE R&D activities the review panel must, the goals, objectives, funding, and milestones of those activities that are relevant to the particular DOE R&D program that is being assessed. Details such as technical and marketing risks of the research sponsored by entities other than DOE must be ascertained.

DOE staff should have some, and in some cases considerable, information on RD&D activities taking place at the state level, or being carried out by foreign governments or by industry. This information is shared with the panels early in the review. In addition, the panel selection process will ensure the selection of knowledgeable professionals involved in other related R&D activities. This should add considerable value to the assessment. There may, however, be occasions where NRC and DOE conflict-of-interest requirements make it impossible to have external experts on the panel who are involved in all of these related activities. On these occasions the panel chair, in consultation with DOE managers and NRC staff, selects external experts to brief the panel during the first 2 days of its deliberation so that members better understand the status of related non-DOE R&D activities.

GENERAL ISSUES

As it assesses a program and reviews DOE activities, a review panel needs to take into account several issues (if the members need additional expertise or information, they may ask for it):

  • Showstoppers

    • Identify projects whose success is absolutely critical for program success and determine whether they are receiving sufficient attention and resources from DOE.

    • Identify other projects and programs that are enabling for or complementary to the program under review. If attainment of the program goal is dependent on parallel DOE programs, the panel requires sufficient information to assess this interdependence.

    • Determine if DOE has a termination strategy in the event that a project is not successful and determine the likely effect of that termination on the program under review.

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
  • Program disconnects

    • Determine if success of the projects that constitute the program will translate easily into achievement of the program goal. Identify gaps in the program that would keep that goal from being achieved.

    • Determine, based on the technical and economic expertise of its members, if the program goal is realistic in light of current and/or expected future funding levels.

    • Evaluate a project’s funding and determine if it is sufficient to permit the project to proceed to a go/no go decision. Determine if this decision point is clearly defined by DOE.

  • Assessment of non-DOE activities

    • Assess industry programs—to the extent they are known to DOE or to the panel itself—that may reach the DOE program goal or its equivalent before or at the same time as the DOE-funded activities.

    • Assess international R&D activities that might support or compete with the DOE program and evaluate their impact on expected benefits.

    • Assess industry projects that DOE is supporting or working on jointly with private industry. If the panel needs proprietary information to do this, it will need to enter into a nondisclosure agreement to access the information.

  • Next-best technology

    • Review the next-best technology that would be competing with the new concepts being developed under the DOE program and track the status of competing concepts throughout the conduct of the DOE program.

QUALITY ASSURANCE

Oversight Committee

An oversight committee, which would be similar to the Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs (Phase Two), will have as its primary role assuring consistency over time across the various assessments of DOE programs being conducted by the expert panels. This and other roles of the oversight committee were discussed earlier in this chapter.

Assurance of Panel-to-Panel Consistency

To be of value to the decision makers at DOE, OMB, and the Congress, there must be consistency among the panels in the conduct of their activities, the use of the methodology, and the products that are delivered. Examples of panel activities that are needed to ensure a consistent approach and quality assurance are the following:

  • Panel selection. The panel members are selected using a procedure similar to NRC’s composition and balance procedure. An effort will be made to balance the different biases of the panel and to ensure that no one panel member has a strong positive or negative bias toward the program and technology under consideration.

  • Decision tree and benefit assessment. A consultant and/or consultants are employed to work with each panel to apply the process/methodology and to work with the panels in their assessments. The consultant(s) perform this function for each panel and assure that the assumptions and data for the assessments are consistent for each panel.

  • Information from DOE. The information flow to the panels from DOE follows a template developed by the committee (and described in Appendix G). Alternatively, DOE provides all the information requested in the template at a level that is consistent from program to program. DOE is asked to run its own benefit calculation using an internal computer program, NEMS. The consultant(s) assess the results of DOE benefits analyses to ensure that the inputs to the panels are as consistent as possible.

  • Role of the panel chairs. Before the first panel meeting, the panel chair meets with the oversight committee chair and is instructed on the application of the process/methodology in program assessment. The chair is instructed to guide its panel’s assessment following the process outlined by the committee. After the first meeting of the panel, its chair communicates with the oversight committee chair and describes how the process is working for his or her panel and any lessons learned that will assist the other panels in applying the process/methodology. After the panel has completed its evaluation, the chair will again meet with the oversight committee chair, discuss the panel’s results, and ensure that the process was applied by each panel in a consistent manner.

Periodically—approximately every 4 months—the entire oversight committee should review for consistency all the panel assessments that were conducted during that time. The consultants and the panel chair should prepare briefings for the committee, highlighting any inconsistencies.

FULL-SCALE IMPLEMENTATION OF THE METHODOLOGY

At the conclusion of the Phase Two assessment, the process/ methodology will be ready for full-scale implementation.

The annual R&D budget for all applied energy programs (energy efficiency, renewable energy, fossil energy, nuclear energy, and electricity delivery and reliability) is about $2 billion. Because this assessment approach is most appropriate for efforts that spend at least $10 million per year, efforts would be reviewed at the program level or the major activity level. The estimation of benefits for programs funded at less than $10 million per year is difficult, particularly when using the NEMS or MARKAL models to calculate benefits. In some cases, however, efforts funded at less than $10 million might warrant assessments, particularly energy efficiency R&D, where specific and narrow expertise might

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

be needed to flesh out the program and its expected benefits. The Industries of the Future Program might be an example where separate panels for chemicals, glass, steel, and so on might have budgets less than $10 million each.

About 40 panels would be needed. Since the assessment should occur every 3 years, there would be 12 or 13 assessments and panels each year. It is suggested that three be initiated every quarter. The oversight committee would meet three times a year to review panel reports as they are completed for consistency and provide feedback to the panels.

There are several options for full-scale implementation. First, the DOE needs to decide whether or not it wants to (1) conduct these reviews in-house using an internal committee or DOE-appointed committee, or (2) use an external third-party institution such as the NRC, which does these types of reviews for the National Institute of Standards and Technology and the Army Research Laboratories, or a contractor in the model of JASON.2

2

JASON is a third-party review of the DOD weapons program, established in the 1950s, managed by MITRE Corporation and funded by the Department of Defense Research and Engineering. JASON was formed by academic scientists to give advice to the U.S. government. A recently published book gives a very good description of the origin of JASON and how it has evolved during the past 50 years (Finkbeiner, 2006).

Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 52
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 53
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 54
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 55
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 56
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 57
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 58
Suggested Citation:"4 Expert Panel Process." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 59
Next: 5 Conclusions and Recommendations »
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) Get This Book
×
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since its inception in 1977 from an amalgam of federal authorities, the U.S. Department of Energy (DOE) has administered numerous programs aimed at developing applied energy technologies. In recent years, federal oversight of public expenditures has emphasized the integration of performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 in response to questions about the value and effectiveness of federal programs. GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which has requested of the National Research Council (NRC) a series of reports using quantitative indicators to evaluate the effectiveness of applied energy research and development (R&D).

The first such report took a retrospective view of the first 3 years of DOE R&D programs on fossil energy and energy efficiency. The report found that DOE-sponsored research had netted large commercial successes, such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization. However, some programs were judged to be costly failures in which large R&D expenditures did not result in a commercial energy technology. A follow-up NRC committee was assigned the task of adapting the methodology to the assessment of the future payoff of continuing programs.

Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic effects. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government did not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement?

This second report continues to investigate the development and use of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. It provides further definition for the development of indicators for environmental and security benefits and refines the evaluation process based on its experience with the six DOE R&D case studies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!