National Academies Press: OpenBook

Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) (2007)

Chapter: Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program

« Previous: Appendix L Report of the Panel on DOE's Light-Duty Vehicle Hybrid Technology R&D Program
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

M
Report of the Panel on DOE’s Chemical Industrial Technologies Program

INTRODUCTION

The Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs, Phase Two, is applying a methodology for benefits evaluation that it developed in its Phase One study. The purpose of Phase Two is to give decision makers useful information on selected programs. Phase Two is also intended to refine the methodology developed in Phase One. The committee selected six programs for this purpose, one of which is the Chemical Industrial Technologies Program, the subject of this report.

The panel’s assignment was to estimate the benefits of the Chemical Industrial Technologies Program, taking into account technical and market risks. The panel also was asked to make observations regarding program risks, benefits, and other factors that could help decision makers in evaluating the program. This report describes the work of the panel and presents its conclusions and recommendations on both the methodology and the program.

The panel members included persons from academia, industry, and government having extensive experience in chemicals processing. In addition, two members of the full committee served on the panel, one as chair. A consultant experienced in decision analysis assisted the panel. Biographies of panel members are presented in Attachment A.

The panel met twice for 2 days each time. On October 10-11, 2005, the panel received information on the program from DOE program representatives and the contractor who assists DOE in estimating benefits for the Chemical Industrial Technologies Program. At that meeting, the panel examined several of the program’s individual projects in detail and from that examination determined how to apply the recommended methodology to this specific situation. Before the next meeting, which occurred on November 3-4, 2005, each panel member independently developed technical and market risk estimates for each of the 22 projects that currently make up the overall Chemical Industrial Technologies Program. At the second meeting, the panel reviewed its risk estimates and calculated the gross risk-adjusted benefits of the program. Following the second meeting, the panel developed a procedure for calculating net benefits. Members of the panel drafted sections of this report, circulated them electronically, and on January 9, 2006, conferred by telephone on the draft report. Based on the results of this conference call, the report was redrafted and circulated for further comment.

The panel is especially indebted to Dickson Ozokwelu, DOE’s manager of the Chemical Industrial Technologies Program, and to Energetics, Inc., the program’s technical assistance contractor, for outstanding support of the panel’s work.

SUMMARY OF THE DOE PROGRAM

The Chemical Industrial Technologies Program, a subprogram of the DOE Industrial Technologies Program, has as its objective to reduce energy use in the chemicals industry. The chemical industry is a major energy consumer, accounting for 23.5 percent of the energy1 among those industries included in DOE’s Industrial Technologies Program.2 The goal of the Chemical Industrial Technologies Program is to implement a successful strategy for DOE research that helps the chemical industry to achieve a 20 percent reduction by year 2020 in energy usage relative to its 2001 energy consumption of 6.6 quadrillion British thermal units (quads). This translates into a reduction in the chemical industry’s energy use by 1.3 quads per year in 2020 and a proportional reduction in emissions.

To achieve its goal, the Chemical Industrial Technologies

1

Dickson Ozokwelu, U.S. Deparment of Energy, “Overview of the chemicals subprogram,” Presentation to the panel on October 11, 2005.

2

ITP is focused on energy-intensive materials and process industries that account for over 55 percent of U.S. industrial sector energy consumption. These industries include aluminum, chemicals, forest products, glass, metal casting, mining, and steel (DOE, 2005e).

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

Program has a portfolio of relatively small projects, all of them competitively awarded and all of them involving 30-50 percent cost sharing. The amount of cost sharing increases as the projects get closer to commercial readiness. Industry partners are involved in planning and funding the research program. DOE has worked with a committee of 15 large chemical producers, the Chemical Industry Vision 2020 Technology Partnership, to select the projects to be funded in the Chemical Industrial Technologies Program. The partnership focused on 10 individual areas of future promise, such as computational fluid dynamics and ionic liquids. An exergy analysis3 identified the major opportunities for energy savings in the chemical industry, focusing DOE project selection on those areas with the largest potential savings. This joint industry-government partnership is ongoing in helping DOE to select and evaluate projects. The primary selection criterion is energy savings in the United States.4

Twenty-two projects are currently being funded at $7 million per year. These projects, if they all were funded to completion and all were successful, are estimated by DOE to achieve a saving of 0.303 quads per year, or 23 percent of the overall program goal for the Chemical Industrial Technologies Program. Previous successful projects and new projects yet to be started will increase the savings.

The current Chemical Industrial Technologies Program as administered by DOE funds projects in three subprogram areas:

  • Reactions (13 projects). This area includes oxidation catalysis, microreactors, new process chemistry, and biocatalysis. The total energy saving goal in the reactions area is 650 trillion British thermal units (TBtu) by 2020, 242 TBtu of which is accounted for by current projects. In several of the funded projects, reaction and separation are carried out in the same step, thus attempting to gain substantial saving in both energy consumption and capital equipment. This work is part of the current worldwide trend toward process intensification.

  • Separations (4 projects). This area includes distillation hybrids, crystallization (not funded at the present time), and membrane separations. The total energy saving goal in separations is 420 TBtu, of which 40 TBtu is represented by current projects. This area also includes projects that combine separation and reaction (e.g., catalytic distillation) in the same equipment.

  • Enabling technologies (5 projects). This area includes materials of construction, with an emphasis on corrosion, computations to improve the efficiency of dense fluidized beds, industrial energy systems such as an energy-conserv

3

The exergy content of a system indicates its distance from thermodynamic equilibrium.

4

For a more complete description and evaluation of the strategic planning process of the Chemical Industrial Technologies Program, see NRC (2004a).

5

Dickson Ozokwelu, U.S. Deparment of Energy, “Overview of the Chemicals Subprogram,” Presentation to the panel, October 11, 2005.

The report presents a positive evaluation of the strategic planning process for the Chemical Industrial Technologies Program. ing burner design, and sensors and control equipment. The total energy saving goal in this area is 260 TBtu, 21 TBtu of which is in current projects.

The budget of the Chemical Industrial Technologies Program has shrunk drastically. From $13 million in FY03, the budget decreased to $9 million in FY05 and $7 million in FY06. There is a clearly apparent contradiction between the ambitious goals of the program and the dwindling resources available to pursue them. The DOE management team seeks to cope with this situation by continuing its portfolio review, described below.

There have been a number of accomplishments from completed projects in FY03-05 in the Chemical Industrial Technologies Program. Examples include in situ analysis, distillation column flooding prediction, dimpled heat exchanger tubes, catalytic hydrogen retrofit reactors, and new alloys for ethylene cracker tubes.5 All of these, with the possible exception of the retrofit reactor, have relatively specialized applications that will produce positive but modest benefits.

However, the program is also pursuing sweeping changes in process design in the hope that they can yield big energy savings. To focus their shrinking budget on the highest payoff projects, the DOE management team is trying to stimulate industry research and to fund projects that would not be possible without federal assistance, and is accelerating progress on projects that might be funded by industry sometime in the future. The portfolio is under continuing review, with the objective of ending projects that are not progressing or that promise only small energy savings.

A major new element complicating the management problem in this program is the surge in energy prices in 2005. This increase worsens the trend that few if any commodity chemical plants consuming large amounts of energy will be built in the United States in the foreseeable future. DOE intends to respond to this trend by emphasizing technologies that can be retrofitted to existing plants to increase their efficiency and lower their costs.

TECHNICAL AND MARKET RISK ASSESSMENT

In assessing the probabilities of technical success for the overall program, the panel needed to decide on whether to proceed from the top down or to rely on the project evaluations to build the program evaluation from the bottom up. Because of their number and heterogeneity, all 22 existing projects were assessed individually. The panel estimated the probability of technical and market success and estimated the benefit for the program as a whole by rolling up the project assessments. The panel was reasonably confident of its technical risk estimates but found market acceptance harder

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

FIGURE M-1 Probability of technical success, probability of market success, and value of benefits if the project is successful. The area of a bubble is directly proportional to the estimated benefit for its associated project.

to evaluate because a number of the factors are difficult to predict at an early stage of research.

Although energy savings is the driver for DOE investment in these research projects, market acceptance of the technologies is determined by many additional factors. For example, industry also considers the total capital involved and its cost, the status of the industrial partners’ manufacturing capacity at the time the technology is ready for consideration as a retrofit or possibly for use in a new plant, future raw material costs and availability, and principal market areas to be served. If all these factors are favorable, company managers will seek to convince themselves that an acceptable rate of return several years after a plant has started up can be achieved (usually about 20 percent return on investment for a new plant and often much more for a retrofit).

Notwithstanding these complexities, the panel believes it arrived at reasonable, and reasonably consistent, estimates of technical and market risk for all projects. The initial estimates from all the members were relatively close,6 and the few outliers were quickly reconciled during the discussions of each project. This was aided by the fact that most projects were in areas where one or more committee members had familiarity or experience.

Overall, the panel assessed the projects in each category to be high risk and thus consistent with DOE’s program strategy. Probability of technical success ranged from 5 to 50 percent, with market success from 10 to 70 percent. Combining these assessments the range for success for the entire portfolio is between 1 and 35 percent with an average of 7.5 percent.7 Figure M-1 is a bubble chart that graphically summarizes the risks and potential benefits of each project. Each project is represented by a bubble on the chart: the size of the bubble is proportional to DOE’s estimated gross economic benefit of the project if it is successful, and the location of the bubble indicates the technical and market risks assessed by the panel for that project.

RESULTS AND DISCUSSION

As noted in the Introduction, above, the Phase Two methodology suggests that an evaluation of the benefits of a specific program should explicitly consider (1) the role of DOE funding, (2) the technical and market risks that can affect the outcome and value of the program’s activities, and

6

The initial estimates were made between the first and second panel meetings and discussed at the second meeting.

7

This result is consistent with research into the success rate of projects of this type. For example, in its presentations DOE compared the Chemical Industrial Technologies Program to studies that found an overall success rate of 16 percent for projects at a similar stage in the research chain.

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

(3) the net economic, environmental, and security benefits that will result from the portfolio. The methodology also requires that benefits be estimated under three global scenarios, representing three different possible future states of the world. The panel considered each of these considerations in their evaluation.

Technical and Market Success

As noted above, the panel estimated a probability of technical success and a probability of market success for each project in the Chemical Industrial Technologies Program. In the panel’s judgment, the probabilities that might be assigned to the technical and market success for the projects would be the same for each of the three global scenarios—that is, in the AEO Reference Case, High Oil and Gas Prices scenario, and the Carbon Constrained scenario.

Benefits Estimation

The economic, environmental, and security benefits of the Chemical Industrial Technologies Program derive directly from the energy savings resulting from the projects. To the degree that energy usage in the chemical industry is reduced, there are net economic benefits to the nation in the form of lower energy demand and lower energy prices, environmental benefits in the form of reduced environmental emissions, security benefits in the form of less use of oil and natural gas and the associated diminished reliance on imports.

Benefits estimation thus begins with an estimate of the energy saving that would result from successful technology development and adoption. DOE’s contractor, Energetics, uses a formal procedure to estimate the energy savings of a project, from which the gross economic benefit can be calculated. The procedure also estimates the reduction in the emissions of criteria pollutants. The panel reviewed this procedure and concluded that it was applied consistently for all projects and that it produced a reasonable estimate of the future energy savings for projects that are in the early stage of research.

Since the DOE estimates assume 100 percent success of each project, the panel calculated the expected total benefit of the portfolio by applying its probabilities of technical and market success to DOE’s estimates of energy savings for each project in the portfolio and rolled up the individual project estimates into an expected value of gross benefits for the overall program.

To get some insight into the uncertainty surrounding this expected value, a probability distribution of the economic benefits of the portfolio was generated using the risk estimates for individual projects. Each project was assumed to succeed with the net probability estimated by the panel (technical and market success) and to yield the full benefits calculated by DOE if successful and no benefits if not successful. By simulating many cases using these probabilities it was possible to create a distribution of expected benefits. The panel recognizes this is a simplification that does not match actual outcomes—each research project could partly succeed and simply yield lower benefits than estimated. It believes, however, that the simplification is a reasonable first-order approximation.

Finally, the panel considered the benefits estimates for each of the global scenarios prescribed by the committee. Although the economic benefits of the research depend on the global scenarios, which have different energy prices, the environmental and security benefits are considered to be constant in all three global scenarios.8 The panel thus calculated economic benefits using the procedure described above but plugged in energy prices appropriate to each scenario.

Role of DOE Funding

In developing their benefits estimates for each project, DOE assumed that the research being funded will accelerate the development and implementation of the identified technology, but that the technology would eventually have been developed by industry if DOE were not supporting the research. The benefit of DOE support is then defined as the energy and other savings that are realized by having the technology available earlier. The acceleration period was estimated technology by technology in the analysis performed by Energetics. For the projects evaluated in this report, they estimated the acceleration usually at about 3 to 5 years; one project claimed a 10-year acceleration, and one a 20-year acceleration.

The panel agreed that in almost all cases DOE support would be a significant accelerating factor, mainly because there are high risk projects. Indeed, in today’s worldwide competitive chemical industry, companies are less likely to spend money on the early exploration of high-risk technology with a low expectation of a large benefit than on safer R&D to gain incremental advances. For this reason, some members of the panel believed that industry is reluctant to make investments in the kind of projects funded by the Chemical Industrial Technologies Program. If so, the acceleration of energy savings due to DOE investment would be much greater. Additionally, some projects in the DOE portfolio, such as developing an industry-wide database on corrosion management, are unlikely to be sponsored by a single company. On balance, the panel concluded that DOE’s acceleration estimates were reasonable and in some cases probably conservative. Accordingly, the panel adopted these estimates as a basis for calculating benefits.

8

Arguably, the market penetration of energy efficient technologies would increase in the High Oil and Gas Prices scenario. On the other hand, higher prices would have the effect of driving domestic chemical production offshore. The panel did not believe the net change in quantities would be small enough to ignore for the purpose of this analysis.

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Decision Tree

In developing the decision tree required by the Phase Two methodology, the panel considered two issues. The first was to estimate the benefits associated with the “no DOE funding” branch of the tree. Because the benefits of the “DOE funding” branch were calculated as the incremental benefit arising from technology acceleration, the “no DOE funding” branch was assigned a value of zero.

The other issue was whether to develop a decision tree and assess probabilities of technical and market success for the overall program as a whole (from the top down) or to rely on the project evaluations to build the program evaluation from the bottom up. As described above, the panel implemented the bottom-up approach, evaluating each project individually and calculating the portfolio-level benefits directly from the project evaluations. To characterize the uncertainty in the benefits of the portfolio as a whole, the panel developed a probability distribution of expected total economic benefits of the portfolio based on the simulation described above. Figure M-2 illustrates the results of this evaluation of uncertainty in portfolio benefits. From this simulation and the assumptions regarding the role of DOE funding, an overall decision tree representing the benefits analysis was derived (Figure M-3). Both figures use gross economic benefits estimates, since the panel determined that net benefits can best be adjusted at the program level.

In the evaluation performed between the two meetings, panelists did estimate the likelihood of each subprogram achieving various levels of technical success. This top-down analysis was compared to the results of the bottom-up analysis described above. For the two smaller subprograms—separations (four projects) and enabling technologies (five projects)—the expected benefits estimated at the portfolio level were quite close to the benefits for the subprogram derived from the individual project evaluations (within 5 percent for the separations projects and within 15 percent for the enabling technologies projects). For the Reactions subprogram, however, with 13 projects, the estimate of benefits at the subprogram level was 10 times greater than the estimate derived from the project-level evaluation. The discrepancy is believed to be partly an artifact of the assessment questions asked, where very low subprogram benefits were not an option, and partly a result of the cognitive difficulties in aggregating a large number of unrelated project evaluations. Based on this result, the panel concluded that bottom-up methodology was more useful for the analysis of a portfolio of diverse and unrelated projects.

Net Economic Benefits Calculation

The economic benefits estimated above, like those estimated by DOE, represent the gross economic benefits of the program–that is, the estimate includes all the benefits potentially flowing from the research but not the full costs of achieving those benefits. Net economic benefits could be estimated for each project by reducing the gross benefits by the R&D costs, the additional development costs of bringing the technology to a point where it could be commercially implemented (weighted by the likelihood the R&D would lead to further development), and the costs of implementing the technologies should a decision be made to do so (weighted by the likelihood a commercialization decision would be made). However, because these projects are at such an early stage of R&D, the panel did not estimate the future development and commercialization costs, nor did it believe it had the information necessary to estimate those costs for each project.

Therefore, a simplified method was employed to estimate net economic benefits from the gross economic benefits calculated by DOE. First, it is assumed that for any technology that reaches market success, a decision will have been made by private industry to implement it. Such a decision is typically made on an economic basis, with the project passing some internal hurdle rate for financial return. A typical hurdle for a cost-reducing project would be a 3-year payback period. Therefore, the panel simply assumed that for any project that “succeeds,” investment costs no more than the net present value of the economic benefits for first 3 years were required.

The net result of these assumptions is to reduce the gross benefits of each project by an amount equal to that associated with a 3-year delay in the gross benefits stream. Accordingly, the net economic benefits are taken as 91.5 percent of the gross economic benefits, corresponding to a decrease in benefits associated with a 3-year delay.9

Results Matrix

Figure M-4 summarizes the results of the panel’s estimation of the benefits of DOE’s Chemical Industrial Technologies program. Benefits are calculated as described in the report, for all three scenarios. Only the net economic benefits change in the three scenarios, as they are partially dependent on energy prices. The figure presents both the probability-weighted average of the net economic benefits (the statistical expectation) and the uncertainty around that estimate. Figure M-4 also presents the panel’s observations about the program.

SUMMARY AND CONCLUSIONS

The panel believes that the benefits estimation methodology developed for Phase Two has successfully applied to the

9

This reduction in estimated benefits is largely modest, because the payback assumption embodies a relatively high private sector hurdle rate, while public benefits are discounted at a lower rate (between 3 percent and 7 percent). The delay of 3 years at the lower public discount rate changes the benefits estimate only slightly.

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

FIGURE M-2 Uncertainty surrounding estimates of program benefits.

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

FIGURE M-3 Implied decision tree representing the panel’s evaluation of the ITP–Chemicals portfolio. This tree is derived from the simulation of total portfolio benefit described in the report. There are several ways to derive a discrete distribution for a decision tree from a continuous distribution. For this figure, a well known shortcut for the bracket-mean approach was used, where the 10th, 50th, and 90th percentiles of the continuous distribution are selected to represent the full distribution and given weights of .25, .5, and .25 respectively.

Chemical Industrial Technologies Program. On the basis of its evaluation, the panel also concludes that the Chemical Industrial Technologies Program is addressing an important opportunity to produce energy savings in a major industrial segment by supporting early-stage research that industry is unlikely to support. However, to realize its potential benefits, the program must adapt to a seriously constrained budget and a changing domestic industrial environment. The following sections discuss more fully these conclusions about the program and the benefits estimation methodology.

Program Conclusions and Recommendations

As noted earlier in this report, the Chemical Industrial Technologies Program faces a very tight and rapidly declining budget. In addition, it seeks to accelerate energy savings in a dramatically changing U.S. chemical industry, notably one in which very few greenfield plants are being constructed in the United States but instead are going overseas to be closer to growth markets and/or raw materials. Rapidly increasing costs for energy and petroleum-derived raw materials could exacerbate these trends. DOE says it is moving toward the view that technologies that can be implemented as retrofits to existing plants will be favored so as to allow this country to directly reap most of the energy-saving benefits. Retrofits are subject to the same financial return tests as greenfield plants, with the possible exception that they are expected to generate earnings sooner after start-up since the primary objective is cost savings, not increased volume.

Against this background, and based on the experience of its members in managing and conducting research in the chemical industry, the panel wishes to underscore how essential it is for DOE management to focus its limited resources on the most promising opportunities available to it. The danger is more than just that of wasting funds on less promising projects. Equally if not more significant is the possibility of losing the benefits of high-priority projects because they were not pursued aggressively enough. This danger is particularly great for early-stage research of the type being funded by DOE, since such projects depend on a few creative and motivated individuals. If these persons cannot work on a project of importance to DOE, they will work on something else—with the result that the project may not be restarted when DOE has the funds to do so.

The panel recognizes that DOE has been pursuing projects in the Chemical Industrial Technologies Program that are of very high risk, but having high payoff. Faced with its severe budget constraint and the changing nature of the domestic industry, DOE should adhere to this fundamental strategy more closely than ever. Based on its members’ experience with similar projects in industry, the panel suggests that DOE frequently review the projects in its portfolio to ensure that each is pursued just long enough—time, and money—to demonstrate feasibility. If it is judged at any time that the

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

FIGURE M-4 Results matrix of the Panel on DOE’s Industrial Technologies Program–Chemicals.

quality of the personnel and/or other resources being applied is not up to the task, or that progress is not up to expectations, the work should be terminated. The freed funds may then be applied elsewhere.

A model for the administration of projects of this kind might be the way Small Business Innovation Research grants (SBIR) are awarded and funded. Following this example, it might be appropriate to view the first year of funding of the chemical industrial projects as a first phase whose purpose is to define and assess the critical technology roadblocks that are potential showstoppers. In the second year-long phase, funds would be applied exclusively to address the most critical of these. Successful outcomes would trigger a request for further and often major increases in funding to demonstrate the approach. Sharing of funding and other resource commitments from industry should increase from phase to phase, reflecting growing confidence. The panel understands that DOE management is thinking along these lines.

Finally, the panel notes that a credible industry commitment to commercialization is crucial to the decision to move from the first phase to the second and third phases. In particular, the third phase must have industrial partners

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

willing to commit considerable resources and have assurances from higher-level company management that there is a good probability of going forward to market. A marketing plan from the commercial partners should be available at this time. Absence of such a commitment should be cause for terminating the project.

Results Matrix

The panel believes it successfully applied the recommended methodology to this portfolio of small projects. This experience suggests that methodology could be applied to other situations in which a portfolio of diverse and technologically unrelated projects is to be evaluated. The panel made several judgments in applying the methodology, however, which should be considered when applying it to similar portfolio analyses. Specifically,

  • The bottom-up analysis described above is more time consuming than a top-down approach, but the panel believes that it provides a more reliable estimate of expected benefits. The distribution of expected benefits at the program level can be inferred from the probability distributions developed by simulating the outcome of the program based on projectlevel risk estimates. Figure M-2 shows the result of this simulation for the Chemical Industrial Technologies Program.

  • For early-stage research, as in the Chemical Industrial Technologies Program, a simplified calculation of the costs of commercialization is appropriate. The panel selected a simple payback model for this purpose. For portfolios comprising of nearer term research, direct calculation of commercialization costs may be more appropriate.10

The methodology developed for Phase Two spells out the information needed from DOE at the beginning of the project. Through its contractor, Energetics, the Chemical Industrial Technologies Program provided a one-page summary of the essential information about each project (see Figure M-5 for an example). This summary proved to be invaluable to the panel because it presented each project in a brief, consistent format, thus simplifying the review of 22 projects. The panel recommends that a similar summary be required for other projects being evaluated. It would be helpful, as well, to add to the summary sheet DOE’s opinion of the technical and market risks and their distribution, the key drivers of uncertainty behind this risk assessment, and the metrics that would be useful for monitoring the progress of research in reducing risk.

Finally, although the Phase Two methodology suggests that benefits be evaluated every 3 years, a more frequent review of projects composed of early-stage research would be useful. This is because the projects are relatively short in duration and because less successful projects must be weeded out as quickly as possible.

Uncertainty About Portfolio Benefits

Figure M-2 illustrates the uncertainty about the total economic benefit of the research portfolio, based on a simulation of project-level risk evaluations. In the top chart, for any given portfolio value along the x-axis, the corresponding value along the y-axis shows the probability that the portfolio value will be equal to or less than that value. For example, there is about a 50 percent probability the value of the portfolio will be $250 million or less. The bottom chart displays a probability density function. The height of each bar represents the relative likelihood that the total portfolio value will lie in the range represented by the column. For example, it is about twice as likely that the total portfolio benefit lies between $0 and $150 million than between $150 million and $300 million.

Comments and Observations

The goal of the Chemical Industrial Technologies Program is to implement a successful DOE research program that helps the chemical industry to use 20 percent less energy in 2020 than in 2001. This translates into a reduction in energy use by the chemical industry of 1.3 quads per year in 2020 and proportional reduction in emissions. To achieve its goal the program has a portfolio of relatively small projects, all of which are competitively awarded and all of which involve 30-50 percent cost sharing. Twenty-two projects are currently being funded at a total of $7 million per year. If all projects were funded to completion and all were successful, DOE estimates they would achieve a saving of 0.303 quads per year, or 23 percent of the overall program goal for the Chemical Industrial Technologies Program.

The panel believes that the Chemical Industrial Technologies Program is seizing an important opportunity to produce energy savings in a major industrial segment by supporting early-stage research that industry is unlikely to support. However, to realize its potential benefits, the program must adapt to a seriously constrained budget and a changing domestic industrial environment. Cutting-edge research that can be done within the limited budget will probably continue to produce valuable but relatively small-scale advances. However, the program is also pursuing sweeping changes in process design in the hope that they can yield big energy savings.

The current portfolio of projects has an expected net economic benefit between $215 million and $534 million in the AEO Reference Case. Because the program is composed of early-stage research projects, the range of benefits is between $0 and $1.55 billion. Benefits in the High Oil and Gas Prices and Carbon Constrained scenarios are somewhat higher.

10

It is important to note that the early-stage research considered here is nonetheless intended to produce discrete technologies to be brought to market. These are not curiosity-driven basic research projects, and the methodology used by the panel would not apply to such projects.

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

FIGURE M-5 Sample project information sheet.

Technical and Market Risk Assessment

The panel assessed all of the projects in this program to be high risk and thus consistent with DOE’s program strategy. Probability of technical success of individual projects ranged from 5 to 50 percent, with market success from 10 to 70 percent. Combining these assessments, the range for success for the entire portfolio is between 1 and 35 percent, with an average of 7.5 percent.

Benefits Estimation

The economic, environmental, and security benefits of the Chemical Industrial Technologies Program derive directly from the energy savings realizable from the projects. DOE estimates that the research will accelerate the development and implementation of the identified technology by 3 to 5 years. The panel agrees that in almost all cases, DOE support would be a significant accelerating factor. It calculated the expected total benefit of the portfolio by applying its probabilities of technical and market success to DOE’s estimates of energy savings for each project in the portfolio and rolled up the individual project estimates into an expected value of gross benefits for the overall program. Net benefits are calculated by assuming that for any project that “succeeds,” investments on the order of the net present value of the economic benefits for first 3 years will be required.

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Program Observations

Based on the experience of its members in managing and conducting research in the chemical industry, the panel wishes to underscore how essential it is for DOE management to focus its limited resources on the most promising opportunities available to it. The danger is more than just the wasting of funds on less promising projects. Equally if not more significant is the possibility of losing the benefits of high-priority projects because they were not pursued aggressively enough. The panel suggests that DOE frequently review the projects in its existing portfolio to ensure that each project is pursued for as little time and money as it takes to demonstrate feasibility or infeasibility.

ATTACHMENT APANEL MEMBERS’ BIOGRAPHIES

Robert W. Fri, Chair, is a visiting scholar and senior fellow emeritus at Resources for the Future, where he served as president from 1986 to 1995. From 1996 to 2001 he served as director of the National Museum of Natural History at the Smithsonian Institution. Before joining the Smithsonian, Mr. Fri served in both the public and private sectors, specializing in energy and environmental issues. In 1971 he became the first deputy administrator of the U.S. Environmental Protection Agency. In 1975, President Ford appointed him as the deputy administrator of the Energy Research and Development Administration. He served as acting administrator of both agencies for extended periods. From 1978 to 1986, Mr. Fri headed his own company, Energy Transition Corporation. He began his career with McKinsey & Company, where he was elected a principal. Mr. Fri is a senior advisor to private, public, and nonprofit organizations. He is a director of American Electric Power Company and of the Electric Power Research Institute and a trustee of Science Service, Inc. (publisher of Science News and organizer of the Intel Science Talent Search and International Science and Engineering Fair). He is a member of the National Petroleum Council and serves on the Advisory Council at the Marian E. Koshland Science Museum and on the Steering Committee at the Energy Future Coalition. In past years, he was a member of the President’s Commission on Environmental Quality, the Secretary of Energy Advisory Board, and the University of Chicago board of governors for Argonne National Laboratory. He has chaired advisory committees of the National Research Council, the Carnegie Commission on Science, Technology and Government, the Electric Power Research Institute, and the Office of Technology Assessment. He served as chairman of the NRC Committee on Benefits of DOE’s R&D in Energy Efficiency and Fossil Energy and of the Committee on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs, Phase One. From 1978 to 1995 he was a director of Transco Energy Company, where he served as chair of the audit, compensation, and chief executive search committees. He is a member of Phi Beta Kappa and Sigma Xi and a national associate of the National Academies. He received his B.A. in physics from Rice University and his M.B.A. (with distinction) from Harvard University.


Anne Chaka is the group leader for computational chemistry at the National Institute of Standards and Technology in Gaithersburg, Maryland. She received her B.A. in chemistry from Oberlin College, her M.S. in clinical chemistry from Cleveland State University, and her Ph.D. in theoretical chemistry from Case Western Reserve University. In 1999-2000, she was Max Planck-Society Fellow at the Fritz-Haber-Institut in Berlin. She spent 10 years at the Lubrizol Corporation as head of the computational chemistry and physics program and previously was technical director of ICN Biomedicals, Inc., an analytical research chemist for Ferro Corporation, and a Cray programming consultant to Case Western Reserve University for the Ohio Supercomputer Center. Active areas of her research include metal oxide surface chemistry, atomistic descriptions of corrosion and materials, pericyclic reaction mechanisms, heterogeneous and homogeneous catalysis, thermochemistry, and oxidation.


Paul H. Kydd is the proprietor of Partnerships, a developer of lithium-ion batteries for applications in boats and automobiles. From 1999 to 2002 he was chairman and chief technology office of Parelec Inc., a firm that commercialized the Parmod technology for additive printing of electronic interconnects. He was president of Partnerships Limited Inc. from 1983 to 1999, a firm that developed chemical technology such as fuels and propellants. Previously, he was vice president for technology of Hydrocarbon Research, Inc., and oversaw approximately $20 million in synfuel and petrochemical process R&D. In addition, he was manager, chemical processes branch, GE Corporate R&D and combustion scientist, GE Research Laboratory. He has an A.B. from Princeton University and a Ph.D. from Harvard University, both in physical chemistry.


Alexander MacLachlan (NAE) retired at the end of 1993 from DuPont after more than 36 years of service. He was senior vice president for research and development and chief technical officer since 1986. In late 1994, he joined the U.S. Department of Energy (DOE) as deputy under secretary for technology partnerships and in 1995 was made deputy under secretary for R&D management. He left DOE in 1996 but remained on its Secretary of Energy Advisory Board, Laboratory Operations Board, Sandia President’s Advisory Council, and the National Renewable Energy Laboratory’s Advisory Council for several more years before resigning a few years ago. He has participated in several studies for the National Research Council, including The Hydrogen Economy: Opportunities, Costs, Barriers, and R&D Needs, Containing

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×

the Threat from Illegal Bombings: An Integrated National Strategy for Marking, Tagging, Rendering Inert, and Licensing Explosives and Their Precursors, Technology Commercialization: Russian Challenges, American Lessons, and Building an Effective Environmental Management Program: An Initial Assessment. Recently he was chair for the Committee for the Review of the Intelligent Vehicle Initiative. He is a member of Phi Beta Kappa and of the National Academy of Engineering. He is a graduate of Tufts University with a B.S. in chemistry and of MIT with a Ph.D. in physical organic chemistry. He currently serves as an outside director for the Millennium Cell Company, a company that strives to develop hydrogen source technology for fuel cells.


Gregory J. McRae is the Hoyt C. Hottel Professor of Chemical Engineering at MIT. His academic education includes a Ph.D. in engineering from the California Institute of Technology. Dr. McRae currently teaches undergraduate and graduate courses in process modeling, control, optimization, and computer-aided design. Another research focus is product and process design to avoid environmental problems and understanding the scientific aspects of problems involving pollutant transport and transformations in multimedia environments. His other interests include computational chemistry, process dynamics, turbulent fluid flow, computational fluid dynamics, reaction engineering, sensitivity/uncertainty analysis of complex systems, nonlinear parameter estimation, parallel computing, numerical analysis, and the design of cost-effective environmental controls. Dr. McRae is the recipient of numerous awards and prizes for his research in environmental and computational science, including the Presidential Young Investigator Award, the George Tallman Ladd Research Prize, the Forefronts of Computational Science Award, and an AAAS fellowship. He is a member of Sigma Xi, the American Chemical Society, and the American Institute of Chemical Engineers.


Michael L. Telson is presently serving as the director of National Laboratory Affairs for the University of California in its Washington Office of Federal Governmental Relations. He previously served as chief financial officer (CFO) of DOE from October of 1997 (after confirmation by the U.S. Senate) through May of 2001. He managed the relationship between DOE and the Office of Management and Budget (OMB), four congressional appropriations subcommittees, DOE’s Inspector General, and the General Accounting Office (GAO). He reported directly to Secretaries Pena, Richardson, and Abraham, advising them on all financial matters, including the preparation and execution of DOE’s nearly $20 billion annual budget, as well as reprogramming requests, in all of DOE’s business lines, including national security, science, energy, and environmental quality. As CFO, Dr. Telson directed a staff of more than 200, also covering a number of other activities, including project management oversight; strategic planning and the Government Performance and Results Act (GPRA); privatization (including the sale of the Elk Hills Naval Petroleum Reserve, the initial public offering of stock in the U.S. Enrichment Corporation, and several environmental management privatization projects); payroll; and financial statement issues. Before the DOE, he served as a senior analyst on the staff of the Committee on the Budget, U.S. House of Representatives. He was responsible for reviewing energy, science, and space issues in the federal budget, including the programs of the DOE, the NSF, and NASA, government-wide R&D policy, and certain user fee programs (including FCC spectrum auction issues). He also served as staff economist to the House ad hoc Committee on Energy created to enact the 1978 National Energy Act. Dr. Telson is a member of Sigma Xi, Tau Beta Pi, and Eta Kappa Nu. He is a fellow of the AAAS, as well as of the APS, and received the Meritorious Service and Superior Performance awards from Energy Secretary Richardson and the Gold Medal for excellence from Energy Secretary Abraham. In 2002, he was named a senior fellow of the U.S. Association for Energy Economics. He holds B.S., M.S., E.E., and Ph.D. degrees in electrical engineering from MIT and an M.S. in management from the MIT Sloan School of Management.


William J. Ward (NAE) is a retired research engineer, GE Research and Development Center, which he joined in 1965. For 10 years he worked full-time in membrane gas separations. In subsequent years he worked part-time with GE and other colleagues on membranes. He did pioneering work on facilitated transport in immobilized liquid membranes and on ultrathin polymeric membranes. The latter resulted in a medical oxygen enrichment appliance. Dr. Ward was a manager from 1976 to 1979, after which he resumed full-time research in catalysis. His catalysis work in the 1980s provided new understanding of, and a much-improved catalyst for, the chemical reaction that is at the heart of the silicone polymer industry. From 1990 until 1995, Dr. Ward worked on understanding and improving the performance of polyurethane foam insulation and on eliminating chlorofluorocarbons as foam blowing agents. From 1996 through 1998 he was the technical leader of a team that made another major advance in the synthesis of silicone polymers. In his last 3 years at GE he was involved in a successful effort to develop a manufacturing process to produce a ceramic metal halide lamp. After retiring from GE in 2000, he has consulted for GE and other companies. Dr. Ward has 29 publications in refereed journals and 40 patents and is a member of the National Academy of Engineering. He holds a B.S. from Penn State and an M.S. and Ph.D. from the University of Illinois, all in chemical engineering.

Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 208
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 209
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 210
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 211
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 212
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 213
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 214
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 215
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 216
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 217
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 218
Suggested Citation:"Appendix M Report of the Panel on DOE's Chemical Industrial Technologies Program." National Research Council. 2007. Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two). Washington, DC: The National Academies Press. doi: 10.17226/11806.
×
Page 219
Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two) Get This Book
×
 Prospective Evaluation of Applied Energy Research and Development at DOE (Phase Two)
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since its inception in 1977 from an amalgam of federal authorities, the U.S. Department of Energy (DOE) has administered numerous programs aimed at developing applied energy technologies. In recent years, federal oversight of public expenditures has emphasized the integration of performance and budgeting. Notably, the Government Performance and Results Act (GPRA) was passed in 1993 in response to questions about the value and effectiveness of federal programs. GPRA and other mandates have led agencies to develop indicators of program performance and program outcomes. The development of indicators has been watched with keen interest by Congress, which has requested of the National Research Council (NRC) a series of reports using quantitative indicators to evaluate the effectiveness of applied energy research and development (R&D).

The first such report took a retrospective view of the first 3 years of DOE R&D programs on fossil energy and energy efficiency. The report found that DOE-sponsored research had netted large commercial successes, such as advanced refrigerator compressors, electronic lighting ballasts, and emission control technology for flue gas desulfurization. However, some programs were judged to be costly failures in which large R&D expenditures did not result in a commercial energy technology. A follow-up NRC committee was assigned the task of adapting the methodology to the assessment of the future payoff of continuing programs.

Evaluating the outcome of R&D expenditures requires an analysis of program costs and benefits. Doing so is not a trivial matter. First, the analysis of costs and benefits must reflect the full range of public benefits that are envisioned, accounting for environmental and energy security impacts as well as economic effects. Second, the analysis must consider how likely the research is to succeed and how valuable the research will be if successful. Finally, the analysis must consider what might happen if the government did not support the project: Would some non-DOE entity undertake it or an equivalent activity that would produce some or all of the benefits of government involvement?

This second report continues to investigate the development and use of R&D outcome indicators and applies the benefits evaluation methodology to six DOE R&D activities. It provides further definition for the development of indicators for environmental and security benefits and refines the evaluation process based on its experience with the six DOE R&D case studies.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!