Prospective Evaluation of DOE Programs: A Preliminary Methodology
Prospective evaluation of R&D programs requires an assessment of program benefits and of the likelihood that the program is successful. The incorporation of uncertainty is the key methodological difference between this analysis and the previous NRC retrospective study of DOE’s FE and EE programs. We have divided the risks into several categories in an attempt to facilitate comparisons and calculations. The suggested methodology for considering program benefits builds directly on the analysis done for the previous NRC study, although, as is discussed below, a number of complications arise in a prospective context that are not critical for a retrospective analysis.1 As in the previous study, we use a matrix to organize the different categories of benefits and costs (Figure E-1). The matrix is presented here, followed by a discussion of the different matrix categories and other issues. This is very much a work in progress and we anticipate major revisions as the panels proceed to evaluate actual programs.
A. THE MATRIX
B. MATRIX DETAILS
A standard expected cost-benefit approach would consider the net benefits of different program outcomes in different states of the world, and take an expectation over all outcomes and possible states of the world.
Our proposed methodology disaggregates the expected benefits of a program into different components: probable benefits in different states of the world (the scenarios); different types of benefits; different types of risks. The disaggregation serves several purposes. First, we hope that the disaggregation aids in assessing the program components by providing a reasonably detailed list of the factors that enter an overall cost-benefit analysis. Second, some of the components are easier to measure (e.g., economic benefits) than others (security benefits). The disaggregation allows precise measures where warranted and avoids oversimplifications in the latter case. Similarly, greater consensus exists about the magnitudes of some risks (e.g., for program success) than others (potential states of the world). We hope to clarify debate over the value of programs by allowing common estimates where appropriate and isolating the less easily measured, or more controversial components of the calculations. Finally, a standard expected cost/benefit analysis equates a high-risk/high-benefit program with a low-risk/low-benefit program (even if the former should be further saddled with a risk-discount). The disaggregated approach clarifies whether a program is particularly risky or a relatively sure thing, and thus aids in assessing whether the DOE’s overall portfolio has a mix that is appropriate for a public applied energy R&D program.
B.2 Program Risks
Risks associated with the programs fall into two broad categories: the Probability of Meeting Program Goals and the Probability of Market Success.
The Probability of Meeting Program Goals
The Probability of Meeting Program Goals is the probability of meeting the specified goals with the resources expected to be employed during the life of the program. These resources include but are not limited to the technical, managerial, and financial support devoted to the project.
Guidance on Assessing the Probability of Meeting Program Goals. The probabilities should be expressed as a single number (a percentage) and should reflect the best judgment of the panel of experts reviewing the project. Consensus assessments are desired, but irreconcilable differences can be noted in the discussion.
It sometimes easiest to assess this probability by considering a series of technical hurdles that must be overcome to achieve success and assigning probabilities to these individual hurdles and then calculating the overall probability of meeting the program goals. For example, the goal of a research project may be to create a system that involves several components, all of which must work to specifications in order for the overall system to be successful. In this case, failure on any one of the components would condemn the overall program. In other cases, there may be a variety of potential technologies for achieving a program goal and success with any technology would ensure success for the overall program. Other cases may be a mix of these “series” and “parallel” cases. When calculating a probability for the overall program goal in terms of these hurdles or subgoals, one must take care to consider potential dependencies between the probabilities for the different hurdles or subgoals. The logic behind the assessment of the program goal should be discussed in the text accompanying the assessment.
An important component of this probability is the management structure of the program. This includes whether the program has the appropriate mix and timing of subcomponents: Is sufficient emphasis and time allocated for basic research? Is the plan flexible enough to account for a variety of outcomes in the early stages? Are the demonstration plans responsive to the outcomes of bench-scale work? Is the program hostage to possible failure of a risky “show-stopper” activity?
Guidance on Defining the Program Goals and Time of Completion. The Program Goal and Time of Completion should be defined so that after the time of completion, one can unambiguously determine whether the goal has been achieved. The unambiguous definition of the goal and time of completion is critical in the project evaluation.
For some programs, the official program goals may be very ambitious and involve large benefits, but the project is unlikely to be totally successful in meeting these goals. For example in the case of the original Partnership for a New Generation of Vehicles, the official goal—to create an automobile with many new component attributes—was viewed as a “stretch goal.” While this stretch goal provided overall focus and inspiration, the program participants understood that though the program was unlikely to produce such an automobile, it was very likely to produce some valuable technologies. Similarly, official estimated completion times are often optimistic. The program goals used in this study should balance the desire to have ambitious goals with large benefits against the desire to be realistic and have a reasonable likelihood of success.
The panel recognizes that complex programs often involve a continuum of outcomes, with associated probabilities of success and ultimate benefits. While a complete expected benefits calculation would integrate the product of program risks and program benefits over all possible outcomes, in this study we focus on a single program goal and time to completion in order to construct consistent and easy-to-understand matrices. The analysis in the matrix based on a single program goal would not recognize potential “upside” benefits if the actual goals are exceeded rather than met or potential benefits that would be obtained even if the stated program goals are not fully realized. Project teams are invited to consider potential upside benefits and benefits associated with incomplete success in the discussion accompanying the matrix.
The Probability of Market Success
The Probability of Market Success is the probability that the technical accomplishments of the program will be commercially used.
Guidance on Assessing the Probability of Market Success. For this assessment, it is assumed that the program technical goals are met. Factors to consider in making this evaluation might include but are not limited to the following:
Market Acceptance—If this is a new product, is it likely to be adopted by the market? Is there a potential for unforeseen pitfalls such as that which has occurred with genetically engineered foods or the use of MTBE in fuels?
Competition—Is there work by others on either the same or different technologies that might achieve the same or equivalent results sooner or cheaper? This would include both work by others in the United States or in other parts of the world. Is this an area of intense competitive activity or is this an area primarily left to the government?
Durable Need—Will the need for the technology at the time the program is expected to be completed still exist? Are there social, environmental, or other factors that might eliminate or reduce the need for this technology? For example, unexpected more-stringent environmental regulations on diesel emissions made obsolete fuel and engine research programs designed to meet anticipated more-modest objectives.
Potential Hazards—Are there any potential hazards
or public safety concerns that might limit the use of this technology? For example, are there any potential by-products that might turn out to be an environmental or biological hazard that might lead to unexpected treatment costs or liability exposure?
Ease of Implementation—Is implementation of this technology easy or does it require changing other systems? For example, it would be easy to substitute a new catalyst into an existing reactor. On the other hand, a hydrogen-powered automobile would require a new fuel distribution system making implementation more difficult, i.e. the “chicken and the egg” problem.
Complementary Technology—Does the implementation of the technology require successful outcomes in other research and development programs?
Capital Intensity—Does this technology require massive capital investment? Programs that result in highly capital-intensive outcomes are less likely to be implemented even if the benefits are positive.
The above comments are for guidance only. The choice of a value for the Probability of Market Success is ultimately a judgment of experts. The panel is expected to develop a consensus and document the factors that led to its assessment of Probability of Market Success. As with the Probability of Achieving Program Goals, irreconcilable disagreements may be noted in the discussion.
B.3 Market Scenarios
The retrospective study employed a matrix with three columns: realized benefits, options benefits and knowledge benefits. Here, no program yet has realized benefits, while the knowledge category is not appropriate as a primary goal of an applied energy program, although program components are expected to yield generic knowledge as well as particular technologies. The columns identified in this case instead expand the “options benefits” category employed earlier, as the programs are intended to yield benefits that vary with the future state of the world—indeed, sometimes different possible future scenarios drive different programs. Three scenarios appear to incorporate futures that are relevant to a wide range of energy programs: a base case; a world with substantially higher oil and/or gas prices, and a world where carbon emissions are deemed a significant environmental hazard in need of regulatory response.
These scenarios are meant to refer to actual states of the world, and not different policy regimes. That is, the “carbon-sensitive” scenario is where carbon emissions are determined to be a hazard, the “high oil/gas price” scenario is due to actual supply or demand conditions, and so on. However, the functional use of the scenarios requires some assumptions about policy responses. We have tentatively decided to use the assumptions employed by DOE’s FE group.
The usefulness of the analysis is enhanced when the categories are standard across programs. Some programs, however, may be intended to respond to other states of the world. The final “other scenario” column covers this possibility. It can describe the scenario that best justifies a program. The column need not be filled in if the program is intended to address one of the standard scenarios. If employed, the characteristics of the scenario should be described in full in a matrix appendix.
The panels should consider whether program risks vary by scenario.
B.4 Program Benefits
The benefit categories included in the retrospective study are appropriate for this analysis as well:
Economic net benefits are based on changes in the total market value of goods and services that can be produced in the U.S. economy under normal conditions … The total market value can be increased as a result of technologies because a technology may cut the cost of producing a given output or allow additional valuable outputs to be produced by the economy … This estimation must be computed on the basis of comparison with the next best alternative, not some standard or average value.
Environmental net benefits are based on changes in the quality of the environment that will (or may) occur as a result of the technology. These changes are possible because the technology may allow regulations to change or it may improve the environment under the existing regulations …
Security net benefits are based on changes in the probability or severity of abnormal energy-related events that would adversely impact the overall economy or the environment. (NRC, 2001, Appendix D)
Guidance on Assessing Net Benefits. Economic net benefits should be expressed in dollar amounts. Future economic benefits (and costs) should be expressed as the present value. Both costs and benefits need to be discounted. The OMB employs an annual rate of 3 percent to discount government research programs. This reflects a time-preference approach to discounting, which is appropriate when the public sector is investing in an activity that would not be undertaken by the private sector.
Environmental benefits can be expressed as quantities of pollution avoided. As is described above, a program will yield environmental benefits when its successful conclusion results in the government instituting a regulation requiring control of a pollutant addressed by the DOE technology. This may accrue to the program even when the technology has poor market acceptance due to the introduction of a less expensive competing technology. The nature of the benefit—facilitation of a regulation or actual deployment of the technology—should be specified in the matrix entry.
The panel considered whether a “five year rule” should
be employed in the period of time during which it is valid to count program benefits. The public program may introduce early technology, or accelerate private development, but in general would not substitute for private activities over long periods of time. However, some of DOE’s activities might justify longer investment-recoupment periods. If the activity is long-term, very risky, and provides a scientific base for technology (some sequestration activities may fit this description), the public program could accelerate technology deployment by considerably longer than five years. If a longer recoupment period is used, the panels should explain why private sector investment would be unusually retarded in the absence of the government program.
Another complication that arises in this study, as in the retrospective study, is allocating credit between the public and private sector. The issue is not materially different in the two cases: important new technology is virtually always the joint product of efforts of different parties. A retrospective study has the advantage of hindsight so that we can allocate credit with more confidence. The allocation needs to be done here as well, although, as with other benefits, on the basis of judgment rather than facts. Note that the allocation of credit is part of the benefits calculation rather than the probabilities of success.
Current energy security concerns center on two scenarios: the classic oil disruption and an electricity disruption. Either can lead to consequential economic disruptions, although the ultimate magnitude of the losses due to disruptions is currently the subject of controversy. We recommend that as an interim procedure a quantity rather than dollar approach be used to characterize benefits in this category, e.g., the extent to which disruptions are mitigated by the programs through energy conservation, reductions in electricity use, reductions in the need for networked transmission, or other activities.
C. BENEFITS ROLL-UP
A problem that arises in assessing RD&D programs comes from the interrelationship of different projects or programs for both technological and commercial success. The technology goals of a system may require success of multiple subcomponents; alternatively, the program may include several projects only one of which need be successful for the technology to work. The former case is usually characterized as “serial” projects and the latter as “parallel.” If we let p1 be the probability of project 1, and p2 the probability project 2 is successful, then the probability that the program is successful is
serial: p1 * p2 (The program succeeds when the projects succeed.)
parallel: 1 − (1 − p1)*(1 p2) (The program succeeds unless both projects fail.)
These are but two ends of a continuum: it may be that the program is most likely to succeed when both (or several) projects succeed, but is complicated, rather than doomed, by the failure of a program component.
A parallel problem exists on the benefits side. Consider programs that are independent technologically (for example: sequestration and wind power). When multiple programs address the same commercial market (alternate ways to generate electricity), and all are successful, the benefits may be identical to the case where only one of the projects succeeded. Alternatively, the magnitude of benefits of a program may depend on several other technologically independent activities. A famous non-energy example is the case of lasers and fiber optics: the value of the former was enhanced by many orders of magnitude when the invention of fiber optic cables enabled a telecommunications application.
As is discussed above, the value chosen for the probability of market acceptance can reflect these concerns. If alternative technologies are critical for market application, or if alternative technologies are under development for the same market application, then the probability of market acceptance, given achievement of technical program goals, will be reduced. Similarly, the probability of technical success can take account of the technological interdependencies described here. However, given the interest of the energy policy community in the DOE portfolio of projects, we recommend a separate calculation when the interdependent programs are within the DOE portfolio. Several options exist to deal with the interdependencies and we expect that these and others are appropriate in different circumstances.
First, projects with highly dependent technological attributes should be considered, if possible, as a single program. This may diverge in important respects from DOE’s organization, which reflects budgetary authority, historical trends, organization of activities among and within the DOE laboratories and a host of other activities. The need to consider disparate projects jointly underscores activities within DOE that would benefit from coordinated management.
Technologically dependent programs will typically have interdependent benefits. If the projects are included within a single program matrix, the latter can be correctly identified or reasonably approximated within the matrix.
Programs with interdependent benefits but technological independence need identification to allow a correct assessment of the DOE portfolio. Two options are to incorporate the dependence within the probability of market assessment, or allowing the probabilities and benefits to presume independence and note the interdependence separately so that matrices can be “rolled up” to allow assessment of the broader portfolio.