Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
6 Findings and Recommendations The prioritization and decision-making processes considered in this report are those used to fund RD&D in support of the DOE-EM cleanup and waste management missions. This chapter presents the major findings end recommendations. Because the committee believes that there are some important recommendations for effective OST decision making that require actions by levels in the DOE-EM organization above OST, the findings and recommendations are divided into two categories, general and specific. Many of the recommendations on general issues are ones that OST may not be able to address directly; nonetheless, the committee believes them to be important and they are included for possible action by DOE-EM. GENERAL FINDINGS AND RECOMMENDATIONS The recommendations that follow are offered to improve RD&D decision making in the current DOE-EM organization,' in which OST's purpose is to develop technology to be used by other EM offices, based on their needs. The recommendations below are generally designed to elucidate the technical needs of He user community for OST's use, to improve the connection between these needs and OST development goals and projects, and to enhance OST decision making with modern methods and good practices. OST efforts to implement these recommendations will, in the committee's view' help to address barriers (see Chapter I) and other current issues (e.g., the right performance measurers)) for the OST program. Importance of a Central RD&D Function ~-Dot ~ Since ends as well as means are important to any decision-making process, brief remarks on these issues are offered. The "ends" are appropriate RD&D activities; the "means" is the OST program office, which in Me current DOE-EM organizational structure is (~) separate from the other EM offices with te~hnoln~v needs and users, and (2) centralized in that it is coordinated Tom DOE headquarters rawer than from any EM site or user office. A legitimate issue is how effective a centralized RD&D function such as OST (as opposed to some over organizational structure) can be in funding technology projects that are developed and deployed on DOE-EM site cleanups and whose use provides benefits as measured in cost, risk, schedule, or other relevant criteria. The committee did not address whether other organizational structures were possible to accomplish RD&D in support of the DOE-EM mission, other than to note, as shown below, the potential advantages of a centralized RD&D fimction for the DOE-EM program. 73
74 Decision Making in the DOE-OST Finding The DOE-EM program faces many difficult technical challenges. Some pervasive cleanup problems (for example, some of those involving ground water contamination) cannot be solved in a practical sense with current technology. Other problems could benefit by technical solutions that reduce cost and/or risks to the public and the environment. Solutions to problems such as these and others may be provided by developing advanced technologies through a central RD&D organization that addresses more broader-based issues than those that might be addressed by a single site or a single management and integration contractor or its subcontractors. In DOE-EM, OST provides such a centralized RD&D function. Recommendation to DOE-EM A centralized RD&D function within DOE-EM should be maintained because of its potential advantage in coordinating potentially duplicative technology development efforts needed at DOE-EM sites and because it is in a better position to address important broader issues (e.g., alternative technologies in the baseline functional flowsheets and alternative functional flowsheets) than more specifically directed RD&D. Deployment of Technologies Finding OST has experienced a significant problem in increasing the percentage of its technologies deployed at the sites. The problem arose in the past for two reasons. The first is that in its early days, OST failed to attempt to solicit in a structured way technology development needs from problem owners on the site. Consequently, some OST programs were often perceived as not being relevant to site needs. Since it is fruitless to have an RD&D program without deployment of the technologies it develops, OST and EM headquarters have attempted to address this problem by establishing first the Focus Areas and STCGs, then the LSDPs, and in 1997, the ASTD program and the TAC. The second reason is that the sites lack strong incentives to use OST-developed technologies, regardless of agreements or understandings OST may believe it has with the sites. This second reason is outside OST's direct control and is addressed in the recommendation below. Recommendation to DOE-EM DOE-EM should continue to seek ways to assist OST in getting its developed technologies deployed at the sites. The ASTD program, which serves a useful function of facilitating the deployment of already demonstrated technologies at DOE-EM sites, is not directed specifically toward the technologies developed by OST. Thus, there is still a need for more aggressive action by DOE-EM in deploying OST-developed technologies, in addition to those available from industry. Recommendation to OST OST's role should be to develop technologies that address DOE-EM site needs. To engender deployments, OST should take the steps listed below to identify current and future DOE-EM site technology needs and should use this information as a guide for tailoring an RD&D program to meet these needs. Use of financial incentives such as those provided to the sites by the ASTD program are not desirable but appear to be necessary at the present time. They should be discontinued as soon as practicable because they commit OST funding that could be redirected with advantage to other OST initiatives. For identification of current needs, OST should continue to stress the involvement of the sites in OST technology development and procurement activities and, reciprocally, should involve itself
Findings and Recommendations 75 to the extent possible in the site activities and problems.2 For identification of needs likely to persist in the long term, OST should identify technology vuinerabilities (i.e., process steps with significant probability of technical failure) in baseline functional flowsheets and use this information to develop backup technologies and alternatives that have been agreed on through discussion with the site problem owners. Balancing Research with Development and Demonstration To have a balanced RD&D program, OST should support research-oriented activities that explore novel and innovative ideas, as well as more development-oriented activities directed at existing processes for specific applications. Research-oriented projects involve relatively high technical risk and are much less than 100 percent successful, if success is measured by large-scale application (i.e., deployment). Finding A balanced RD&D program will have a less than 100 percent success rate, if the measure of success is deployment in the relatively near term. Recommendation to DOE-EM The percentage of OST technologies that reach the deployment stage should not be the sole figure of merit used in judging the OST program, although it is an important one. A long-term view should be adopted wherein the direct use of OST technologies or the use of derived technologies is also considered in the evaluation of OST's portfolio of technology development projects. Site Baseline Remediation Functional Flowsheets Finding Decisions on site technology development needs are derived from the so-called baseline functional flowsheets developed at the sites by the site remediation problem owners. These flowsheets are fundamentally important in determining process steps and related technology needs. In effect they dictate the technology needs to which OST must respond with technology development programs. As a consequence of the current DOE-EM organizational structure, OST has no direct role in establishing the baseline functional flowsheets, which are developed by other EM organizations and contractors at the site level. At present, to influence decisions on technologies to be adopted, OST must undertake studies that compare existing baseline technology costs with more favorable costs of OST-proposed technologies and , . ~ _ . ~ · . . ~ · . , _ ~ ~ _ 1 _ 1 ~ 1 ~ ~ ~ ^1_ _ _ _ .1~ _ _ 1 ~ _ _ 1 _ in this way try to persuade site managers to adopt technologies oltterent from those they are already committed3 to use. Recommendation to DOE-EM The expertise of technology developers supported by OST could be of value to the site problem owners in formulating and maintaining technically sound and practicable cleanup functional flowsheets. Therefore, efforts should be made to have substantial involvement of appropriate OST and OST contractor personnel in reviews of fimctional flowsheets. Such participation 20ne practice in the past to enhance this involvement has been for the site user representatives of the Tanks Focus Area to endorse the consolidated, prioritized list of technology needs (see Appendix C). 3If these commitments are represented in legal agreements or in site contracts, opportunities to make decisions on alternative technology may be significantly restricted.
76 Decision Making in the DOE-OST would have the benefits of (1) ensuring that OST technology developers fully understand the site problem owners' technical needs and their bases and (2) increasing the sites' confidence in OST's dedication and ability to meet their needs. Technical Alternatives to Baseline Remediation Functional Flowsheets Finding Good decision-making practices (e.g., hedging against technical uncertainty and insisting on alternatives) imply that DOE-EM should plan for alternatives to the site baseline functional flowsheets, especially when the baseline flowsheet involves high cost, high or poorly defined risk, and/or substantial probability of technical failure. Uncertainties about the future, both in funding and those related to possible failures of existing functional flowsheet process steps, highlight the importance of developing technology alternatives. in addition, the possibility of failure of large-scale privatization contracts to achieve site cleanup goals reinforces the value of technology developments targeted at alternative functional flowsheets. At present, there is little or no DOE-EM funding for developing alternative functional flowsheets, and consequently, no OST activity is expended on technology development directed toward alternative fi~nctional flowsheet technology needs. Recommendation to DOE-EM The development of alternative functional flowsheets is the responsibility of DOE-EM offices other than OST, but they should seek OST's input. It is highly desirable that the problem-owning EM offices should seek out and acknowledge the potential vuinerabilities in cost, risk, and technological failur~of the baseline functional flowsheets and processes and, with OST's assistance, develop alternative flowsheets as appropnate. OST should encourage this course of action and seek to collaborate in it. Recommendation to OST OST should attempt to make input to alternative functional flowsheets and, in particular, should advocate their development when the baseline functional flowsheet involves high cost, high or poorly defined risk, and/or substantial probability of technical failure. OST should identify specific technology development opportunities aimed at supporting alternative functional flowsheets and processes designed to enhance the overall probability of remediation successes and to minimize program delays. In practice, this means that OST should be allowed to commit a portion of its resources to developing technologies to address needs derived from such alternative functional flowsheets, in addition to developing technologies to meet the needs derived from the baseline flowsheets. Tn(lependent, External Reviews Finding Peer review~hat is, reviews by technical experts who are independent of and external to the program of work being reviewe~iare a vital part of a credible decision-making process. OST programs and projects are subjected to many, often resource-intensive reviews but, according to recent NRC reports (1997b, 199Sb), to few true peer reviews. To address this shortcoming, in August 1996 OST established a peer review system using the American Society of Mechanical Engineers (ASME, 1997a) to review works in progress. The result is an OST program with many different reviews, which
Findings and Recommendations 77 are not fully effective in the important decisions of the initial selection and subsequent continuation of meritorious technology development projects. Reviews of Technology Development Projects Recommendation to OST Peer reviews of technology development projects should be part of OST's decision-making process. These project peer reviews should occur as necessary and in a way that is not an overly burdensome commitment of OST resources. Therefore, the OST review system should be streamlined by reducing the number and types of reviews based on an analysis of the objectives being served by each. Reduction in the number of reviews could be accomplished in part by combining reviews where practicable. Reviews of Program Budget Targets and Their Rationales Review of technology development projects is part of RD&D project monitoring and evaluation, which is but one step in the OST decision-making process (see Figure 4.~. The committee therefore considered whether other steps of the process merited review. Finding The formulation of budget targets for the venous OST program units (e.g., Focus Areas and Crosscutting Programs) is an important decision point in OST's process. At this decision point an independent external review is needed but is not now being undertaken. These funding targets are made by the OST upper management levels at headquarters based on input from DOE site managers and other OST program managers. These targets are proposed to higher authorities (and ultimately to the Congress) through the annual federal budget process. Recommendation to OST An independent, external review should be held on the basis of, and rationale for, decisions on funding targets within OST. One goal of this review should be to identify the technical areas of greatest need, where improvements over existing conventional approaches would have the greatest benefit to DOE-EM. This review and its outcome should take into consideration such factors as DOE-EM programmatic strategies, political pressures, stakeholder pressures, risk to human health and the environment, safety, cost-benefit, and timing. Such a review might be carried out by an already- constituted authoritative external body such as Me Environmental Management Advisory Board (EMAB), or by a group created specifically to conduct the review. OST Involvement in Reviews of Site Remediation Functional Flowsheets Finding The site remediation functional flowsheets, whose development and review are in the domain of the site problem owners, are important for OST because they define the user plans from which technology needs are derived.
78 Decision Making in the DOE-OST Recommendation to DOE-EM Site remediation functional flowsheets should be subjected to independent, external review before they are adopted, and periodically dunug development of the technologies that are to implement them. The EM offices developing these flowsheets should have them peer reviewed, that is, reviewed by technical experts who are independent of and external to the program. This expertise may be found in academia, private industry, and national laboratones. The purpose of such reviews is to identify possible vuInerabilities or uncertainties in the functional flowsheet assumptions and technology selections. the committee understands that the other bM ottlces already sponsor Such peer reviews of functional flowsheets for the most part, but it would recommend the practice for all important functional flowsheets. Recommendation to OST OST should try to work in conjunction with other DOE-EM offices to participate to the extent possible (e.g., by establishing a role for OST contractors) in the schedule of peer reviews of the site baseline functional flowsheets. The purpose of this OST involvement is to make the results of these reviews and the rationale for the technology development needs available to OST. This OST role in reviews of site baseline functional flowsheets will also, in the committee's view, help strengthen the interaction between OST and the DOE- EM user community. SPECIFIC FINDINGS AND RECOMMENDATIONS The recommendations presented below are within OST's purview and control to implement. This is not to say, however, that exigencies of the moment regarding such constraints as limited funding may not prevent OST from taking immediate action on some of them. OST's Institutional Environment Affects Customer Interactions, Relevance to Site Needs, and Deployment Finding OST operates in a complex, ever-changing, and politicized" environment. An important complication for OST is that the sites and the EM offices that are responsible for cleanup activities have a great deal of autonomy in selecting baseline remediation processes and technologies to deploy. Furthermore. theY are under no obligation to use OST-developed technologies, even if they have made , i, _ . .. . . . · ~ ~ _ ~ . ~ 1_ 1 _ ~ _ ~ 1 ~ ~ ~1~ ~ it: _ _ earlier commitments to do so. As new cleanup and waste management problems are Iouna al one sues, new technology needs arise. This creates a very difficult situation for OST in keeping abreast of technology needs and getting its technologies accepted and deployed. Recommendation To the extent possible, OST should increase its efforts to identify site technology needs on a current basis and to anticipate future needs. Regularly scheduled meetings with site problem owners should be considered. More discussions of technical issues and their implications for technology development needs should be held with the working-level scientists and engineers.
Findings and Recommendations 79 Finding OST has more than one "customer" to satisfy. Other EM offices such as EM-30 and EM-40 are obvious end users of OST technology developments, but the U.S. Congress must also be satisfied that a reasonable fraction of OST products are useful and worth their cost. Furthermore, parts of OST expenditures are congressionally mandated (U.S. Congress, 1994; 199Sa-b). Consequently, the type and quality of the information provided to Congress (and to other interested review groups) are critically important to OST. Recommendation OST should ensure that the decisions underlying the technologies it develops are well documented, traceable to customer needs, and supported by sound technical reviews. Records should be kept of the reasoning by which the deciding factors were evaluated, including whatever methodist were used in their evaluation. Finding Although the user focus is necessary, not all RD&D needs come from user requests. As an example, alternative technical approaches to site remediation baseline functional flowsheets are a potential source of technology development needs, and these alternatives would not be derived from user requests based on baseline functional flowsheets. Therefore, there is a need for exploratory RD&D with a shorter time frame than the Tonger-range basic research projects of the Environmental Management Science Program, to meet the need of alternatives to baseline functional flowsheets (see Chapters 3 and 5 and Appendix E). Recommendation Although the technology development projects should be based primarily on specific needs at the sites, some should be of an exploratory nature to meet the need for backups and alternatives to the baseline functional flowsheets. Top-Level Strategic Planning and Goals Fincling OST's strategic goals do not provide an adequate level of guidance for program managers as they attempt, in collaboration with users, to assign priorities to technology needs. In times of limited funding, it is especially important for managers to have the type of guidance and direction that come only from strategies developed by top OST managers, whose perspective is much broader than that of personnel further down the management chain. Recommendation OST managers, in conjunction with other top-level EM managers,4 should produce strategic goals and plans that define explicitly the technical problems that the program will (and will not) address, and use these goals and plans effectively within OST program units to assist them in making technology development decisions. Although EM "strategic goals" are now available (DOE, 1997j), these are obtained through a user- driven and therefore "bottom-up" decision-making process of other EM offices in which most remediation decisions are made at individual sites (DOE, 1998a; 1998b). Therefore, these strategic goals, 4Input from other EM offices is recommended since OST goals should be derived from user plans and needs.
80 Decision Making in the DOE-OST which target the sum of the work to be achieved at all sites in a bottom-up approach, differ from the top- level goals recommended here for OST that are specified by a centralized, coordinated, top-down process to drive the technology development program. Of course, any top-level strategic goals developed by OST should be consistent with the EM mission and be derived in concert with user plans and needs. That is, any such strategic goals for OST would be guided and constrained by priorities established within other DOE-EM offices, priorities that have varied over the years and would be represented by their top-level goals and other statements of their high-priority tasks. A particularly important area for current attention relates to the implications of privatization for OST's technology development program. This goal-oriented approach, to be successful, would have a significant impact on the way in which OST operates. Obtaining the "right" goals is a challenge, as is the orientation of program activities in support of these goals. For the goals to work, everyone in the organization that can have an impact on goal achievement must be measured on the success of reaching the goal (or at least the part of the goal that their job impacts). If these broadly written goals are in place and directions of important efforts can be derived from them, then this framework would provide a way to readily assess the value of each RD&D project. The committee chooses not to be prescriptive, but instead believes that OST management should be responsible for formulating its goals and criteria. These would include (~) adequate cleanup targets (e.g., level of decontamination to be achieved in various media, for use as program goals by Focus Areas and other OST program units in soliciting and selecting projects) and (2) a set of objective criteria to be used in evaluating technology development proposals and projects. This effort could solicit and use inputs from the users of technology (i.e., site contractors) and all interested parties. Use of a Structured Decision-Making Process The main reasons for a structured decision-making process are to ensure its comprehensiveness, objectivity, and consistency. Often, these qualities can best be realized when data and other information are expressed in quantifiable terms. Quantifiable inputs include not only measurements that are already in quantitative form (e.g., contaminant concentrations or dollar values), but also inputs such as expert judgment estimations (e.g., scores expressed as high, medium, or lows that can be scaled and converted into quantitative form. Both quantitative input (such as cost estimates) and expert estimates involve judgment; the aim of a quantifiable approach is to make these judgments as objective and transparent as possible. Finding Structure to the OST decision process is warranted, in part because of the complexity of OST's institutional environment and in part because of the potential for change over time in technology needs (which serve as the basis for the technology development projects that OST funds) and top-level · . . priorities. Recommendation For decisions involving the allocation of significant resources, OST should institute a decision-making structure wherein projects and/or proposals are evaluated against consistently defined criteria such as project cost, probability of technical success, probability of implementation on field applications, potential cost savings, and human health risk reduction. This structure should be applied broadly throughout the organization, with each OST program unit evaluating projects against the
Fir~clings arid Recommendations 81 same criteria. An important criterion would be the project's relevance to site activities, that is, all projects should have one or more specific objectives related to the cleanup and waste management goals at sites. For an ongoing OST project, its history of past perfo~ance in meeting appropriate developmental objectives would be useful information to gauge its likely future success. The quantities mentioned in the preceding paragraph are recommended to be estimated in order to provide more discipline and guidance to decision making than if these estimates were not done. It is a challenge to perform estimates well with the degree of rigor that is possible depending on the state of maturity of the concept. Initial estimates of early-stage development projects would likely be subject to data limitations, large uncertainties, and limited sophistication. Refinements to these calculations would be likely over time as a technology development project progresses and should be documented. The quantification of project benefits and probabilities of success is necessarily uncertain, but the committee believes that a rigorous attempt to estimate and use these quantities and other available data is preferable to the less structured methods that OST has used in the past. The recognition and evaluation of uncertainties is itself a key part of a decision process. This structured system is proposed as a useful decision support methodology to organize inputs that are used by the OST managers who make decisions. Any such decision support too! has limitations, particularly if it fails to account for all significant factors; 5 therefore, managerial flexibility to "override the numbers" in occasional situations is necessary. There should also be sufficient managerial flexibility to enable responses to late-breaking development opportunities. Industrial RD&D Decision-Making Practices Applicable to OST Finding Good private-sector decision-making practices include the use of formal decision-making processes that employ quantifiable, measurable goals and follow-up procedures such as those discussed in Chapter 3 for RD&D decisions. Recommendation OST should adopt, where applicable and appropriate in the OST environment and to the extent practicable, the basic principles of private-sector formal decision-making and follow-up practices that are presented in Chapter 3. In particular, an attempt should be made to assess the following factors and adopt them consistently where applicable across the entire organization: Understand, focus on, and monitor changes in customer needs and requirements. Agree on clear and measurable goals. Use a formal (i.e., common, consistent, structured, and rational) technology development decision making process and apply it uniformly. Think strategically (i.e., long-term and high impact). Measure and evaluate to guide resource allocation. sIndeed, because of the potential for complexity in Me various considerations relevant to a decision on technology RD&D, the phenomenon of "rank reversal" (Corbin and Marley, 1974; Buede and Maxwell, 1995; Saaty, and Vargas, 1993; Tversky and Simonson, 1993) is possible, resulting in a decision in which the outcome that is preferred is not the one that scores most favorably against all the criteria used in the ranking or rating exercises. The committee suggests that managerial flexibility be invoked to provide for this kind of situation (i.e., to make final decisions by choosing the options that do not necessarily have the highest scores as evaluated against the criteria used) in the absence of a more rigorous method.
82 Decision Making in the DOE-OST Communicate across organizational boundaries (i.e., with technology users). Continually Improve the research and development (R&D) management process. Hire the best people possible and maintain expertise. The committee prefers to leave to OST management the task of translating how these practices could best be adapted. For example, "hire the best people possible and maintain expertise" would refer in private industry to the cadre of researchers within the company; for OST, which is a collection of program units run by program managers who contract out the technology development work to PIs, the application of this concept might be best translated as "select the most competent contractors." The general recommendation is that OST seek to implement those practices that have been identified as improving decision quality. The support for this recommendation comes from published literature on broad surveys of company practices. To lend support to the validity of these decision-making practices for OST, the committee sought confirming evidence that they were useful concepts in organizations analogous to OST in their structure, institutional environment, and/or the kinds of technology within their environmental remediation RD&D programs. The industries cited in Chapter 3 and Appendixes G-l provide examples of institutions that have some similarity to OST. The institutions described there do not represent an exhaustive list of such analogous institutions, but were chosen as specific examples to illustrate the utility and practical application of the good decision-making practices and principles described in Chapter 3. These institutions should not be construed as constituting a representative sample of all institutions employing successful decision-making practices; many such institutions exist, some employing different (e.g., non-utility theory based) methodologies. Specific Methodologies Finding The quantifiable approaches used by OST (see Appendixes B-E) are simple multicriteria scoring techniques in which criteria are developed with relative weights assigned to them. A project or proposal is rated against these criteria by reviewers who assign numerical scores. The scores and relative weights are used to calculate an overall score, which is used for planning and management purposes in comparing various projects or proposals. This simple scoring technique is one example of multi-attribute utility analysis. Utility analysis has its strengths but also its weaknesses. For example, utility analyses can be criticized on several accounts, such as I. the inability to formulate a meaningfully weighted utility function that adequately synthesizes many, often disparate, "customer" views (Zeleny, 1982~; 2. the possible interdependence (in practice) of criteria that, in utility analysis treatments, are treated as independent (Machine in Stigum and Wenstop, 1983~; 3. the problem of expressing preferences for these criteria in ways that cannot be adequately captured by numerical scales of numbers that are added or multiplied together (e.g., translation of measurement scale values, such as those of ratio scales, into interval scale representations) (Saaty and Vargas, 1993~; 4. the combination of tangible and intangible factors (an example of the latter is the valuations applied to expert judgments), especially in hierarchical prioritizations (Saaty and Vargas, ~ 9981; and
Findings arid Recommendations 83 5. the possible violations of the "principle of invariance," also known as regularity, the independence from irrelevant alternatives, and "rank preservation," by which is meant that the rank order of selections may change as new criteria are added or old ones deleted (Tversky and Simonson, 1993; Tversky, Slovic, and Kahneman, 19904. Other decision-analytic methods, such as the analytic hierarchy process, have been advanced as alternatives to utility analysis that treat these complicating issues in a different way (see, for example, Saaty, 1994; 1996~. The issue for OST applications is which of all possible decision methodologies is adequate and convenient to use for making the requisite decisions. One way to answer this question is to judge whether a method's shortcomings, such as those enumerated above, are important "fatal flaws" that prevent the best decision from being made. The committee makes no finding on a relative comparison of various decision methodologies, or which is "best" for OST, but would instead offer the following recommendation: Recommendation OST should examine the efficacy of the sets of criteria and scoring techniques currently used by OST program units (e.g., STCGs, Focus Areas, and Crosscutting Programs). This could be done by (~) using one or more contractors with suitable expertise to survey alternative decision- making analytical methods and (2) using the considered judgment of OST management to identify those analytical methods that are well-suited to OST's various needs. The judgments for OST management are whether, for OST decisions, the shortcomings of any decision methodology are important practical concerns and whether the decision methodology provides a useful organization of input information to help guide a decision among several candidates. The current OST quantitative schemes are multi- attnbute systems with criteria and relative weights, and rely on expert (e.g., reviewer) judgment to score projects and project proposals against the criteria. in practice, the recommendation offered above would evaluate whether this process is sufficiently adequate or whether an alternative process would provide a more useful output. For example, one way to minimize "rank preservation" problems (issue 5 above) is with the use of pairwise comparisons of alternative proposals (i.e., rank ordering them by having reviewers compare them using their judgment, experience, and knowledge) rather than rating each proposal against a set of criteria and comparing the numerical outputs of the rating and scoring exercise. One such comparison of multi-attribute value theory and the analytical hierarchy process methodologies showed that the decision outcome depended to a lesser extent on which methodology was used and to a greater extent on issues associated with the way the problem was structured and valuations and weighting factors were elicited (Buede and Maxwell, 19951. As noted above, the general recommendation of this report is for a method with structure, documentation, and quantifiable attributes, without a recorded preference on which specific method with these attributes OST should adopt. The recommendation is primarily that decisions be structured, by which is meant that the goals, factors, and criteria believed to influence the decision should be clearly specified. Records should be kept of the reasoning by which the deciding factors were evaluated, including whatever method(s) were used in their evaluation. Most decisions on the selection and continuation of any research and development project are revisited several times (i.e., at least annually), particularly if the project is funded during more than one fiscal year. Record keeping would support systematic refinement of decisions as more information is developed and would provide the basis for "lessons learned" to help improve decision quality.
84 Decision Making in the DOE-OST Summary A major goal of this report is to outline the institutional environment in which OST operates, as the "macrolevel" context within which OST prioritization and decision making occur. To suit this context, a decision method is proposed in a previous section of this chapter that offers structure, quantifiable attributes, and documentation. A "microlevel" recommendation is not offered on which of any of the specific methods available in the literature to meet these general criteria is the best for OST to select and adopt. Rather, a familiarity with various decision-making approaches, accompanied by the considered judgment of OST management, is recommended to direct the selection and adaptation of specific decision-making methods. Project Selection and Evaluation Criteria Finding There are no general, OST-wide guidelines for setting criteria for the selection and prioritization of technology development needs, although individual OST program units have developed their own guidelines. Moreover, the general criterion that technologies should be applicable to multiple sites, while useful, is flawed when applied without exception because it may lead to the failure to develop technologies for potentially very important problems that exist at only one or two sites. Recommendation To the extent practicable, and with input from its various organizational elements, OST headquarters should establish general selection and prioritization criteria, and guidelines for applying these criteria, to include allowance for instances in which exceptions to the criteria may be appropriate. Procurement of Externally Demonstrated Technologies Finding The present approach to technology procurement wherein both OST's Industry Program and other OST organizational units such as the LSDPs perform some aspects of technology selection and procurement from industry is cumbersome and duplicative, and impairs OST's deployment initiatives. Recommendation A better-coordinated, less duplicative, and less cumbersome system should be established for integration of technology procurement activities. Since procurements should be made only if warranted in a make-or-buy decision, the ability to assess available technology is crucial. These assessments should be done, by the use of a comprehensive database of demonstrated and commercially available technologies. The purpose of these comments is to engender a coordinated program in which information flow is effective and cumbersome or duplicative elements are reduced.
Findings and Recommendations 85 Project Monitoring Finding OST monitors technology development projects in several different ways, depending on the specific project. One established formal process that OST has adapted and used is the stage-and-gate system. In this system, funded projects are reviewed as they progress through their various stages of development from early exploratory studies to engineering development and finally to deployment. At the end of each stage there is a gate, which is in actuality a decision point on whether to proceed to the next level of technology development. However, the stage-and-gate system does not help to critically review the initial decisions to select projects for funding and entry into the tracking system. In addition, the system has an excessive number of stages and gates to use them all for review points in the OST application, and the scoring system has some flaws. Recommendation OST should use the minimum number of stages needed to track projects. This will reduce the administrative load and will lead to better decisions by producing better-defined decision points and clearer lines of demarcation between them. Recommendation In selecting a new technology development project for funding, OST should base this decision on both technical merit and quantifiable estimations of the project's probable value to site cleanup activities. OST has developed this latter concept as part of the criteria of the stage-and-gate system, but OST program units do not uniformly adopt and use these criteria to guide their selection of new projects for Finding. Recommendation OST should correct the additive scoring system to account better for threshold criteria. One way to do this is to multiply scores in keY categories rather than add them. , O . Cost Estimates Finding OST has attempted to gain acceptance of new and/or improved technologies by showing cost savings. However, it is difficult to assess the claimed savings because they require knowledge of both the cost of the baseline technology to be replaced and the cost of the OST technology. Often the baseline technology cost estimates, if they exist at all, are uncertain. The same thing is true of the proposed replacement technologies. Nonetheless, these cost comparisons can be an important decision- making criterion and can serve a useful purpose if they can be trusted. Uniformity in cost calculations (both information and methodology) is a need that has recently been acknowledged by the Decontamination and Decommissioning Focus Area through its use of the U.S. Army Corps of Engineers as an independent, external organization to perform cost calculations. However, further steps are needed to establish the credibility of cost estimates for OST technologies. Recommendation OST should do cost avoidance or ROT calculations on its more expensive technologies in a more uniform and credible manner than in the past and should communicate results to the potential technology users in the most effective way possible. Initial estimates of costs and benefits should be developed at the inception of large RD&D projects, and periodic refinements of the estimates should be a part of the project as it progresses.
86 Decision Making in the DOE-OST Exoloratorv Development v Part of the decision-making process is the gathering of technology needs. The recommendation above on using site remediation functional flowsheets, including consideration of the baseline version as well as alternatives, to best derive these needs will lead to related technology development opportunities. Finding A iSraction of OST's budget is devoted to exploratory research through the EMSP, which is a long-range basic research program that stresses research related to environmental cleanup problems (discussed further in Appendix E). Although this program is important, it is not sufficient to meet the need for exploratory development studies related to alternatives to the baseline functional flowsheets. These studies are intermediate in timing between the long-range time frame of EMSP projects and the immediate need for work on existing functional flowsheet technologies. -~7 Recommendation Additional funding should be sought (or some existing funding redirected) for exploratory development that is directed to technologies for alternative functional flowsheets. Because exploratory development is recommended, the issue is raised of how best to fund and manage such efforts. One relatively straightforward way to fund small-scale, exploratory projects in a manner similar to the ways in which some OST funds (snecificallv. the research oDDortunitv i- ~ ,# , ~ ~ announcements and the EM science Program tunas; see Appendix E) are dispersed and managed at present is to set aside some uncommitted funds (e.g., a fraction of the total OST budget) in a program that disperses them by having an open solicitation to which prospective PIs can respond, with frequent (e.g., quarterly) reviews of submitted proposals by teams of independent reviewers assembled for this purpose. An approach of this kind can capture some of the key features of exploratory decision making in a mechanism that will work in the federal budget system. The recommendation is for OST to adopt a suitable method that will disperse uncommitted funds in a reasonably timely and well-defined way for exploratory development activities. CONCLUDING PERSPECTIVE Consideration of OST's decision-making process begs the issue of what decisions are and should be made, the answer to which requires consideration of the role and fimction of an RD&D organization such as OST. There is a role for RD&D activities in providing economical, effective, acceptable, and practicable technologies for use in DOE-EM site cleanups. Although OST accounts for a small part of the DOE-EM budget, its work can have substantial and beneficial impacts on reducing the costs, risks, and probability of technical failure associated with environmental remediation activities, which are estimated to represent in excess of $100 billion dollars. OST is an RD&D office within DOE-EM that is separate from the other EM offices that bear responsibility for addressing cleanup problems. This organizational structure is a reasonable one for conducting RD&D and is mirrored in utility consortia such as the GR! and the EPRI, in private industry, and in the Defense Advanced Research Projects Agency of the Deparunent of Defense. Although the findings and recommendations presented in this report are directed to OST, they could, in principle, be applied to any RD&D office that manages project work in support of a broader federal program, particularly if organizational similarities to OST and DOE-EM exist.
Findings and Recommendations 87 RESPONSES TO ISSUES IN STATEMENT OF TASK The statement of task for this study asked the NRC study group "to evaluate the effectiveness of the OST decision-making process and make specific recommendations to improve it, if appropriate." To accomplish this task the pane! was asked to address four points. The material to address these points has been presented in the preceding chapters. This material was used to provide input to the responses to the four points, which are reproduced below. I. The appropriateness and effectiveness of decision-making processes currently in use by OST to select, prioritize, and fund RD&D activities, both at sites and at headquarters: The current OST decision-making practices, which change with the passage of time, are embodied in its program structure. The program structure of OST's core RD&D functions is based primarily on STCGs, Focus Areas and Crosscutting Programs, and a headquarters oversight activity. A summary of the elements of this program structure is presented in Box 2. ~ . The process functions reasonably well to prioritize technologies to propose for funding at individual sites and to make decisions for funding among sites within the framework in which OST operates. However, as pointed out in this report, some of the process steps for example, technology prioritization by the STCGs are cumbersome and ill-conceived, and Here has been little effort to apply a carefully thought-out system uniformly across OST. This leads to difficulties in comparing and evaluating priorities across the diverse technology areas that OST's activities encompass. Also, the stage-and-gate process for evaluating technology developments at various stages of maturity is unnecessarily complex. More direct input by OST to functional flowsheet review at the site level is desirable. However, a major point made in the report is that key decisions about which technologies to support lie outside OST's direct control. Input to technology and functional flowsheet selection at the sites is one of these key decision areas. Another critical point at which peer review is needed and is presently absent is at headquarters where final funding decisions are made. 2. The technical factors appropriate to consider in the decision-making process for selection, prioritization, and the development of cleanup technologies, and the adequacy with which these factors can be measured: The committee has identified three technical factors to consider: (~) the degree of maturity of the technology, (2) the ease of integration of the technology into a total system, and (3) the impact of the technology. The degree of maturity of a technology is represented bY a developing technolo~'s remaining --I -- - ~ , ~ __ _ assumptions and uncertainties that require testing, and can be measured in terms of the estimated time required to bring it to a sufficiently large-scale (e.g., pilot-scale) demonstration. It must be recognized that OST is not constrained to use technologies that are mature and, in fact its, greatest contributions may be in the development of new technologies. This report points out that OST should earmark a part of its funds for exploratory development of new technologies that have easily recognizable applications. The stage-and-gate process is designed to assess the maturity and likelihood of success of a developing technology and to terminate technology developments found to be lacking promise of successful completion or application. The ease of integration of the technology into a total system is a question that cannot be answered solely by considering OST and its decisions. This integration depends strongly on the attitudes and
88 Decision Making in the DOE-OST constraints of the sites with respect to what they are willing to accept or what they can accept because of regulatory controls and agreements that may have been made with state or other influential bodies, as has been stressed in this report. These factors often control what technologies can be integrated in cleanup systems and therefore represent valid consideration in RD&D decision making. The intended impact of a technology is to reduce costs, schedules, and/or risks of various kinds, and/or to enable a difficult or intractable task to proceed. These intended beneficial impacts imply tradeoffs; they are not necessarily mutually reinforcing. One way to assess these impacts is to measure the number of functional flowsheets needing that technology, the number of times it is needed in each flowsheet, and the importance of the flowsheet need addressed by the technology. The practice of making quantitative assessments of this kind would lessen the need for OST to rely upon guidelines as a basis for decision making. For example, one of the most important guidelines OST uses to decide on funding technologies is their applicability at more than one site. Although the committee agrees that this is indeed an important consideration, there may be instances when a technology is very important for a single site. The Quantification of technology impacts would enable an informed decision to be made in 1 C7 ~' -- - -- - ---- - - - ---- - - - - -- - - -- - .. .. . .. . . .. . . . . . , . , . .. Am, .. . . . .. . . . estimating whether a smgle-slte technology nac greater potential nene~lt than a technology that nac uses at multiple sites. 3. Recommendations, if appropriate, for improving the decision-making process: Recommendations are addressed in several ways in this report. Specific recommendations are given above in this chapter, separated into general considerations and those specific to the current OST decision-making process and program structure. Chapter 5 presents general principles derived from the private sector that, if followed, will lead to improved decision making. As stated in Chapter I, the committee did not undertake to prescribe or recommend one methodology or school of thought in decision making over any other in its study. 4. The role and importance of effective peer reviews in the decision-making process: The committee recognizes the crucial role played by peer review at appropriate points in the decision- making process, and subscribes to the process and opinions described in earlier NRC (1997b, 1998b) reports. These reports also acknowledges that OST is subjected to many reviews (many of which have not been carried out in the manner recommended by NRC, 199Sb) and suggest that the number be reduced by combining reviews whenever possible. The DOE-EM RD&D decision-making process has two key decision points other than technology development project reviews that are opportunities for reviews. These reviews, although not true peer reviews in the sense of NRC (1997b, 1998b), nevetheless can embody many of the features of peer review. This report recommends (~) a "policy review" at the headquarters level where budget targets are formulated for the various OST program units, and (2) OST participation in (and/or access to) external reviews of the baseline flowsheets developed by the other EM offices. ----I r-~J
