Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Put Discipline into Decision Making DOE-EM has spent billions of dollars to introduce technologies to remediate environmental problems at DOE's weapons complex sites and yet has continued to be subjected to strong criticism for lack of performance (i.e., few deployed technologies) (GAO, 1996, 1998~.2° BROOM reports have pointed to a lack of discipline in DOE-EM and OST's decision-making processes as a significant cause of this problem. In particular, We reports indicate Mat Me decision-making process often has been ad hoc and frequently has varied from site to site and from decision to decision. Unless decision making can be improved significantly, criticism likely will intensify. The findings and recommendations from these reports address three components of an effective decision-making process: (1) framing decisions, (2) making decisions, and (3) prioritizing needs and evaluating results. FRAMING DECISIONS The six reports emphasize two broad issues related to framing decisions about technology development: defining goals and considering alternatives. The reports provide three main examples of how OST's technology development 20 The Subsurface Contaminants Committee pointed out that DOE is not alone in its limited application of innovative technologies. For example, the committee cited an EPA estimate that innovative remedies have been selected for contaminated ground water at just 6 percent of all CERCLA sites as of 1995 (Subsurface report, p. 203~. The committee also noted that despite slow progress in deploying innovative remediation technologies, the SCFA has helped to develop a number of technologies that have shown considerable promise (Subsurface report, p. 11~. 22
PUT DISCIPLINE INTO DECISION MAKING 23 goals could be better defined: (1) through the use of an end state based methodology, (2) through consideration of waste forms, and (3) through the use of baseline functional flowsheets.2i The reports also emphasize the importance of developing alternative technologies and alternatives to baseline functional flowsheets. Defining Goals To be effective, a technology development program must begin by defining and specifying particular problems to be solved,22 rather than by developing a solution and then looking for a problem (D&D report, p. 24~. An important theme from the six reports is that such an approach had not been used by DOE- EM to manage its technology development activities. For example, the Tanks Committee concluded that DOE's process for screening and formulating technology needs lacks a systematic basis (Tanks report, p. 1~. Similarly, the D&D Committee found that the prospective uses of facilities that will undergo decontamination and decommissioning have not been defined adequately, and this has prevented OST from assessing the technology needs, cost, and schedule for D&D projects (D&D report, p. 3~. The committee found no evidence of significant progress in defying or even proposing goals to be targeted by new technologies (D&D report, p. 3~. End State Based Methodology Both of the aforementioned reports (the Tanks report and the D&D report) make frequent reference to the concept of "end states." As used in the Tanks report, "an end state can be expressed as the desired composition, configuration, performance, and location of a particular waste product at the completion of remediation activities, frequently wastes emplaced in a disposal facility. If the phased-decision approach previously recommended by the National Research Council (1996bJ were to be used, the end state my be that associated with the end state of one of the phases" (Tanks report, p. 15-16, emphasis added). As the italicized passage makes clear, within this conceptual approach an end state is not necessarily a final state. In other words, further phases with new end states may occur. 2} A baseline functional flowsheet is the sequence of steps that comprise the waste treatment process from the initial waste configuration to the final state. 22 Of course, the technology program must base these problems on technology needs identified by the sites (see discussion on "Input from Sites/Users"~.
24 TECHNOLOGIES FOR ENVIRONMENTAL MANAGEMENT The Tanks Committee recommended that DOE adopt a systems engineering approach for identifying technology requirements to remediate stored tank wastes (Tanks report, p. 7~. The recommended approach involves the analysis of remediation scenarios23 to identify the technologies needed to achieve specified goals. The committee concluded that such an approach would serve two fundamental objectives: (1) greatly facilitate the efficiency and visibility of EM's efforts to provide the technologies required to remediate high-level waste tanks throughout the DOE complex, and (2) clearly expose the underlying basis of the technology development program (including the ability to reach a prescribed end state in a cost-effective manner), which is critical to gaining public understanding and support for the program (Tanks report, p. 2, 6~. The committee argued that this can be done even though substantial uncertainties exist in the end states (Tanks report, p. 6~. The committee further suggested that such an approach should be generally applicable to any of the waste tank farms throughout the DOE nuclear weapons complex and possibly other DOE- EM problems (Tanks report, p. 1~. Waste Forms As an example of how OST should better define goals, Me board uses the results of Me Mixed Waste Committee. In its report, the committee evaluated Me state of development of waste forms for mixed wastes and found that currently available waste forms are adequate (sufficiently developed) to enable proper disposal of DOE's known and expected mixed waste inventory. These waste forms resulted mainly from Me intensive worldwide efforts and experience in developing waste forms for high- and low-level radioactive waste, and include grout, glass, ceramics, polymers, and compacted waste (Mixed Waste report, p. 3~.24 The committee therefore recommended Cat waste form development no longer be a primary focus of the Mixed Waste Focus Area (MWFA) (Mixed Waste report, p. 98~. Instead, the committee recommended Cat Me MWFA emphasize engineering design, integration, and scale-up of its proposed treatment processes and Weir demonstration and deployment at the DOE sites (Mixed Waste report, p. 98~. 23 For this report, a scenario is loosely defined as the sequence of events that takes wastes in their current status to desired end states. 24 The Mixed Waste report did not give explicit credit to new OST technology development. A similar conclusion was reached in the 1996 report: "The MWFA has determined that 90 percent of the mixed-waste streams can be treated with technologies that currently exist or that can be modified" (NRC, 1996a, p. 69~.
PUT DISCIPLINE INTO DECISION MAKING 25 The Mixed Waste Committee also concluded that conceptual design of treatment processes and waste form selection can proceed, but the upfront characterization is inadequate for detailed engineering design of treatment processes or process optimization (Mixed Waste report, p. 100~. To address this issue, the committee recommended that the MWFA: (1) develop simplified methods to characterize the waste up front, with emphasis on nondestructive examination and assay techniques; (2) continue to develop and implement techniques and procedures to ensure that all new waste streams are characterized adequately; and (3) strive for a balance between the effort and cost of up-front characterization and the effort and cost to develop more robust technologies (Mixed Waste report, p. 1004. The MWFA assigned improved waste characterization first priority in its list of technology needs. Flowsheets The Decision Making Committee discussed the use of baseline functional flowsheets (i.e., the sequence of steps that comprise the waste treatment process from the initial waste configuration to the final state) as fundamentally important to determining technology needs.25 The committee found that one major problem with the system has been that OST has had no direct role in establishing such flowsheets, which have been developed at the sites by the site remediation problem owners (Decision Making report, p. 75~. The committee concluded that the expertise of OST and its contractors could be valuable to the site problem owners in formulating and maintaining technically sound and practicable cleanup functional flowsheets and recommended that efforts be made to have substantial involvement of OST and OST contractors in reviews of functional flowsheets (Decision Making report, p. 75~. The committee argued Mat such OST participation would have Me dual benefits of (1) ensuring that OST technology developers fully understand Me sites problem owners' technical needs and Weir bases, and (2) increasing Me sites' confidence in OST's dedication and ability to meet their needs (Decision Making report, p. 75-76~. 25 For OST-developed technologies to be adopted at the sites, OST must persuade site managers and contractors to adopt different technologies from those they are already committed to use (i.e., its baseline technologies). OST therefore needs to undertake studies that compare existing baseline technology costs with more favorable costs of OST-proposed technologies (Decision Making report, p. 75~.
26 TECHNOLOGIES FOR EMlIRONMENTAL MANAGEMENT Alternatives In the face of technical uncertainty, sound decision making demands that multiple paths to achieve goals be explored. The importance of considering alternatives in technology development was emphasized in both the Tanks and Decision Making reports, as well as other recent NRC reports (e.g., NRC, 1996b). The Tanks Committee found that one major weakness of EM's approach to technology development for the Hanford Site tanks was a lack of consideration of end states other than those baseline scenarios codified in various site-specific compliance agreements (Tanks report, p. 6~. The committee concluded that scenarios involving alternative end states may need to be considered for such reasons as life cycle costs, technical failures, and delays in meeting schedules when originally selected end states present problems (Tanks report, p. 5~. The committee did recognize that such an approach would require a major change in DOE-EM technology development philosophy because alternative end states are outside the present plans of both the remediation programs and the technology development organization (Tanks report, p. 5~. Indeed it could require either renegotiation of some of DOE's current legally binding agreements, or legislation by the Congress. For this reason, recommendations related to alternatives (in the Tanks report and included in this report) are directed primarily to the Assistant Secretary for EM and the Secretary of DOE. The Decision Making Committee found that good decision making practices (e.g., hedging against technical uncertainty and insisting on alternatives) imply that DOE-EM should plan for alternatives to the site baseline functional flowsheets, especially when the baseline flowsheet involves high cost, high or poorly defined risk, and/or substantial probability of technical failure (Decision Making report, p. 76~. To address this issue, the committee recommended that EM seek out and acknowledge the potential vulnerabilities in cost, risk, and technological failure-of the baseline functional flowsheets and processes, and with OST's assistance, develop alternative flowsheets as appropriate. The committee recommended that OST encourage this course of action and seek to collaborate in it. In particular, the committee recommended that OST identify specific technology development opportunities aimed at supporting alternative functional flowsheets and processes designed to enhance the overall probability of remediation successes and to minimize program delays (Decision Making report, p. 76~.
PUT DISCIPLINE INTO DECISION MAKING 27 Recommendations DOE-EM should implement an end state based methodology26 (similar to that described in the Tanks report) to identify the technology needs and research and development required to achieve specific remediation goals. Such an approach would include the following characteristics: · DOE management and legislative decision makers should allow for consideration of a wider range of alternative end states that may be needed in the future, and this should be reflected in the DOE remediation and technology development programs (Tanks report, p. 5~. · Alternative end states and alternative scenarios should be considered for remediation scenarios that involve high uncertainty or high risk. · OST should be allowed to commit a portion of its resources to developing technologies to address needs derived from alternative functional flowsheets, in addition to developing technologies to meet the needs derived from the baseline flowsheets (Decision Making report, p. 76~. · If initial conditions cannot be adequately characterized and end states cannot be adequately specified, a plan leading to the timely resolution of open items should be prepared and executed. In the interim, assumptions that allow scenario development and identification of technology needs to proceed should be emplaced and clearly stated, preferably by problem owners, but by technology providers if necessary (Tanks report, p. 7). MAKING DECISIONS The Decision Making Committee found that, as OST's decision-making process has matured, it has functioned reasonably well to prioritize technologies for funding at individual sites and to make decisions for funding among sites within OST's operational framework. However, some of He process steps are 26 As noted previously, with this conceptual approach, the term "end state" does not necessarily connote a final disposition of the waste or waste sit; further phases with new end states may occur.
28 TECHNOLOGIES FOR ENVIRONMENTAL AL4NAGEMENT cumbersome and ill conceived. Furthermore, there has been little effort to apply a carefully considered process uniformly throughout OST (Decision Making report, p. 874.27 Other committees, including the Peer Review Committee, the Mixed Waste Committee, the Subsurface Contaminants Committee, and the D&D Committee, also analyzed parts of OST's decision-making process. Based on its review of these reports, the board has identified four main areas where OST can improve its decision-making process: (1) adequate documentation and communication of results; (2) use of independent, external expert review; (3) use of input from sites and other affected parties; and (4) increased transparency. Documentation and Communication of Results The importance of adequate technical documentation and communication of results has been emphasized in several recent board reports. For example, the Decision Making Committee noted that the type and quality of information provided to Congress and other interested review groups are critically important to OST. The committee recommended that OST ensure that the decisions underlying the technologies it develops are well documented and supported by sound technical reviews (Decision Making report, p. 79~. The Peer Review Committee commended OST's peer review program for modifying Me documentation required for peer reviews to focus on technical issues. However, Me committee pointed out Cat Me program does not require a detailed technical proposal or statement of work, and recommended Hat such a document be required for every peer review (Peer Review report, p. 10~. The Subsurface Contaminants Committee found Mat Subsurface Contamination Focus Area (SCFA) periodic technology summary reports were not sufficiently detailed to evaluate potential technology performance and effectiveness relative to other technologies, and recommended Hat such reports include well-documented 27 OST recently pilot tested a new system (its Work Package Ranking System) to create an integrated priority list of "work packages" (i.e., a group of related projects) at the OST headquarters level. The new process employs five criteria to rank the work packages: site needs, project value/need, future technology deployments, technological risks, and technology cost savings (DOE, l999a,d). In this new system, a numerical score for each work package is produced through a process stated to be based on multi- objective decision analysis methodology. Beyond this generality, the processes used to determine the scoring criteria, ranking factors, and weighting factors have not been specified. The Decision Making Committee noted that in principle, this prioritization process could be used to set budget targets for OST program units. However, the committee also pointed out that this system was not yet developed to a point where OST staff used it as the sole basis for prioritization (Decision Making report, p. 481.
PUT DISCIPLINE INTO DECISION MAKING 29 performance data, detailed cost estimates, design information useful to practitioners, and lessons learned (Subsurface report, p. 246~. The committee also recommended that a key future role for the SCFA be the development of design manuals for technologies that could be used across the DOE weapons complex (Subsurface report, p. 2164. In addition, the D&D Committee concluded that one factor that limited the acceptance of OST-demonstrated technologies was the lack of prompt dissemination of technical and cost data on these projects (D&D report, p. 2~. The committee recommended that the D&D Focus Area communicate its program results in a more effective manner (D&D report, p. 4~. Independent, External Expert Review The Peer Review Committee conducted a detailed evaluation of the peer review28 process that OST established in 1996 to perform independent technical assessments of technology projects. During the course of the study, OST continued to change its peer review procedures, partly in response to comments in the committee's interim report. The changes were intended to improve the peer review program and were generally found by the committee to be steps in the right direction. In particular, OST (1) revised its review criteria to focus on four technical issues; (2) developed a more systematic approach for selecting projects to be reviewed; and (3) modified its list of documentation required for peer reviews (Peer Review report, p. 6~. The committee also noted that OST senior management appears to be committed to this improvement process (Peer Review report, p. 7~. Based on this report and its contrast with prior critical reviews of OST's peer review efforts (NRC, 1996a), it appears that OST has made significant improvements In this program. Despite these improvements in the procedures used to conduct peer review, however, the Peer Review Committee identified several important issues Cat remain to be addressed: peer reviews are not being incorporated adequately into decision making (Peer Review report, p. 7~; OST is overwhelmed with a backlog of needed peer reviews (Peer Review report, p. 11~;29 for peer reviews to make a difference, new procedures, while important, are not enough- instead a culture change also is needed (Peer Review report, p. 13~; and unless OST uses peer reviews judiciously, it runs the risk of adding yet more reviews and paperwork, even as its budget continues to shrink (Peer Review report, p. 13~. 28 Peer reviews are defined as evaluations by technical experts who are independent of, and external to, the program of work being reviewed. 29 As of May 1998, only 43 of 226 active projects had been peer reviewed.
30 TECHNOLOGIES FOR ENVIRONMENTAL MANAGEMENT The committee made a number of recommendations to address these issues. First, OST program managers should be required to clearly identify the upcoming decision or milestone for which the peer review results will be used, before a project is chosen to be reviewed (Peer Review report, p. 7~. Second, prioritization should be applied to the selection of projects for peer review and how the peer reviews are conducted; for example, OST should consider adopting a triage approach30 to allow more effective in-depth reviews (Peer Review report, p. 11~. Third, OST leadership should develop a strategy to accomplish a change in its organizational culture so that OST staff recognize and accept the value of independent expert advice (Peer Review report, p. 14~. The Decision Making and Subsurface Contaminants Committees also considered the role of independent external review in OST's decision-making process. The Decision Making Committee concluded that independent expert reviews are a vital part of a credible decision-making process (Decision Making report, p. 76), and that peer reviews of technology projects should be part of OST's decision-making process. lathe committee also pointed out that the evaluation of technology development projects is but one step in OST's decision-making process (Decision Making report, p. 84. The Subsurface Contaminants Committee recommended that the SCFA significantly increase the use of peer review for determining technology needs and for evaluating projects proposed for funding (Subsurface report, p. 9~. Both the Peer Review and Decision Making Committees identified additional areas to which external independent reviews could be applied usefully. The Peer Review report included a detailed discussion on how peer reviews could be used to assess We technical merit of programs (Peer Review report, p. 46-47~. The Decision Making Committee recommended Rat OST use an external independent body to review Me bases of annual decisions Mat establish budget targets for OST program units (Decision Making report, p. 81. The committee also recommended Rat OST have a role in reviewing site remediation fimctional flowsheets (Decision Making report, p. 3~. Bow committees cautioned, however, that before adding any more reviews, OST should carefully assess Me purpose and value of the many reviews already being used (Decision Making report, p. 77; Peer Review report, p. 131. The Decision Making Committee pointed out Mat peer reviews of projects should not require an overly burdensome commitment of OST resources, and therefore 30 Such an approach involves a formal prescreening of projects by peer reviewers based exclusively on the project's written documentation. The results of this prescreening review could then be used to determine (1) highly ranked low-budget projects that should be considered for funding without additional review; (2) highly ranked projects that should receive a more detailed evaluation; and (3) technically weak projects that should not be considered for funding (Peer Review report, p. 11-12~.
PUT DISCIPLINE INTO DECISION ASKING 31 recommended that the peer review of projects be streamlined by reducing the number and types of reviews based on an analysis of the objectives being served by each (Decision Making report, p. 77~. Input from Sites and Users The necessity of incorporating input from sites and users in OST's decision-making process also has been a prominent theme in recent board reports (Decision Making, D&D, Mixed Waste, Subsurface reports), as well as GAO's recent report.3i OST has had limited success in getting the technologies it developed or procured deployed at the sites with cleanup problems. The Decision Making committee pointed out that this problem is due in part to conditions outside OST's control (see Chapter 5~. However, the committee found that the problem also was the result of the way OST operated in the past when it developed technologies without adequate input from site problem owners (Decision Making report, p. 9~. In more recent times, these site inputs have been obtained through Site Technology Coordination Groups (STCGs)32 (Decision Making report, p. 9), but the Decision Making committee found weaknesses in the STCG structuring of criteria and in the STCG evaluative and prioritization methods (Decision Making report, p. 3~33. To address this issue, the committee recommended that OST use the best available information on DOE-EM site technology needs as a guide to tailoring program goals and RD&D projects. As one way to acquire this information, the committee recommended that OST establish (or increased its direct contact with site personnel at the problem-solving and decision making levels (Decision Making report, p. 3~. The Decision Making Committee also concluded Hat longer-term technology needs be derived from OST's consideration of the fi=ctional flowsheets for site remediation Hat over DOE-EM offices already develop, use 3] In its evaluation of the extent to which innovative technologies developed by OST have been deployed, GAO concluded that lack of user involvement is one of the major remaining obstacles to more widespread use of technologies developed by OST (GAO, 1998). 32 OST has formed STCGs at each major DOE-EM site to interact with local contractor personnel and others to obtain that site's environmental restoration and waste management technology needs (Decision Making report, p. 21. STCGs are responsible for developing and prioritizing a list of site problems and technology needs based on the environmental management issues relevant to a specific site (Decision Making report, p. 105~. Each STCG evaluates and prioritizes (i.e., ranks or rates) technology needs according to a set of criteria established by the STCG (Decision Making report, p. 3~. 33 The criteria were different at each site and at some sites were not rigorously constructed (Decision Making report, p. 3~.
32 TECHNOLOGIES FOR ENVIRONMENTAL MANAGEMENT in their planning, and subject to reviews (Decision Making report, p. 3). At present, OST has no direct role in establishing or reviewing these flowsheets, which are activities conducted by other EM organizations and contractors at the site level. The committee recommended that, in conjunction with the other DOE-EM offices responsible for site cleanups, OST participate to the extent possible (e.g., by establishing a role for its contractors) in a review of site remediation functional flowsheets. OST's technology development projects should be responsive to technology needs identified from baselines remediation plans and their alternatives (Decision Making report, p. 3~. Similarly, the Mixed Waste Committee recommended that OST continue to address technology deficiencies identified by the STCGs (Mixed Waste report, p. 69~. The Subsurface Contaminants Committee found that a major barrier to deployment of SCFA's technologies is a lack of demand from individual DOE sites (Subsurface report, p. 7), and that one important factor in limiting demand for SCFA technologies (among other factors) is insufficient involvement of technology end users in setting SCFA's technology development priorities (Subsurface report, p. 9~. The committee recommended that the SCFA strive to increase the involvement of technology end users in planning the technology demonstrations it funds (Subsurface report, p. 12~. Transparency The Tanks Committee concluded mat although DOE has a participatory process for screening and formulating technology needs, this process lacks transparency (in terms of being easily understood by all concerned decision makers and other interested and affected parties34) (Tanks Report, p. 1~. The Peer Review Committee discussed at some length the pros and cons of conducting peer reviews in an "open" manner (i.e., identifying reviewers, fully informing Me public of Me nature of Me reviews, and employing a known process) (Peer Review report, p. 35-36~. The committee concluded Mat Me strengths of open reviews (e.g., enhanced credibility of Me process, Me potential for more constructive evaluations) far outweigh Me potential weaknesses (e.g., possible lack of candor by some reviewers when evaluating weak proposals), especially for Me peer review of projects or programs (Peer Review report, p. 79~. The committee emphasized Cat openness does not imply 34 "Interested and affected parties" can include members of the public, technology users, the affected state or tribal nation, and regulators. All of these should, on a selective case-by-case basis, have the opportunity to be involved in decisions concerning remediation technologies. See Understanding Risk: Informing Decisions in a Democratic Society (NRC, 1996c).
PUT DISCIPLINE INTO DECISION MAKING 33 Mat all deliberative sessions are held In public, however. The ability of an evaluating body to discuss frankly He merits and weaknesses of a project and to reach consensus In a closed session is an important attribute of some open reviews (Peer Review report, p. 361. The committee encouraged OST to continue to promote openness of its peer reviews and to fully inform members of Me public and others attending Be reviews of Heir nature (Peer Review report, p. 111. The importance of transparency to OST's decision-making process also was noted by the Decision Making Committee, which recommended Hat OST ensure that He decisions underlying He technologies it develops are well documented, traceable to customer needs, and supported by sound technical reviews (Decision Making report, p. 791. The board believes Cat such transparency should be a standard characteristic of much of DOE's decision-making processes. Recommendations . OST's decision-making process should be transparent; include participation from relevant interested and affected parties (such as the technology users, the surrounding community, the affected states and tribal nations, and regulators); incorporate adequate documentation; involve peer review; and lead to setting priorities. The decision points at which (1) budget allocations are made and (2) user-defined technology needs are established are important to shaping the OST program and are therefore opportunities for independent external review (Decision Making report, p. 81. Before adding any additional reviews, however, OST should carefully assess the purpose and value of the many reviews already being conducted (Decision Making report, p. 77, Peer Review report, p. 13). OST should use the best available information on DOE-EM site technology needs as a guide to tailoring program goals and RD&D projects. As one way to acquire this information, OST should establish (or increase) its direct contact with site personnel at the problem-solving and decision making levels (Decision Making report, p. 3~.
34 TECHNOLOGIES FOR ENVIRONMENTAL MANAGEMENT · OST should continue to address technology deficiencies identified by the Site Technology Coordination Groups, but prioritized among the sites by upper-level DOE-EM management. PRIORITIZING NEEDS AND MEASURING RESULTS There have been no general OST-wide guidelines applied among the OST programs for setting criteria for the selection and prioritization of technology development needs, although individual OST program units have developed their own guidelines (Decision Making report, p. 84~. The Decision Making Committee recommended that OST establish general selection and prioritization criteria, and guidelines for applying these criteria, to include allowance for instances when exceptions to the criteria may be appropriate (Decision Making report, p. 84~. Similarly, the D&D Committee recommended that OST and the D&D Focus Area (DDFA) develop and apply a consistent approach to comparative technology assessment across all projects (D&D report, p. 31~. The Decision Making Committee emphasized that the decisions underlying the technologies OST develops should be traceable to user needs (Decision Making report, p. 79~. However, this does not mean that all RD&D needs should be derived from user requests. For example, alternative technical approaches to site remediation baseline flowsheets are another important source of technology development needs (see "Alternatives" earlier in this Chapter). The committee therefore concluded that there is a need for exploratory RD&D to meet the needs of alternatives to baseline flowsheets (Decision Making report, p. 79~. Moreover, the Decision Making Committee emphasized that the general criterion that technologies should be applicable to multiple sites, while useful, is flawed when applied without exception because it may lead to a failure to develop technologies for potentially very important problems that exist at only one or two sites (Decision Making report, p. 84~. The Decision Making Committee also emphasized the importance of establishing quantifiable attributes and follow-up procedures to measure (and hopefully improve) organizational results (Decision Making report, p. 3-4~. Based on its review of private sector decision-making practices, the committee recommended that OST adopt relevant principles of private sector decision making, including the need to agree on clear and measurable goals and to measure and evaluate results as a guide to resource allocation (Decision Making report, p. 74. The committee found that OST's stage-and-gate system for monitoring ongoing projects has more development stages than necessary, and it does not seem to assist effectively in making decisions to select a project for funding or to terminate a project. The committee therefore recommended that OST use the minimum number of stages and gates needed to track a project and
PUTDISCIPL1NE INTO DECISION MAKING 35 use peer reviews at key decision points (gates), especially in Me selection of a new project (Decision Making report, p. 5). Recommendations . . . . For decisions involving the allocation of significant resources, OST should institute a decision-making structure wherein projects and proposals are evaluated against consistently defined criteria, such as project cost, probability of technical success, probability of implementation, potential cost savings, and human health risk reduction (Decision Making report, p. 801. To the extent practicable and with input from its various organizational elements, OST should establish general selection and prioritization criteria, and guidelines for applying these criteria, including allowance for instances where exceptions to the guidelines may be necessary (Decision Making report, p. 841. Although the technology development projects should be based primarily on specific site needs, some should be of an exploratory research nature (Decision Making report, p. 791. OST should establish measurable goals that can be used to quantify its success in meeting organizational objectives (Decision Making report, p. 7~.
