Click for next page ( 12


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 11
1 Introduction Decisions that any technology-based organization makes for research, development, and demonstration (RD&D) result in allocations of resources to achieve certain ends. The decision-making process is the ongoing multiparty conversation that leads to those allocations. Better decision making can contribute to better use of an organization's resources in many ways, including better coupling to "customer" needs, better selection of high-productivity options, faster achievement of results, and more successful deployment of results. The quality of decisions can always be refined, with improvements sought at various conceptual levels: philosophical, process, and procedural. To do this for the decision-making process, like other management processes, it is necessary to define the process, He players, and the steps involved. Decision processes provide structure to the evaluation that is needed when l. desired results or goals are defined, 2. alternatives to meet these goals are identified, 3. relevant information on alternatives is collected and made available, and 4. values are applied, as in evaluations using prioritization criteria or practical experience, to select among the alternatives (see March, 1999~. A successful decision process results in the development of decisions and the execution of efforts that support an organization's goals. Decision making in simplified situations (e.g., bounded by clearly defined goals, limited alternatives, and limited access to applicable information) may not require much formality or structure, in which case informal decisions (such as those made by an individual's subjective judgment) may be adequate. However, He degree and frequency of success for informal decision making diminishes sharply the further the circumstances depart from ideal or simplistic situations. Large organizations such as government agencies customanly deal wid1 complex situations, uncertainties, long time honzons, and a range of stakeholder1 concerns. Such organizations would therefore stand to benefit Tom sound, well-defined, structured decision-making processes that direct the collection and use of information, allow for transparency and stakeholder access, permit the use of medics to assess and redirect efforts over time, iA "stakeholder" as used here is anyone with a stake in the program or process involved, that is, an "interested and affected party" (NRC, 1996d). For DOE-EM, the U.S. taxpayer is in principle a stakeholder, but the term is commonly used to denote constituencies local to any of the DOE-EM sites. 11

OCR for page 11
I2 . learned," and Decision Making in the DOE-OST provide the opportunity to conduct retrospective analyses to elucidate and apply any "lessons address alternatives as a hedge against failure. The complexity of the Department of Energy's (DOE's) RD&D decisions and the context in which these decisions are made warrants the use of formal procedures. However, some managerial flexibility is warranted in fielding decisions, particularly in response to late-breaking development opportunities, since the nominal Office of Science and Technology (OST) procedures of assessing technology needs and initiating suitable development projects are managed on an annual time frame to correspond with federal budget provisions. The DOE's Environmental Management (DOE-EM) program has RD&D needs- to address radioactive waste management activities, the long-term disposition and dismantlement of nuclear facilities with no further mission, and the environmental cleanup of contaminated DOE sites. The Office of Science and Technology within DOE-EM fiends RD&D projects on technologies having application to these waste management and remediation challenges. The complexity of OST decision making stems in part Tom the diversity of technical areas and in part from its institutional environment and other considerations discussed later in this report. PURPOSE OF THIS REPORT This report examines the decision-making processes of the Office of Science and Technology within the DOE-EM program. Several National Research Council (NRC) reports have identified issues associated with He DOE-EM program and with OST in particular (see Appendix A).2 The nature and quality of the decisions made in allocating resources to RD&D activities is a topic that has been addressed in those reports. At the request of DOE, the NRC undertook the present study "to evaluate the effectiveness of the OST decision-making process and make specific recommendations to improve it, if appropriate" (see statement of task in Box ES.~. As is customary in NRC studies, committee members did not represent the views of their institutions but formed an independent body to author this report. The purpose, scope, and activities of OST and DOE-EM are discussed briefly below. DOE-EM AND OST The DOE's Office of Environmental Management was established in November 1989 to address environmental contamination and other cleanup problems resulting Dom nearly 50 years of nuclear weapons production, reactor fuel and target rod processing, and RD&D activities at DOE sites. One of several program offices within EM,3 the Office of Science and Technology (commonly referred to by its "mail stop" 2Appendix A summarizes the findings and recommendations of past NRC reports that relate to decision making within DOE-EM. They are mentioned here to show the statements that have already been made on the subject. In general, this report is in agreement with the essential points made in the earlier reports and is a natural extension of them. The findings and recommendations offered in this report are intended to augment them by a more in-depth examination of OST than these earlier studies had the opportunity to conduct. slather separate EM program offices are the Office of Waste Management (EM-30), the Office of Environmental Restoration (EM40), and the Office of Nuclear Material and Facility Stabilization (EM-60~.

OCR for page 11
Introduction 13 as EM-50 and known prior to 1996 as the Office of Technology Development) supports project work in basic environmental science, risk assessment, technology development, and technology deployment. OST has no environmental liabilities of its own; hence it is not directly subject to compliance requirements or cleanup agreements. OST has been the object of reviews and scrutiny by several entities interested in its value and accomplishments (GAO, 1992; 1994; 1996; 1998; Holt and Day, 1997; NRC, 1996b; Rezendes, 1997 Suries, 1997~. In particular, questions have been raised about how to assess the OST program with respect to its return on investment and whether the return is worm the program's cost. The answer to these questions resides in the nature and magnitude of the contributions made (and a projection of contributions that will be made) by OST's technology developments to actual site cleanup and other DOE operations. Analysis of OST's performance requires that records be kept of expenditures made and technologies deployed at sites and that some credible approach be devised for evaluating the actual and potential contributions of OST's technologies. Because technologies are often slow to mature and their applications may come about over many years, meaningful present-day evaluations of their worth are difficult to make. Nonetheless, at least some products of a truly worthwhile technology development program should have some acceptance and impact on DOE- EM site cleanups in the relatively short term. If the decision-making processes employed by OST for selecting the waste management and cleanup technologies it supports (and for encouraging their development and deployment) are sound and are embedded in a sound framework of good decisions by its parent organization, then OST can play a valuable role In site cleanup. OST PROGRAM UNITS FOR RESEARCH, DEVELOPMENT, AND DEMONSTRATION Since 1989, OST has selected, funded, and managed RD&D projects whose successful completion and deployment should result in improvements to the technologies underpinning DOE-EM cleanup work. These activities have been carried out in several OST program units: four Focus Areas; three Crosscutting Programs; an Industry Program; a University Program; a Technology Integration Systems Application (TISA) International Program; and an Environmental Management Science Program (EMSP) (see Table A.. The names of the four Focus Area programs (Subsurface Contaminants, Mixed Waste, Decontamination and Decommissioning, and Tanks) and the three Crosscutting Programs (Efficient Separations and Processing; Robotics; and Characterization, Monitoring, and Sensor Technology) represent seven major categories in which technology needs and corresponding development work exist. OST managers in each of these program units make decisions, based on an assessment of priorities, to initiate and to continue (or terminate) finding technology development projects. The result of these decisions is a suite of technology development projects intended to address DOE-EM cleanup and waste management problems. In addition to the Focus Areas and Crosscutting Programs, OST inaugurated the Industry Program to enhance technology input from private industry; the University Program to establish research and development centers at a few select universities; and the TTSA International Program to involve foreign cleanup expertise, especially that of the former Soviet Union. In 1995, the EMSP was mandated by Congress to ensure that a fundamental knowledge base would be developed and maintained from which technologies could be generated to meet the longer-range DOE site cleanup problems for which there may not now be efficient, effective, economical, and/or acceptable solutions.

OCR for page 11
14 TABLE I.1 Major OST Program Units in FY 1997-1998 OST Program Unit Decision Making in the DOE-OST Management Location Focus Areas Decontamination and Decommissioning Focus Area Tanks Focus Area Subsurface Contaminants Focus Area Mixed Waste Focus Area Crosscutting Programs Robotics Crosscutting Program Efficient Separations and Processing (ESP) Crosscutting Program Characterization, Monitoring, and Sensor Technology (CMST) Crosscutting Program Industry Programs University Programs Technology Integration Systems Application (TISAJ TISA Domestic Program TISA International Program Other OST Programs and Groups Environmental Management Science Program (EMSP) Accelerated Site Technology Deployment (ASTD, formerly Technology Deployment Initiative (TDI)) Program Technology Management System (TMS) Site Technology Coordination Groups (STCGs) Federal Energy Technology Center (FETC) Morgantown, West Virginia Richland Operations Office Richland, Washington Savannah River Operations Office Aiken, South Carolina Idaho Operations Office Idaho Falls, Idaho Albuquerque Operations Office Albuquerque, New Mexico Oak Ridge Operations Office Oak Ridge, r ~ . . ennessee Nevada Operations Office Las Vegas, Nevada Federal Energy Technology Center (FETC) Morgantown, West Virginia Federal Energy Technology Center (FETC) Morgantown, West Virginia DOE Headquarters Washington, D.C. DOE Headquarters Washington, D.C. DOE Headquarters Washington, D.C. DOE Headquarters Washington, D.C., and Idaho National Engineering and Environment Laboratory, Idaho Falls, Idaho DOE Headquarters Washington, D.C. Each major DOE-EM site

OCR for page 11
Introduction 15 Expansion of OST's Role in Deployment In addition to selecting, funding, and managing RD&D projects, OST supports work to facilitate deployment of the resulting technologies on DOE-EM problems.4 Because OST performs RD&D activities on behalf of (and to be deployed by) other DOE-EM program offices, the deployment of OST- developed technology is in part a function of the interactions of these program offices. OST has learned that it cannot stop with the development or even the demonstration of a technology and expect that the technology will be used by the problem owners at the sites. There exists a need for additional action to encourage the site "end user" (i.e., the combination of the DOE site manager with responsibility for the problem for which the technology is intended and the site contractor responsible for implementing a remediation approach) to adopt and deploy any new technology. Consequently, OST has proposed and, with congressional approval, implemented several programs whose scope appears to be outside a narrow interpretation of its original charter (U.S. Congress, 1989) of engaging in technology development, to perform this "deployment facilitation" function. In general, these programs were proposed to facilitate OST interactions with the technology user to engender the user's deployment of OST's technologies and those available from other sources. Such deployments must occur if OST is to play a useful role by introducing efficient, effective; acceptable, and/or economical new technologies into site cleanups. Examples of programs initiated by OST to increase technology deployment include the following: .. . . The TISA Domestic Program, intended to facilitate regulatory and stakeholder knowledge, communication, and acceptance of new technology applied to DOE-EM problems; The Large-Scale Demonstration Projects (LSDPs) of the Decontamination and Decommissioning Focus Area (DDFA), intended to bring industrial technologies into DOE-EM for use on decontamination and decommissioning (D&D) applications; and The Accelerated Site Technology Deployment (ASTD, formerly the Technology Deployment Initiative, or TDI) Program, intended to facilitate technology deployment. The ASTD program is an especially noteworthy result of OST's determination to have improved, innovative technologies deployed at the sites. The mission of the ASTD program is to cofund DOE-EM site proposals to use already-demonstrated technologies to expedite site cleanups, in situations where the intended deployment would achieve a significant cost savings. ASTD was not directed at developing new or improved technologies, but rather was designed to overcome barriers to the use of technologies new to DOE- EM. The appendixes to this report provide further description of the ASTD and TISA Domestic Program (Appendix E) and the LSDPs of the DDFA (Appendix C). Tracking the Benefits of OST Technology Development Activities In an attempt to monitor its programs better, and thus gain insight into the validity of its decision-making practices, as well as demonstrate that it was doing a worthwhile job of developing and implementing new technologies, in 1997 OST put into place the Technology Management System (TMS), a database to keep track of the technologies it developed, their costs and cost savings, and their deployments. This system 4Although deployment of OST-developed technology outside DOE-EM is possible, the deployments within DOE-EM provide accomplishments that are directly related to the OST and DOE-EM missions.

OCR for page 11
16 Decision Making in the DOE-OST provides a tool for OST to use in documenting, examining, and evaluating its performance. The number of deployments is an important measure of OST performance that is used by Congress and other interested parties, and reflects in part on the quality of decisions that have been made. An additional step taken by OST was to employ the U.S. Army Corps of Engineers to prepare estimates of cost savings over baseline process costs (DOE, 1997j; U.S. Army Corps of Engineers, 19971. A serious difficulty with this approach is in obtaining good estimates of baseline costs (NRC, 1998a). However, relative costs~hat is, differences in cost between competing technologies or processe~may be estimated more reliably than absolute costs of total projects and are of value in providing direction on which technologies to support. IMPORTANCE OF BUY-IN BY TECHNOLOGY USERS OST's decisions on which technologies to develop and attempt to implement at the sites is strongly affected by influences originating outside OST. Two primary external influences are (~) the budget it receives Dom Congress and (2) the problem owners, or site managers who are customers who both define the technology needs provided to OST and use the technologies from OST. These two external influences are not independent of each other. Failure of the site problem owners to use the technologies can make OST's efforts appear to be without value, which could adversely affect congressional funding actions. Consequently, OST must fund projects that are needed by the problem owners at the sites, who must be convinced of the merits of the technologies and must use them. This must be done often in the face of reluctance on the part of the sites to make changes to a baseline process that might result in inconvenience, liability (financial or professional), delay, adverse stakeholder reaction, or problems with regulatory agencies. There also may be a desire by the problem owner to receive RD&D fimds directly rather than have them go to another party, such as OST, particularly if the latter is perceived as not being as responsive to problem owner needs as the problem-owning office would be if it received the funds directly. Although OST's operating environment is different in many ways from that of most industnes, there are good industrial decision-making practices for technology RD&D that are helpful in achieving problem owner buy-in. One such practice is the strong emphasis in industry on customer involvement in decisions on technologies to develop. This involvement works in both directions: the industrial RD&D organization obtains input from the customer on needs, and the customer learns from the RD&D organization about technologies and the industry's technology development capabilities. In the early 1990s, OST did not fully recognize the necessity of obtaining customer (problem owner) buy-in. However, OST is now acutely aware of the importance of paying close attention to customer needs and, in 1994, put into place mechanisms for unproved interactions with site personnel regarding site needs. For example, a program reorganization in 1994 created the current user-~iven approach to identifying technology needs as the basis for investment in technology development projects. Additional OST efforts to interact with the end customer and to overcome resistance to deployment of its technologies are discussed in Chapter 2.

OCR for page 11
Introduction 17 REPORT SCOPE AND ORGANIZATION The approach taken by the committee for this study was first to gather information by visiting the DOE EM sites where the major OST program units are managed (see Table ~ . ~ for a listing of the program units). The prioritization and decision-making processes of each of these major OST program units are described in Appendixes B-E, summarized in Chapter 4, and evaluated in Chapter 5. After the site visits, the committee . developed an historical overview of DOE-EM and OST and the evolution of OST to its present organizational structure; investigated how some large private industries and non-profit institutions make technology RD&D decisions; and developed a decision-making mode! (see Figure 4.~) that embodies the essence of the context in which OST makes its decisions but is, in fact, a generic mode! for technology RD&D in He federal government environment. Chapter 2 presents an historical overview of the evolution of DOE-EM and OST in response to a recognition of the ineffectiveness of past practices for getting technologies deployed. This evolutionary development led to the current organizational structure and decision-making practices, which are themselves being refined in a continuing evolutionary effort to improve them. Chapter 3 and Appendix F identify relevant "best practices" by industry in technology development decision making. To suggest how OST might improve its prioritization and decision-making processes, the committee considered the decision-making practices of some other organizations, including many of the leading RD&D companies in private industry. This was largely done using the results of a multiyear, multicompany RD&D decision quality benchmarking study carried out by Strategic Decisions Group (SDG), the RD&D Decision Quality Association, and the Industrial Research Institute (see Appendix F). The committee also interviewed five companies with substantial environmental remediation programs. In addition, site visits were made to the Gas Research Institute (GRI) and the Electric Power Research Institute (EPRI), institutions that face many of the same organizational complexities as DOE, to study their RD&D project selection and prioritization processes. Summaries of information acquired at three of these interviews are contained in Appendixes Gel. Finally, the committee drew upon published literature describing the use of decision and risk analysis, probabilistic risk analysis, and cost-benefit analysis to help make similar RD&D decisions in a wide variety of settings. Chapter 4 presents a model process for technology decision making in the federal government environment, addressing important constraints such as the congressional budgeting process and the interactions among OST, other DOE-EM entities, and the contracted site operators. Chapter 5 contains discussion supporting the committee's findings and recommendations, which are presented in Chapter 6, on Me nature, appropriateness, and effectiveness of OST's decision-making processes for selecting and finding RD&D activities. This chapter concludes win a discussion of the way the subject of decision making, particularly in RD&D contexts, is formulated, analyzed, and practiced. 5The committee concluded its information gathering from DOE sources in April 1998.

OCR for page 11
18 Decision Making in the DOE-OST DECISION MAKING AS A DISCIPLINE Structured decision making involves setting goals, defining alternatives to achieve these goals, and collecting information about the likely consequences of each alternative (March, 1999; and Zeleny, 19821. In manY real-world situations. relativeIv simple circumstances often apply. For DOE-EM, these is defined situations may occur, for example, after a given development project scope is defined, goals are agreed upon, and measurement milestones and funding are set and well documented. Then the further set of decisions involved in executing and monitoring the oroiect is straiahtforwar~rovided the coals or measurement milestones do not chance. ~., , ,& do, O In many decision-making contexts, and in most DOE-EM situations, these simple circumstances rarely prevail. As expounded in many business courses and books, it is no longer practical for a business organization to have a single dominant and unchanging goal (such as bottom line return) or a simple vision (such as making the best "widget" of its type). The goals and missions themselves must be subject to continuing critical review, explicit articulation, and refinement (e.g., by a change in regulations). In the public sector, this kind of review can be more complex because of the larger number of stakeholders often involved and the existence of multiple, potentially conflicting goals. In addition to changing goals and missions, other refinements over time can occur in the criteria used to select and evaluate RD&D proposals, the relative weighting of these criteria, and the information available for evaluation against these criteria. In industry, a basic criterion such as the relative cost- benefit of several options commonly serves to screen out weak items. In government RD&D, the simplification of having one or two dominant factors is rare; many relevant factors are often evident, and commonly more than two of them will appear to have comparable importance or weighting. Moreover, the information on some choices may be fragmentary or evidently unreliable, but this may be a reason for doing some exploratory RD&D. Development and Growth of Decision Disciplines The evident deficiencies in business, military strategies, government regulation, and legislation-of oversimplified approaches to complex decisions has contributed to the development of the discipline of operations research, which involves modeling the system or the situation in order to recognize a framework for decisions. As part of this development, a scientific and engineering approach to a formal, prescriptive theory of decision making has also been created. Seminal books and publications (e.g., von Neumann and Morgenstern, 1947; Edwards, 1954; Luce and Raiffa, 1958; Chernoff and Moses, 1959' Fishburn 1964; Howard 1966;6 1983; Pratt, Raiffa, and SchIaifer, 1965; Raiffa, 1968; Tribus, 1969) set the foundations of structured decision making. More modern references (e.g., Cooper, 1993; Hammond, Keeney, and Raiffa, 1999; Keeney, 1992; March, 1994; 1999; Martino, 1995; Matheson and Matheson, 1998; Saaty, 1994; 1996) also describe decision-making practices. Some references (Simon, Smithburg, and Thompson, 1991; Simon, 1997) connect decision making to other closely related disciplines or to other components and issues (e.g., organizational structure, administration and policy, and efforts to promote communication and teamwork between different units) within an organization. The committee's approach in conducting this study was 6This is the first article to use the term "decision analysis."

OCR for page 11
Introduction 1 19 not to produce a report similar in character to these treatments. instead, the committee sought to identify the important decisions facing the sponsoring program office (OST), its current decision-making processes, and the important influences on these processes. The result is a series of findings and recommendations intended as general guidance on decision making for OST (or any organization in a similar context), given its important constraining features. The description and consideration of OST decision processes were difficult to separate from descriptions and considerations of organizational structure, institutional procedures, and program management. indeed, the committee had to consider these and other program elements because of their relevance to the prioritization and decision processes. Therefore, some report language, although relevant to decision making, is descriptive of and applicable to these other programmatic elements. These comments are offered only insofar as they impact decision-making processes, which the committee was obliged to consider (see statement of task in Box ES.~. Risk Aspects of RD&D Decisions A closely related development, probabilistic risk analysis (PRA), has occurred in parallel to the growth of methods and applications of decision analysis. The logic, data, and arithmetic processes involved in the disciplines of risk assessment and decision analysis have close similarities. Both disciplines involve realistic modeling and require systematic treatment of the inherent uncertainties in the information and data available. Risk evaluations commonly are among the most heavily weighted factors in many kinds of decisions. For environmental RD&D, three kinds of risk evaluation are commonly considered. The first is the probability of success or failure of the RD&D to meet its intended performance goals (i.e., a risk of failure to meet a technical specifications. The second is the extent to which the expected result of the RD&D can serve to reduce significant environmental hazards to workers and the public (i.e., a risk to the environment or to people). The third is the extent to which a successful RD&D result will be used in the field (i.e., a risk of failure to be deployed). Even if the work is technically sound and solves an important problem, it may fad! to get much use for reasons unrelated to the technical quality of the RD&D output. The context in which "risk" is used in this report makes clear which of these three meanings is intended.