Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 73
6
Findings and Recommendations
The prioritization and decision-making processes considered in this report are those used to fund
RD&D in support of the DOE-EM cleanup and waste management missions. This chapter presents the
major findings end recommendations. Because the committee believes that there are some important
recommendations for effective OST decision making that require actions by levels in the DOE-EM
organization above OST, the findings and recommendations are divided into two categories, general and
specific. Many of the recommendations on general issues are ones that OST may not be able to address
directly; nonetheless, the committee believes them to be important and they are included for possible
action by DOE-EM.
GENERAL FINDINGS AND RECOMMENDATIONS
The recommendations that follow are offered to improve RD&D decision making in the current
DOE-EM organization,' in which OST's purpose is to develop technology to be used by other EM
offices, based on their needs. The recommendations below are generally designed to elucidate the
technical needs of He user community for OST's use, to improve the connection between these needs
and OST development goals and projects, and to enhance OST decision making with modern methods
and good practices. OST efforts to implement these recommendations will, in the committee's view'
help to address barriers (see Chapter I) and other current issues (e.g., the right performance measurers))
for the OST program.
Importance of a Central RD&D Function
~-Dot ~
Since ends as well as means are important to any decision-making process, brief remarks on these
issues are offered. The "ends" are appropriate RD&D activities; the "means" is the OST program office,
which in Me current DOE-EM organizational structure is (~) separate from the other EM offices with
te~hnoln~v needs and users, and (2) centralized in that it is coordinated Tom DOE headquarters rawer
than from any EM site or user office. A legitimate issue is how effective a centralized RD&D function
such as OST (as opposed to some over organizational structure) can be in funding technology projects
that are developed and deployed on DOE-EM site cleanups and whose use provides benefits as measured
in cost, risk, schedule, or other relevant criteria.
The committee did not address whether other organizational structures were possible to accomplish RD&D in
support of the DOE-EM mission, other than to note, as shown below, the potential advantages of a centralized
RD&D fimction for the DOE-EM program.
73
OCR for page 74
74
Decision Making in the DOE-OST
Finding The DOE-EM program faces many difficult technical challenges. Some pervasive cleanup
problems (for example, some of those involving ground water contamination) cannot be solved in a
practical sense with current technology. Other problems could benefit by technical solutions that reduce
cost and/or risks to the public and the environment. Solutions to problems such as these and others may
be provided by developing advanced technologies through a central RD&D organization that addresses
more broader-based issues than those that might be addressed by a single site or a single management and
integration contractor or its subcontractors. In DOE-EM, OST provides such a centralized RD&D
function.
Recommendation to DOE-EM A centralized RD&D function within DOE-EM should be maintained
because of its potential advantage in coordinating potentially duplicative technology development efforts
needed at DOE-EM sites and because it is in a better position to address important broader issues (e.g.,
alternative technologies in the baseline functional flowsheets and alternative functional flowsheets) than
more specifically directed RD&D.
Deployment of Technologies
Finding OST has experienced a significant problem in increasing the percentage of its technologies
deployed at the sites. The problem arose in the past for two reasons. The first is that in its early days,
OST failed to attempt to solicit in a structured way technology development needs from problem owners
on the site. Consequently, some OST programs were often perceived as not being relevant to site needs.
Since it is fruitless to have an RD&D program without deployment of the technologies it develops, OST
and EM headquarters have attempted to address this problem by establishing first the Focus Areas and
STCGs, then the LSDPs, and in 1997, the ASTD program and the TAC. The second reason is that the
sites lack strong incentives to use OST-developed technologies, regardless of agreements or
understandings OST may believe it has with the sites. This second reason is outside OST's direct control
and is addressed in the recommendation below.
Recommendation to DOE-EM DOE-EM should continue to seek ways to assist OST in getting its
developed technologies deployed at the sites. The ASTD program, which serves a useful function of
facilitating the deployment of already demonstrated technologies at DOE-EM sites, is not directed
specifically toward the technologies developed by OST. Thus, there is still a need for more aggressive
action by DOE-EM in deploying OST-developed technologies, in addition to those available from
industry.
Recommendation to OST OST's role should be to develop technologies that address DOE-EM site
needs. To engender deployments, OST should take the steps listed below to identify current and future
DOE-EM site technology needs and should use this information as a guide for tailoring an RD&D
program to meet these needs. Use of financial incentives such as those provided to the sites by the ASTD
program are not desirable but appear to be necessary at the present time. They should be discontinued as
soon as practicable because they commit OST funding that could be redirected with advantage to other
OST initiatives. For identification of current needs, OST should continue to stress the involvement of the
sites in OST technology development and procurement activities and, reciprocally, should involve itself
OCR for page 75
Findings and Recommendations
75
to the extent possible in the site activities and problems.2 For identification of needs likely to persist in
the long term, OST should identify technology vuinerabilities (i.e., process steps with significant
probability of technical failure) in baseline functional flowsheets and use this information to develop
backup technologies and alternatives that have been agreed on through discussion with the site problem
owners.
Balancing Research with Development and Demonstration
To have a balanced RD&D program, OST should support research-oriented activities that explore
novel and innovative ideas, as well as more development-oriented activities directed at existing processes
for specific applications. Research-oriented projects involve relatively high technical risk and are much
less than 100 percent successful, if success is measured by large-scale application (i.e., deployment).
Finding A balanced RD&D program will have a less than 100 percent success rate, if the measure of
success is deployment in the relatively near term.
Recommendation to DOE-EM The percentage of OST technologies that reach the deployment stage
should not be the sole figure of merit used in judging the OST program, although it is an important one.
A long-term view should be adopted wherein the direct use of OST technologies or the use of derived
technologies is also considered in the evaluation of OST's portfolio of technology development projects.
Site Baseline Remediation Functional Flowsheets
Finding Decisions on site technology development needs are derived from the so-called baseline
functional flowsheets developed at the sites by the site remediation problem owners. These flowsheets
are fundamentally important in determining process steps and related technology needs. In effect they
dictate the technology needs to which OST must respond with technology development programs. As a
consequence of the current DOE-EM organizational structure, OST has no direct role in establishing the
baseline functional flowsheets, which are developed by other EM organizations and contractors at the site
level. At present, to influence decisions on technologies to be adopted, OST must undertake studies that
compare existing baseline technology costs with more favorable costs of OST-proposed technologies and
, . ~ _
. ~ · . . ~ · . , _ ~ ~ _ 1 _ 1 ~ 1 ~ ~ ~ ^1_ _ _ _ .1~ _ _ 1 ~ _ _ 1 _
in this way try to persuade site managers to adopt technologies oltterent from those they are already
committed3 to use.
Recommendation to DOE-EM The expertise of technology developers supported by OST could be of
value to the site problem owners in formulating and maintaining technically sound and practicable
cleanup functional flowsheets. Therefore, efforts should be made to have substantial involvement of
appropriate OST and OST contractor personnel in reviews of fimctional flowsheets. Such participation
20ne practice in the past to enhance this involvement has been for the site user representatives of the Tanks
Focus Area to endorse the consolidated, prioritized list of technology needs (see Appendix C).
3If these commitments are represented in legal agreements or in site contracts, opportunities to make decisions
on alternative technology may be significantly restricted.
OCR for page 76
76
Decision Making in the DOE-OST
would have the benefits of (1) ensuring that OST technology developers fully understand the site problem
owners' technical needs and their bases and (2) increasing the sites' confidence in OST's dedication and
ability to meet their needs.
Technical Alternatives to Baseline Remediation Functional Flowsheets
Finding Good decision-making practices (e.g., hedging against technical uncertainty and insisting on
alternatives) imply that DOE-EM should plan for alternatives to the site baseline functional flowsheets,
especially when the baseline flowsheet involves high cost, high or poorly defined risk, and/or substantial
probability of technical failure. Uncertainties about the future, both in funding and those related to
possible failures of existing functional flowsheet process steps, highlight the importance of developing
technology alternatives. in addition, the possibility of failure of large-scale privatization contracts to
achieve site cleanup goals reinforces the value of technology developments targeted at alternative
functional flowsheets. At present, there is little or no DOE-EM funding for developing alternative
functional flowsheets, and consequently, no OST activity is expended on technology development
directed toward alternative fi~nctional flowsheet technology needs.
Recommendation to DOE-EM The development of alternative functional flowsheets is the
responsibility of DOE-EM offices other than OST, but they should seek OST's input. It is highly
desirable that the problem-owning EM offices should seek out and acknowledge the potential
vuinerabilities in cost, risk, and technological failur~of the baseline functional flowsheets and
processes and, with OST's assistance, develop alternative flowsheets as appropnate. OST should
encourage this course of action and seek to collaborate in it.
Recommendation to OST OST should attempt to make input to alternative functional flowsheets and,
in particular, should advocate their development when the baseline functional flowsheet involves high
cost, high or poorly defined risk, and/or substantial probability of technical failure. OST should identify
specific technology development opportunities aimed at supporting alternative functional flowsheets and
processes designed to enhance the overall probability of remediation successes and to minimize program
delays. In practice, this means that OST should be allowed to commit a portion of its resources to
developing technologies to address needs derived from such alternative functional flowsheets, in addition
to developing technologies to meet the needs derived from the baseline flowsheets.
Tn(lependent, External Reviews
Finding Peer review~hat is, reviews by technical experts who are independent of and external to
the program of work being reviewe~iare a vital part of a credible decision-making process. OST
programs and projects are subjected to many, often resource-intensive reviews but, according to recent
NRC reports (1997b, 199Sb), to few true peer reviews. To address this shortcoming, in August 1996
OST established a peer review system using the American Society of Mechanical Engineers (ASME,
1997a) to review works in progress. The result is an OST program with many different reviews, which
OCR for page 77
Findings and Recommendations
77
are not fully effective in the important decisions of the initial selection and subsequent continuation of
meritorious technology development projects.
Reviews of Technology Development Projects
Recommendation to OST Peer reviews of technology development projects should be part of OST's
decision-making process. These project peer reviews should occur as necessary and in a way that is not
an overly burdensome commitment of OST resources. Therefore, the OST review system should be
streamlined by reducing the number and types of reviews based on an analysis of the objectives being
served by each. Reduction in the number of reviews could be accomplished in part by combining reviews
where practicable.
Reviews of Program Budget Targets and Their Rationales
Review of technology development projects is part of RD&D project monitoring and evaluation,
which is but one step in the OST decision-making process (see Figure 4.~. The committee therefore
considered whether other steps of the process merited review.
Finding The formulation of budget targets for the venous OST program units (e.g., Focus Areas and
Crosscutting Programs) is an important decision point in OST's process. At this decision point an
independent external review is needed but is not now being undertaken. These funding targets are made
by the OST upper management levels at headquarters based on input from DOE site managers and other
OST program managers. These targets are proposed to higher authorities (and ultimately to the Congress)
through the annual federal budget process.
Recommendation to OST An independent, external review should be held on the basis of, and
rationale for, decisions on funding targets within OST. One goal of this review should be to identify the
technical areas of greatest need, where improvements over existing conventional approaches would have
the greatest benefit to DOE-EM. This review and its outcome should take into consideration such factors
as DOE-EM programmatic strategies, political pressures, stakeholder pressures, risk to human health and
the environment, safety, cost-benefit, and timing. Such a review might be carried out by an already-
constituted authoritative external body such as Me Environmental Management Advisory Board (EMAB),
or by a group created specifically to conduct the review.
OST Involvement in Reviews of Site Remediation Functional Flowsheets
Finding The site remediation functional flowsheets, whose development and review are in the
domain of the site problem owners, are important for OST because they define the user plans from which
technology needs are derived.
OCR for page 78
78
Decision Making in the DOE-OST
Recommendation to DOE-EM Site remediation functional flowsheets should be subjected to
independent, external review before they are adopted, and periodically dunug development of the
technologies that are to implement them. The EM offices developing these flowsheets should have them
peer reviewed, that is, reviewed by technical experts who are independent of and external to the program.
This expertise may be found in academia, private industry, and national laboratones. The purpose of such
reviews is to identify possible vuInerabilities or uncertainties in the functional flowsheet assumptions and
technology selections. the committee understands that the other bM ottlces already sponsor Such peer
reviews of functional flowsheets for the most part, but it would recommend the practice for all important
functional flowsheets.
Recommendation to OST OST should try to work in conjunction with other DOE-EM offices to
participate to the extent possible (e.g., by establishing a role for OST contractors) in the schedule of peer
reviews of the site baseline functional flowsheets.
The purpose of this OST involvement is to make the results of these reviews and the rationale for the
technology development needs available to OST. This OST role in reviews of site baseline functional
flowsheets will also, in the committee's view, help strengthen the interaction between OST and the DOE-
EM user community.
SPECIFIC FINDINGS AND RECOMMENDATIONS
The recommendations presented below are within OST's purview and control to implement. This is
not to say, however, that exigencies of the moment regarding such constraints as limited funding may not
prevent OST from taking immediate action on some of them.
OST's Institutional Environment Affects Customer Interactions,
Relevance to Site Needs, and Deployment
Finding OST operates in a complex, ever-changing, and politicized" environment. An important
complication for OST is that the sites and the EM offices that are responsible for cleanup activities have a
great deal of autonomy in selecting baseline remediation processes and technologies to deploy.
Furthermore. theY are under no obligation to use OST-developed technologies, even if they have made
, i, _
. .. . . . · ~ ~ _ ~ . ~ 1_ 1 _ ~ _ ~ 1 ~ ~ ~1~ ~ it: _ _
earlier commitments to do so. As new cleanup and waste management problems are Iouna al one sues,
new technology needs arise. This creates a very difficult situation for OST in keeping abreast of
technology needs and getting its technologies accepted and deployed.
Recommendation To the extent possible, OST should increase its efforts to identify site technology
needs on a current basis and to anticipate future needs. Regularly scheduled meetings with site problem
owners should be considered. More discussions of technical issues and their implications for technology
development needs should be held with the working-level scientists and engineers.
OCR for page 79
Findings and Recommendations
79
Finding OST has more than one "customer" to satisfy. Other EM offices such as EM-30 and EM-40
are obvious end users of OST technology developments, but the U.S. Congress must also be satisfied that
a reasonable fraction of OST products are useful and worth their cost. Furthermore, parts of OST
expenditures are congressionally mandated (U.S. Congress, 1994; 199Sa-b). Consequently, the type and
quality of the information provided to Congress (and to other interested review groups) are critically
important to OST.
Recommendation OST should ensure that the decisions underlying the technologies it develops are
well documented, traceable to customer needs, and supported by sound technical reviews. Records
should be kept of the reasoning by which the deciding factors were evaluated, including whatever
methodist were used in their evaluation.
Finding Although the user focus is necessary, not all RD&D needs come from user requests. As an
example, alternative technical approaches to site remediation baseline functional flowsheets are a
potential source of technology development needs, and these alternatives would not be derived from user
requests based on baseline functional flowsheets. Therefore, there is a need for exploratory RD&D with a
shorter time frame than the Tonger-range basic research projects of the Environmental Management
Science Program, to meet the need of alternatives to baseline functional flowsheets (see Chapters 3 and 5
and Appendix E).
Recommendation Although the technology development projects should be based primarily on
specific needs at the sites, some should be of an exploratory nature to meet the need for backups and
alternatives to the baseline functional flowsheets.
Top-Level Strategic Planning and Goals
Fincling OST's strategic goals do not provide an adequate level of guidance for program managers as
they attempt, in collaboration with users, to assign priorities to technology needs. In times of limited
funding, it is especially important for managers to have the type of guidance and direction that come only
from strategies developed by top OST managers, whose perspective is much broader than that of
personnel further down the management chain.
Recommendation OST managers, in conjunction with other top-level EM managers,4 should
produce strategic goals and plans that define explicitly the technical problems that the
program will (and will not) address, and
use these goals and plans effectively within OST program units to assist them in
making technology development decisions.
Although EM "strategic goals" are now available (DOE, 1997j), these are obtained through a user-
driven and therefore "bottom-up" decision-making process of other EM offices in which most
remediation decisions are made at individual sites (DOE, 1998a; 1998b). Therefore, these strategic goals,
4Input from other EM offices is recommended since OST goals should be derived from user plans and needs.
OCR for page 80
80
Decision Making in the DOE-OST
which target the sum of the work to be achieved at all sites in a bottom-up approach, differ from the top-
level goals recommended here for OST that are specified by a centralized, coordinated, top-down process
to drive the technology development program. Of course, any top-level strategic goals developed by OST
should be consistent with the EM mission and be derived in concert with user plans and needs. That is,
any such strategic goals for OST would be guided and constrained by priorities established within other
DOE-EM offices, priorities that have varied over the years and would be represented by their top-level
goals and other statements of their high-priority tasks. A particularly important area for current attention
relates to the implications of privatization for OST's technology development program.
This goal-oriented approach, to be successful, would have a significant impact on the way in which
OST operates. Obtaining the "right" goals is a challenge, as is the orientation of program activities in
support of these goals. For the goals to work, everyone in the organization that can have an impact on
goal achievement must be measured on the success of reaching the goal (or at least the part of the goal
that their job impacts). If these broadly written goals are in place and directions of important efforts can
be derived from them, then this framework would provide a way to readily assess the value of each
RD&D project.
The committee chooses not to be prescriptive, but instead believes that OST management should be
responsible for formulating its goals and criteria. These would include (~) adequate cleanup targets (e.g.,
level of decontamination to be achieved in various media, for use as program goals by Focus Areas and
other OST program units in soliciting and selecting projects) and (2) a set of objective criteria to be used
in evaluating technology development proposals and projects. This effort could solicit and use inputs
from the users of technology (i.e., site contractors) and all interested parties.
Use of a Structured Decision-Making Process
The main reasons for a structured decision-making process are to ensure its comprehensiveness,
objectivity, and consistency. Often, these qualities can best be realized when data and other information
are expressed in quantifiable terms. Quantifiable inputs include not only measurements that are already in
quantitative form (e.g., contaminant concentrations or dollar values), but also inputs such as expert
judgment estimations (e.g., scores expressed as high, medium, or lows that can be scaled and converted
into quantitative form. Both quantitative input (such as cost estimates) and expert estimates involve
judgment; the aim of a quantifiable approach is to make these judgments as objective and transparent as
possible.
Finding Structure to the OST decision process is warranted, in part because of the complexity of
OST's institutional environment and in part because of the potential for change over time in technology
needs (which serve as the basis for the technology development projects that OST funds) and top-level
· . .
priorities.
Recommendation For decisions involving the allocation of significant resources, OST should
institute a decision-making structure wherein projects and/or proposals are evaluated against consistently
defined criteria such as project cost, probability of technical success, probability of implementation on
field applications, potential cost savings, and human health risk reduction. This structure should be
applied broadly throughout the organization, with each OST program unit evaluating projects against the
OCR for page 81
Fir~clings arid Recommendations
81
same criteria. An important criterion would be the project's relevance to site activities, that is, all projects
should have one or more specific objectives related to the cleanup and waste management goals at sites.
For an ongoing OST project, its history of past perfo~ance in meeting appropriate developmental
objectives would be useful information to gauge its likely future success.
The quantities mentioned in the preceding paragraph are recommended to be estimated in order to
provide more discipline and guidance to decision making than if these estimates were not done. It is a
challenge to perform estimates well with the degree of rigor that is possible depending on the state of
maturity of the concept. Initial estimates of early-stage development projects would likely be subject to
data limitations, large uncertainties, and limited sophistication. Refinements to these calculations would
be likely over time as a technology development project progresses and should be documented. The
quantification of project benefits and probabilities of success is necessarily uncertain, but the committee
believes that a rigorous attempt to estimate and use these quantities and other available data is preferable
to the less structured methods that OST has used in the past. The recognition and evaluation of
uncertainties is itself a key part of a decision process.
This structured system is proposed as a useful decision support methodology to organize inputs that
are used by the OST managers who make decisions. Any such decision support too! has limitations,
particularly if it fails to account for all significant factors; 5 therefore, managerial flexibility to "override
the numbers" in occasional situations is necessary. There should also be sufficient managerial flexibility
to enable responses to late-breaking development opportunities.
Industrial RD&D Decision-Making Practices Applicable to OST
Finding Good private-sector decision-making practices include the use of formal decision-making
processes that employ quantifiable, measurable goals and follow-up procedures such as those discussed in
Chapter 3 for RD&D decisions.
Recommendation OST should adopt, where applicable and appropriate in the OST environment and
to the extent practicable, the basic principles of private-sector formal decision-making and follow-up
practices that are presented in Chapter 3. In particular, an attempt should be made to assess the following
factors and adopt them consistently where applicable across the entire organization:
Understand, focus on, and monitor changes in customer needs and requirements.
Agree on clear and measurable goals.
Use a formal (i.e., common, consistent, structured, and rational) technology development decision
making process and apply it uniformly.
Think strategically (i.e., long-term and high impact).
Measure and evaluate to guide resource allocation.
sIndeed, because of the potential for complexity in Me various considerations relevant to a decision on
technology RD&D, the phenomenon of "rank reversal" (Corbin and Marley, 1974; Buede and Maxwell, 1995;
Saaty, and Vargas, 1993; Tversky and Simonson, 1993) is possible, resulting in a decision in which the outcome
that is preferred is not the one that scores most favorably against all the criteria used in the ranking or rating
exercises. The committee suggests that managerial flexibility be invoked to provide for this kind of situation (i.e.,
to make final decisions by choosing the options that do not necessarily have the highest scores as evaluated against
the criteria used) in the absence of a more rigorous method.
OCR for page 82
82
Decision Making in the DOE-OST
Communicate across organizational boundaries (i.e., with technology users).
Continually Improve the research and development (R&D) management process.
Hire the best people possible and maintain expertise.
The committee prefers to leave to OST management the task of translating how these practices could
best be adapted. For example, "hire the best people possible and maintain expertise" would refer in
private industry to the cadre of researchers within the company; for OST, which is a collection of
program units run by program managers who contract out the technology development work to PIs, the
application of this concept might be best translated as "select the most competent contractors."
The general recommendation is that OST seek to implement those practices that have been identified
as improving decision quality. The support for this recommendation comes from published literature on
broad surveys of company practices. To lend support to the validity of these decision-making practices
for OST, the committee sought confirming evidence that they were useful concepts in organizations
analogous to OST in their structure, institutional environment, and/or the kinds of technology within their
environmental remediation RD&D programs. The industries cited in Chapter 3 and Appendixes G-l
provide examples of institutions that have some similarity to OST. The institutions described there do
not represent an exhaustive list of such analogous institutions, but were chosen as specific examples to
illustrate the utility and practical application of the good decision-making practices and principles
described in Chapter 3. These institutions should not be construed as constituting a representative sample
of all institutions employing successful decision-making practices; many such institutions exist, some
employing different (e.g., non-utility theory based) methodologies.
Specific Methodologies
Finding The quantifiable approaches used by OST (see Appendixes B-E) are simple multicriteria
scoring techniques in which criteria are developed with relative weights assigned to them. A project or
proposal is rated against these criteria by reviewers who assign numerical scores. The scores and relative
weights are used to calculate an overall score, which is used for planning and management purposes in
comparing various projects or proposals.
This simple scoring technique is one example of multi-attribute utility analysis. Utility analysis has
its strengths but also its weaknesses. For example, utility analyses can be criticized on several accounts,
such as
I. the inability to formulate a meaningfully weighted utility function that adequately synthesizes
many, often disparate, "customer" views (Zeleny, 1982~;
2. the possible interdependence (in practice) of criteria that, in utility analysis treatments, are treated
as independent (Machine in Stigum and Wenstop, 1983~;
3. the problem of expressing preferences for these criteria in ways that cannot be adequately
captured by numerical scales of numbers that are added or multiplied together (e.g., translation of
measurement scale values, such as those of ratio scales, into interval scale representations) (Saaty and
Vargas, 1993~;
4. the combination of tangible and intangible factors (an example of the latter is the valuations
applied to expert judgments), especially in hierarchical prioritizations (Saaty and Vargas, ~ 9981; and
OCR for page 83
Findings arid Recommendations
83
5. the possible violations of the "principle of invariance," also known as regularity, the
independence from irrelevant alternatives, and "rank preservation," by which is meant that the rank order
of selections may change as new criteria are added or old ones deleted (Tversky and Simonson, 1993;
Tversky, Slovic, and Kahneman, 19904.
Other decision-analytic methods, such as the analytic hierarchy process, have been advanced as
alternatives to utility analysis that treat these complicating issues in a different way (see, for example,
Saaty, 1994; 1996~. The issue for OST applications is which of all possible decision methodologies is
adequate and convenient to use for making the requisite decisions. One way to answer this question is to
judge whether a method's shortcomings, such as those enumerated above, are important "fatal flaws" that
prevent the best decision from being made. The committee makes no finding on a relative comparison of
various decision methodologies, or which is "best" for OST, but would instead offer the following
recommendation:
Recommendation OST should examine the efficacy of the sets of criteria and scoring techniques
currently used by OST program units (e.g., STCGs, Focus Areas, and Crosscutting Programs). This
could be done by (~) using one or more contractors with suitable expertise to survey alternative decision-
making analytical methods and (2) using the considered judgment of OST management to identify those
analytical methods that are well-suited to OST's various needs. The judgments for OST management
are whether, for OST decisions, the shortcomings of any decision methodology are important practical
concerns and whether the decision methodology provides a useful organization of input information to
help guide a decision among several candidates. The current OST quantitative schemes are multi-
attnbute systems with criteria and relative weights, and rely on expert (e.g., reviewer) judgment to score
projects and project proposals against the criteria. in practice, the recommendation offered above would
evaluate whether this process is sufficiently adequate or whether an alternative process would provide a
more useful output. For example, one way to minimize "rank preservation" problems (issue 5 above) is
with the use of pairwise comparisons of alternative proposals (i.e., rank ordering them by having
reviewers compare them using their judgment, experience, and knowledge) rather than rating each
proposal against a set of criteria and comparing the numerical outputs of the rating and scoring exercise.
One such comparison of multi-attribute value theory and the analytical hierarchy process
methodologies showed that the decision outcome depended to a lesser extent on which methodology was
used and to a greater extent on issues associated with the way the problem was structured and valuations
and weighting factors were elicited (Buede and Maxwell, 19951. As noted above, the general
recommendation of this report is for a method with structure, documentation, and quantifiable attributes,
without a recorded preference on which specific method with these attributes OST should adopt. The
recommendation is primarily that decisions be structured, by which is meant that the goals, factors, and
criteria believed to influence the decision should be clearly specified. Records should be kept of the
reasoning by which the deciding factors were evaluated, including whatever method(s) were used in their
evaluation.
Most decisions on the selection and continuation of any research and development project are
revisited several times (i.e., at least annually), particularly if the project is funded during more than one
fiscal year. Record keeping would support systematic refinement of decisions as more information is
developed and would provide the basis for "lessons learned" to help improve decision quality.
OCR for page 84
84
Decision Making in the DOE-OST
Summary
A major goal of this report is to outline the institutional environment in which OST operates, as the
"macrolevel" context within which OST prioritization and decision making occur. To suit this context, a
decision method is proposed in a previous section of this chapter that offers structure, quantifiable
attributes, and documentation. A "microlevel" recommendation is not offered on which of any of the
specific methods available in the literature to meet these general criteria is the best for OST to select and
adopt. Rather, a familiarity with various decision-making approaches, accompanied by the considered
judgment of OST management, is recommended to direct the selection and adaptation of specific
decision-making methods.
Project Selection and Evaluation Criteria
Finding There are no general, OST-wide guidelines for setting criteria for the selection and
prioritization of technology development needs, although individual OST program units have developed
their own guidelines. Moreover, the general criterion that technologies should be applicable to multiple
sites, while useful, is flawed when applied without exception because it may lead to the failure to develop
technologies for potentially very important problems that exist at only one or two sites.
Recommendation To the extent practicable, and with input from its various organizational elements,
OST headquarters should establish general selection and prioritization criteria, and guidelines for
applying these criteria, to include allowance for instances in which exceptions to the criteria may be
appropriate.
Procurement of Externally Demonstrated Technologies
Finding The present approach to technology procurement wherein both OST's Industry Program and
other OST organizational units such as the LSDPs perform some aspects of technology selection and
procurement from industry is cumbersome and duplicative, and impairs OST's deployment initiatives.
Recommendation A better-coordinated, less duplicative, and less cumbersome system should be
established for integration of technology procurement activities. Since procurements should be made
only if warranted in a make-or-buy decision, the ability to assess available technology is crucial. These
assessments should be done, by the use of a comprehensive database of demonstrated and commercially
available technologies.
The purpose of these comments is to engender a coordinated program in which information flow is
effective and cumbersome or duplicative elements are reduced.
OCR for page 85
Findings and Recommendations
85
Project Monitoring
Finding OST monitors technology development projects in several different ways, depending on the
specific project. One established formal process that OST has adapted and used is the stage-and-gate
system. In this system, funded projects are reviewed as they progress through their various stages of
development from early exploratory studies to engineering development and finally to deployment. At
the end of each stage there is a gate, which is in actuality a decision point on whether to proceed to the
next level of technology development. However, the stage-and-gate system does not help to critically
review the initial decisions to select projects for funding and entry into the tracking system. In addition,
the system has an excessive number of stages and gates to use them all for review points in the OST
application, and the scoring system has some flaws.
Recommendation OST should use the minimum number of stages needed to track projects. This will
reduce the administrative load and will lead to better decisions by producing better-defined decision
points and clearer lines of demarcation between them.
Recommendation In selecting a new technology development project for funding, OST should base
this decision on both technical merit and quantifiable estimations of the project's probable value to site
cleanup activities. OST has developed this latter concept as part of the criteria of the stage-and-gate
system, but OST program units do not uniformly adopt and use these criteria to guide their selection of
new projects for Finding.
Recommendation OST should correct the additive scoring system to account better for threshold
criteria. One way to do this is to multiply scores in keY categories rather than add them.
, O .
Cost Estimates
Finding OST has attempted to gain acceptance of new and/or improved technologies by showing
cost savings. However, it is difficult to assess the claimed savings because they require knowledge of
both the cost of the baseline technology to be replaced and the cost of the OST technology. Often the
baseline technology cost estimates, if they exist at all, are uncertain. The same thing is true of the
proposed replacement technologies. Nonetheless, these cost comparisons can be an important decision-
making criterion and can serve a useful purpose if they can be trusted. Uniformity in cost calculations
(both information and methodology) is a need that has recently been acknowledged by the
Decontamination and Decommissioning Focus Area through its use of the U.S. Army Corps of Engineers
as an independent, external organization to perform cost calculations. However, further steps are needed
to establish the credibility of cost estimates for OST technologies.
Recommendation OST should do cost avoidance or ROT calculations on its more expensive
technologies in a more uniform and credible manner than in the past and should communicate results to
the potential technology users in the most effective way possible. Initial estimates of costs and benefits
should be developed at the inception of large RD&D projects, and periodic refinements of the estimates
should be a part of the project as it progresses.
OCR for page 86
86
Decision Making in the DOE-OST
Exoloratorv Development
v
Part of the decision-making process is the gathering of technology needs. The recommendation
above on using site remediation functional flowsheets, including consideration of the baseline version as
well as alternatives, to best derive these needs will lead to related technology development opportunities.
Finding A iSraction of OST's budget is devoted to exploratory research through the EMSP, which is
a long-range basic research program that stresses research related to environmental cleanup problems
(discussed further in Appendix E). Although this program is important, it is not sufficient to meet the
need for exploratory development studies related to alternatives to the baseline functional flowsheets.
These studies are intermediate in timing between the long-range time frame of EMSP projects and the
immediate need for work on existing functional flowsheet technologies.
-~7
Recommendation Additional funding should be sought (or some existing funding redirected) for
exploratory development that is directed to technologies for alternative functional flowsheets.
Because exploratory development is recommended, the issue is raised of how best to fund and
manage such efforts. One relatively straightforward way to fund small-scale, exploratory projects in a
manner similar to the ways in which some OST funds (snecificallv. the research oDDortunitv
i- ~ ,# , ~ ~
announcements and the EM science Program tunas; see Appendix E) are dispersed and managed at
present is to set aside some uncommitted funds (e.g., a fraction of the total OST budget) in a program that
disperses them by having an open solicitation to which prospective PIs can respond, with frequent (e.g.,
quarterly) reviews of submitted proposals by teams of independent reviewers assembled for this purpose.
An approach of this kind can capture some of the key features of exploratory decision making in a
mechanism that will work in the federal budget system. The recommendation is for OST to adopt a
suitable method that will disperse uncommitted funds in a reasonably timely and well-defined way for
exploratory development activities.
CONCLUDING PERSPECTIVE
Consideration of OST's decision-making process begs the issue of what decisions are and should be
made, the answer to which requires consideration of the role and fimction of an RD&D organization such
as OST. There is a role for RD&D activities in providing economical, effective, acceptable, and
practicable technologies for use in DOE-EM site cleanups. Although OST accounts for a small part of the
DOE-EM budget, its work can have substantial and beneficial impacts on reducing the costs, risks, and
probability of technical failure associated with environmental remediation activities, which are estimated
to represent in excess of $100 billion dollars.
OST is an RD&D office within DOE-EM that is separate from the other EM offices that bear
responsibility for addressing cleanup problems. This organizational structure is a reasonable one for
conducting RD&D and is mirrored in utility consortia such as the GR! and the EPRI, in private industry,
and in the Defense Advanced Research Projects Agency of the Deparunent of Defense. Although the
findings and recommendations presented in this report are directed to OST, they could, in principle, be
applied to any RD&D office that manages project work in support of a broader federal program,
particularly if organizational similarities to OST and DOE-EM exist.
OCR for page 87
Findings and Recommendations
87
RESPONSES TO ISSUES IN STATEMENT OF TASK
The statement of task for this study asked the NRC study group "to evaluate the effectiveness of the
OST decision-making process and make specific recommendations to improve it, if appropriate." To
accomplish this task the pane! was asked to address four points. The material to address these points has
been presented in the preceding chapters. This material was used to provide input to the responses to the
four points, which are reproduced below.
I. The appropriateness and effectiveness of decision-making processes currently in use by OST
to select, prioritize, and fund RD&D activities, both at sites and at headquarters:
The current OST decision-making practices, which change with the passage of time, are embodied in
its program structure. The program structure of OST's core RD&D functions is based primarily on
STCGs, Focus Areas and Crosscutting Programs, and a headquarters oversight activity. A summary of
the elements of this program structure is presented in Box 2. ~ .
The process functions reasonably well to prioritize technologies to propose for funding at individual
sites and to make decisions for funding among sites within the framework in which OST operates.
However, as pointed out in this report, some of the process steps for example, technology prioritization
by the STCGs are cumbersome and ill-conceived, and Here has been little effort to apply a carefully
thought-out system uniformly across OST. This leads to difficulties in comparing and evaluating
priorities across the diverse technology areas that OST's activities encompass. Also, the stage-and-gate
process for evaluating technology developments at various stages of maturity is unnecessarily complex.
More direct input by OST to functional flowsheet review at the site level is desirable. However, a
major point made in the report is that key decisions about which technologies to support lie outside OST's
direct control. Input to technology and functional flowsheet selection at the sites is one of these key
decision areas. Another critical point at which peer review is needed and is presently absent is at
headquarters where final funding decisions are made.
2. The technical factors appropriate to consider in the decision-making process for selection,
prioritization, and the development of cleanup technologies, and the adequacy with which these
factors can be measured:
The committee has identified three technical factors to consider: (~) the degree of maturity of the
technology, (2) the ease of integration of the technology into a total system, and (3) the impact of the
technology.
The degree of maturity of a technology is represented bY a developing technolo~'s remaining
--I -- - ~ , ~ __ _
assumptions and uncertainties that require testing, and can be measured in terms of the estimated time
required to bring it to a sufficiently large-scale (e.g., pilot-scale) demonstration. It must be recognized
that OST is not constrained to use technologies that are mature and, in fact its, greatest contributions may
be in the development of new technologies. This report points out that OST should earmark a part of its
funds for exploratory development of new technologies that have easily recognizable applications. The
stage-and-gate process is designed to assess the maturity and likelihood of success of a developing
technology and to terminate technology developments found to be lacking promise of successful
completion or application.
The ease of integration of the technology into a total system is a question that cannot be answered
solely by considering OST and its decisions. This integration depends strongly on the attitudes and
OCR for page 88
88
Decision Making in the DOE-OST
constraints of the sites with respect to what they are willing to accept or what they can accept because of
regulatory controls and agreements that may have been made with state or other influential bodies, as has
been stressed in this report. These factors often control what technologies can be integrated in cleanup
systems and therefore represent valid consideration in RD&D decision making.
The intended impact of a technology is to reduce costs, schedules, and/or risks of various kinds,
and/or to enable a difficult or intractable task to proceed. These intended beneficial impacts imply
tradeoffs; they are not necessarily mutually reinforcing. One way to assess these impacts is to measure
the number of functional flowsheets needing that technology, the number of times it is needed in each
flowsheet, and the importance of the flowsheet need addressed by the technology. The practice of
making quantitative assessments of this kind would lessen the need for OST to rely upon guidelines as a
basis for decision making. For example, one of the most important guidelines OST uses to decide on
funding technologies is their applicability at more than one site. Although the committee agrees that this
is indeed an important consideration, there may be instances when a technology is very important for a
single site. The Quantification of technology impacts would enable an informed decision to be made in
1 C7 ~' -- - -- - ---- - - - ---- - - - - -- - - -- -
.. .. . .. . . .. . . . . . , . , . .. Am, .. . . . .. . . .
estimating whether a smgle-slte technology nac greater potential nene~lt than a technology that nac uses
at multiple sites.
3. Recommendations, if appropriate, for improving the decision-making process:
Recommendations are addressed in several ways in this report. Specific recommendations are given
above in this chapter, separated into general considerations and those specific to the current OST
decision-making process and program structure. Chapter 5 presents general principles derived from the
private sector that, if followed, will lead to improved decision making. As stated in Chapter I, the
committee did not undertake to prescribe or recommend one methodology or school of thought in
decision making over any other in its study.
4. The role and importance of effective peer reviews in the decision-making process:
The committee recognizes the crucial role played by peer review at appropriate points in the decision-
making process, and subscribes to the process and opinions described in earlier NRC (1997b, 1998b)
reports. These reports also acknowledges that OST is subjected to many reviews (many of which have not
been carried out in the manner recommended by NRC, 199Sb) and suggest that the number be reduced by
combining reviews whenever possible.
The DOE-EM RD&D decision-making process has two key decision points other than technology
development project reviews that are opportunities for reviews. These reviews, although not true peer
reviews in the sense of NRC (1997b, 1998b), nevetheless can embody many of the features of peer
review. This report recommends (~) a "policy review" at the headquarters level where budget targets are
formulated for the various OST program units, and (2) OST participation in (and/or access to) external
reviews of the baseline flowsheets developed by the other EM offices.
----I r-~J
Representative terms from entire chapter:
baseline functional
