5

The Analytic Hierarchy Process and Expert Elicitation

As the committee has previously noted, the team has a very difficult task: on the one hand, the team must provide detailed information and analysis of alternative approaches in a way that a decision-maker can readily and accurately compare the alternatives; on the other hand, the team must avoid allowing itself to assign weight to relevant criteria or itself to make the comparison, as this is the province of the decision-maker. The committee’s overall view is that the draft report does not provide a sufficient orientation toward providing information in terms that are informative for decision-makers. As this review has already noted, the basic questions that need to be answered for each of the options under consideration are: “How much will each treatment process cost?”; “Can the process be expected to be implemented successfully?”; and “What will be the relative performance of the resulting waste forms, and will they meet regulatory compliance?” Although other attributes may be useful to summarize as well, these central questions also represent the foundation for a cost-benefit evaluation, which is mandated by Section (Sec.) 3134. The preceding sections of this review make it clear that the Federally Funded Research and Development Center’s (FFRDC’s) current draft report does not provide clear answers to these questions.

At the same time, however, the team, as discussed in Appendix F of the draft report, undertook a detailed exercise that resulted in a hierarchically arranged comparison of approaches, using factors and weights of the team’s choosing. Moreover, the committee is left to believe the FFRDC intends to provide a comparison of the SLAW options based on the team’s Analytic Hierarchy Process (AHP) exercise, given that Appendix F contains only the documentation of the outcomes of that exercise (which the FFRDC also refers to as its “expert elicitation”). The committee notes that two of its members and the National Academies of Sciences, Engineering, and Medicine’s study director were welcomed by the FFRDC to observe the AHP exercise, and the following peer review comments are thus informed by more than just a reading of Appendix F. However, the committee emphasizes that the following observations and findings regarding how the FFRDC needs to craft a useful comparative analysis section reflects the consensus views of the entire committee, not just those who observed the AHP. Based on the direct observations of the AHP exercise and the resulting discussion in the draft report, the committee believes that the exercise was conducted in good faith, without motivational biases, and without any effort to skew the results toward a particular conclusion. The committee further notes its comments are concerned with the appropriateness of AHP for providing decision-makers with a useful comparative analysis of alternative SLAW options. In brief, the committee believes the team’s draft report provides too little information in meaningful comparative formats useful to support decision-makers’ evaluations, while its use of its AHP results would supplant (or at least anticipate) the decision-makers’ evaluation by performing one of its own.

OBSERVATIONS REGARDING THE APPROPRIATENESS OF AHP FOR THE SECTION 3134 REPORT

AHP is a well-known process to help decision-makers create a structured numerical framework reflecting their objectives in choosing an option from among several where the choice involves trade-offs among multiple attributes or “criteria” (Saaty, 2008). The standard steps of AHP are helpfully outlined in the figure on slide 41 of the FFRDC’s presentation materials for the July meetings in Richland (and as reproduced in Figure 5-1). The hallmark of AHP is that once a set of evaluation-structuring steps are completed, AHP guides each decision-maker through a sequence of one-on-one comparisons of the criteria,

eliciting from the decision-maker what he or she feels is the relative importance of each criterion in maximizing the defined objective. Once all possible pairwise ratings are completed, the AHP software computes a set of relative weights that it will apply to the scores on each criterion (which are assigned in the following step of the process) to produce a rank-ordering of the decision options in terms of how well they meet the overall objective for the decision. Importantly, these weights—and the resulting rank-ordering—necessarily reflect the preferences of the decision-maker.¹

The FFRDC team and several affiliated experts met in person in an office building at Savannah River Site for three full days (May 1-3, 2018) to work through an AHP process aimed at comparing the relative merits of identified SLAW immobilization and disposal options. Approximately 20 FFRDC individuals participated (excluding the committee’s observers, who did not participate except to occasionally ask clarifying questions). The expressed purpose of the exercise, consistent with use of AHP methodology, was to identify the specific treatment/disposal options to be considered, and to establish a ranking of them based on consideration of multiple different attributes.

___________________

¹ In the case of a public policy decision (as opposed to a private-sector or personal decision), a societal perspective on the desirability of the various trade-offs is needed. The preference elicitation process of AHP is not well suited to supporting decisions on major policies because its ranking outcomes lack transparency regarding the trade-offs that are implicitly being made. A more transparent way of supporting a public policy decision would be for the experts to present information about key options and their respective key criteria in a direct and comparative manner, enabling the decision/policy-makers to understand the nature and extent of the trade-offs faced, and to be able to articulate the justification for their decisions in terms of the trade-offs they consider to be in the public interest.

The question of concern to this committee’s review is whether AHP is a useful tool for responding to the requirements of Sec. 3134. On that question, the critical point is that the AHP method is intended for use by the decision-makers themselves, perhaps with supporting input from technical experts, to rank alternatives to support making a specific choice among them. As described above, this is not the role of the FFRDC study (or, indeed, of this committee’s review). The role of the FFRDC team is not to choose or recommend an option but only to provide decision-relevant information about the costs, benefits, and risks/uncertainties of the options. However, in its use of AHP, the FFRDC team has had to incorporate its own set of preferences into the process.

Furthermore, the committee observes that the AHP software used by the FFRDC only allows for a single point estimate of a score, and does not allow for representation of uncertainty about how well an option will perform with respect to a criterion. Thus, it is not a good tool for handling situations marked by significant uncertainty in the outcomes, and is rather more designed for handling decision situations that are made complex by the need to grapple with multiple different attributes of concern. The committee expects that comparisons of the options of concern in the Sec. 3134 report will be strongly affected by uncertainty in (i.e., lack of precise knowledge about) costs and waste form performance, in addition to the fact that there are multiple types of trade-offs to be considered. Thus, lack of features in the tool to accommodate direct representation of uncertainty in criteria scores is another reason the committee believes that the AHP tool is not well suited for responding to the requirements of Sec. 3134. Moreover, although it might be possible to modify an AHP process to input uncertainty ranges on scores, and then to produce uncertainty ranges on rankings, this was not done. Rather, the FFRDC team defined a number of its criteria to reflect concerns with uncertainty on cost and schedule, to name two important issues. This is not a standard method for addressing uncertainty in a cost-benefit or risk analysis.

Thus, while the FFRDC’s AHP exercise has been characterized as “expert elicitation,” the committee notes that it was not a traditional expert elicitation, which is a set of formal procedures to help experts extract from their professional knowledge and experience a subjective judgment about the potential range over which the true value of some parameter or future outcome may fall. Uncertainty was not what was elicited, but merely a “best point estimate.” The fact that the point estimate needed to be only a score of 1 to 5 exacerbates the inappropriate precision that is usually associated with any point estimate on an unknown value, but the scores are nevertheless deterministic values without accompanying uncertainty ranges.

Relatedly, the results of the AHP presented in Appendix F give an unwarranted appearance of authoritativeness and certainty due, respectively, to the numerical complexity of the calculations and the ordinal simplicity of the conclusions. In fact, the complex numerical calculations leading to the rankings is viewed by the committee as a step in the wrong direction for a report that is supposed to identify and communicate the key attribute differences that imply trade-offs that decision-makers ultimately must make when choosing among SLAW options. To an extent, these concerns exist simply because of the appearance that the FFRDC is relying on numerical calculations rather than plain English summaries of what they have ascertained about the differences among the treatment options in terms of the relative magnitudes of their dollar costs (however uncertain) and their risks (however uncertain). Such plain English summaries are what the committee would expect to ultimately appear in Chapter 7 of the final report, rather than further synopsis of the AHP outputs themselves.

Finding 5-1

The FFRDC’s role in the Sec. 3134 congressional mandate is to produce a comparison of the costs, benefits, and risk trade-offs necessary to support a well-informed public policy decision-making process by the U.S. Department of Energy and others, but the implementation of the AHP did not lead to a defensible ranking of the alternatives.

Although the committee is uncomfortable with the use of the AHP process as the basis for the comparative analysis that should be provided in the final report, it does feel that this was probably a very useful exercise for the team members to have gone through, individually and as a group, to help them recognize where there remained important, decision-relevant gaps in their information as of that point in their investigation, and what aspects of the multiple criteria they discussed may be most important to communicate to the decision-makers who will be a primary audience for their final report. The committee urges the FFRDC to now put those insights to use to organize its “Comparative Assessment” chapter (see Chapter 7) to provide a structured comparison of the options according to the central questions outlined above, providing direct cost and risk estimates (with quantified representation of the magnitude and direction of uncertainties) for each option rather than mere scores.

Recommendation 5-1

In its final report, the FFRDC should focus on the decision factors identified by Sec. 3134 as the basis for its analysis. The remainder of the main body of the final report should be structured so as to permit direct comparison of the approaches (including SLAW treatment and pre-treatment variants) according to direct estimation of what is known about each of those factors.

Recommendation 5-2

While ranking alternative approaches according to individual criteria, as the FFRDC has done in the draft report, may inform the decision-maker, the FFRDC’s final report should refrain from attempting or presenting a single or unified ranking of alternatives, or assigning priorities or weights to the criteria—and thus avoid supplanting the role of the decision-maker.