2

The Committee’s Major Observations and Overarching Assessment of the FFRDC’s Draft Report

This chapter has two parts: overall observations concerning the Federally Funded Research and Development Center’s (FFRDC’s) draft report of July 15, 2018, including identification of work still to be done, and discussion of the importance of and need for a comparative analysis of the supplemental low-activity waste (SLAW) treatment options, including a cost-benefit analysis. The committee received the team’s second draft report on July 16, 2018. This draft report formed the basis for discussion at the public meeting at Richland, Washington, held July 23-24, 2018. Table 2-1 lists the contents of the draft report, and Table 1-1 lists the FFRDC’s presentations shown at the public meeting.

OVERALL OBSERVATIONS

The committee offers several general observations about the (1) overall structure of the draft report and its completeness, (2) compilation and presentation of information, (3) clarity of the writing for decision-makers, (4) adequacy of the executive summary, and (5) identification of key risks and characterization of those risks.

Table 2-1 lists the contents of the FFRDC’s draft report. The committee’s first observation is that the draft report clearly remains a work in progress. It is also clear that the FFRDC has done significant work, under tight deadlines, since the first set of draft working documents the FFRDC sent the committee in mid-February 2018. Nevertheless, a great deal of work has yet to be done, primarily in the area of integrating a large amount of information for enabling the comparison of alternatives in a decision-relevant format—the central purpose of the FFRDC’s report.

The overall structure of the draft report is that it first briefly discusses “Parameters of the Analysis,” including sections on “Strategy,” “Scope,” “Uncertainties,” “Technical Challenges,” and “Cost-Estimation Summary.” It then gives an overview of the proposed processes at Hanford for treatment of LAW and SLAW. Next, it has a section on “Analysis Risk Assessment,” which is essentially an outline of the components of the intended risk assessment for the types of risk to be considered (project, alternatives, and environmental); it does not contain the results of any actual risk assessments.

These initial chapters are followed by chapters that will address, in summary form, the specific elements of the task before the FFRDC: the technologies assessed by the team (i.e., pre-treatment, vitrification, steam reforming, grout, and other approaches) (Chapter 4), disposal site (Chapter 5), transportation (Chapter 6), and finally a comparative analysis of approaches (Chapter 7). These summaries do not appear in the report and had yet to be written by the time of the committee’s public meeting on July 23-24, 2018. Instead, placeholder phrases refer the reader to a series of appendixes, A-K, which provide detailed discussions of the relevant materials, technologies, regulations, cost, and comparison methodologies. The appendixes are descriptive, as opposed to evaluative or analytical. Thus, the appendixes contain more than 150 pages of valuable supporting information and a foundation for the summary chapters; however, each appendix functions essentially as a stand-alone document, and its essential information still has to be integrated into the main body of the report.

In the chapters to be completed, the FFRDC would have to integrate, analyze, and compare the alternatives being considered in its analysis. At the July public meeting, the FFRDC team leader told the committee that the FFRDC will provide a complete final draft report before the next public meeting on November 29-30, 2018.

The committee’s second set of observations is about the compilation of information in the draft report. The report contains an impressive amount of information, but the authors have not prioritized this information according to either the general significance of the information, or—more importantly—the extent to which particular factors distinguish between alternatives and thus form the basis for choosing among them. For example, facts and findings are not distinguished according to their significance or influence on the choice among approaches. In addition, this information lacks meaningful integration. For example, the approaches to SLAW treatment are presented separately and not as a comparative analysis. A comparative analysis—presumably to be supplied in the future report in the summary chapters—would most usefully serve the needs of decision-makers.

The committee’s third area of observations concerns the ways in which the draft report communicates information and analysis. This is especially true for the choices on whether to primarily use very technical language or laypersons’ terms. This is important because many readers of the final report will not be engineers and scientists. Moreover, many decision-makers and stakeholders are not technical experts in the complex subject matter of SLAW treatment. Thus, decision-makers would benefit from a final report that uses transparent plain English (as much as possible), with clear explanations of complex technical issues where required.

TABLE 2-1 List of the Chapters and Appendixes in the FFRDC Draft Report, “Report of Analysis of Approaches to Supplemental Treatment of Low-Activity Waste at the Hanford Nuclear Reservation,” Dated July 15, 2018

The committee’s fourth observation is that the executive summary neither reflects key findings of the draft report, nor does it adequately characterize the alternatives, explain systemic risks, or in any way provide the important information needed by decision-makers. The executive summary would have to serve this purpose, as it is often the most widely read part of a document.

The committee has also observed that the FFRDC team has done a credible job identifying individual key risks (and assumptions—which by their nature can represent uncertainties in the risks) associated with the overall tank farm cleanup and its potential impact on SLAW composition and feed rates. The team has also identified key risks and assumptions with the three primary SLAW processing approaches and variants within the approaches. Significant work, however, remains to be done in characterizing those risks and their impacts on the alternative approaches to treating SLAW both individually and comparatively. It is particularly important for the FFRDC to explain, characterize, and transparently present information on alternative approaches to SLAW treatment (and their relationship to the larger tank remediation effort) to help decision-makers understand the highly complex nature of the SLAW treatment system and how their decisions can impact future outcomes. The FFRDC’s draft report as currently configured does not achieve this goal.

The introductory chapter, “Parameters of the Analysis,” could usefully be structured as a more traditional introduction to present what the team is trying to do in the rest of the report (focusing on comparing SLAW treatment alternatives), why the team is doing it (meeting the congressional mandate in Section [Sec.] 3134), and how the team is doing it (defining the scope of the SLAW treatment, identifying main process alternatives, developing flowsheets for each alternative, obtaining or creating data for each criterion of each alternative, juxtaposing the data, and evaluating the juxtaposition to support the comparison among alternatives). Some of these steps are listed in the draft report. But the trail leading to the evaluation of the juxtaposition of alternatives is too brief and is camouflaged by the early introduction and extensive discussion of the GAO “best practices.” Some specific suggestions for improving this section are as follows:

• Here and in the rest of the report the sequence of thought and analysis need to be transparent to readers who may be unfamiliar with the details of the Hanford cleanup effort or possibly even nuclear technology.
• Move the “best practices” table to an appendix with a short description of how they were used, including discussion of why the U.S. Department of Energy (DOE) orders, guides, and standards were not used.
• Associate the discussion of uncertainties, technical challenges, and cost estimation with related sections in the main body or appendixes.
• Identify previous studies of and experience with SLAW treatment and pre-treatment technologies, summarize their results, and discuss any important differences between the previous and current results. For example, specifically, previous bulk-vitrification work has been explored at Hanford. While there are important differences between the bulk and currently proposed vitrification processes, that experience could yield valuable insights into technical and cost aspects of the proposed SLAW process.

THE NEED FOR A COMPARATIVE ANALYSIS

According to Sec. 3134 (see Appendix A), the task set by Congress for the FFRDC team is to perform “an analysis of approaches” that will provide the technical and analytical basis on which DOE, in consultation with the Washington State Department of Ecology and other stakeholders, can make a well-informed choice among the technologies available for treating SLAW for permanent disposal. This is not a simple task. Within a very constrained amount of time, the team is required to gather and present a very large amount of information, created over many years, about three basic approaches to treatment of SLAW; and to present that information in a way that enables decision-makers to make an informed and respected choice in a context of intense public scrutiny and even skepticism. At the same time, because the team is neither the decision-maker, nor has it been asked to recommend a particular choice among alternatives, the team’s

report has to be careful to provide an analysis in which the particular strengths and weaknesses of each approach are clearly and fairly represented. The team’s role is to facilitate the choice, but not preordain or make the choice. To facilitate decision-making, the committee offers this overarching guidance for all future reports of the FFRDC:

• Assemble and frame information in ways that will clearly convey useful information to decision-makers.
• Find and use communications experts and a technical editor in assembling and framing information to be accessible to non-technical readers.
• Provide analyses that will make it relatively easy for decision-makers to compare the various aspects of the treatment approaches without preordaining the conclusion.
• Ensure that the graphs and images are designed and selected to facilitate interpretation and guide decision-makers, for example, by showing a projected timeline of construction and operations for each approach with projected costs per year.

More fundamentally, the committee suggests that the final report provide a structured comparative analysis that can form the basis for selection among alternatives. Sec. 3134 (see Appendix A) provides criteria for the FFRDC to use:

• The risks of each approach,
• Their costs and benefits,
• The anticipated schedules,
• Compliance of each approach with applicable laws and regulations, and
• Any obstacles to implementation.

While a comparative table appears in Appendix F (see Table F-2) of the draft report, comparative analysis is the essence of the congressionally mandated task, and the committee suggests that the FFRDC consider it as the organizing structure. A useful final report would be fundamentally structured around a common set of factors. For example, there is no clear parallelism in the presentation of each approach in the appendixes in accordance with these criteria. This is more than a rhetorical misdemeanor or a mere convenience for the reader. It deprives the user of the report—and indeed the authors of the report—of a direct side-by-side comparison of alternatives that informs deciding among them. If this is not done, key features are likely to be missed, and comparisons are likely to be inconsistent. While detailed descriptions of each approach are unquestionably important, they would most usefully be presented with the comparative task and the ultimate objective.

Likewise, the criteria would need to be characterized so as to respond to decision-makers’ questions, and not general exposition. The committee suggests that the final report consider the sources, consequences, and probabilities of the several types of risks (health, environment, social, and regulatory) posed and avoided by each approach; assess the reliability (the likelihood that it can be made to operate and then continue to produce waste forms of the intended quality, at the expected rate, on a sustained basis) of the technology underlying the approach; and estimate the full lifetime costs of approaches.

A comparative analysis would have to recognize that not every criterion is equally important, and that some criteria are more meaningful discriminators among approaches than others. Up to the point of the draft report, the FFRDC team has, quite reasonably, focused on comprehensive information gathering and technical assessment of each approach, as the summary-and-appendix structure of the draft report clearly reflects. This structure is both sensible and commendable, as comprehensiveness has utility and is essential to the added value and the credibility of the present study.

The committee suggests, however, that in the forthcoming final report, the team prioritize the information and analysis for decision-makers by asking some relevant questions:

• What information will be of most salience to DOE and its stakeholders?

• What information would be “show-stoppers” or trigger no-go decisions?
• What information makes little difference to the final decision, either because it is relatively unimportant, or because there is little meaningful difference between the alternatives in that respect?
• Where do uncertainties overwhelm the results of the analysis?

This kind of analysis is not foreign to the team, because in its comparison section the team rejected for further consideration several variants of the three basic approaches. This was a useful and desirable step in the comparison process, in order to winnow down the alternatives to a manageable number for decision-makers. According to its presentation at the July public meeting, the team used criteria such as exorbitant cost, technological infeasibility, and not differentiating among the approaches or variants as a basis for winnowing. The team’s winnowing results seem sensible; however, the winnowing process is not well described and is essentially tacked onto, rather than being integral to, the draft report.

Cost-Benefit Analysis

An essential part of the comparative analysis is a cost-benefit analysis. Sec. 3134 requires the FFRDC to analyze “the benefits and costs of such approaches,” where the approaches are the various technologies for the treatment of SLAW. The committee notes that the FFRDC’s draft report does not explicitly discuss cost-benefit analysis. However, the draft report does present cost estimation, which is a necessary input to a cost-benefit analysis. A cost-benefit analysis sums the total costs of an activity (in this study, a treatment approach) and compares it against its total benefits. For ease of comparison, a cost-benefit analysis typically places a value on each benefit. But in practice, many of the key benefits are often not possible to monetize, and that is the case here too. Nevertheless, all important benefits need to be identified and estimated, even if not in a monetized form, so that one can assess the nature and degree of trade-offs between the costs of alternative options and the benefits (or risks) associated with those options. Indeed, a clear and well-structured list of the important categories of costs and benefits that could arise from consideration of each treatment approach can provide very useful input for decision-making and for strategizing; providing an analysis-based summary of their relative magnitudes (even if in non-monetized form) is, however, a hallmark of good cost-benefit analysis.

On the benefits side of the ledger, there will be many commonly shared benefits for the different treatment approaches. For example, each approach will provide the benefit of a vastly safer and healthier environment in the Hanford region compared to current conditions. It is clear that this benefit can vary depending on the waste form, chosen disposal site, and disposal performance, and the challenge for decision-making is in understanding the qualitative and quantitative differences in these benefits for different SLAW options. Thus, the main qualitative dimensions of these benefits need to be explained in understandable terms, and within those benefit dimensions the relative performance of each option needs to be reported in terms that allow direct comparison to each other. It is important to explain not just that the options differ in their expected waste performance, but by how much. For example, risk to the most exposed individual over time should be illustrated for each option, even if broad uncertainty bands are placed on each. The same can be done regarding differences in cleanup schedule and its uncertainties. Only in this way can decision-makers make informed decisions about how much additional cost is “worth it” from a societal perspective.

As the FFRDC has already recognized, the costs can certainly vary significantly for different treatment approaches (as discussed in Appendix G of the draft report and Chapter 3 of this review). However, an important and essential technique of cost-benefit analysis is to present the present value of the costs, and to present the timeline of the costs that comprise that present value to understand its implications from a budgetary risk, as well. Present value is a quantitative means to recognize that costs occurring in the future are weighted less than costs incurred today. This is accomplished by discounting future costs in the spending timeline to compare them to near-term costs. The U.S. Office of Management and Budget recommends that

cost-benefit analyses use both a 3 percent and 7 percent discount rate when comparing alternative regulatory options, to help understand how sensitive the total costs are to a range of discount rates that may be appropriate for societal decisions. In section 1.1, page 13 of the draft report, it is noted that the U.S. Government Accountability Office (GAO) best practice is to give “the life-cycle cost estimate in present value terms” and discuss the reason for choosing a specific discount rate. The team mentions that it “presented life-cycle costs in present value terms,” but it does not mention what discount rate it chose and why a particular discount rate was chosen or not chosen. Also, the report does not provide the associated timeline of spending, which would have to be exactly consistent with the present value estimates, and are important for understanding if some of the options involve very different budgetary challenges.

The above discussion of present values is stated only in terms of the cost element, because the committee does not expect that the benefits that the FFRDC needs to characterize will or can be done in a monetized format. If they could be, then discounting would apply to benefits estimates, and the present value of net benefits (i.e., benefits minus costs) would be appropriate to compute as well. However, note that the suggestion to present risks over time reflects a similar informational objective.

Finding 2-1

The information compiled by the FFRDC team does not guide the reader to the Sec. 3134 factors and criteria that are most likely to distinguish one treatment approach from another. Especially in view of the volume of information (in more than 150 pages of appendixes) and the number of approaches and variations, it will be difficult for DOE and stakeholders to make the most effective use of the report as presently configured, which has all of the “summary” chapters labeled as “To Be Determined.”

Finding 2-2

The FFRDC’s draft report emphasizes information and analysis concerning the treatment approaches in isolation from each other, making a direct comparison of them difficult for decision-makers. At this point in the drafting process, the information presented in the draft report lacks meaningful integration for the purpose of comparison. For example, the approaches to supplemental waste treatment are presented separately and not as a comparative analysis. Because the fundamental purpose of the FFRDC analysis is to form the basis for making choices among alternatives, a direct comparison for each factor will greatly improve the utility of the report for decision-making.

Recommendation 2-1

In order to distinguish one treatment approach from another, in its final report the FFRDC should identify the distinctive aspects of the chosen approaches in a comparative analysis. Moreover, the FFRDC needs to present that information in a manner consistent with cost-benefit analysis practice, adapted to the case where benefits are presented in non-monetized terms.

Recommendation 2-2

The final report should emphasize and describe in detail the most important differences among the alternatives with respect to the decision-relevant attributes, and present the areas of difference in a parallel format that enables ready and accurate comparison for the purposes of the selection of a preferred alternative by the decision-maker.