2

Committee’s Review of the FFRDC’s Draft Methodologies

The first charge in the committee’s Statement of Task (see Appendix B) is to evaluate the technical quality and completeness of the methods used by the Federally Funded Research and Development Center (FFRDC) team to conduct the risk, cost estimation, schedule, and regulatory compliance assessments and their implementation.

RISK ASSESSMENT

Key Points in the FFRDC’s Work

In Document 2, the FFRDC describes three risk assessment methods that it is planning to apply to its analysis: probabilistic risk assessment (PRA), semi-quantitative risk assessment, and qualitative hazards analysis. As to PRA, the FFRDC notes the U.S. Nuclear Regulatory Commission’s defines its purpose as being “to estimate risk by computing real numbers to determine what can go wrong, how likely is it, and what are its consequences. Thus, PRA provides insights into the strengths and weaknesses of the design and operation of a nuclear power plant” (USNRC, 2018). Document 2, p. 2, mentions that PRA “usually involves a very structured, systematic, and quantitative analysis explicitly accounting for uncertainties through probabilistic methods. In comparison, semi-quantitative methods are intermediate between a fully numerical PRA and a textual, qualitative risk assessment.” Semi-quantitative methods “provide a structured approach to ranking risks with numeric scores, frequently using expert input versus mathematical models.” The FFRDC describes qualitative hazards assessment methods as producing “non-numerical estimates of risk, and may use a risk matrix to organize levels of impact and likelihood, and prioritize or rank risks for future action.”

While the FFRDC states in Document 2, p. 2, that each of these risk assessment methods is being applied in different areas, the FFRDC highlights (Document 2, p. 3) that it will apply, for its main risk assessment method, an expert elicitation technique to perform the semi-quantitative analysis and specifically will use Expert Choice^® software¹ to collect the elicitation input and help the team perform its analysis. Also, the FFRDC team members, who are experts from multiple U.S. Department of Energy (DOE) national laboratories, will comprise the subject matter experts needed to carry out the expert elicitation and that a subject matter expert in risk analysis will support the expert elicitation’s development and implementation.

In Document 2, pp. 3, 5-6, the FFRDC discusses that it has defined or will use pre-defined lines of inquiry (LOIs), such as safety and waste form performance, in its expert elicitation. Each LOI has criteria associated with it. For example, the LOI for safety includes nuclear safety, chemical safety, accident/hazards analysis, and hazards requiring controls. Each criterion will have a metric from one to five to allow the experts to rank whether a treatment technology has low to high confidence for meeting that criterion.

___________________

¹ For the Expert Choice^® software’s website, see https://expertchoice.com.

Committee’s Observations

The committee notes that Congress has provided guidance about specific risk assessment methods in the congressional conference committee’s report on Sec. 3134. That conference report states that an amendment to the act removed the requirement that the FFRDC should use a specific risk assessment approach. But the conferees drew attention to Section 3161 of the National Defense Authorization Act for Fiscal Year 2013, which specifically calls for “probabilistic or quantitative risk assessment if sufficient data exist.” The conference report further points out that the Sec. 3134 conferees “expect that, to the extent practicable and appropriate, the analysis shall be conducted using state-of-the-art assessment practices such as probabilistic risk assessment” (Congress, 2016). The committee also notes that DOE published in 2013 a standard on “Development of Probabilistic Risk Assessments for Nuclear Safety Applications”; this publication was based on the standards, guides, and best practices used across a number of industries and was designed to be used to complement qualitative and deterministic methods for developing hazard assessments, hazard controls, and safety management programs (DOE, 2013).

Based on the committee’s review of Document 2, it is unclear to the committee how lines of inquiry and associated criteria are determined and defined. Also, the expert elicitation technique and where exactly it will be applied in the analysis are not fully described. During May 1-3, 2018, two committee members and the study director observed the expert elicitation session that used the Analytic Hierarchy Process (Saaty, 2008) for pairwise comparison among criteria and options. Expert Choice^® software was used to record these expert assessments and allowed the experts to perform some sensitivity analyses. During the public meeting in July 2018, the observers will give a summary of their observations. The full committee will review the results of this elicitation that the committee anticipates will be described in the forthcoming FFRDC report.

The committee noted some gaps in the draft risk assessment methodology, as described in Document 2 and Presentation 8, as well as Presentation 7, which mentions issues related to transportation of wastes off site. For instance, the committee notes that in the FFRDC’s consideration of shipping waste forms outside of Washington State, some additional risks were not explicitly mentioned, in particular, the possibility of transportation accidents would lead to consideration of health risks to populations near the accident sites. In addition, in the transportation routing, several state boundaries would have to be crossed and would introduce programmatic risks (for example, legal or political opposition) into the evaluation.

While tank waste management is not within the scope of the FFRDC’s analysis, tank failure is a primary concern given the fact that in recent years DOE had discovered water intrusion in several tanks and leaks from a couple of tanks (GAO, 2014). The committee stresses the likely increased risk of tank failure if there is no or delayed action on waste processing. The longer the treatment process will take, the longer the waste remains in tanks whose useful life expectancy is finite; the 149 single-shell tanks were built between 1943 and 1964 (NRC, 2006a). The committee also notes that this concern could be relevant for the FFRDC’s analysis if the throughputs of competing supplemental low-activity waste (SLAW) treatment approaches and the expected lives of the various tanks being sequenced for a particular treatment are significantly different. As to potential changes in throughput, Document 8 discusses the baseline assumptions for feed material to be processed in SLAW treatment:

Changes in the required throughput of Supplemental LAW could occur if the schedule for completion of LAW immobilization changes from the current assumption of 40 years after the start of HLW process (to allow the LAW mission end to coincide with HLW mission end). It is noted that acceleration of the mission is not simply a matter of building [a] bigger immobilization facility; tank farm operations would need to be scaled similarly to allow retrieval of waste to meet the processing needs of the larger facility. (Document 8, p. 2)

This would appear to indicate that the FFRDC will assume that all treatment alternatives will have the same lifetime (40 years after the start of high-level waste [HLW] treatment) for the purpose of performing its analyses, but it is unclear whether the FFRDC’s analysis will consider the benefits of higher throughput

with respect to the expected lives of the tanks and the impacts on SLAW operations of having to quickly empty an unexpectedly failed tank. The risks of the present configuration of the waste material, including the need to transport the waste a substantial distance from the tanks to the Waste Treatment and Immobilization Plant (WTP), are relevant to DOE’s overall management decisions. However, they do not affect the choice among the primary options, except inasmuch as the difference in duration of treatment is relevant to the integrity of the tanks that are past their expected useful life. The committee notes that consideration of new tank construction is beyond the scope of this study.

Committee’s Suggestions

While the FFRDC has provided useful information about its proposed analytic techniques, the FFRDC has not discussed specifically how and where it will apply PRA techniques. Also in light of congressional interest in application of PRA, to the extent practicable, the committee suggests that the FFRDC discuss in its forthcoming report what type of PRA will be used, the sources of uncertainties in risk assessment data, the parts of the SLAW system to which it will be applied, and the basis for not applying it to other parts of the SLAW system, as well as the basis for selecting the risk analysis techniques applied to other parts of this system and to other risks such as schedule, technology, and cost. It will also be useful for the FFRDC to discuss which risks can be quantified and which can be analyzed using qualitative assessments, e.g., legal and regulatory risks.

The FFRDC has not fully described in detail how it will perform the expert elicitation and on what parts of the SLAW system. While as noted above, the committee had three observers at the May 1-3, 2018, expert elicitation session, the committee still believes that it is useful for the team to give a full description of the elicitation process to be used in its forthcoming report.

As to understanding lines of inquiry, the committee suggests that the forthcoming analytic report discuss more fully how lines of inquiry and associated criteria are determined and defined. It would be useful for the results in the FFRDC analytic report to summarize, compare, and contrast relevant results from previous reports concerning supplemental treatment.

The committee, in addition, suggests that the FFRDC take into consideration the additional health risks and high-level legal issues that could challenge transportation of wastes off-site. While the committee notes that political risks will be an important consideration for DOE, the committee understands that there is little, beyond identifying such risks, that can be done within the FFRDC’s scope.

COST ESTIMATION

Key Points in the FFRDC’s Work

Document 7 and Presentation 9 outline the FFRDC’s preliminary considerations for the estimation of costs. Notably, the FFRDC intends to provide a Class 5 planning estimate, which is useful at the concept screening phase of a project to develop an order-of-magnitude estimate. To make the cost estimates, the FFRDC can draw on expert judgment, expert opinion, and specific analog projects (DOE, 2011a). Document 7, p. 2, notes that such estimates “have the least project definition available (from 0 to 2 percent) and therefore have very wide accuracy ranges. They are the fastest of the five types of estimates to complete, but they are also the least accurate.” For its cost estimates, the FFRDC mentions that it will make use of cost data of similar processes and analogous facilities that were designed, developed, and deployed at other sites.

In its draft report and presentations, the FFRDC team states that it is using the One System Integrated Flowsheet (Cree et al., 2017) to estimate feed vectors for the notional supplemental treatment approaches. As to cost estimation, the FFRDC team mentions in Document 1, p. 1, that this flowsheet will be used “to determine the scale of the facilities during cost estimation.” The other primary basis of the FFRDC’s ongoing analysis is the River Protection Project System Plan’s Revision 8, often referred to as “System Plan 8” (ORP, 2017). The foreword to System Plan 8 states that the plan “is a computer modeling exercise, which

evaluates a set of 11 technical scenarios and provides rough cost and schedule estimates” and is “not intended as a decision document or budget document.” But it “will aid discussions with regulators and other stakeholders” (ORP, 2017).

Committee’s Observations

While System Plan 8 states that its cost estimation figures are not detailed or precise enough for decision-making, the information in System Plan 8 is still useful for the FFRDC’s analysis, even if this would only produce order-of-magnitude estimates. The committee notes that it is too early to conclude that cost estimation cannot be adequately based on the data available to the FFRDC. Also, the committee recognizes that the FFRDC will obtain other useful cost data from similar processes and analogous facilities.

Committee’s Suggestions

The committee suggests that the FFRDC team in its forthcoming analysis discuss how order of magnitude (which is significantly uncertain) cost estimates could be useful to decision-makers including analysis of how the quality and uncertainty in the cost estimates affect their usefulness to decision-makers. Also, the committee suggests that the team consider use of DOE’s “Cost Estimating Guide” because it provides guidance on methods and procedures that are to be used in programs at DOE for preparing cost estimates (DOE, 2011a).

SCHEDULE ASSESSMENT

Key Points in the FFRDC’s Work

On the assessment of the anticipated schedules for supplemental treatment approaches, including the time needed to complete necessary construction and to begin treatment operations, Presentation 8 states that the team will assess the time needed to implement each treatment technology by comparing against the current DOE’s Office of Environmental Management baseline liability profile and by evaluating each treatment option’s opportunities to improve the schedule. In addition, the team will review previous scheduling estimates for each disposition technology. This presentation specifies two questions the team is considering:

• Can the disposition technology accelerate the baseline schedule?
• How likely is meeting the estimated schedule?

In Presentation 8, the team, in addition, outlines its considerations of technological complexity and process risks. The team mentions that an assessment of technological complexity will include evaluating “the level of difficulty in operating and maintaining required facilities and unit operations for each disposition technology” to include:

• Number and type of unit operations
• Expected life of processing equipment
• Secondary waste generation/disposition
• Packaging operations
• Ability to handle process upsets (such as off-spec products)

The team notes that considerations of process risks will include whether the process would fail to make acceptable immobilized product by having the product out of specification. Another process risk is whether the throughput is met. The presentation also mentions generation of excessive secondary wastes as a process risk.

Committee’s Observations

The schedule for completion of waste treatment is estimated to take at least four to five decades based on System Plan 8. Most of the scenarios for tank waste treatment in System Plan 8—which are driven by initial assumptions defining each scenario—extend to the 2060s, and two scenarios project tank waste treatment to be finished in 2079 and 2081 (ORP, 2017). The committee notes that the issue areas of technological complexity and process risks are not explicitly linked in Presentation 8 to evaluation of scheduling risks. Given the complexities involved in the treatment scenarios and the technical challenges involved in treating 177 tanks filled with approximately 56 million gallons of hazardous and radioactive wastes, the committee observes that assessments of the impact of technological complexity and process risks are useful to more fully understand schedule risks.

Presentation 8 does not include in the schedule risk the potential that some materials and specialized components needed for certain SLAW treatment options could become less available or more expensive. For example, fly ash is proposed to be used for the grouting approach. Although there are many stockpiles of fly ash, a byproduct of coal-fired generation of electricity, not all fly ash is suitable for the grouting approach. Thus high quality fly ash might become scarce if coal-fired plants become less available. To minimize this aspect of scheduling risk, alternative materials such as calcined clay and natural pozzolans can be considered. Early warnings of changes of availability of materials can be obtained by connecting with industry associations such as the Portland Cement Association and the American Coal Ash Association.

The committee also notes the importance of receiving continuous funding at needed levels to meet the planned schedule. It is well known that budgets can have significant impacts on schedule performance. Schedules can be accelerated or delayed depending on the budgetary assumptions and actual receipt of funds to manage a project.

Committee’s Suggestions

While the committee views the proposed scheduling evaluation as a good starting point, the committee suggests that the FFRDC’s assessments of process risks at unit operation and system level (specifically the interactions among the units), as well as overall technological complexity connect to its assessment of the risks to achieving the SLAW treatment schedule for the various treatment approaches. Given the importance of schedule as a reason for commissioning the FFRDC study and the increasing danger of tank leakage over time, schedule risk must be a central consideration.

The committee notes that additional considerations could result in better estimates of schedule risks, by adequately understanding the interplay between technical and schedule risks. Such considerations could include considering an incremental approach that would have the SLAW choice be made after the WTP is operational, keeping in mind the WTP’s start date, and performing a sensitivity analysis of the optimistic assumption of 70 percent availability for the feed material into the SLAW facility. That is, it is worth considering performance of a sensitivity analysis of the likely downtimes or failures of essential equipment as well as of the potential for using modular equipment that could be switched out in the event of equipment breakdown.

While schedules can be accelerated or delayed depending on the budgetary assumptions and actual receipt of funds to manage a project, the FFRDC team should consider whether it would be helpful to acknowledge the existence of such risks and their impacts on its analysis, so that DOE can take them into account in its decision-making.

REGULATORY COMPLIANCE ASSESSMENT

Key Points in the FFRDC’s Work

In Presentation 8, the FFRDC notes the draft status of the Integrated Disposal Facility’s (IDF’s) waste acceptance criteria (WAC) and indicates that the draft criteria are mostly comparable to the WAC at the

Waste Control Specialists (WCSs) disposal site in Texas. The IDF’s draft criteria are based on disposal of vitrified LAW and still being reviewed by DOE as of April 2018. In Presentation 8, the FFRDC mentions that it will conduct a mini-performance assessment to compare alternative waste forms and to assess whether a waste form will meet long-term performance objectives. That assessment will account for differing release mechanisms for various radionuclides from each waste form and the rates of transport to groundwater.

Committee’s Observations

The committee observed, during the most recent public meeting, that significant disagreements were expressed about fundamental aspects of the regulatory environment, which could make the feasibility of some alternatives highly uncertain. One important disagreement is that two of the treatment approaches depart from the widely assumed use of vitrification as a waste form. Although the State of Washington appears to accept the HLW-LAW-SLAW waste products as the appropriate regulatory-treatment structure, it does not accept deviation from what it believes is an agreement to use vitrification for SLAW, unless the alternative is “as good as glass.” This difference introduces uncertainty into the acceptability and potential timelines for alternatives to vitrification technologies. Although this may not be readily quantifiable by the FFRDC analytic report, the committee notes this issue as well as the consequences (delays, waste disposition elsewhere, etc.) of rejection of an alternative to vitrification technologies.

As to other features that would help with compliance assessments and other analyses of SLAW treatment alternatives and their implementation, the committee notes that the FFRDC has not specified comparable alternatives (starting with the same feed vector, accounting for all effluents and wastes and including pre-treatment) that will be analyzed from the numerous options identified in various parts of their draft report.

Similarly, there appears to be a time-of-compliance disagreement between DOE and the Washington State Department of Ecology. DOE’s Order 435.1 concerning waste disposal has a compliance period of 1,000 years while the Washington State Department of Ecology focuses on the time when the calculated dose will reach its peak, which is projected to be several thousand years after waste disposal (Smith, 2018). However, at the recent public meeting, the Washington River Protection Solutions presentation on the performance assessment of the IDF showed graphs with dose estimate calculations to 10,000 years and with a broad peak in calculated doses occurring over several thousand years (Lee, 2018). Resolution of this disagreement might well affect the selection of the final treatment technology.

The committee understands that the results of the FFRDC’s performance assessment of the disposal facility could also help with the identification of potential regulatory obstacles that might affect the need for or desirability of SLAW feed processing. Sec. 3134 specifically called for the FFRDC to examine “further processing [conditioning] of the low-activity waste to remove long-lived radioactive constituents, particularly technetium-99 and iodine-129, for immobilization with high-level waste” and to assess compliance with applicable laws and regulations under this and the other treatment options being evaluated. Removal of technetium-99 and iodine-129 could significantly affect the disposal options and even open consideration of other sites for disposal.

Committee’s Suggestions

The committee suggests that the FFRDC’s analysis discuss whether the non-vitrified waste forms being assessed could be considered “as good as glass,” because this standard is a de facto element of the decision-making environment at Hanford. The committee also suggests that the FFRDC’s analysis discuss how consideration of processing alternatives to remove technetium-99 and iodine-129 could affect disposal options, taking into account compliance with applicable laws and regulations. In addition, the committee suggests that the FFRDC’s analysis discuss differences in requirements concerning time-of-compliance, and the impacts of various assumptions on lines of inquiry such as cost, schedule, and risk.

Finally, in this chapter, the committee suggests that the FFRDC’s analysis report define the flowsheets that would be compared from the waste receipt tank showing the steps of blending, conditioning, treatment, transportation, and disposal. Also, the committee would like to know the material balances for the key radionuclides in each flowsheet.