Review of Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts (1997)

This review and commentary by the National Research Council's Panel on Seismic Hazard Evaluation presents the panel's evaluation and critique of the report titled Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts (U.S. Nuclear Regulatory Commission, NUREG/CR-6372, Washington, DC, 1997). The reviewed report was prepared by the Senior Seismic Hazard Analysis Committee (SSHAC), a committee created and sponsored by the U.S. Nuclear Regulatory Commission (USNRC), the U.S. Department of Energy, and the Electric Power Research Institute. The panel was appointed at the request of the USNRC to provide an independent interactive review of the results of SSHAC's efforts.

SSHAC's charge from its sponsors' perspective was to provide an up-to-date procedure for obtaining reproducible results from the application of probabilistic seismic hazard analysis (PSHA) principles established in past practice, not to advance the foundations of PSHA or develop a new methodology. This focus led to an emphasis on procedures for eliciting and aggregating data and models for performing a hazard analysis, rather than an examination of the earth science foundations of PSHA. SSHAC focused on process because previous PSHA studies have shown that different groups of experts can produce highly discrepant results. A second major theme in the SSHAC report is the treatment of uncertainties in data and models in arriving at stable estimates of seismic hazard at a selected site.

With this in mind, the panel found that the SSHAC report offers substantial contributions to the foundations and practice of PSHA. In particular, the panel commends SSHAC for emphasizing the need for critical evaluation of expert opinion. But the panel also identified some limitations in both the report and the recommended procedures, of which

potential users should be aware. Only certain key points are highlighted here in the summary; the rest are included in later chapters.

As stated above, the SSHAC report focuses on procedures for using experts in probabilistic seismic hazard analysis and for determining uncertainties at key stages of the analysis process. In its treatment of the use of expert opinion, SSHAC outlines four possible levels of effort and complexity. But the SSHAC report is strongly flavored by emphasis on hazard analysis for nuclear and other critical facilities, and SSHAC therefore discusses at great length its highest-level (level 4) procedure for evaluating expert opinion. And although SSHAC includes proper disclaimers the unwary reader could gain the incorrect impression that the high-level (level 4) PSHA procedure is needed for every hazard analysis.

The panel agrees that all PSHA projects should share the same basic principles and goals, but that the elaborate level 4 methodology is not required for every PSHA study. SSHAC does indeed recognize that alternate simpler methods are probably adequate for less critical facilities, but the simpler methods are not discussed in detail and the reader is not fully advised about other sources of information. Adequate disclaimers in the SSHAC report should protect the analyst who chooses to use procedures other than those recommended by SSHAC from the need to defend that decision in a regulatory setting.

SSHAC's contributions to PSHA methodology include the testing and full explication of the technical facilitator/integrator (TFI) entity, which is the essential ingredient in implementing SSHAC's high-level (level 4) analysis.¹ The TFI approach was found to be very effective in two workshops on ground motion estimation and led to an unexpected degree of agreement among the experts consulted, who began with many diverse viewpoints. The panel notes that TFI elicitation procedure is not

synonymous with PSHA methodology. Nor is the TFI approach recommended by SSHAC for every PSHA study.

In outlining its four levels of complexity, SSHAC visualizes three distinct roles that experts should play at various stages of the process. First, an expert may start out as the proponent of a particular position (data or model). Then the expert is asked to become an objective evaluator of the positions of the other experts in the group. Finally, the expert becomes an integrator and aggregates all the positions to arrive at a putative position of the whole informed scientific community. This estimation of the position of the whole informed community by integration of the positions of a sample of well-qualified experts is the primary goal of the more complex SSHAC procedure. The panel questions whether any group of experts can truly assess the view of the whole informed scientific community on the entire range of relevant issues.

SSHAC sponsored workshops on seismic source characterization, ground motion estimation, and earthquake magnitudes. These workshops are documented in detail in Appendixes A,B C, and H, of the SSHAC report. The workshops contributed both to the development of the procedures SSHAC recommends and to advancement of our knowledge of the earth science elements of PSHA for the eastern United States. Because SSHAC focused on procedures for PSHA rather than technical issues, some of these valuable results are presented but not highlighted. They deserve more attention.

The SSHAC report emphasizes the importance of how uncertainty is treated because the results of a PSHA can be influenced heavily by uncertainties in the data, the models, or both. SSHAC's treatment distinguishes and emphasizes the difference between two types of uncertainty: aleatory (i.e., uncertainty due to variability inherent in the phenomenon under consideration) and epistemic (uncertainty due to our limited knowledge of the phenomenon). After separation, these two

components must be quantified for the model or parameter under consideration. The panel has more trouble with this element than any other in the SSHAC report.

Recognition of the two kinds of uncertainty is useful initially when eliciting and combining expert inputs. Experts need to be aware of the sources of uncertainties (e.g., limitations of available data) so that they can make informed assessments of the validity of alternative hypotheses, the accuracy of alternative models, and the value of data and then transmit those uncertainties to the TFI. However, as detailed in Chapter 3 of this report, the panel believes that the statistical analysis and uncertainty separation procedures recommended by SSHAC may in some cases be more sophisticated than is warranted by the data or the purposes for which the results are to be used.

During the planning of a PSHA, a detailed analysis of uncertainty would be helpful but typically is not available. It may be sufficient for planning purposes to conduct limited sensitivity analyses, using bounding hypotheses, and to consider the level of effort that would be required to reduce the associated uncertainty.

In addition, the value of an epistemic/aleatory separation to the ultimate user of a PSHA is doubtful. In particular, it is not clear that such a separation would be more helpful than the display of expert-to-expert variability of a mean hazard at the time of an analysis, with an explanation of the source of the differences.

The panel also notes that the SSHAC report's discussions and recommendations on uncertainty and the use of experts are quite independent of PSHA and can be applied to other types of risk analysis. The panel believes that the SSHAC report makes a solid contribution to the methodology of hazard analysis, especially in the use of expert opinion.