Review of Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts (1997)

Committee on Seismology, National Research Council

Comments on SSHAC Draft Report of 11 November 1994

Based on the Panel Meeting of December 9-10, 1994

The Panel on Seismic Hazard Evaluation of the Committee on Seismology, National Research Council (NRC), is charged with reviewing the report to be produced by the Senior Seismic Hazard Analysis Committee (SSHAC) under the sponsorship of the U.S. Nuclear Regulatory Commission (USNRC), the Department of Energy (DoE), and the Electric Power Research Institute (EPRI). The USNRC prescribed that the Panel provide feedback to SSHAC as they prepare their report, but in such a way as not to compromise the objectivity of the Panel in providing its review of the final product. SSHAC submitted for review a draft of their report in mid-November, 1994, and the Panel met, with all SSHAC members present, on December 9, 1994, for discussion of the draft.

Unfortunately the draft was not complete, missing some key appendices, some sections of text, and an executive summary. It should be understood that the Panel may have comments with regard to the missing material when it is available for the final review. The discussions of December 9 were carried out in the presence of representatives of the sponsoring organizations. The Panel met in executive session on December 10 to continue its review. The resulting comments and recommendations are submitted to the USNRC.

The suggestions made are offered as guidance to SSHAC on the issues at this stage of their work, in accord with the request of the USNRC. They should not be interpreted as a substitute for the final report to be developed by the Panel.

The Panel believes that the draft report is a basis for a useful final product that has the potential to advance the process of Probabilistic Seismic Hazard Analysis (PSHA). However, the Panel feels that the introduction to the report must be expanded to make clear the purpose and scope of the report, and specifically to state what the report is not. As it stands, the report implies that the methodology is applicable to a broader range of facilities than can be justified. The full range of alternative approaches is not discussed, let alone taken into account.

From the discussions, it appears that there may be a conflict between the expressed needs of the USNRC for a single unified, fully prescribed regulatory method of seismic hazard analysis (SHA) and the attempt by SSHAC to produce a general consensus methodology. The USNRC wants a prescribed procedure that is based on what has been learned from past PSHA experiences. The USNRC recognizes that the way in which input from experts was obtained is a main reason for the discrepencies between the analyses made by Lawrence Livermore and EPRI.

The Panel recognizes the strengths of the report and the significant contributions it offers to PSHA. As applied to nuclear regulations the SSHAC report breaks new ground in its discussion of the Technical Integrator (TI)/Technical Facilitator Integrator (TFI) approaches. However, as discussed in detail below, the presentation of these ideas needs to be made more clear to eliminate some apparent contradictions and advise the users of the report when the full TFI treatment is called for. The TI/TFI approach has the potential to overcome some aspects of past PSHA applications that have led to objections by critics of the whole process.

Because the focus of the report is on process for PSHA, rather than on the underlying earth science, the detailed attention to the treatment of uncertainty is appropriate. However, as discussed below, the motivation for this careful treatment of uncertainty and the way in which the results will be applied are not made clear to the potential user.

Again without yet having the benefit of full discussion of the subject, the Panel feels that the recommendation that behavioral aggregation of expert input be employed is sound, because mechanical aggregation algorithms, if used as “black boxes,” may lead to poor results.

Some suggestions for revision and restructuring of the report were given orally to SSHAC during the Panel meeting. The most essential of these, which the Panel feels cannot be neglected during revision of the report, are repeated here for completeness of the record.

The word “Consensus” should be removed from the title, perhaps replaced by a more appropriate adjective.

An excellent executive summary is essential for the success of this report. The report is lengthy and detailed. The key findings and recommendations of SSHAC must be assembled in concise, easily understandable form if they are to be accessible to others than the experienced practioneer of PSHA.

The draft as submitted is overly repetitious. Unnecessary redundancy should be eliminated, to reduce the length substantially without loss of content.

The specific criticisms to follow all can be categorized as due to one or more of the following: inadequate focus of the report, absence of the history of evolution of the key concepts and recommendations, or lack of a presentation of the context within which the report was developed and is to be understood and applied.

Motivation. The reader should be offered better motivation for adopting the procedures required or recommended in the report. In addition, the context for the procedures should be framed in such a way that the PSHA analyst who follows other procedures for any of a number of valid reasons is not put in a position of having to defend in a regulatory situation the failure to carry out the SSHAC prescription in every detail.

PSHA methodologists often have sound reasons for introducing new concepts and approaches, but have not always included in their reports the background reasoning that has led to these innovations. Where it exists in this report, this shortcoming must be overcome if the final SSHAC product is to be widely accepted and applied. In particular, the report should say how the results are to be used as motivation for the great emphasis on the distinction between aleatory and epistemic uncertainty and the need to separate the two in SHA.

Space and emphasis devoted to the TFI approach. Scattered through the text, and asserted by SSHAC members at the December 9 meeting, is the key idea that the full TFI approach is required only for some complex issues for which a review of the published literature cannot produce satisfactory input to the PSHA process. However, the great detail in which the recommended TFI approach is depicted tends to obscure this principle. The reader is left with the impression that the use of the TFI is dominant in a properly executed PSHA.

• SSHAC must carefully set out the criteria for deciding if an issue requires a TFI. What are the operational criteria for deciding if an issue is of type A, B, C?

• SSHAC must state its perception of the qualifications required of the TFI. The recommendation for use of a strong TFI for prescribed issues, without clearly expressed qualifications, contradicts one of the stated criteria for success: that the recommended methodology, when applied independently by different groups, should always yield comparable results.

• The Panel is concerned that the TFI is empowered to act as a “super expert,” able to overrule the diverse views of the experts from whom input is elicited. It is not prudent to generate an apparent consensus unless consensus among the experts is really achieved. It is not necessary that the TFI agree with the outcome of the process; the TFI can stand behind that outcome as the result of thorough interaction among experts.

The issue of breadth. The statement on breadth of application on page 1-7 of the draft report and other statements related to the intended breadth of application of the recommended methodology are the cause of much uneasiness among the Panel. A clear statement of the purpose and scope of the report should be included early in the introduction.

• It should be made clear that the recommended methodology is based on a study of the experiences with LLNL and EPRI procedures. This should be brought out in the history-context material called for above. In the appropriate places, specific references to the lessons learned by examination of previous PSHA projects should be cited. The studies from which the recommended methodology was derived should be clearly described, even though the intent of the report is not to address the reconciliation of the LLNL/EPRI studies. The reader should be made

aware of the lessons learned from the evaluation of those (and other?) studies that have gone into the formulation of this report. The reader should be told explicitly that alternate PSHA approaches were not assimilated and that this report is not based on a consensus of a broad sample of practioneers.

• Some statement of costs would be in order. What a hazard evaluation can deliver is often a matter of how many dollars are available. Cost estimates may be beyond SSHAC's scope, but even this could be mentioned.

• The Panel anticipates that the full procedure recommended in this report will not be applied to the seismic regulation of all critical facilities. It is not a general methodology that will be applied step-by-step in all situations. Therefore, criteria or guidelines are needed in the report, to aid the project sponsor and the PSHA analysts in deciding when the full procedure is justified. A statement is needed about what can be delivered with different levels of PSHA, so the buyer can make an informed decision as to what will and will not be produced. As stated above, the analyst who chooses for sufficient reasons to use other procedures should not be put by this report in a position of having to defend that decision in a regulatory setting. He or she, of course, must be prepared to defend the procedures that were adopted.

The Panel questions whether the links between SSHAC's recommended methodology and its applications are spelled out in sufficient clarity. Although SSHAC is not charged with specifying the use of hazard numbers in engineering design, a brief treatment is needed pointing to how the results can be used, and, in particular, what the knowledge of highly refined uncertainty estimates contributes to applications. A clear and unequivocal definition of aleatory and epistemic uncertainty is needed, as well as a clear and readily applied prescription

for separating the two. This is needed because of the emphasis on this subject in the report.

Although not as yet the subject of full panel evaluation, the following example illustrates the need for SSHAC to be very clear on the value and the method of application of their categorization of uncertainty. “What should count for decision is not the aleatory/epistemic distinction, but the temporal variation in the total uncertainty (in the total or predictive distribution of A_T, maximum peak ground acceleration and spectral values at the site in the next T years) during the lifetime of the project.” According to this viewpoint:

• There is no need to label uncertainty as epistemic or aleatory.

• If one sees total uncertainty as being contributed by different sources (e.g., by uncertainty on model type or on various parameters), then it is reasonable to expect that the uncertainty associated with each source will evolve in its own way in time. Making a binary distinction between epistemic and aleatory uncertainty corresponds to assuming that each source will be either explained totally (epistemic components) or will remain constant over the lifetime of the system (aleatory components.)

• One can formulate rational ways to make decisions accounting for the possible temporal evolution of uncertainty. The Panel member responsible for these comments is not, on the other hand, aware of any convincing method to make decisions based on the aleatory/epistemic decomposition. The amount of conservatism displayed by decisions under time-varying uncertainty depends on the nature of the problem (essentially on the degrees of asymmetry in the rewards and penalties associated, respectively, with future possible decreases and increases in the calculated risks).

The SSHAC report will be strengthened by addressing these concerns in a straightforward way.

Intensity data from historic strong earthquakes in the central and eastern United States is not incorporated in the ground motion models. The relation between m_bLg and intensity in the eastern United States, first established by Nuttli, should not be ignored.

“Seismic source zones”, a key concept in the prescribed source characterization procedure, should be explicitly recognized as an artificial construct introduced to make hazard calculations tractable. They are not real physical entities.

Some issues important for applications are not touched on. For example, the value of the availability of computed full seismic waveforms, site response analysis, and the importance of non-linear soil response in hazard calculations are not mentioned. Assessment of non-linear behavior is at least as important as uncertainties in bedrock motion. So also is the relationship between hazard-consistent time series and magnitude-distance parameters. And, as we have noted above, SSHAC 's recommended methodology is based on the LLNL-EPRI experience and therefore focuses properly on power plant design. It is not necessarily suited to structures that respond to longer-period motions (e.g., suspension bridges).

Though the full treatment of these and related topics is outside the scope of the SSHAC task, they should at least be mentioned to show SSHAC's awareness that seismic hazard analysis encompasses more than the issues and procedures recommended in this report and that these problems are important in the total process. This will give greater credibility to the final report. A full report on site response analysis should be prepared in a future effort. A handbook on USNRC-approved procedures for doing PSHA, presumably restricted in scope to projects in the jurisdiction of that agency, would be helpful. This handbook would be based on this SSHAC report and additional reports on the issues outside of SSHAC's responsibility. However, as stated above, such a handbook should not be used to require a specific procedure in all hazard assessments.

Mention is made of earthquake prediction in two places in the Introduction, pages 1-1 and 1-11. The comments give a misleading impression of the goals and potential benefits of a prediction methodology, if one is eventually developed. In particular, it is implied that the ability to produce accurate short-term predictions would somehow lessen the importance of reliable long-term earthquake hazard assessments, on which sound engineering decisions must be based.

The reader should be informed where the software needed to carry out the recommended procedures can be obtained.

The discussion of the effects of correlation of parameters is apparently based on lessons from experience. The whole point should be made clearer, as the draft presentation is not readily followed by a reader who has not been through the process.