Review of Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts (1997)

By its own definition, the main emphasis of the Senior Seismic Hazard Analysis Committee's (SSHAC) report is on the procedural rather than the technical aspects of probabilistic seismic hazard analysis (PSHA). SSHAC argues that many of the major potential pitfalls of PSHA are procedural and therefore goes to great efforts to outline what it views as an appropriate process. In SSHAC's view the important aspects of “process” have to do primarily with experts, their interaction, and methods for translating their views into useful input for a PSHA. Of particular significance is the role assigned to the facilitation/integration team that organizes and directs a PSHA project and its use of experts. SSHAC lays out two basic principles underlying the PSHA process and its results:

1. Regardless of the scale of a PSHA study, the goal (as stated by SSHAC) is “to represent the center, the body, and the range of technical interpretations that the larger technical community would have if they were to conduct the study.”

2. “It is absolutely necessary that there be a clear definition of ownership of the inputs into the PSHA, and hence ownership of the results of the PSHA.”

The panel supports these principles as ideological guidelines for planning and executing a PSHA study, at least in the case of critical facilities. The first is, or should be, the goal of a sponsor in initiating a PSHA, the assumption being that using the collective input of the informed technical community would be the best, and most defensible, way of defining seismic hazard. That principle also has an enabling effect because, as discussed later, it allows experts to transcend the role of being proponents of models (the usual mode in scientific discourse) into the roles

of objective evaluators and integrators. The extent to which this goal can reasonably be pursued in a particular case should depend on the scope and importance of the project and the resources available to support the study.

The second principle is important because it assigns to an identified entity, the “owner,” clear intellectual or scientific responsibility for the conduct and results of a PSHA. This does not necessarily mean that the “owner” agrees with every particular input or result but that the owner feels confident that the PSHA has fulfilled the purpose of representing the larger technical community and can be defended in scientific and regulatory arenas, as necessary. These principles underlie the primary recommendations of the SSHAC report that deal with the PSHA process.

SSHAC recognizes that a PSHA can be carried out at different levels of effort and emphasizes that the effort expended should match the importance of the facility, the degree of controversy, uncertainty, and complexity associated with the relevant scientific issues, and external decision factors, such as regulatory concerns and the resources available. This is shown in Table 2.1, taken from Chapter 3 of the SSHAC report.

Four levels of study are defined, the first three of which rely on a single entity called the technical integrator (TI), who is responsible for all aspects of the PSHA, including specifying the input. Although experts may be involved on a consulting basis, there is no formal elicitation of their views. The highest level of study (level 4) makes use of formally elicited expert judgment. As such, a new entity called the technical facilitator/integrator (TFI) is needed. The role of the TFI is discussed below. A large part of the SSHAC report is devoted to defining what is necessary to carry out a level 4 study and explaining the function of the TFI because the ideas are new, not because this level of effort is required for every seismic hazard assessment. It would be inappropriate to infer that all PSHAs require the considerable resources needed to carry out the level 4 PSHA described by SSHAC. ²

The Panel endorses the conceptual framework embodied in Table 2.1, recognizing that the application of PSHA to engineering and regulatory problems is varied and that the level of effort needed should also vary.

SSHAC points out that most site-specific studies make use of some type of TI approach. The TI performs analyses, accumulates information relevant to each issue, and develops a representation of the technical community's views on the relevant input models, parameters, and their uncertainties. At the lowest level of effort (level 1) the technical community's views are determined primarily by a literature search. At higher levels the TI makes use of outside technical researchers and proponents to gain insight into different data sets and models.

The panel emphasizes that a TI must still be guided by the principles of representation and ownership described above.

The importance of peer review is discussed below, but the panel stresses its particular significance when the TI mode is used. Reliance on a single entity (TI) to characterize the input of the whole technical community may be a very efficient mode of operation, but additional assurance is needed to provide confidence that the results are a reasonable representation of the community's views.

The TFI process views experts as acting in different roles—proponents, evaluators, and integrators. The proponent role is one in which the expert explains, and argues for, the choice of a particular model or set of parameters. The aim is to make sure that the different views in the technical community are presented and discussed by the expert panel. If necessary, individuals outside the expert panel may be brought in to argue points of view with which panel members may not be comfortable. The next role the experts are asked to assume is that of independent evaluators representing their own views of the information presented. Mean estimates of model, component, or parameter values are elicited, along with their uncertainties as appropriate. The result should be the group's composite views of the issues at hand. The experts are encouraged to evaluate their own and other models according to their own technical judgment, without regard to who originally proposed the models. In the past, most PSHAs

that have relied on formally elicited expert judgment have strived to get experts to think in this manner. The hope was that the experts ' composite view also represented the composite view of the technical community as a whole.

To more truly represent the technical community's view, the SSHAC report recommends that the experts be specifically asked to assume the role of integrators and to characterize their perception of how the technical community as a whole would view the issues at hand. Thus, although the expert may view his/her assessment as being the most correct, he/she is explicitly thrust into the role of trying to fulfill the first principle of PSHA as outlined above and must be willing to do so. This mode of expert behavior may not be achievable in all issues. Also, the

panel is not aware of any objective way to test the assumption that a whole technical community's views can be accurately determined from the interactions of a small group of experts.

SSHAC introduces some useful concepts in its discussion of the interaction among experts. One is that in the process of eliciting, aggregating, evaluating, and integrating the opinions of experts the TFI (discussed in the next section) should create an atmosphere in which there will not be “winners” and “losers.” Another useful idea is the avoidance of unintended dissent or consensus. Apparent disagreement may arise because of lack of communication and understanding among those disagreeing; the process of “active listening,” in which a listener is asked to give back what he/she has just heard, is a step toward eliminating disagreement where it really does not exist. At the other extreme is the development of an apparent but false consensus; the TFI should strive for consensus among the experts only if it is really agreed on.

The panel views the role of expert as integrator as important and worthwhile. However, successful implementation of the integrator role of the experts should be viewed more as a goal to strive for than a uniformly and demonstrably achieved measure of success.

The SSHAC report implies four basic criteria for the identification and selection of experts: (1) technical expertise, (2) strong communication skills, (3) willingness to assume the role of independent evaluator, and (4) willingness to commit the time and effort to participate actively in the study. The choice of disciplines to be represented and the breadth of knowledge of each expert depend on the issues to be addressed and whether or not interdisciplinary subgroups of experts will be formed to provide input. SSHAC also strongly recommends a formal nomination process based on consulting the literature and asking technical societies, government organizations, and knowledgeable individuals to submit the names of potential experts. Whatever the issue or structure of elicitation, the panel believes that the credibility and quality of an elicitation-based PSHA depend very much on the choice of experts. The panel supports the need for careful attention to the selection process and finds the criteria suggested by SSHAC to be reasonable and likely to be effective.

One of SSHAC's main contributions to PSHA methodology is the introduction of the technical facilitator/integrator (TFI) concept. The SSHAC report describes this new function in Section 3.3.1 as follows:

The TFI is a single entity who has the responsibility and is empowered to represent the composite state of information regarding a technical issue of the scientific community.... The TFI process is centered on the precept of thorough and well-documented expert interaction as the principal mechanism for integration.

As SSHAC acknowledges, a major stimulus for its charge was the need to resolve the differences in hazard estimates between the Lawrence Livermore National Laboratory and the Electric Power Research Institute studies. SSHAC's investigation revealed that the process of elicitation and the procedures for integration allowed room for considerable misunderstanding and potential misinterpretation. Six areas in which improvements could lead to a better outcome are detailed in Section 3.3.2.2 of the SSHAC report:

1. Overly diffused responsibility

2. Insufficient face-to-face expert interaction

3. Inflexible aggregation schemes

4. Imprecise or overly narrow objectives

5. Outlier experts

6. Insufficient feedback

The TFI concept was designed to resolve these procedural issues. This approach is described in detail in Chapters 3 through 5 and Appendix J of the SSHAC report. The panel concurs that, in cases in which decisions about a critical facility of major complexity depend on controversial and uncertain inputs, the TFI approach offers an effective mechanism for capturing the best of what is known about the particular issues.

The seven steps proposed by SSHAC for the TFI approach (Section 3.3.4) were first suggested by Keeney and von Winterfeldt (1991), based on their experience in eliciting expert judgment for probabilistic risk assessment of nuclear power plants. The steps are:

1. Identification and selection of technical issues

2. Identification and selection of experts

3. Discussion and refinement of technical issues

4. Training for elicitation

5. Group interaction and individual elicitation

6. Analysis, aggregation, and resolution of disagreements

7. Documentation and communication

A flow chart of the process as applied to ground motion elicitation by SSHAC is reproduced here as Figure 2.1. Appendix J of the SSHAC report spells out the background, evolution, and details of the TFI process as developed by SSHAC. Appendix J must be read carefully; readers may need to consult additional references in order to fully understand some of the issues discussed, such as the weighting of individual expert inputs.

The TFI process requires careful and time-consuming setup procedures to ensure that all participants are clear on the objectives of the study, their roles in the study, and the intended results. The TFI (an individual or, perhaps, a team of two or three people) must be highly competent in the relevant subject areas, adept at elicitation and group process, and thorough. Because a strong TFI will have a major influence on the outcome of the elicitation/aggregation process, it is essential that, if more than one TFI is assigned to work on a particular analysis project, they all be equally well qualified.

The panel concludes that for appropriate issues the TFI process holds significant promise for PSHA. This process was developed by SSHAC as part of its effort to overcome limitations of previous PSHA studies. The panel cautions, however, that this process is expensive, time consuming, and demanding of all participants. SSHAC's criteria for identifying the issues for which the full TFI process is justified (Table 2.1) must be understood by project sponsors and their analysts.

As discussed in the next chapter, each element of a seismic hazard analysis may involve high degrees of uncertainty. Many situations arise in

which competent experts may legitimately disagree in their interpretation of extant data and theory. In view of the complexity of the issues and models involved in PSHA, SSHAC concluded that an improvement in the process of elicitation would help focus attention on the technical issues by reducing previously observed problems in “consensus,” unintended agreement, and unintended disagreement.

At each step of the elicitation process, the TFI strives for complete understanding by each expert of all technical issues. The goal is that all experts are “on the same page.” The results of two ground motion workshops conducted by SSHAC and documented in Appendixes A and B of its report indicate that investment in the TFI process bore substantial results.

The panel is aware that the TFI process, as implemented in these workshops, has rarely been used in the earth sciences. An example of the application of the process in a related subject field is provided by a probabilistic volcanic hazards analysis (Coppersmith et al., 1995).

SSHAC correctly points out that in theory it is always possible to formulate the expert integration problem as a Bayesian inference problem in which the opinions rendered by the experts are viewed as “noisy observations” of the quantities of interest (e.g., parameter values, distributions). Difficulties lie in the formulation of an “observation model” tailored to each expert combination task and sometimes in implementing the Bayesian analysis to produce a posteriori uncertainties. A discussion of combination problems and models is given in Appendix J of the SSHAC report. SSHAC repeatedly warns against blindly using any specific model and stresses that the models described in Appendix J are only examples for illustration. The panel agrees with these warnings and adds the following comments:

• In essence, Appendix J presents two very different types of models: (1) the so-called classical models, which emphasize the “noisy observation ” interpretation of expert opinion, and (2) the TFI model, which regards each expert as being potentially correct, with a probability

FIGURE 2.1 Roadmap of ground motion elicitation process (Figure 5-5 of the SSHAC report).

proportional to an assigned weight. Although this interpretation of the TFI model is not given in the SSHAC report, the fact that the community distribution is defined as a weighted sum of the expert distributions is equivalent to saying that each expert is correct with a probability equal to his/her assigned weight. At the end of Appendix J, the two approaches are compared numerically and shown to produce very different results. Without an in-depth discussion of when each type of model (or neither) is applicable, Appendix J may leave the reader confused. The classical models combine distribution functions with the meaning of uncertainty on the value of an unknown parameter. Hence, in this case the object of estimation is an unknown scalar quantity and the distributions express uncertainty on that quantity according to different experts. The TFI model, on the other hand, combines distribution functions that express the state of uncertainty of the scientific community according to different experts. In this second case the object of estimation is the distribution function itself. Therefore, while the inputs to, and results from, both

models are in the form of probability distributions, such distributions have different meanings in the two cases and should not be compared.

• The community distribution, which the TFI model estimates, is defined in Appendix J, Section 5, of the SSHAC report as “the mixture of the distributions of the individual experts if [the decision maker] believed that the experts . . . in this ‘perfect community' were effectively equally informed on the issue of interest and equally interdependent. . . .” As the entire SSHAC procedure revolves around this distribution, the panel believes that its definition should have been given in the main report, with a detailed explanation and justification.

• SSHAC gives expressions for the mean and variance of the community distribution after stages 1 and 2 of the TFI process. Given the approximate nature of the results for the variance and the fact that distributions, not just mean values and variances, are needed, a much simpler and basically as accurate combination rule would be to take the weighted average of the distributions provided by the experts. The statement in Appendix J that “determination of the predictive (i.e., a posteriori) distribution follows a straightforward but cumbersome Bayesian statistical analysis” indicates that SSHAC knows how to perform a fully nonparametric Bayesian estimation of the community distribution function. This panel could think of no straightforward procedure to do so (one would need to consider the expert distribution estimates as random processes given the true community distribution function, with serious practical and conceptual implications). Because determination of community distribution and its uncertainty is at the core of the SSHAC approach, the report should have been more explicit about such a procedure.

• SSHAC favors an equal weighting integration scheme, unless there are clear indications that different weights should be used, for example, to reduce the influence of outliers. Linear combination rules with equal (unequal if necessary) weights are applied to parameter estimates (classical models) as well as to the probability distributions that, according to the panel of experts, quantify uncertainty in the scientific community (TFI model). Conditions for “equal weights ” are set forth in the report. The panel believes that there may be some confusion about linear combination with equal weights and symmetrical (but possibly nonlinear) treatment of the expert assessments. The conditions quoted in the SSHAC report apparently lead to symmetrical treatment, not necessarily to averaging. There is a brief reference to nonlinear

combination rules in the section on nonequal weights in Appendix J, with little discussion. Analysts are advised to verify whether the conditions of linearity and normality of the observation model apply before using a linear combination rule. Contrary to what SSHAC states (e.g., Figure J-6), in some cases it would be better to combine the parameters of the distributions provided by the experts rather than the distributions themselves (combining the parameters results in a nonlinear combination of the distributions.) For example, if the experts agree on all distribution characteristics except for a location parameter, combining the estimated locations would be the right thing to do.

In view of these limitations and the objective difficulties in properly combining expert opinions, the panel recommends the following:

1. Use the models in Appendix J of the SSHAC report for reference, not as prescriptive or even recommended combination procedures.

2. Do not accept the results of a mechanical combination rule unless they are consistent with judgment.

3. If a mechanical combination rule is used, a general way by which to derive that rule is to view experts as noisy observers of the quantity being estimated. This approach is always the correct one from a Bayesian viewpoint, irrespective of the problem at hand. What differs in different cases is the nature of the observation errors, which need not necessarily be normal, additive, or independent.

4. When combining expert opinions on distribution functions, the correct Bayesian approach requires the use of a random process formalism, unless the problem can be reduced to a discrete one through appropriate parameterization. In all but the simplest cases a formal analysis becomes prohibitive, and the panel recommends primary reliance on judgmental combination procedures.

One of the more problematic aspects of PSHA has always been the aggregation of input from different experts, especially when one or more expert opinions are outliers relative to the views of the rest of the

participants. This problem has led to consideration of weighting of different experts' opinions based on quantitative or qualitative assessments of the degree of expertise (typically a highly subjective exercise). The extensive interactive education and elicitation process proposed by SSHAC is intended to bring all expert participants to parity. This process should make it more reasonable to use equal weighting of all the experts. Appendix B of the SSHAC report states that equal weights were used for the combination of expert opinions and concludes that the TFI “integration process is robust.”

The panel concurs that equal weighting of experts should be the clearly preferred target in a multiple-expert PSHA. To achieve this, proper choice of experts and group interactions should be emphasized, as outlined in Chapter 4 and Appendix H of the SSHAC report. In the case in which a different weighting scheme is applied, the burden of proof rests with the TFI; nevertheless, every effort should be made to obtain expert concurrence on the weights used or modification applied.

A related aggregation problem, dependency among experts, is, on the surface, exacerbated by the TFI process. The overall community is composed of a finite number of experts who rely on a finite number of models and methodologies. While one or more of the participating experts may not be thoroughly familiar with the entire range of such models and methodologies at the beginning of the exercise, such familiarity is an objective of the TFI process. As shown in the second SSHAC ground motion workshop, this interactive process narrowed the range of estimates as the experts increased their knowledge and understanding of issues and methods. One goal of a well-executed TFI process is that all participating experts are better able to make informed independent judgments.

SSHAC requires that peer review be an integral part of the PSHA process. The panel concurs. SSHAC defines two types of review: (1) participatory and (2) late stage. Participatory peer review involves “full

and frequent access throughout the entire project” by the reviewers. The advantage of a participatory review is the opportunity to subject interim results and deliberations to independent feedback. This provides the PSHA team with an opportunity for adjustment and limits the possibility that a lengthy and costly effort might be found to have serious flaws in the end. SSHAC recognizes that a limitation of participatory peer review is that “peer reviewers might lose their objectivity as they interact with the project over time.” The panel views a participatory peer review as equivalent to a backup group of experts who provide oversight of the work of the primary team. Safeguards must be established to preserve the objectivity of the review process. As explained in the introduction to this report, this panel was asked to provide participatory peer review to SSHAC, and the panel insisted on a process by which it would not become so deeply involved in the preparation of its report that its objectivity would be compromised. The panel believes that this is also a necessary precaution for peer review of any PSHA study.

The late-stage review is closer to the traditional academic review in that it occurs near the end of a project. SSHAC strongly recommends participatory peer review on the grounds that a late-stage review can be risky, especially with regard to the process aspects of a PSHA study. Table 3-2 in the SSHAC report summarizes its recommendations on how to structure the peer review process.

The panel concludes that participatory review, as part of a PSHA process, would serve to improve the quality of a study insofar as it is another step toward incorporating the views of the broad informed scientific community. Other considerations—for example, the requirements of regulatory bodies—might call for a late-stage review also.

Chapter 7 of the SSHAC report puts much emphasis on the importance of fully documenting every PSHA study. The guidelines on documentation are intended to ensure that each step of the PSHA process is not only completely recorded but also that the records are stored in accessible formats that permit the technical community to review all operations and decisions. This documentation also greatly facilitates later reanalysis and update as new information becomes available, perhaps eliminating the necessity of redoing the entire PSHA.

The panel believes that the calculated seismic hazard derived from each individual expert's input needs to be presented. It is not clear whether this is included in SSHAC's recommendations. Regardless of how the aggregation is carried out, it is important to be able to compare results caused by each expert's input with those of the composite produced by aggregating the individual inputs. This comparison provides users with a good indicator of the diversity of input and its impact on the final calculations, as discussed in Chapter 3.

SSHAC proposes that this documentation follow a two-tiered approach that is to be applied to every element of a PSHA. Tier 1 documentation is defined as all documentation that must be published as part of the main report or its appendixes, so that it is widely accessible. Simply stated, tier 2 is everything else that constitutes background material for the analysis. SSHAC's prescription for what materials should go into the two tiers is spelled out for each of the elements of a PSHA (i.e., seismic source characterization, ground motion attenuation, and the methods used to produce the PSHA results).

The SSHAC report specifically states that the computer software used should be identified and archived. This would include any relevant programs and code that would be necessary for an independent analyst to replicate the study. Should problems be identified later with either the computer code or the input data, reanalysis is greatly facilitated. The panel recommends that specialized computer programs needed to implement the SSHAC procedures be readily accessible to any group that wants to engage in seismic hazard evaluation as part of a research program or business venture. The availability of these programs becomes especially important if the procedures recommended by SSHAC are so successful that they become the standard adopted by governmental regulatory bodies and the major engineering concerns of the nation.

To facilitate the accurate and timely documentation of PSHA projects, the panel recommends that an individual or small team be designated as the Project Archivist and that a documentation plan be in place at the beginning of each project. The thoroughness and complexity of the SSHAC approach, especially when the TFI is used, require that all participants have ready access at any time to materials generated previously. This implies a documentation process that keeps current with the rest of the project.

The panel concludes that the discussion of the documentation process in Chapter 7 of the SSHAC report provides thorough and useful guidance for numerous other applications in addition to seismic hazard assessment. Documentation is not one of the more glamorous aspects of the scientific enterprise, but it is essential to the full realization of the benefits of the large investment in data acquisition, analysis, and interpretation that are characteristic of large projects.