tive to the problem at hand. Normative goodness, on the other hand, refers to the expert’s ability to express that knowledge in accordance with the calculus of probabilities and in close correspondence with his or her actual opinions. Depending on the situation, one or the other type of goodness predominates.
Questions that need to be considered with regard to the use of expert opinion fall into two categories: (a) elicitation (e.g., how to select the experts, how many to select for a given issue, and how to elicit their opinions) and (b) how to use the elicited opinions and information about the experts to estimate the unknown quantity.
It has been widely documented that judgmental estimates are subject to a number of potential biases. Two biases that are particularly important in the practice of risk assessment are (a) the possibility of systematic overestimation or underestimation and (b) overconfidence, or the tendency for people to give “overly narrow confidence intervals which reflect more certainty than is justified by their knowledge about the assessed quantities” (Tversky and Kahneman 1974).
The following are helpful points to consider when expert opinion is used:
It is important to select good domain experts and train them in normative aspects of the subject.
Aggregating the opinions of multiple experts tends to yield more accurate results than using the opinion of a single expert.
Mathematical methods of aggregation are generally preferable to behavioral methods for reaching consensus.
The quality of expert judgments can be substantially improved by decomposing the problem into a number of more elementary problems.
Significantly better overall results are obtained if the initial problem definition and decomposition are performed with care and in consultation with the experts.
Expert opinions are subject to bias and overconfidence. Effective means of reducing overconfidence are (a) using calibration techniques and (b) encouraging experts to actively identify evidence that tends to contradict their initial opinions (see the discussion below on bias).
Sources of strong dependency among experts should be identified. Weak dependency does not appear to have a major impact on the value of multiple expert judgments.