Conclusions and Recommendations of the Prince William Sound Study
Base Case Results (Section 5), Risk Reduction Evaluation (Section 6), and Assessment of the Effectiveness of Risk Reduction Measures (Section 7) of the PWS Study contain the most important conclusions and recommendations. The base case was used to establish the parameters against which risk reduction measures will be evaluated. Section 5 presents the results of the GWU simulation and the DNV fault tree/MARCS models. Point estimates are presented without uncertainty ranges. The study expresses confidence in the results because both the simulation and fault tree/MARCS approaches yielded similar results. However, the NRC committee believes that the results are not independent of each other because the same data sets and some common assumptions were used and, in some cases, were calibrated against each other. Therefore, similarities, especially without estimates of uncertainty, do not demonstrate the validity of the results. Because the models had different capabilities, adjustments had to be made. For example, because the simulation model did not consider groundings in the same detail as the MARCS model, some simulation results were allocated to different geographical sections (PWS Study 5:12–5:14).
Section 5 (Base Case Results) includes runs for inbound tankers, which were not included in the MARCS model, indicating an accident rate of one every ten years (PWS Study 5.27). Although inbound tankers are not laden, the bunker oil and the relative high frequency of accidents led to a potential average oil outflow of 30 tons/year, which is not significantly different from many estimates for outbound accidents (PWS Study 5.23, 5.28). The results are presented in chart form, which is easy to read. However, because no uncertainty analysis is included, the reader may be misled by the apparent precision of the results.
Section 6 (Risk Reduction Evaluation) describes the process by which measures of risk were developed. The NRC committee was not able to determine from the PWS Study how the public record was reviewed or which documents were used to develop the data-
base of risk reduction measures. In response to a question from the NRC committee, the PWS study team provided a list, but proprietary documents were mentioned only briefly (PWS Study 3.2; TD 1.4). The TD indicates that the PWS steering committee developed risk reduction measures based on a hazard identification exercise but does not indicate which measures or which stakeholder group provided them. The open-ended questionnaire asked for suggestions for risk reduction measures, but the study does not indicate which of the candidate measures came from questionnaires or how the suggestions were used. From discussions during meetings with the PWS Study team, the NRC committee learned that the TAPS companies did not all agree on recommended measures, but there is no indication in the PWS Study of how that information was used, other than that self-reported failure rates were not used for companies with low management scores.
The PWS steering committee and study group were involved regularly with the study, but it is not clear how or if the risk reduction measures were modified to satisfy requests from the steering committee. These modifications may have been appropriate, but the development of the list of risk reduction measures should have been explained in the PWS Study. It is clear, for example, that some potential risk reduction measures were not included in the published list, such as using non-U.S. flag ships (which may or may not reduce risk); straightening out the path through the central sound, which would decrease transit time and, therefore, reduce opportunities for interactions with other traffic; and removing the worst performers from the TAPS trade. The study mentions that the list of risk reduction measures was “presented to the steering committee for comments and corrections” (PWS Study 6.3) but does not indicate the results of the consultation. This made it difficult for the NRC review committee to evaluate the thoroughness of the list.
Section 6 (Risk Reduction Evaluation) includes an extensive list of risk reduction measures, not all of which are included in the models (PWS Study 6.10–6.36). The following reasons are given: the models were incapable of treating enough detail; data were not available; the measure was redundant; the measure will be tested in follow-on analysis; and an equivalent measure was tested. Nevertheless, the reader is given the impression that most of the measures can be modeled, including many human factor measures, like the ones listed below:
drug tests before transit (PWS Study 1:6.22)
standard job descriptions (PWS Study 1:6.22)
extra mates on tankers (PWS Study 5:6.28).
The study notes that “risk reduction measures…had to be translated into changes in modeling parameters…" (PWS Study 6.36). The changes were made to do the following things: to decrease the parameters of operational errors by 10 or 20 percent (PWS Study 6.41); to improve performance parameters by 40 percent to account for an additional officer on the bridge (PWS Study 6.42–6.43, Rule 9); and to reduce fault tree probabilities by 20 percent (PWS Study 6.44, Rule 20). These percentage changes, which are appropriate for sensitivity analyses, are used to indicate the effect of implementing the recommended risk reduction measures. No explanations are given for how the percentages were determined or how they could be accomplished.
Section 7 (Assessment of the Effectiveness of the Risk Reduction Measures) is based
on evaluations of 18 cases, one combined case, and the base case.1 The cases are clearly identified, and the modeling assumptions are clearly described. The results are presented in tabular and graphical form, with more extensive presentations in the TD (5.1, 5.2). The results are presented to the reader in an easily understandable fashion.
Section 7 includes some appropriate caveats. With respect to reducing human error and mechanical failures by 40 percent, as modeled in some of the cases, the PWS Study notes that it “may be hard to achieve and even harder to verify successful implementation…" (PWS Study 7.5). With regard to system interventions that are modeled as preventing 40 percent of all incidents, the study notes that “these interventions may be easy to verify, but it may be difficult to capture 40 percent of all incidents” (PWS Study 7.5). With regard to representing human and organizational error, the study notes that “the modeling changes…are best estimates determined by judgment of the project team based on limited data, prior studies, and personal expertise. The percentage reductions in human error used as modeling inputs are estimates. These error reductions may not actually occur if the risk reduction measures represented are implemented” (PWS Study 7.8).
An attempt to model the value of an extra person on the bridge illustrates the modeling difficulties. The simulation team assumed a reduction of 40 percent for each additional officer. The fault tree team assumed a reduction of 75 percent for the first additional officer and 34 percent for each additional officer (PWS Study 7.11). The study notes that “the area of human error reduction was the primary area where analysis had to stop before a desired level of detail was reached due to an inability to determine model parameter changes that could represent very specific interventions…" (PWS Study 7.7).
One of the major recommendations of the PWS Study is to improve the capability of tugs stationed at the Hinchinbrook Entrance. The NRC committee notes that the value of these tugs is assumed in the modeling: “Procedures ensure that the pre-positioned tug is capable of saving all vessels transiting under allowed conditions” (emphasis added) (PWS Study 7.16 [Table 7.2–9]). The PWS Study notes that “the risk reduction achieved by this change is almost entirely due to the improved ability to assist a disabled tanker at Hinchinbrook Entrance ensured by the provision that the standby vessel at Hinchinbrook was always capable of saving any tanker making an allowable transit” (PWS Study 7.18).
For the final results, the GWU model was used for 18 of the 20 cases, the DNV models for 10 (PWS Study 7.25–7.26 [Table 7.3–1]). The results “show that the most significant reduction in accident frequencies could be attained through interventions that effectively prevent human errors or vessel failures from occurring…or from ‘capturing’ human error when it occurs…" (PWS Study 7.20). However, the reader must keep in mind the weaknesses in the data on which these results are based: “Historical data does not adequately support a detailed analysis of the contribution of human error to incidents and accidents or the estimation of the effect of specific interventions designed to mitigate human and organizational error. In addition, historical data for vessel repair times is inadequate to support detailed risk analyses” (PWS Study 8.7). Table 7.3–5 shows that the most effective mea
sure for reducing accidents is revising traffic rules (PWS Study 7.33). The study also concludes that the two most effective measures for reducing oil outflow are improving human and organizational performance.2
Despite the description of weaknesses in the data and the arbitrary modeling assumptions, the NRC committee believes that the conclusions about revising traffic rules to reduce the probability of accidents were supported by the data and the models. However, based on weaknesses in the probabilities of human error, including the lack of reliable data and the necessarily arbitrary assumptions used to incorporate human error into the models, the NRC committee believes that the results regarding measures of human performance were not supported. The PWS Study itself warns the reader not to rely on these results: “Reducing human and organizational error and reducing vessel failure rates provide the most consistent and largest risk reductions. There are, however, problems with relying on these interventions as a cure-all. The experience and data to substantiate the modeling assumptions that produced these results is very sparse " (PWS Study 7.51). The committee believes that improved training, better management practices, and increased vigilance are worthwhile goals to pursue. However, unless their effect on the variables in the risk analyses can be accurately assessed, their value can best be described qualitatively and subjectively. The modeling approach gives the appearance of scientific credibility to what are, in reality, assumptions.
In Section 7, the PWS Study warns that preventing oil in the water may not be an appropriate, single measure of risk reduction. “Since the collision interactions potentially involve vessels with large numbers of persons on board (cruise ships, ferries, tour boats), a risk intervention that trades a decreased frequency from grounding for an increased frequency of collision based on a single metric of reduced oil outflow may not be a sound policy (PWS Study 7.39).
Anyone considering using this methodology should read the summary comments in Section 8 (Conclusions and Recommendations), carefully. Although no uncertainties are indicated in the presentation of the results, the summary comments note that assumptions had to be made to estimate the effects of risks reduction measures. “Each of the estimates has a high degree of uncertainty. The risk reduction results based on these estimates are also uncertain” (PWS Study 8.4). The NRC committee agrees.
LIMITATIONS OF RECOMMENDATIONS
The recommendations were made with an eye toward the overall limitations of the analytic methods. Recommendations were formulated to be as specific as possible without recommending particular solutions (or detailed risk reduction measures). However, they are based on comparisons with risks for 1995, the baseline year of operations, which is probably not a representative year of PWS operations. This was suggested to the NRC committee in discussions with the PWS Study team. In 1995, for example, the weather
was mild, and there was little ice in PWS. Apparently no attempt was made to determine if 1995 was a representative year, if another year was representative, or if a fictitious year could have been developed that would have been representative. Probably no year would probably have been representative, and an artificial, fictitious year would have had little meaning.
If the overall objective of the PWS Study was to improve the risk situation, finding a representative year was not essential, and comparisons with 1995 may have yielded distortions in the ranking of risks. If the objective of the PWS Study was to rationalize numerically and justify a risk reduction investment strategy, the use of a representative year raises many concerns. For example, one might conclude that no additional risk reduction measures should be undertaken. The PWS Study does not establish guidelines or procedures for determining the effectiveness of risk reduction measures as they are implemented.