Conclusions and General Comments
The PWS Study is an ambitious effort to combine several probabilistic modeling approaches and site-specific data with international data to arrive at estimates of risk and reach conclusions on risk mitigation measures. In addition, the study approach involved close and continuous interactions with a nongovernmental citizens group (the RCAC) representing most of the stakeholders in the region. Despite these strengths, the PWS Study does not meet the NRC peer review standards of clarity and support for conclusions. The report is difficult to read without frequent references to the two large volumes of technical documentation, and parts of these are unclear.
The most significant weaknesses of the report are: (1) it has no overarching risk assessment framework to ensure the consistency and logic of the analyses; (2) it lacks a clear description of how the models were implemented, how the probabilities were calculated, and how the results were reached; (3) because of proprietary commitments, NRC reviewers could not examine the processes or much of the data on which the results were based; (4) the treatment of human and organizational errors is inadequate; and (5) it gives the false impressions that conclusions were both precise and logical.
The PWS Study and the technical documentation represent a substantial effort in data collection, modeling, methodology, and involving stakeholders, and the results have been well received by the organizers. The NRC review committee was not charged with assessing whether or not the recommended risk measures were appropriate. Therefore, we have not made that assessment, although all of the proposed measures meet the test of common sense.
The objectives of a risk analysis are (1) to compute the overall risk from accidents and determine if they are tolerable or acceptable (the PWS Study, obviously, began with the assumption that they were not) and (2) to rank possible mitigation measures. The options in the PWS Study were identified before the computations were made, and new options were rarely introduced during the analyses. Logic and consistency of the methodologies
and data are essential for obtaining meaningful and comparable results for different failure modes and different risk mitigation measures. The objective of a risk analysis not to make the audience feel secure. If the methodology is flawed, some stakeholders may dispute the conclusions.
The NRC committee was charged with conducting a peer review of the methods used in the PWS Study. This is not the same as assessing whether the study sponsors were satisfied. A peer review is a rigorous analysis of the quality of the process by which results were determined and of whether those results are supported scientifically by the analysis. The NRC review committee found that the PWS Study does not measure up to these standards.
The completion of the PWS Study was largely the result of the PWS community banding together to guide, assist, and support the PWS Study team. Stakeholder participation took the form of a highly interactive and cooperative PWS steering committee comprised of representatives of various subcommunities interested in, and affected by, the PWS TAPS trade. “Unanimous approval for all decisions taken by the PWS steering committee was required. Thus, the PWS steering committee and the contractor study team, which was deferential to the PWS steering committee and its subunit, the PWS Study group, interacted closely and frequently.
The PWS steering committee contributed to the PWS Study to the degree that the PWS Study team must be defined as a combination of the contractor analysts and the PWS steering committee. This arrangement has advantages and disadvantages. One advantage was that the arrangement facilitated timely provision of essential information and the resolution of disagreements about the analyses and analytic methods. It encouraged trust among and brought together significant elements of the PWS community so that actions based on the PWS Study might be taken without significant arguments over the analysis. The PWS steering committee’s role and mode of operation were described in the report: “Requiring unanimous agreement at all stages of the project has made each party try just a little harder to achieve consensus without abandoning strongly held convictions. The PWS steering committee’s resolution to work through difficulties in a constructive manner is one of the project’s more enduring successes” (PWS Study, 1.13).
However, there is a also a down side to the steering committee’s intimate involvement in the study process and the agreement/approval process. Disadvantages include:
a possible loss of the PWS Study team’s objectivity
compromises on issues for which a common denominator should not determine the decision
the disproportionate influence of some steering committee members who represented their interests more forcefully or persuasively than others
a defensive attitude in discussions of the study with the RCAC (Regional Citizens’ Advisory Council) and the PWS community as a whole
the appearance of conscious and unconscious biases in the conduct and results of the PWS Study
inadequate representation of the views of stakeholders who were not represented on the PWS steering committee
a loss of independence on the part of the contractor analysts
Thus, the PWS Study may have a number of built-in deficiencies that were not identified by the PWS Study team and were unknown to the NRC reviewing committee (NRC 1989, 1996).
CANDIDATE RISK REDUCTION MEASURES
The PWS Study team, including the PWS steering committee, attempted to be exhaustive and to get their compilation of (community-supplied) candidate risk reduction measures (interventions) “right.” The underlying philosophy of choosing candidate measures was apparent to the NRC review committee and was broadly confirmed by the PWS Study team. This philosophy dictated that risk reduction measures fall within a loosely defined boundary would that allow the TAPS trade to proceed without significant disruptions in current operations; risk reduction measures had to meet subjective, unspecified criteria for “practicality.” In other words, only limited changes in operations were considered. Radical changes were not considered.
The participating stakeholders understandably assumed this posture because of their investment in current operating procedures. Thus, a directive, positive-control scheme for TAPS vessels, as well as for other vessels operating in the PWS, was not considered. Major changes in vessel traffic patterns and mandatory separations between vessels were not analyzed (e.g., vessel separation schemes for the Hinchinbrook Entrance were not considered). Changes in vessel speeds (e.g., matching tanker speeds and escort speeds) were not analyzed. The current dynamic-escort approach was not tested tested against a prepositioned-escort or combined approach.
Omission of Key Factors
The PWS Study focused for the most part on oil spills originating with TAPS vessels. For other vessels in PWS (e.g., process ships, cruise vessels, and ferry boats), only traffic information was used. No assessment was made of the likelihood of a mechanical failure or human error leading to a collision of one of these vessels with a TAPS vessel. A complete assessment of PWS should have involved the other vessels in the area, especially the faster, larger vessels.
The NRC committee recognizess that the PWS Study team has compiled a large database, probably one of the largest databases on marine incidents involving TAPS vessels. Preserving this database would be beneficial to the industry for further studies on PWS.
LACK OF AN OVERARCHING PROBABILISTIC MODEL
The PWS Study had no explicit overarching probabilistic framework to ensure the logic of the analyses and the consistency of their application and of the interpretation of
outcomes.1 The PWS Study has three possibly four, “independent parts” that seem to have been arbitrarily defined, with no explicit link to ensure the internal consistency of the global model. Because the PWS Study had no overarching probabilistic framework, the committee could not confirm the following statements:
All plausible scenarios were included.
The probabilities of failure modes were properly computed and assembled (including dependencies, common causes of failures, and the effects of external events).
There was no overlap or double counting in the probability computations.
The absence of an overarching model casts doubt on the usefulness of both the MARCS model and the simulation model.
Human errors are mentioned many times in the PWS Study, and a global estimate was used that 80 percent of the failure probability could be prevented through better personnel management. In the final analysis, however, some human errors may also have indirectly influenced the results through the databases that were used (e.g., the statistics for groundings, some of which must have been caused by human errors). Therefore, in some cases, there may have been double counting. In other cases, common causes of errors, such as couplings and dependencies, may have been missed.
Dynamic analysis is a useful tool if the rates at which events develop influence the outcomes (e.g., speeds, rate of deterioration in weather conditions, time lapse between a warning and an event, etc.) and if the outcomes can be modified through risk mitigation measures. A sophisticated dynamic analysis may not be necessary, however, if the same information can be obtained through direct treatment of the random variables (e.g., the probability that the signal detection time exceeds the time between its appearance and the system failure). A simpler probabilistic analysis in the PWS Study, instead of the dynamic analysis, would have yielded results that were just as robust.
Fault trees are generally used to compute the probabilities of subsystem failures as part of an overall event tree analysis. Fault trees show the logical links between occurrences of basic events and occurrences of the top event through Boolean relationships; computation of the probability of the top event is a function of the probabilities of the basic events.
The PWS analysts did not construct real fault trees, however. They assessed the probability of the top event directly (assuming that they had sufficient statistics), and then they allocated the probability of the top event among the basic events, thus using the fault tree “backwards” and reversing the logical process. Their goal was to assess the effects (on the top event) of risk mitigation measures that affected the basic events. Calling this a “fault tree analysis” is therefore misleading.
An assumption in the PWS Study was that ships did not exceed the legal speed limit. However, since the authors were convinced that human errors were important elements of
An overarching framework is the basic model that ensures, at the outset of the analysis, that all known failure scenarios have been included, that they have been included only once, and that all dependencies among events have been systematically accounted for. The framework can be, for example, a basic event tree that includes the major classes of initiating events and the corresponding accident sequences. The probabilities of this event tree generally need further analysis using other tools, such as fault tree analysis, simulation, stochastic processes, etc. A sample framework is given in Appendix B.
risk, this assumption may be overly optimistic. Excessive speed could be an important element in the probability of groundings (if there is a loss of propulsion) or of the probability of collisions (especially in the fog.) Excessive speed could also be an important factor in other kinds of accidents. If speed affects the risk, it could also affect the benefits of some risk mitigation measures and should have been included in the analysis
Gathering expert opinions is an art as well as a science because experts have to understand what probabilities mean and that they have to be comparable. Gathering opinions from questionnaires requires appropriate interaction among experts and between experts and analysts to ensure that judgments are comparable. In the PWS Study, experts did not have an opportunity to modify their opinions based on the opinions of their colleagues. The variables in the PWS Study were not defined carefully or precisely enough to ensure that the encoding was meaningful, and some of the questions in the study appendices were vague. A more significant problem was created by gathering expert opinions through pairwise comparisons. This method has the advantage of simplicity, but the proposed ratios may not be ratios of probabilities. There was no way to “anchor” the whole process in probabilistic reality.
In principle, the three chosen methodologies were appropriate for the PWS Study. Notwithstanding their mathematical and scientific value, however, the way they were implemented raises questions about accepting the results of the study without further confirmation. Although the study team made great efforts to “get things right,” the team was hampered by a number of problems. The absence of appropriate information caused analytic and modeling uncertainties and prompted the team to adopt a number of creative approaches to make up for the deficiencies. These corrective measures, particularly the use of expert judgments, led to even greater uncertainties in the data and analyses. Nevertheless, the study did not include an uncertainty analysis, which raises many questions about the limitations, bounds, and overall applicability of the numerical results. Because the accident frequency in PWS was very low, the study team established a “ground truth” against which to compare the analytic results, which added to the uncertainty.
Because of the constant interaction of the PWS Study team and PWS steering committee, the NRC committee concluded that the steering committee was, in effect, part of the study team and that the study was conducted jointly. This situation may have injected some stakeholder biases into the analyses (such as the emphasis on maintaining current operations and the least costly risk reduction measures. At the behest of the PWS steering committee, many important assumptions and simplifications were made that were not well supported in the study or the TD. The influence of these changes on the results was not evaluated in the study and could not be evaluated by the review committee because they were not traceable through the analyses. Furthermore, the committee was not convinced that an appropriate range of risk reduction measures was considered, perhaps because of the desire to maintain “business as usual” and the unquestioned assumptions about the practical costs of implementing risk reduction measures that would require significant changes in the status quo.
The committee has doubts that the fault tree analysis contributed reliable results. The MARCS and simulation analyses are more credible. But not all operating conditions were modeled, and the ones that were modeled were not necessarily the most appropriate ones with respect to some parameters, such as weather, ice, earthquakes, TAPS vessel speed and momentum, and the potential oil outflow from non-TAPS trade vessels. The numerical results of the three analyses are separated by half an order of magnitude or less, which is intuitively satisfying. But the mutual validation is more apparent than real because a large number of the inputs and conditions were the same for all three.
The relative rankings in the results are probably more meaningful than the numerical results. The results do not provide a suitable basis for cost-benefit analyses to support investment decisions for risk reduction measures. Nor does the study provide guidelines for measuring the effectiveness of implementing risk reduction measures. The committee believes the PWS Study should be considered a first step and a basis for lessons learned rather than a completed work. The results should be interpreted and used very cautiously, especially in support of investment decisions for risk reduction measures.
The results of a risk analysis are always site-specific and system-dependent. The NRC committee cautions against using the recommendations in the PWS Study anywhere else in the world. Furthermore, because this study is only a first step and its structure is shaky, it should not be replicated without significant modifications. In fact, a much simpler, but more carefully executed analysis (starting with a better overall framework and more careful encoding of expert opinions), could be done more effectively and at less cost.
The PWS Study represents considerable efforts by the two contractors of the study team, who were assisted on a regular basis by the PWS steering committee. As a product of a group of consultants, the study apparently met the expectations of its sponsors. The NRC review committee, however, concluded that the PWS Study did not meet either the peer review criteria set forth by the NRC or the expanded criteria developed by this committee. The following summary is an assessment of the PWS Study in terms of the two sets of peer-review criteria.
NATIONAL RESEARCH COUNCIL CRITERIA
Are the conclusions and recommendations adequately supported by evidence, analysis and argument? Because the committee was unable to review the data (the evidence) or determine weaknesses in data collection (e.g., the absence of information on weather conditions and human factors), the committee concluded that this criterion was not satisfied.
Are the data and analyses handled competently? Although the analyses are presented well, the data remain hidden. The committee was unable to resolve doubts about the treatment of human error and uncertainties.
Are sensitive policy issues treated with proper care? Insofar as the PWS steering committee appears to have been satisfied, this criterion has been satisfied.
Are the exposition and organization of the report effective? The committee concluded that lack of clarity was a significant weakness of the study and that this criterion had not been satisfied.
Is the report fair? This criterion appears to have been satisfied.
Is there a clear statement of the constraints placed on the study team and of the impacts of these constraints? In briefings to the NRC committee, the PWS Study team mentioned some constraints, such as the exclusion of the effects of earthquakes, that are also mentioned in the study, although the impact of these constraints is not discussed. Other constraints, often described as ground rules (e.g., not eliminating weak performers in the TAPS trade as a risk reduction measure), were not described.
Are the data collection methods clearly explained? Are they based on established procedures? Because the data were not available for review, the NRC committee was unable to answer this question.
Are all key factors included in the analyses? Is a credible explanation given for any that are not? The committee could not answer these questions because the characteristics of the key factors were never explained. The speed of ships, for example, was not included as a factor in the analyses, and no explanation for this omission was given. The data collection method appears to have been as good as it could have been under the circumstances.
Are all assumptions identified and explained? Can the effects of these assumptions be traced through the analyses? The NRC review committee concluded that, although the assumptions were mentioned in the PWS Study, it was impossible to trace their effects through the analyses.
Are the methods based on established procedures, or, if not, are they clearly explained and supported? Do they connect with the “real world”? The methods used in the PWS Study were based on established procedures. If they had been based on appropriate data, they could have connected with the “real world.” However, substantial changes will be necessary before the methods can be used elsewhere.
Can the logic be followed readily? Are the influence of specific inputs and methodologies, such as simulations, identified? From the NRC review guidelines: “Are the data and analyses handled competently?” The logic of the PWS Study was not transparent.
Are any major uncertainties, due to data collection or other factos, identified? Although many of the uncertainties are discussed in the final section of the PWS Study, the
study is written as though these uncertainties were not important. In addition, the study does not include a comprehensive uncertainty analysis.
Were sensitivity analyses done regarding key assumptions and uncertainties? No, only one sensitivity analysis was done.
Do the results follow from the methodologies used, the data presented, and the assumptions identified? In view of the weaknesses the NRC committee identified in the data and methodology, particularly the unreviewable data and the lack of analyses of human error, the committee could not determine that the results followed from the data and analytic methods.
Are the conclusions and recommendations consistent with the study results? Are the study conclusions consistent with the results of the sensitivity analyses? From the NRC guidelines: “Are the conclusions and recommendations adequately supported by evidence, analysis, and argument?” Weaknesses in eliciting expert judgments, the unavailability of data, and problems with the analytic methods, raised serious questions. The best that could be said was that, to the extent that the results of the analyses were valid, the conclusions and recommendations are consistent with the study results.
What limitations, uncertainties, or other weaknesses should be identified to the PWS steering committee? This NRC review committee has identified many weaknesses that should be brought to the attention of the PWS steering committee and any other group evaluating the usefulness of the PWS Study. If this approach were used elsewhere, the scope of the study should be expanded to include other outcomes, such as loss of life and damage to a broader class of vessels.
CLARITY OF PRESENTATION
The presentation lacks clarity, consistency, and simplicity in many respects. The models, data sets, and analyses are admittedly difficult to digest and to describe succinctly, but in many of the technical areas, the study merely refers to the TD rather than providing lucid summaries. Assumptions and assertions can only be supported, in some cases, by the reader’s diligent search of, and inferences from, the TD. Frequent references to other parts of the study interrupt the logical flow of the text. The use of many figures and tables is commendable, but the text generally states only the results and does not explain the underlying analyses (e.g. why the results are presented in this form, what phenomena or analysis/ model led to this result, etc.) The analytic rationale and logic for evaluating risk reduction measures are not explained.
CORRELATION BETWEEN THE RESULTS AND REAL DATA
Because the frequency of incidents and accidents in the PWS is very low, ordinary statistical analyses are extremely problematic. The actual oil outflow incidents and acci-
dents may well be anomalous and may depend on factors that were not modeled. None of the methods used in the PWS Study explore the actual PWS experience or correlate the study results with real data. The correlation between the study results and PWS experience is not known. Thus, the absolute values for the frequency of accidents and oil outflow are not known. The absolute values used for the probabilities of incidents and accidents and for the amounts of oil outflow are intuitively reasonable in that the study results do not contradict the real data.
The PWS Study is a significant step forward in the use of probabilistic methods in marine risk analysis, even though, in the judgment of the NRC committee, it is not complete and should not be replicated. But if the weaknesses identified in this NRC review report are addressed, a systems approach based on the PWS Study could be used to analyze risks in other marine systems. The substantial involvement of stakeholders, who appear to have become owners of the PWS Study, may be particularly valuable. Unfortunately, sometimes a sense of ownership can cause serious issues to be overlooked, and rigor can be compromised for the sake of consensus.