Models Used in the Prince William Sound Study
The PWS Study addresses problems of petroleum transport between the Port of Valdez and the Hinchinbrook Entrance to the Gulf of Alaska. This transit is broken down into six sections: Port of Valdez, Valdez Narrows, Valdez Arm, Central PWS, Hinchinbrook Entrance, and the Gulf of Alaska. (PWS Study, 2.1) Because of major differences in geography, weather, and traffic in these sections, the PWS Study addresses each one separately.
The PWS Study used three modeling approaches: a static statistical model developed by DNV called the marine accident risk calculation system (MARCS); fault tree analyses, also developed by DNV; and a dynamic simulation model, developed by the GWU team. Oil spill volumes for all three approaches were calculated using a model developed by DNV. Other critical aspects of the modeling approaches were the collection of data and the development of probabilities based on questionnaires and expert judgments (data collection, including the use of questionnaires, is discussed in detail in Chapter 3 of this report). The work plan of the PWS Study specified that the risk measure of “oil in the water” would be the final stage of the calculations, so no consideration was given to the environmental impact of oil spills (or to loss of life from collisions).
MARINE ACCIDENT RISK CALCULATION SYSTEM (MARCS)
MARCS was originally developed by DNV for risk assessments of shipping around the United Kingdom. Section 3.8 of the PWS Study briefly describes MARCS. Substantially more information is presented in Section 4.1 of the TD.
MARCS treats all ships alike and assumes that they stay in assigned shipping lanes, using a Gaussian density to determine the probability distribution of a ship’s distance from the center of the lane. Because the model calculates a statistical distribution of shipping traffic over a whole year, it does not treat seasonal variations, although, in principle, shorter periods of time could be used for model runs. The MARCS model calculates the probabil-
ity of collisions using fault trees based on expert judgments for collisions of vessels that pass within a ship’s length of each other. The model can include weather and other environmental factors, such as currents, sea states, and wind, as well as geographical features. MARCS also includes a powered grounding model, a drift grounding model, a structural failure/foundering model, and a fire and explosion model. All models are based on worldwide data.
The MARCS approach as used in the PWS Study has the following potential weaknesses:
Models are not dynamic
All ships are assumed to travel at an average speed (TD 4.1:1.10)
All ships are assumed to adhere to the collision avoidance rules (i.e., there are no “rogue” ships.) (TD 4.1:1.10)
Human factors are not explicitly included.
The powered grounding model does not include cases caused by failure to make required course changes. (TD 4.1:1.4).
The last omission is especially important because, of the four possible causes for powered grounding considered by DNV, failure to make a required course change is estimated to have a frequency three to four times greater than the other three causes (TD 4.1:1.24 [Table 5.2]). The other three causes are hard-over rudder failure; errant behavior of the attached tug; and wind or current from the side with crew inattention.
Because of the weaknesses of the MARCS approach listed above, the PWS Study team warns that it would be incorrect to consider the results delivered by MARCS to be a true and complete picture of oil spill risks in the Prince William Sound (TD 4.1:1:37). The final PWS Study does not use MARCS for many results, and never relies on MARCS alone (although in one case, for spills caused by fire and explosion, MARCS is used with the fault tree without the simulation model) (PWS Study, 5.3 [Table 5.2]). MARCS does have one major advantage, however, over the simulation model. It can include the characteristics of tugs, although it only uses the characteristics of the most powerful tug if more than one tug is involved (TD 4.1:1.28).
FAULT TREE ANALYSES
Fault tree analyses are widely used in failure and risk analyses of technological systems, such as satellite systems, launch vehicle systems, nuclear power plant systems, and chemical plant systems. One of the leading practitioners of this approach, Norman Rasmussen, has cautioned that relevant data are necessary for fault trees to be effective:
Fault tree analysis is a technique used to predict the expected probability of failure of a system in the absence of actual experience of failure…The technique is applicable when the system is made up of many parts and the failure rate of the parts is known…The fault tree analysis always starts with the definition of the undesired event whose probability is to be determined…[T]he tree is then developed to lower and lower levels, to the lowest events, called primary faults. For the fault tree method to work…primary faults must be events whose probability can be determined from experience (Rasmussen, 1981).
Unfortunately, relevant data were not available in the PWS Study, although this problem was acknowledged by the study team: ‘The basic events of a fault tree are those events that make up the bottom line of the fault tree structure. To perform calculations of the top frequency or probability of a fault tree, these basic events needs [sic] to be quantified” (TD 4.2:1.1). In other words, strictly speaking, fault tree analysis was only used in a metaphorical sense in the PWS Study.
A minimal description of the fault tree approach is given in Section 3.9 of the PWS Study. A more detailed description is given in Section 4.2 of the TD, and the results are given in Section 5.3 of the TD. The TD presents the fault tree diagrams for several cases, as well as the results of the calculations using fault trees. Although the fault trees appear to be reasonably complete, they were not developed with real data about the basic events. In discussions with the members of the NRC committee, the fault tree modelers said they had essentially used expert judgments to fill in the top boxes, which they then used to calculate risk.
For example, the powered grounding fault trees (i.e., while the motors are operating) have 46 blocks, which were filled in as follows (TD 4.2:1.23–1.33):
21 based on expert judgments (three DNV employees with substantial maritime experience)
10 based on an unpublished thesis for inattention or failure to perform for officers of tugs and tankers (Haugen, 1991)
6 based on actual data
6 based on estimates or calculations
3 deemed not applicable to PWS
The NRC committee believes that describing this as true fault tree modeling is erroneous and misleading because there was no logical analysis relating basic events to accidents. It might have been better if the PWS Study team had used only the top blocks and labeled the estimates expert judgments instead of implying that a real fault tree analysis had been done.
Another potential weakness in this fault tree analysis is the assumption that the failure rates for steering, propulsion, and radar for all ships in PWS are the same as for the tanker fleet. Thus, estimated failure rates for ferries, cruise ships, processing vessels, and fishing vessels are the same as those for the TAPS (Trans-Alaska Pipeline System) tankers (TD 4.2:1.43).
The committee recognizes that data were not available to fill in the fault trees and that expert judgments had to be used. But the estimates do not even include uncertainty ranges and thus give the reader a false sense of completeness, which could lead to underestimating or overestimating the risks and overlooking possibly effective risk reduction measures.
The model used for most of the risk analyses in the PWS Study is the simulation model developed by the GWU team (35 out of 43 cases) (PWS Study 5.3 [Table 5.2–1]). The simulation model is briefly described in Section 3.7 of the PWS Study and in more detail in Section 4.5 of the TD. A simple description follows.
Using a tanker as a reference point, the model calculates the probability that one or more other vessels are within two miles and ten miles of the tanker. All vessels within two miles are included, and vessels in the two to ten mile range are included or excluded using a triangular probability distribution. This calculation is updated every five minutes. From a separate set of calculations, probabilities are developed for a set of possible “incidents” at each five minute snapshot. The probabilities of an accident, given an incident,1 are then used to estimate if an accident will occur. If an accident occurs, the oil spill outflow model is used to estimate the amount of oil in the water. The results require 25 separate calculations.
The simulation model includes a weather model, transit routes for tankers and SERVS vessels, and a traffic model. The simulation also includes traffic rules for six different types of vessels (fishing, ferry/tour, cruise, tug with tow, SERVS, and tanker). The model first calculates whether another vessel is within 10 miles of the tanker. If not, it moves on to the next five minute snapshot. A vessel within 10 miles creates an opportunity for an incident. If there is a vessel within 10 miles (and there may be several), the probabilities are used to determine if an accident occurs. For an accident to occur, the model requires one of the following: a failure of the propulsion or steering system of the tanker, an operational error by the tanker, or a failure on a nearby vessel. The model assumes that two errors of the same kind cannot happen at the same time (TD 4.5.2:3–2.4).
The traffic model appears to be relatively straightforward and has the capability to handle the large number of fishing boats and the smaller number of other vessels in PWS. The traffic model can also consider the need for a tanker to remain at anchor if a dock is not available when it reaches Valdez (inbound) and the availability of tugs and SERVS vessels for a tanker to leave dock. The traffic model also includes weather calculations updated hourly and imposes closure conditions in both the Narrows and the Hinchinbrook Entrance when winds are greater than 45 knots, in which case the inbound tanker drops anchor and the outbound tanker circles.
At the heart of the simulation are the probabilities of incidents and the conditional probabilities of accidents, given an incident. In the opinon of the NRC committee, unusual, and questionable, aspect of the simulation model is in the way the probabilities were developed. As the developer of the simulation states, “Most parameters in the conditional accident probability are obtained through expert judgment” (TD 4.5:16). The expert judgments for the PWS Study were obtained from questionnaires given to 162 people involved in PWS maritime affairs (as described in Chapter 3).
Experts only provided relative probabilities. To determine absolute probabilities, the probability of incidents caused by propulsion failure were calibrated based on DNV worldwide data. To calibrate the probability of operational errors, it was assumed that 80 percent of incidents were caused by human error and 20 percent by mechanical failure. This assumption is based on a similar assumption said to hold true for accidents (TD 4.5.2:17). This so-called “80–20 rule” is widely quoted but has not been credibly substantiated; and
the NRC committee has many concerns about oversimplifications that may arise from applying this rule (see Appendix D).
The simulation model’s accident rate calculations were benchmarked by comparing the simulation results with the accident calculations for the MARCS model, using the 80–20 assumption about the relative frequencies of human and mechanical errors.
Appropriateness of Methods
There are several possible approaches to structuring a risk analysis model. An important consideration is to start with the right initiating events and follow with the proper probabilistic conditioning of the variables in different accident scenarios. One can first divide the problem into accident types, assuming, for example, that they are either probabilistically independent or not. For each accident type, one can then structure scenarios starting with the initiating event and considering the subsequent events and variables sequentially. Event trees represent these probabilistic dependencies; fault trees are logical tools that allow computing some of the failure probabilities shown in the event trees. The order of the variables in event trees is somewhat arbitrary and depends, in practice, on how the information is structured.
Although other approaches and analytic tools might have been selected (and could have been implemented differently), the PWS Study team’s choices of a dynamic simulation (the GWU model), a static model (MARCS), and fault tree analysis appear to be reasonable. The rationale for the last two selections is not clear, however. Nor are the anticipated advantages or disadvantages of using three apparently independent and unrelated approaches. However, one rationale for using more than one model is that the simulation model does not, by itself, show traceable cause and effect.
The PWS Study team might have been better off using one approach and applying all available resources to making that approach as complete and accurate as possible. Nevertheless, as a learning experience and a preliminary assessment, the implementation of these different approaches (including the construction of the model, the adaptation of the model to objectives and purposes, the unique application to PWS, and the interrelationships of the several analyses) led to a number of useful insights into the comparative strengths and weaknesses of each method. These insights offer valuable lessons into the application of risk assessment methods to the maritime industry.
Each of the chosen analytic methods is theoretically reasonable and appropriate for use in the PWS Study. A dynamic simulation model is particularly applicable to situations like PWS that have time-dependent elements that interact in response to complex stimuli. The MARCS model, which is based on a historically-derived statistical representation (static) model, is best suited for a computer-resource-limited study for which there are adequate data. The fault tree analysis, in principle, traces cause and effect.
The logic of the PWS Study, however, was to create a unified approach linking the three methods. Unfortunately, this unity appears to be artificial, and the several methods seem to reflect the assignment of specific analytic execution to separate contractors. In
defense of a unified approach, however, one can point out that certain information and models are common to all three methods, including the frequencies of events and the calculations of oil outflow. The significance of the differences and similarities in the numerical results of the three models are not explored in the PWS Study; only contradictions and inconsistencies of a definitional or procedural nature are explored.
The various elements of the analyses and assertions about the analyses are based on many assumptions about the available data and about operations in PWS. The PWS study team informed the NRC review committee that these assumptions had been coordinated with, and agreed to by, the PWS steering committee. However, at best the study weakly supports most of the assumptions, and agreement by the steering committee does not lend them scientific credibility (e.g., assumptions that ship speeds are constant and lane violations do not occur.) In general, the impact of the assumptions on the analyses and on the results is not discussed in the PWS Study.
Assertions about consistency checks and intermethod validation (through comparisons of the three methodological approaches) appear to be merely that, assertions without analytic foundation. Evaluations of candidate risk reduction measures depend on fundamental assumptions about the direction and magnitude of proposed changes. For example, it is assumed that more bridge officers will reduce the probability of mistakes and, hence, of accidents. However, this logic is not supported analytically. If one officer is inadequate and it is assumed that two will be better, will five officers improve operations further, or will they make observations, decision making, and giving commands more complicated?
Independence of the Three Analytic Methodologies
The PWS Study asserts that the relative closeness of the numerical results for all three models indicates the correctness of the results and the validity of the methods. This conclusion assumes that the methods are independent. However, they all used the synthesized PWS database (described in Chapter 3), they used a common traffic image and oil outflow model, and they relied on expert judgments made by members of the oil and maritime community, who, it may be presumed, shared many experiences, had similar knowledge of industry literature, had similar cultural biases, and communicated with each other. The analyses were also based on shared knowledge of the worldwide body of scientific and analytic literature. This is not to say that the expert judgments were wrong. But the three methods, as implemented, had a good deal in common, both in terms of input data and modeling assumptions. Consequently, a reasonable case can be made that the results would obviously be roughly comparable.
Oil Outflow Model
The same oil outflow model was used with each methodology to calculate oil outflow from an accident. The oil outflow model included principal accident types, hull types and
loading conditions, and accident severity and location. Several important assumptions about the effects of hull type, the energy required for hull penetration, and the consequences of hull damage were made to simplify the oil outflow model. The basis for calculating oil outflow, in the absence of PWS data, was Lloyd’s Casualty Returns (worldwide data) and other analyses, as well as assumptions based on judgments of members of the PWS steering committee. The oil outflow model was characterized by uncertainties, break-points in the probabilities of oil outflow, and discontinuities caused by the phenomenological and simplifying assumptions (e.g., the assumption that there is no oil overflow in 50 percent of tanker/smaller vessel collisions and that there is always outflow in tanker/tanker collisions, given sufficient energy), as well as by the inherent nature of oil release mechanisms related to the receiving environment (e.g., the release differences under various sea conditions.)
Even the definition of a collision was inconsistent. The MARCS model, for example, defines a collision as passing another vessel within one ship length. But the MARCS model might have underestimated the number of collisions because it undercounted collisions during traffic peaks. The simulation model may have overestimated the probability of collisions by overcounting multiple interactions. The assumptions regarding hull penetration that yield oil outflow, which were used to simplify calculations, were not well justified (e.g., 15 megajoules is assumed to be the threshold kinetic energy of a ship perpendicularly colliding with a tanker with resulting penetration). The PWS Study team claims that its judgments were conservative, and the oil outflow model does appear to be a reasonable approximation accidents. But the conservatism and accuracy of the modeled events were not documented.
Sensitivity Analysis and Traceability
The PWS Study contains no sensitivity analyses or discussions of uncertainties, with the exception of the uncertainty in the fault tree modeling of powered grounding in the Narrows. The results for the analytic methods all span a similar range, up to a factor of five, and this level of agreement is said to validate the methods. However, no analysis is presented to enable the reader to understand the impact of uncertainties and assumptions and, therefore, to establish the confidence limits of the results or the probability distribution of the results.
A common, but simplified, assumption is that 80 percent of incidents and accidents are caused by, or are directly related to, organizational and personnel performance (i.e., human factors). However, a rigorous risk assessment should be based on much more than this simplified assumption. The PWS Study does not model human factors separately. They are implicit and hidden in the incidents and accidents database.2 Explicit considerations of human factors and associated risk reduction measures, based on real data, are not included.
Moreover, the consideration of human factors in the PWS Study (principally in the GWU system simulation analysis) is based on conjectures, such as that a failure of ship navigation due to personnel performance can be corrected if the bridge is manned by more than one officer. However, quantitative evidence for this failure mode and for the corrective action is not presented. In general, specific risk reduction measures for improving human performance are not presented.
Limitations of the Analytic Approach
The NRC review committee found that the fault identification logic was both questionable and redundant. Because event frequency values were generally assigned and used at the highest level in each tree chain, the fault trees do not provide real logical analyses. Where lower levels were used, frequency values were assigned based on the judgments of DNV experts using North Sea and worldwide experience. No attempt was made to update that information for PWS. Expert judgments were based on personal experience, which generally did not include experience in PWS. Thus, the committee does not find the fault tree analysis, as implemented, highly credible.
The MARCS model was designed to provide a system-level, steady-state, average-effect picture. It does not, therefore, readily reflect short-term, time-dependent, anomalous, or micro events. Because it is designed to handle paired (i.e., two object) interactions, it may understate the effects of interactions that, for example, occur at peak traffic levels. In the PWS Study, the MARCS model does not model inbound traffic.
The simulation model is specifically designed to investigate the dynamics of the transportation system (i.e., time-dependent, dynamic event sequences). It models vessels as point objects with step functions in speed and course changes. Some elements of the simulation model were simplified to ease the computational burden (e.g., using a five-minute time interval and fewer course changes than usual). The study team assumed that the affects of these simplifications were insignificant.
The simulation does not contain detailed models of ship control, handling, maneuvering, or other behaviors that interact with advances in track, nor does it contain models of human behavior. Because the simulation is based on judgments by experts who were queried about specific scenarios, it is satisfactory (to the degree that it is valid) for only a small range of parameters for each scenario. Large deviations from the norm could not be reliably handled. Like other models, the simulation model depends on its implementation and may overstate or understate risk or miss an unknown number of significant events.
In principle, the simulation model is capable of absorbing all of the attributes of the other methods. However, the computational burden of simulation can increase rapidly, and simulation may be limited by practical constraints like funding and the availability of computer resources.
The simulation in the PWS Study incorporated assumptions that were assumed to have little significance. No consideration was given to changes in TAPS traffic in response to changes in North Slope field production or other conditions or to changes in the vessel fleet. Specific TAPS ships were modeled, but a future fleet was not modeled. Therefore, the results may be unique to specific vessels and not applicable generally. The oil outflow
from collisions in the Narrows may have been overestimated because of the operational requirement that tracks be parallel, thus reducing the probability of collisions even for close passages or near misses; nevertheless the oil outflow model assumes collisions at right angles. Tracing cause and effect through the simulation analytic process is very difficult and was not done in the PWS Study.
Some details of the analytical methods were not adequately treated in the simulation as a consequence of inadequacies in the models or gaps in the data. For example, the influence of currents (treated in the MARCS model), tides, wave heights, visibility, and ice were not adequately treated in the simulation. Other potentially significant phenomena were not treated at all, including earthquakes, which occur as frequently as some events that were treated.
Speed and momentum were not included in the MARCS models for drift groundings in critical locales. By way of explanation, the PWS Study team stated that the modeling of oil outflow probability was adequate because (1) the available residual steerage way in the case of propulsion failure makes evasive maneuvers possible, thus lowering the probability of grounding, and (2) in case of a steering loss, the probability and amount of oil outflow during drift grounding is reduced because the vessel could be maneuvered using propulsion steering. The NRC committee found these arguments to be superficial and speculative.
The PWS Study focuses on potential oil releases from TAPS trade vessels. Estimates of oil releases from incidents and accidents involving other vessels, notably cruise vessels, were similar to estimates for oil tankers. Non-TAPS vessels were not included in the analysis.