6
Assessment of Simulator Technology and Results

As described in the previous two sections, simulation involves an array of physical and mathematical components, each with their own limitations and inaccuracies. In addition, real-time simulation uses a pilot who introduces human variations into the simulation. If the results of the simulation are to be interpreted sensibly in the waterway design process, it is important to determine how well this array will predict the track of a ship in a given situation. Discussion in the preceding chapters shows that it is difficult, if not impossible, to treat the question quantitatively and scientifically. With this caveat, the following discussion assesses simulator technology from an engineering point of view, that is, in the context of its application to waterway design.

ACCURACY

Simulation has only recently become a feature of some waterway design initiatives, although use of the technology is increasing. Interpretation of the simulated vessel tracks provides insight into the various navigation factors (principally turn characteristics, channel width, and depth). The assessment presented here addresses the related concepts of accuracy and validity of simulation in the context of the waterway design process. For this discussion, a simulation will be considered accurate if it can produce piloted track predictions that are useful as a basis for a design decision



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 59
Shiphandling Simulation: Application to Waterway Design 6 Assessment of Simulator Technology and Results As described in the previous two sections, simulation involves an array of physical and mathematical components, each with their own limitations and inaccuracies. In addition, real-time simulation uses a pilot who introduces human variations into the simulation. If the results of the simulation are to be interpreted sensibly in the waterway design process, it is important to determine how well this array will predict the track of a ship in a given situation. Discussion in the preceding chapters shows that it is difficult, if not impossible, to treat the question quantitatively and scientifically. With this caveat, the following discussion assesses simulator technology from an engineering point of view, that is, in the context of its application to waterway design. ACCURACY Simulation has only recently become a feature of some waterway design initiatives, although use of the technology is increasing. Interpretation of the simulated vessel tracks provides insight into the various navigation factors (principally turn characteristics, channel width, and depth). The assessment presented here addresses the related concepts of accuracy and validity of simulation in the context of the waterway design process. For this discussion, a simulation will be considered accurate if it can produce piloted track predictions that are useful as a basis for a design decision

OCR for page 59
Shiphandling Simulation: Application to Waterway Design concerning navigation and risk. Accepted guidelines for this accuracy apparently do not exist, and the accuracy requirement varies depending on the exact nature of the design problem. Construction tolerance for horizontal dimensions of the waterway is about 10 feet for fairways. Therefore, comparable accuracy for simulation is a reasonable goal for channels or approaches to berths involving dredging; greater accuracy is only of academic interest. However, for berthing or lock operations with very tight maneuvering tolerances, the level of precision required for the simulation is correspondingly higher. Validity of a simulation can be expressed in the form of two narrow but scientific assessments. First, does the predicted track of a given ship accurately reproduce the real ship track when the pilot or autopilot performs at the simulator exactly as either would perform on a real ship? More scientifically, is the output of the simulator the same as that of the ship when the input to both is identical. If the simulator meets this criterion, then, will the pilot (or autopilot) make the same maneuvers at the same times in the simulator environment as would be made in the shipboard environment given the same transit conditions? If the answers to both questions are affirmative, then the simulation can clearly be considered valid. That is, the predicted tracks will compare well with the full-scale results for a piloted ship. For discussion purposes, the first of these is identified as the mathematical modeling problem and the second as the pilotage modeling problem. In the past, these two very different aspects of model validation have often been intermingled. Assessing the validity of a simulation in terms of the separate accuracies of the mathematical model and of the pilot model is used here for convenience of discussion. The mathematical modeling described above is the open loop response of the ship (that is, without the use of corrective steering measures). A mathematical model that is accurate by these terms can perform dead reckoning, a computation of the track, given only the history of the commands. This significantly more sensitive problem poses a particularly severe demand on accuracy. In either the full-scale or simulated transit, the pilot takes corrective action when the ship appears to deviate from the strategy for transit, no matter what the cause. Thus, the pilot in a simulator will attempt to correct any deviation from the planned track resulting from an error in the mathematical model, just as if some real deviation was caused by the proximity of a bank, vessel, or other waterway feature. As a result, the tracks of all types of ships tend to be close to one another, independent of their inherent maneuvering behavior. The pilot's skill enables anticipation of the ship's behavior and its interaction with the environment so that commands are given expeditiously. This result is achieved, even for ships that are difficult to steer. Thus, the errors in the mathematical modeling are particularly

OCR for page 59
Shiphandling Simulation: Application to Waterway Design difficult to discover from an analysis of piloted tracks. On the other hand, the accuracy demand for the mathematical model used in a shiphandling simulation can be less than that required fo dead-reckoning simulation. That is, feedback provided by the pilot may minimize the effects of inaccuracy. During this study, no research was identified that can provide guidance for relating these two accuracies. In the sections below, a critique is presented for the state of practice of the two aspects: the mathematical modeling problem and the pilotage modeling problem. These discussions, however, remain rather general because, as the above discussion reveals, the accuracy required of these components cannot be specified within the current state of practice. CRITIQUE OF MATHEMATICAL MODELING TECHNOLOGY The discussion in Chapter 5 demonstrates that many acceptable data base frameworks for the waterway and its environment are possible. Identification of the constants for the data base is direct for bathymetry, but may require either physical or computational fluid dynamics (CFD) modeling for the currents or winds. Thus, tools are available for developing these data bases, but the cost of determining the appropriate input data and of exercising these tools limits their use. Mathematical models for ship behavior are well known in the case of steering and maneuvering in deep water. The measurement of turning performance in deep water is usually included in the trials of new ships. As a result, many comparisons with actual ship data have been made between simulator predictions for deep-water maneuvering and full-scale gross measures of maneuvering performance (for example, advance, transfer, tactical diameter, directional stability). The state of practice is such that theoretical predictions of deep-water turning performance are typically within 10 to 20 percent for these measures when the coefficients in the particular mathematical framework are identified using scaled physical experiments (captive model tests and extensive propeller-rudder interaction model tests). Unfortunately, the performance of physical model maneuvering tests on new ship designs is not common. The identification of coefficients in the mathematical framework for new ships is often performed by interpolating within a data base of coefficients for similar ships (that is, without using physical experiments). This approach appears to be successful if the new ship is indeed similar to those in the data base, and the degree of success depends critically on both the size and quality of the database and on a careful review of the resulting coefficients by a knowledgeable practitioner. The frameworks and coefficient identification process in use for both unrestricted and restricted shallow water vary from simulation facility to simulation facility. Almost all of these mathematical models are considered

OCR for page 59
Shiphandling Simulation: Application to Waterway Design proprietary to the individual facilities and were not available for scrutiny or detailed comments in this report. As discussed in Chapter 5, the technology to identify coefficients in any such framework is available if the under-keel clearance is greater than 10 percent. Only a small part of this technology is ever involved in any simulation because of the extremely high cost of the required experimental setup and of the data collection. For very small under-keel clearances, the physical phenomena remain unclear. In the current state of practice, many frameworks and associated coefficients for models that deal with details specific to waterway design (shallow water, banks, passing ships, variable currents, and so on) are constructed heuristically by using an amalgam of available theoretical developments, by using results of the limited available model tests of ship performance in idealized waterway configurations, and by bold assumption. In addition, there appears to be little scientific basis for the usual, quasi-steady treatment of highly time-dependent events, yet these are critical situations in many simulations. Scientific verification of the accuracy of available models by comparison with full-scale results generally is also missing. Full-scale measurement of maneuvering tracks in unrestricted shallow water is limited. The most extensive set of tests appears to be those conducted on the Esso Osaka 10 years ago (Abkowitz, 1984; Ankudinov and Miller, 1977; Bogdonov et al., 1987; Dand and Hood, 1983; Eda, 1979b; Fujino, 1982; Gronarz, 1988; Miller, 1980). Even so, these tests were performed at under-keel clearances that are still large compared to those tolerated in many waterways. Thus, the parts of the mathematical modeling process that are critical to simulation results but lack scientific precision are precisely those aspects that differentiate simulation for waterway design from simulation for deepwater maneuvering. These include: modeling of the hydrodynamic forces and moments for situations where the under-keel clearance is small; determination of the forces on a ship passing near waterway sides (banks); determination of the forces on a ship in essentially unsteady conditions (approaching banks, approaching and passing other ships, moving into regions with sharp current or bathymetric gradients); and the implicit assumption in most frameworks that the forces resulting from the various phenomena (for example, bank effects, propulsion, rudder effects) can be superimposed without considering their interaction. Because of all the additional assumptions, it is unreasonable to expect that state-of-the-art mathematical models for maneuvering in restricted shallow water will be as accurate as those for deepwater maneuvering. However, as discussed above, the accuracy required for the mathematical model (open

OCR for page 59
Shiphandling Simulation: Application to Waterway Design loop) may be considerably less than the accuracy needed for the waterway design (closed-loop). This consideration, together with the significant costs involved, has inhibited the development of more accurate mathematical models. In practice, the validity of the mathematical model is established by comparing it with one or more of the following: real-world measurements or data, such as ship test trials; results from tests conducted using measurements deived from scale-model tank testing; performance estimates derived through mathematical extrapolation or interpolation using accepted theoretical models; the performance expected and evaluated by experts on the system the simulation has been designed to model; and the performance expected and evaluated by an interdisciplinary team of participants in the design process, including sponsors, planners, designers, and pilots, for the system the simulation has been designed to model. CRITIQUE OF PILOTAGE MODELING TECHNOLOGY A properly developed mathematical model will predict with acceptable accuracy the motions performance of a ship plying a waterway in response to commands from its pilot. The pilotage problem is this: Will the pilot perform on the simulator as if on board an actual ship? This question is much different from, and more difficult than, the mathematical modeling aspect because it is, to a major extent, a physiological and psychological question. If the pilot is a human pilot, then one can anticipate that slightly different commands will be given during each transit, even if the conditions during the passage are exactly the same. Thus, the sets of commands for a series of like passages by a single human pilot will be only approximately alike. Variations of this type are not encountered in standard fast-time simulation. If wide variations in shiphandling do result from man-in-the-loop simulations, the implications for waterway design may be important. If it can be determined that a particular waterway configuration is highly sensitive to pilot performance, then it would be prudent to search for and consider an alignment that is less sensitive: for example, a more desirable alternative would show little variation in swept path amongst different pilots. The bridge, visual scene, and radar contribute to an observer's judgment of the face validity (also referred to as apparent validity) or realism of a simulation. A full-size ship's bridge, a high-fidelity visual scene, and a ''stimulated'' real radar set have high face validity. In turn, such a judgment contributes to the observer's acceptance of the simulator, design study, and eventual implementation of the findings. Physical surroundings may also

OCR for page 59
Shiphandling Simulation: Application to Waterway Design contribute indirectly to pilot-ship-environment system performance by affecting the shiphandler's motivation. Thus, a relationship exists between fidelity of the simulator and pilotage accuracy, but the relationship is difficult to pin down. The omission of real-world elements may influence the pilot to make a command that would not be made on board ship, or vice versa. Such an omission may be a major element (for example, the absence of a compass repeater in a place where it can easily be referred to) or the omission of a seemingly small item (for example, a distant church steeple that the pilot normally uses as a navigational reference). Because it is difficult to know a priori what is important to portray in a simulation and what is not, designers and pilots generally have more confidence in a high-fidelity, real-time simulator than in a low-fidelity one. Whether or not fidelity adds to the accuracy of the pilotage modeling, it certainly adds considerably to the cost of simulation. For example, including a prominent water tower next to the church steeple in the visual scene mentioned above may only contribute to face validity, the immediate impression of realism. However, face validity can contribute to user acceptance of the simulator and its results. If the steeple is not used by local pilots as a visual cue in the piloting process, however, its inclusion may add little or nothing to the accuracy or fidelity of the simulated pilotage. The question of how much fidelity is needed to achieve accurate simulation is different from how much fidelity is needed to make the pilot perform realistically in the simulator. Pilots, by the nature of their profession, need to be quick learners and exceedingly adaptable. For instance, pilot skill includes the safe piloting of vessels from the pilot's very first pilotage on vessels of that class. This flexibility helps pilots interpret and use modest simulation that can only be called low fidelity. A typical low-fidelity simulator may have, for instance, only one crude television-sized display that can be switched from a synthetic radar view to a low-resolution, dead-ahead view. In these situations, pilots with experience in operating ships and knowledge of the waterway in question may be able to fill in the missing information and produce a track similar to that achieved on a high-fidelity simulator. Moreover, it is possible, even in a low-level simulation, to provide the pilot with a much more accurate view of the situation than will be available on board ship (for instance, with an accurate bird's-eye view). In this case, the results of the simulation may underplay the safety factors (Perdok and Elzinga, 1984; Schuffel, 1984). Because a high-fidelity simulation can be quite costly, the demonstration of validity and user acceptance for a low-fidelity simulation could lead to increased use of the latter. The accuracy of pilotage modeling in a real-time simulator is much more difficult to ascertain than that with mathematical modeling of the

OCR for page 59
Shiphandling Simulation: Application to Waterway Design basic ship behavior. No objective measure was found for it during the study. The accuracy of pilotage modeling of fast-time simulation is an issue that is separate from that of real-time simulation, and it is most often accomplished either by comparison with man-in-the-loop results or by simply presenting results to pilots familiar with the area in question and eliciting their comments and criticism. If the mathematical model of the ship track is known to be accurate, then the focus of the comparison is on the command sequence and timing produced by the autopilot. However, in the typical uses of fast-time simulation (that is, sensitivity studies to determine effects of changes in the waterway environment), a high degree of accuracy may not be needed. VALIDATION Because it is not possible to assess scientifically the accuracy of either the mathematical model (for fast-time or man-in-the-loop simulation) or the pilotage performance of individuals relative to the simulation, an overall validation of the simulation is typically conducted instead. Currently, a simulation is considered valid if pilots conclude that it accurately reproduces the modeled ship's behavior in a particular waterway (Eda et al., 1986; Hwang, 1985; Moraal, 1980; Puglisi et al., 1985b; U.S. Maritime Administration (MARAD), 1979; van de Beek, 1987; Williams et al., 1982). The idea is that if several pilots familiar with the waterway, the modeled ship, or both are all satisfied with the simulation, then there is reasonable confidence in the results. However, since this process is highly subjective, great care must be exercised to assure that preconceived views and experience with vessel behavior do not bias the evaluation of simulation results (J. P. Hooft, personal communication, 1992). The process for developing a valid simulation (mathematical model plus pilotage model) is iterative. Various procedures are used to screen out unintended bias that might result from "ownership" in simulation runs, preconceived views about vessel behavior, and other performance and technical factors. Typically, simulator facilities use an interdisciplinary team approach for validation although the process is often not made formal in facility procedures. Pilots participating in preliminary simulations to validate simulator performance are either part of the validation team or provide information for the validation process. Other pilots, qualified facility staff or other experts are sometimes used to observe and evaluate simulation runs to guard against bias (J. P. Hooft, personal communication, 1992; Puglisi, 1985; Puglisi and D'Amico, 1985). The mathematical model of vessel behavior, the physical representation of the visual scene, and for fast-time simulation, the autopilot model, are modified until the validation team is satisfied that the simulator performs realistically. Modifications that are

OCR for page 59
Shiphandling Simulation: Application to Waterway Design made represent an acceptable balance between theoretical considerations and practical experience. Considering the subjectivity that is involved in this process, the validation team must be carefully composed to include expertise for key governing factors in the simulation. Guidelines for validation team composition have not been established. The committee believes that for waterway design, expertise in vessel operations, hydrodynamics, mathematical modeling (of the physical environment and vessel behavior), and waterway design would be prudent. As part of the iterative validation process, simulation facilities typically conduct interviews with pilots participating in real-time simulation before any simulations are performed and after each simulated transit. These interviews serve several purposes. The pilots' knowledge of subtle aspects of a waterway is especially important because these pilots may be the only source from which these features can be detected, as evident from simulation case studies (Appendix C). The interview agenda is structured to obtain the pilot's subjective interpretation of how the simulated vessel behaved under the conditions tested, such as realism of vessel response to wind. Their broad experience in shiphandling helps identify flaws in the mathematical model but, according to many simulation facilities, is less useful for identifying the cause of these flaws. Pilots also have individual general strategies for making a transit and use their critical visual cues to navigate according to these strategies. Omission of pilot-specific cues can lead to less than satisfactory simulations. Interviews help uncover omissions so that they can be corrected. Finally, when analyzing a modification to an existing waterway, many facilities will first use a model of the existing waterway together with a model of a ship that currently uses the waterway. This simulates a situation familiar to the pilots and is useful for determining how much fidelity is needed to uncover errors in the basic mathematical model and to gain pilot confidence. This process also establishes a baseline from which to measure the effect of proposed changes. Introduction of either new ships or waterway configurations can then be made with greater confidence. In developing a new harbor or waterway, rather than modifying or upgrading an existing one, it is more difficult to determine simulation validity, even subjectively. For new waterways, pilots (as well as others participating in the design process) have no local knowledge as a reference for assessing the simulator's performance. Similarly, pilots confronted with unfamiliar hull forms or vessel sizes would be constrained in their assessment of validity. When similar waterway configurations or ships are used elsewhere, experts familiar with them are sometimes invited to work with regional experts to determine simulation validity. Considering the highly subjective nature of simulator validation in the maritime sector, can validation procedures be adapted from the aviation

OCR for page 59
Shiphandling Simulation: Application to Waterway Design sector where simulation is widely used for design and training? The committee found substantial differences between simulation in the aviation and maritime sectors that make transfer of technology and procedures highly problematic. Aircraft flight simulators are an integral element of the aircraft design and evaluation process. They are used to support design decisions, assess the design validity, train pilots (initial and proficiency training), and support mission planning and analysis. For commercial aviation, both aircraft and flight simulators are certificated by the Federal Aviation Administration (FAA). Military operational flight trainers and weapon system trainers are accepted by the using organizations for new designs as the new aircraft enters flight operations. Attention to fine detail is integral to designing high fidelity simulation mathematical models and recreating the aircraft's cockpit. Very good simulation fidelity is obtained by using proper models of the aircraft's aerodynamics, propulsion system, control system, weight and inertias, and by including a high fidelity cockpit. The aircraft design process is structured to provide this information that concurrently supports development of flight simulators, including their validation (Appendix F). Pilots are brought into the airframe development process early to provide operational perspectives for airframe design, and participate in final testing of flight simulators, primarily adjustments to the mathematical model to gain pilot approval. Unlike the aviation sector, shiphandling simulations are not developed as an integral part of vessel design. The design process for ships provides limited quantitative information for developing mathematical models for commercial ship behavior. Commercial ships are often one-of-a-kind or constructed in limited classes. Even if a class vessel, ship behavior varies significantly relative to other ships in the class with loading (which can radically change ship hydrodynamics) and other operational factors. Hydrodynamic testing is not performed extensively and aerodynamic testing is rarely conducted for commercial vessels. Ship trials data are available for certain vessels, but the operating envelope for testing is almost exclusively unrestricted deep water, providing no insight on variations (usually substantial) between ship behavior in deep and shallow water. Substantial differences also exist between the aviation and maritime sectors relative to modeling the operating environment, particularly the effect of external boundaries. In particular, the forces on a ship are strongly affected by the details of waterway geometry. Modeling the aviation operating environment (such as the atmosphere and atmospheric disturbances) is more straightforward. Aircraft are mostly operated out of ground effect. Even if operated in ground effect for longer periods of time, modeling change in ground effect is much easier to predict than what is required to model the forces in relation to other vessels, shore structures, and bathymetry (which can vary dramatically in contour). Marine simulations also in-

OCR for page 59
Shiphandling Simulation: Application to Waterway Design volve the effects of aerodynamic forces on a vessel that vary with draft and deck loads. Data on these effects are often insufficient or not available. Consequently, marine simulations for training and channel design lack the quantitative data that forms the basis for developing and validating aircraft flight simulators. Characterizing ship-operating environment interactions remains a challenge in applying marine simulation technology to waterway design. Until a firm quantitative basis is developed, validation of marine simulations will continue to rely on subjective evaluations by expert marine pilots and other parties involved. INTERPRETING THE RESULTS As discussed in Chapter 3, the current state of practice in waterway design is to focus on ships and situations that will strain the safety of a waterway the most. Interpreting such results remains problematic. No formal basis was found to relate the results of simulation to a numerical measure of risk for the waterway. In other contexts, similar estimates are made for engineering projects by summing the products of probabilities of each possible accident in the project's lifetime and the cost of the consequence of the particular accident. This computation yields an estimate (expectation) of the risk exposure during the lifetime. Performing such a computation from the results of shiphandling simulation seems difficult, because only a few ships, pilots, waterway environments, and traffic situations are studied. The lack of objective validation of the simulation further compounds this analysis. As a result, the current practice is limited to a subjective judgment that the waterway design is or is not satisfactory based on the limited simulations. Many facilities indicated to the committee that the judgments formed on this basis predict a much greater accident rate than is seen in practice. Whether this anomaly is due to mathematical model inaccuracy, lack of personal consequences to the pilot (including absence of liability and discipline that could result from mistakes in real life), or some other cause is not known. Often, simulation studies will be carried out on several alternative designs for the selected ship. Clearly, the judgments resulting from these comparisons may have more value, because fewer variables are introduced. That is, the trends of these comparisons may be more correct than their absolute tracks and may provide sufficient information for selecting one alternative over another.

OCR for page 59
Shiphandling Simulation: Application to Waterway Design SUMMARY The accuracy of simulation for restricted shallow water almost certainly is lower than that obtained for deepwater maneuvering, because the latter situation has a much larger research base and does not require the use of so many heuristic models. Currently, no guidelines are available for assessing the accuracy required from the mathematical model to develop results useful for waterway design. Likewise, there is no numerical measure available for determining the accuracy of pilotage modeling. In particular, no guidelines are available for determining the level of simulation required for a particular situation or for an appropriate analysis of its results. The state of practice is to use subjective measures (for example, interviews with pilots) to validate the overall simulation and subjective interpretations of the simulation results in terms of overall risk corresponding to the waterway design. Although questions about accuracy, validation, and interpretation cannot be resolved objectively, simulation has proven extremely useful in some applications (see Chapter 7).