Simulator-Based Training and Sea-Time Equivalency
INTERNATIONAL SEA-TIME REQUIREMENTS
Onboard experience has traditionally been required as a practical necessity for learning the skills of a merchant marine deck officer. The international Standards for Training, Certification, and Watchkeeping (STCW) guidelines standardized the sea-service requirements for deck officer licenses. It requires a minimum period of service in the deck department with prerequisite service in bridge watchkeeping duties before a prospective deck officer can be certified as officer in charge of a navigational watch.1 There are additional onboard experience requirements for deck officers to upgrade to chief mate or master or renew a marine license or competency certificate. The International Maritime Organization's (IMO) role and the STCW guidelines are discussed in Appendix B.
The minimum period of onboard time, commonly referred to as sea time, is considered essential to ensure that mariners are exposed to actual operating conditions, over an adequate period of time, to prepare them for positions of increasing shipboard responsibility. Currently, the STCW guidelines require a minimum of three years of sea service for an original mate's license. Two of the required three years can be substituted by "special training" that marine licensing authorities are satisfied "is at least equivalent in value to the period of seagoing
service it replaces..." (IMO, 1993). The STCW guidelines do not define the nature, form, or content of special training, an issue that is under review by the IMO.
The amount of time and structure of at-sea experience required to qualify for an original license varies widely, from the structured sea project required of U.S. Merchant Marine Academy cadets during their year of service aboard merchant vessels, to the three annual roughly 60-day training ship cruises of the state maritime academies, to the normally unstructured periods of at least 3 years at sea by unlicensed personnel qualifying for an initial license examination. Other than a requirement for documenting the duration of service, no national or international standards exist on the type of experience to be garnered during this period.
It is important to recognize that no baseline has been established as a frame of reference for determining adequacy of the current sea-time requirements for building the necessary mariner knowledge and skills. The minimum sea-time requirements have not been validated by either scientific research or documented by empirical evidence. Although analyzing the adequacy of the sea-time requirements is beyond the scope of this study, it is important to note that all discussions of sea-time equivalency are based on the assumption that existing requirements are adequate.
DEFINITION OF SEA-TIME EQUIVALENCY
The U.S. Coast Guard (USCG) and other certification authorities have begun to modify license requirements for onboard experience and have granted mariners sea-time equivalency—reductions in onboard service requirements—for completion of onshore training. The term sea-time equivalency, as used in marine licensing and this report, refers to a formal judgment concerning the relative value of simulator and other structured training as compared with actual hands-on experience gained aboard ship.
The concept behind sea-time equivalency is based on the perceived value of structured training, especially simulator-based training. During the past few years, the majority of mariners who have advanced to the level of master have done so through a combination of structured academic and onboard on-the-job training. The use of sea time equivalency represents a conclusion by the USCG that structured, simulator-based training is equivalent to certain on-the-job, shipboard experience.
The granting of credit for simulator training as a substitute for required sea time is referred to as remission of sea time. Recently, the USCG stated that it will encourage training and simulator use by expanding the practice of sea-time remission (USCG,1993).
The STCW guidelines do not specifically define, though they do imply, that seagoing service is service aboard an actual ship. Remission policies are permitted,
though not encouraged, under the guideline's present wording. As a matter of policy, the USCG has interpreted the existing language of the STCW guidelines as a sufficient basis for substituting time spent in ship-bridge simulator-based courses for sea-time remission ratios up to 6 to 1.
The application of a specific ratio (such as 6 to 1) implies that a judgment has been made that each day of the simulator-based training course(s) specified is equal to six days of on-the-job experience aboard ship. The USCG has extended the remission policy to include, within certain limits, portions of the minimum one-year sea time required by the STCW guidelines.2
During this study, the committee could find no strong support among mariners or employers for the desirability of remission of sea time. The committee also found that most groups were skeptical and ambivalent about the current policy. Pilot groups explicitly criticized the policy and recommended that it not be applied to pilots under any circumstances. This lack of support implies that, before additional measures are taken to increase sea-time remission, answers should be sought to the questions of whether, or to what degree, and under what terms and conditions simulation should be substituted for a sea time in the professional development and qualification of mariners. The following discussions address these questions by considering unresolved issues, application of systematic approach in the development of criteria for decisions to grant remission of sea time, and potentially useful applications of remission of a sea time.
SEA-TIME EQUIVALENCY AND MARINER COMPETENCY
Chapter 2 through 4 discuss current simulator-based training and its applications to mariner skills improvement and professional development. There is, however, a basic, and quite important, distinction between the use of simulators as training tools and the use of simulator-based training for granting remission of sea time. Thus far, simulator-based has been discussed in this report primarily in the context of a supplement to existing training practices. In the application of simulator-based training to the remission of required sea time, however, simulator-based training becomes a substitute for on-the-job skills and knowledge acquisition.
The primary goal of all mariner training is to improve mariner competency (including safety and efficiency of operations). That competency comes from both structured and unstructured training experiences. Current requirements for onboard service are based on the concept that certain professional skills development should take place through exposure to actual operating conditions and the resulting learning by experience and colleague-assisted training discussed in Chapter 1. When a decision is made to substitute simulator-based training for current practice, it is important to ensure that there will be no degradation of mariner competency and that safety will not be compromised. The ultimate test of every decision for remission of sea time, therefore, should be the test of the types of experience and skills being replaced by the simulator-based training and the possible effects of this substitution on mariner competency and the safety of the vessel, crew, and cargo.
BASIS FOR SEA-TIME EQUIVALENCY
Scientific Basis for Equivalency
The use of the word equivalency implies that simulator-based training replicates and equals on-the-job experience in terms of the development and reinforcement of the knowledge, skills, and abilities needed to perform effectively. A number of experiments conducted on the effectiveness of simulator-based training over the past 20 years concluded that the training is effective in its own right (Williams et al., 1980; Kayten et al., 1982; Multer et al., 1983; Miller et al., 1985; Reeve, 1987; Drown and Lowry, 1993). The committee could not, however, find any studies that directly addressed whether there is any equivalency between simulator training in selected skills and the skills learned ad hoc on board a ship.
A recently published study by MarineSafety International Rotterdam and TNO Human Factors Research Institute (DGSM, 1994) did attempt to determine what amount of simulator training is required to compensate for the first 30 days of cadets' sea time . The study compared two groups of students—one that had received only three weeks of simulator training and one that had completed their full sea time on board a ship. The study concluded that "the only ratio of simulator time to sea time that is based on observable data points, and not on extrapolations, is the ratio based on the 50 percent performance level. "That ratio was 1 to 7.25 with a 95 percent confidence interval of 1 to 3.3 to 1 to 11.15. The application of this yielded a conclusion that, for cadets, the 30 days of sea time could be replaced by 40 hours of simulator time (DGSM, 1994).
There is also some possibly supporting research in the commercial air carrier sector. Commercial air carrier research suggests that training in appropriate tasks can be accomplished more quickly, with effectiveness equivalent to on-the-job training, using well-founded, limited-task and full-mission simulator
programs (Hays and Singer, 1989). It remains uncertain, however, that either the research in the commercial air carrier field, or the conclusions formed as a result of such research, are directly transferable to the maritime industry or other industries (Hays and Singer, 1989).
Anecdotal Basis for Equivalency
Acceptance of the concept that simulator-based training is equal to or better than certain on-the-job experience appears to be based almost exclusively on anecdotal information. In reviewing that information, it appears that all parties involved in simulator-based training share the commonly held conviction that it does improve job performance.
Professional mariners who have operated simulators as experienced trainers have expressed a strong conviction that "they know the method works." Students (the "end users") have provided feedback through course critiques, indicating that they believe that "simulator training is effective." Feedback from past students based on field experiences is however, limited. As stated earlier, there is no information that effectively relates skills developed through simulator training to skills acquired on board ship.
ISSUES AFFECTING SEA-TIME EQUIVALENCY DECISIONS
The USCG has approached the issue of sea-time equivalency from a programmatic, not a technical, basis. USCG decisions to grant remission of sea time in ratios, such as 6 to 1, have been based on achievement of licensing objectives, a stated interest in encouraging training, and use of simulators or other program objectives. No technical basis has been developed for these ratios.
The USCG is using remission of sea time as an element of its licensing program. Its remission policies may have been ad hoc, based primarily on a perceived value of simulator-based training. It has authorized sea-time remission to assist the maritime academies in meeting the STCW sea-service guidelines and to encourage training.
Remission of sea time is, in fact, a complex problem that should be addressed much more systematically. Many of the issues discussed earlier in this report apply to considerations of sea-time remission. For example, issues raised in use of simulators for training, such as the need for a systematic approach to simulator training course development and the need for instructor qualification, should be considered in developing a rational approach to evaluating simulator-based training programs and their applicability to sea-time equivalency. Many of the issues related to performance evaluation with simulators and the application of simulation to licensing assessment should also be considered in decisions concerning sea-time equivalency.
Before any additional decisions are made in remission of sea time, these issues affecting possible implementation need to be addressed.
Mariner Tasks and On Board Equivalency
Earlier discussions of effective training and performance evaluation with simulators included an expression of the committee's concern that results of studies of the tasks and subtasks of mariners (in this case, deck officers and pilots) have not been used effectively and often do not provide information at the level of detail to adequately characterize the knowledge, skills, and abilities a mariner must develop and the behaviors and detailed steps involved in carrying out specific, duties and responsibilities. To be able to define training objectives, then to measure through performance evaluation whether those objectives have been met, tasks and subtasks must be defined and matched to the ability and limitations of the simulators to train them.
It is important in the understanding of the tasks of mariners that all tasks are understood. Simulator-based training is not suitable for training every job skill needed on board ship. The random, sometimes high-stress situations that can develop during at-sea time are important for developing confidence, command presence, and interpersonal skills and for learning about the ship's business.
Sea time aboard ship provides a broad range of experiences that would be enhanced and complemented, but not replaced, by specific simulator training. As in other work environments, many tasks are often learned as part of routine duties without structured training. Deck officers and pilots learn these skills aboard ship at no additional expense to the mariner or operator.
In considering sea-time equivalency and the decision to identify specific simulator-based training as being equivalent to some period of on-the-job experience, this detailed understanding of the character of onboard tasks and subtasks being simulated is especially important. Without this understanding, it may not be possible to ensure that the substitution being permitted will result in achieving the level of mariner competency currently provided through onboard experience.
Training Course Standards
A related issue to sea-time equivalency is the quality of the specific simulations-based training courses that will substitute for sea time. If simulator-based training is accepted as equivalent to some amount of on-the-job training, it is necessary to be able to rely on that training as having accomplished specific training objectives as well as specific licensing objectives (see Chapter 5).
In the remission of sea time for cadets, the Maritime Academy Simulator Committee (MASC) has taken an active role developing quantitative data on which to base its recommendations that credit for 60 days of sea time be granted for successful completion of the 40-hour cadet watchstanding course. The proposed
course structure would spread the 40 hours over 10 weeks to improve the effectiveness of the training. MASC is currently conducting a survey to compare shipboard and full-mission ship-bridge procedures to validate this proposed equivalency of 12 to 1.
Because the instructor must ensure that all course objectives are met, course instructor quality is also an important consideration. Certification or some other mechanism for ensuring instructor quality should be an element in any plan for determining sea-time equivalency. A part of that certification should be a requirement for instructor training. The three sample instructor training courses discussed in Chapter 3 are examples of efforts in this area.
Sea-Time Equivalency and License Levels
Because mariner knowledge and skill requirements vary by position and tasks, remission of sea time should be examined in the context of range of tasks and types of licenses. Granting remission of qualifying sea time for initial and upgraded licenses, for example, is entirely different from granting remission for recency of sea time for license renewal. An initial or upgraded license entitles the license holder to work in a new of different and, in most cases, more difficult position. A renewed license only permits a licensed mariner to continue working in the same position.
Consideration should be given, therefore, to remission of sea time in several different dimensions, including:
- original licenses issued to entry-level officers,
- original licenses issued to senior-level officers,
- upgraded licenses issued to senior-level officers,
- renewal of a currently held license,
- certificates of competency for specialized service, and
- credit for onboard trips to qualify for pilot licenses and extension of route.
Transfer and Skills Reinforcement
Before any major decisions are made concerning substitution of simulator-based training for sea-time requirements, the issues of transfer, retention, and reinforcement of learning from simulator-based training discussed in Chapter 3 need to be addressed. Transfer does not have to be complete for simulator-based training to be useful in improving skills. There should, however, be some mechanism to assess the effectiveness of the transfer and retention rate of skills learned. If transfer is incomplete, it may be necessary to ensure that the training
is followed by some type of onboard reinforcement, such as an apprenticeship, to ensure continuing high levels of mariner competence.
Validation of Simulations and Simulators
In substituting simulator-based training for sea-time requirements, consistency, in results can be crucial. Simulations are often modified in the regular course of doing business at many simulator facilities. Chapters 5 and 7 discuss the need for validation of simulators and simulations used in training and licensing assessment.
To ensure that training and licensing objectives are met, it is important to replicate simulator-based training results across platforms for like simulators. The training courses themselves, as well as the simulators and simulation scenarios used to substitute for sea time, need to be measured against established criteria to ensure that minimum standards are met. It is important that appropriate agencies recognize the need for standards and develop a mechanism to ensure that appropriate standards are set and enforced (see Chapter 7).
Cost and Availability of Simulators
Chapter 5 discusses the issues of availability of simulator resources and who would pay for simulator-based training that may be required in mariner licensing . Any mandated requirements for simulator-based training will result in a cost to the employer, union, candidate, or some other source. Part of planning for the expanded use of simulation in sea-time remission should be consideration of the potential impact of the cost of any mandated training.
The impact on the simulator facility "marketplace" should also be considered. Any proposed changes should be phased in to provide time for the marketplace to provide the resources needed to meet the increased demand that might result from substituting simulator-based training for sea-service requirements.
A SYSTEMATIC APPROACH TO DETERMINING SEA-TIME EQUIVALENCY
The decision to grant remission of sea time for specific simulator-based training is a decision to modify current licensing requirements. The committee could find no substantive technical basis for current USCG decisions that have been made in granting remission of sea time. It may be, however, that some decisions can be made despite the lack of a strong technical basis. It may be possible for the USCG to develop elements within its licensing program that grant remission of sea time in specific, well-defined areas, provided that the USCG program is based on a structured, systematic approach that includes a mechanism for ensuring that mariner competency and safety are not compromised.
The USCG cannot afford to continue to approach those decisions in an unstructured, ad hoc manner. Decisions to grant remission of sea time should be based on research that demonstrates the benefits and only after considering the issues outlined above. In determining whether to remit sea time, the agency should ensure that:
- Sea-time remitted clearly would not have provided as much benefit as the simulator-based training.
- There are some real, demonstratable benefits to the industry, public, or deck officer for the remission.
- It can be demonstrated that there is no degradation of other skills as a result of reducing sea time.
- Equivalency serves some purpose besides inducement to using simulator-based training.
It is important that the USCG develop a systematic approach based on research and an in-depth understanding of the type of experience replaced by the simulator-based training. To define the need for a policy of remitting sea time based on equivalency, the USCG should consider taking the following steps:
- identifying the license level and type (e.g., original licenses issued to entry-level officers) for which remission of sea time is being considered;
- outlining licensing objectives and competency requirements in terms of task and subtask descriptions, based on the license level and type defined;
- characterizing the tasks and subtasks most effectively developed through simulator-based training;
- developing criteria for simulator-based training based on licensing objectives;
- evaluating proposed simulator-based training programs against criteria;
- evaluating the full range of onboard experiences being replaced by simulator-based training;
- determining the relative importance of simulator-based skills in the mariner's overall work aboard ship;
- evaluating the degree of equivalency of each specific course and the specific transferred skills and determine how much time, if any, should be credited for the training;
- determining the skills learned aboard ship that are not covered by the simulator and the extent to which they deteriorate, if at all, as a result of sea-time remission policy; and
- establishing a mechanism for ongoing program monitoring through regular data collection and analysis to ensure maintenance of mariner competency and vessel, crew, and cargo safety.
Insofar as the value of the training course can be determined, it seems appropriate to vary the amount of sea time credited for a given course in accordance
with its relative value in satisfying STCW guidelines and federal licensing requirements, as determined by qualified experts.
POSSIBLE EQUIVALENCY APPLICATIONS
As simulator-based training becomes more sophisticated and the objectives of the individual courses are more clearly delineated, the arguments become more persuasive that there is a basis for substituting simulator training for some presently required sea time. In fact, a number of applications of simulators could provide superior professional development. These possible applications need to be considered in any analysis of sea-time remission for simulator-based training.
Training for which Simulations is Well Suited
As discussed in Chapter 2, there are a number of attributes of simulator-based training that make it particularly effective in training certain skills. These attribute include:
- the ability to simulate adverse operating conditions safely,
- the ability to play back and repeat scenarios, and
- the ability to permit and use mistakes and accidents for teaching.
Because of competing demands of ships' scheduling and business, subjects such as emergency procedures, maneuvering in traffic, shiphandling, and bridge team and bridge resource management are sometimes given cursory treatment, at best, aboard many commercial ships. Simulator-based training could be superior to on-the-job training in these areas.
Encourage More Appropriate Training
Developing knowledge, skills, and abilities for prospective third mates is an area where simulator-based training may be considered an effective substitute for on-the-job training. Simulator training, for example, is substituted for a portion of the sea time required to prepare U.S. Merchant Marine Academy cadets for entry-level third mates' licenses. This training is considered preferable to traditional sea experience because the type of experience being replaced—service as unlicensed seaman in the deck department—is more effectively trained on the simulator. The ship-bridge simulator-based training develops watchkeeping skills and relates directly to the duties and responsibilities of a third mate's license (Hammell et al., 1980; Kayten et al., 1982). This substitution of sea-time requirement is applicable because the able-bodied seaman—especially aboard modern, more technically sophisticated ships—performs only a minimum of traditional seafaring tasks.
Simulator-Based Training to Meet Refresher Requirements
The STCW guidelines pertaining to frequency of training or refresher training are minimal, although there is an international trend toward more frequent training as a requirement for license upgrades and renewals. Current STCW guidelines are reflected in USCG regulations. The ship-bridge simulator can potentially be used to refresh knowledge, skills, and functional responsibilities in many areas for both license maintenance and recency purposes.
Simulator testing is much more performance-based than traditional, written examinations, particularly multiple-choice examinations used for license testing for the past two decades. Scenarios could be developed to demonstrate all desired skills and abilities—an advantage over shipboard checks (in those rare circumstances where they may be made) which can only demonstrate ability to deal with random situations. Furthermore, simulators could be used to train and check skills in emergency and hazardous situations that would never be deliberately created aboard ship.
Prior to recent developments in simulation, refresher training, which is now practical for many professional skills, could only be practiced aboard ship. Subjects that might be considered for refresher training using computer-based and manned-model simulation include:
- proficiency checks on nonshiphandling aspects of piloting for pilots;
- proficiency checks on shiphandling skills for masters and deck officers;
- new ratings (e.g., masters accepting their first-time appointments to very large crude carriers, pilots upgrading to bigger vessels or new ship types, masters appointed to vessels with nonstandard handling characteristics);
- bridge team management (masters and pilots, in particular, could be targeted for refresher training in human factors and bridge teamwork);
- bridge watchkeeping (for cadets and all watchkeeping personnel, to include rules of the road, navigation, voyage planning, and other skills inherent to watchkeeping);
- rules of the road (refresher training for masters, deck officers, pilots); and
- shiphandling updating for pilots (manned models are preferred method).
Equivalency Standards for License Renewal
By using simulator-based examinations to test the renewing of mariner's skills in realistic situations, it may be possible to eliminate some or all recency requirements for officers renewing a marine license. Within the current system, it is assumed that a mariner's skills are current if he or she has a minimum period of service aboard ship, or in affiliated industries, in the years immediately preceding the five-year anniversary date of the mariner's license. As the U.S.-flag fleet shrinks, it is becoming increasingly difficult for ships' officers to maintain a shipboard
career and obtain sufficient service to meet the recency requirements. Many mariners who want to maintain their licenses may be forced to let them expire.
In these instances, it might be reasonable to expect that a mariner who can demonstrate proficiency in realistic underway scenarios on a ship-bridge simulator has retained the basic proficiency level needed to renew an existing license. Recency may be academic, in this case, if continued proficiency is demonstrated in the simulator and a strong case can be made that a reasonably thorough demonstration of proficiency can be substituted for recent service.
In fact, substituting simulator training for recent sea time for license renewal may actually raise the standard of professional competence, while simultaneously reducing sea-time requirements, since credit is presently granted for service in rather loosely affiliated industries that often do not include any actual underway service or bridge watches aboard ship.
Providing Structured Training
At sea, on a commercial ship, a cadet or mariner gains experience randomly as individual situations develop—situations that are seldom selected to provide specific training. Although this real-world experience can be crucial in many training situations, the opposite may also be true for other training objectives.
In commercial at-sea training there is no structured instruction and often no structured monitoring or performance critique. On a simulator, the instructor and evaluator can teach and demonstrate preferred techniques and evaluate performance under a particular set of circumstances. Poor work habits can be corrected immediately.
On training vessels, operated by maritime academies, there is structured instruction and monitoring of cadet performance. The cadet is immersed in the total environment similar to a commercial ship. A training cycle on a training vessel would typically encompass a day of maintenance, a day of instruction, and a day of watch. Skills taught during class instruction on a training vessel and ashore at the academy are uniform. Modeling of the ship-bridge simulator after the training vessel and documents (voyage plans, check-off sheets, logs, standing orders, etc.) further enhances the training effectiveness.
Other Possible Applications of Sea-Time Remission
A modest level of sea-time remission might be granted for certain professional development courses taught using full-mission ship-bridge simulators. Given the lack of data on the effectiveness of the transfer of simulator-based training skills discussed in Chapter 3, any sea time granted in this instance would probably be based more on an intuitive relationship between onboard experience and simulator-based training than on quantifiable data.
Summary of Findings
Strong maritime tradition has supported the position that onboard experience can be an effective way for mariners to develop the skills and knowledge necessary to advance to officer positions aboard merchant vessels. Recently, however, many mariners who achieve the position of master have advanced through a combination of structured and on-the-job training.
The international STCW guidelines have recommended minimum sea-service requirements for deck officers. The USCG, as the national license authority, has granted a remission of sea time in several areas—including license upgrades—for successful completion of specified simulator-based training. In granting these remissions, the agency has approached the issue of sea-time equivalency from a programmatic, not a technical basis. USCG decisions to grant remission of sea time in ratios, such as 6 to 1, have been based on decisions related to the achievement of licensing or other program objectives.
Recent work in the Netherlands concluded that in cadet training a ratio of approximately 1 to 7.25 would result in the substitution of 40 hours of simulator-based training for the first 30 days of sea time, at a 50 percent proficiency level. Beyond this study, the committee could find no other technical basis for these ratios.
It may be that for some specific applications, however, none is necessary. It may be possible for the USCG to develop an effective program for remission of sea-time by applying a systematic approach. Any decision to use simulator-based training for remission of sea-time requirements, however, needs to be treated similarly to use of simulation for licensing assessment. The same level of systematic analysis and assessment needs to be applied to both decisions.
A primary consideration of a sea-time equivalency program should be to ensure that mariner competency and marine safety are not compromised or degraded by substituting simulator-based training for sea service. The USCG should ensure that skills learned aboard ship that cannot be trained in the simulator do not deteriorate as a result of the remission of a sea-time system. The agency should demonstrate the benefits of sea-time remission through research and develop a system for granting such remission that includes consideration of all issues discussed and all steps outlined in this chapter, including mechanisms for:
- characterizating and comparing onboard tasks and subtasks to the objectives of the simulator-based training;
- developing training course criteria and performance standards,
- validating training simulators and simulation scenarios; and
- monitoring and improving programs continuously by ongoing collection and analysis of performance data and statistics.
There are a number of areas where simulator-based training may be more effective at achieving licensing objectives than on-the-job experience. These include:
- areas where onboard training may be limited by concerns over ship safety, such as maneuvering in traffic and shiphandling;
- cadet training;
- refresher requirements; and
- recency requirements in license renewal.
Until sufficient research is conducted to demonstrate the benefits of sea-time remission, it should be applied only to original third mates' licenses (for which simulator-based training has been demonstrated to be superior to the onboard experience) and to the renewal of licenses without grade increases (for which adding structured training to onboard experience would be a significant benefit).
Implementing a sea-time equivalency program will require a well-defined, ongoing monitoring program that will collect and analyze data to ensure that program objectives are met and mariner competency levels are maintained. Before there is any full-scale use of simulator-based training to meet certain refresher requirements, research should be conducted to determine the objectives of the current requirements and the type and length of training that could be substituted.
DGSM (Director-General of Shipping and Maritime Affairs). 1994. Simulator Time and Its Sea-Time Equivalency. DGSM Project No. 634(70/3/017). MarineSafety International Rotterdam and TNO Human Factors Research Institute. Rijswijk, Netherlands: DGSM.
Drown, D.F., and I.J. Lowry. 1993. A categorization and evaluation system for computer-based ship operation training. Pp. 103–113 in MARSIM '93. International Conference on Maritime Simulation and Ship Maneuverability, St. Johns, Newfoundland, Canada, September 26–October 2.
Hammell, T.J., K.E. Williams, J.A. Grasso, and W. Evans. 1980. Simulators for Mariner Training and Licensing. Phase 1: The Role of Simulators in the Mariner Training and Licensing Process (2 volumes). Report Nos. CAORF 50-7810-01 and USCG-D-12-80. Kings Point, New York: Computer Aided Operations Research Facility, National Maritime Research Center.
Hays, R.T., and M.J. Singer. 1989. Simulation Fidelity in Training System Design: Bridging the Gap Between Reality and Training. New York: Springer-Verlag.
IMO (International Maritime Organization). 1993. STCW 1978: International Convention on Standards of Training, Certification, and Watchkeeping, 1978. London, England: IMO.
Kayten, P., W.M. Korsoh, W.C. Miller, E.J. Kaufman, K.E. Williams, and T.C. King, Jr. 1982. Assessment of Simulator-Based Training for the Enhancement of Cadet Watch Officer. Kings Point, New York: National Maritime Research Center.
Miller, W.C., C. Saxe, and A.D. D'Amico. 1985. A Preliminary Evaluation of Transfer of Simulator Training to the Real World. Report No. CAORF 50-8126-02. Kings Point, New York: Computer Aided Operations Research Facility, National Maritime Research Center.
Multer, J., A.D. D'Amico, K. Williams, and C. Saxe. 1983. Efficiency of Simulation in the Acquisition of Shiphandling Knowledge as a Function of Previous Experience. Report No. CAORF 52-8102-02. Kings Point, New York: Computer Aided Operations Research Facility, National Maritime Research Center.
Reeve, P.E. 1987. Mariner skills transfer by simulation. Pp. 35–38 in Proceedings of Problems of the Developing Maritime World, Sharjah, United Arab Emirates, January 26–28.
USCG (U.S. Coast Guard). 1993. Personal communication from USCG OPA90 staff to Committee on Ship-Bridge Simulation Training, National Research Council, July 23.
Williams, K., J. Goldberg. and A. D'Amico. 1980. Transfer of Training from Low to High Fidelity Simulators. Report No. CAORF 50-7919-02. Kings Point, New York: Computer Aided Operations Research Facility, National Maritime Research Center .