APPENDIX C Professional Licensing Infrastructure for U.S. Merchant Mariners
U.S. COAST GUARD MARINE LICENSING PROGRAM
Competency Determinations Through Essay and Multiple-Choice Examinations
From 1942 until the mid-1970s, the U.S. Coast Guard's (USCG) license test consisted of essay examinations. The licensing administration system consisted of hundreds of examination cards, with different questions for each major topic. Each marine inspection officer was provided the same set of cards, containing a separate question. Some cards were designed for individual license levels, others applied to all license levels. The licensing examiner would prepare an individualized test for each person by pulling cards from the file. Thus, each test was different. The consistency of the examinations varied with the licensing examiner's ability to organize and correlate question content, level, and difficulty and attitude toward the individual being tested Examinations also varied among licensing examiners. The number of questions given varied from 5 to about 20, depending on the topic.
The licensing examiner was empowered to use his or her professional expertise to evaluate the license candidate's response. Options available to the examiners were to:
- fail the individual,
- give the candidate additional questions to provide a broader base for evaluation, or
- prepare an alternate response if it was clear that the candidate did not understand the question.
This approach was necessary, in part, because many questions retained in the updated system were prepared before World War II. Licensing examiners were often individuals who had sailed in the merchant marine and who held deck or engineer merchant marine licenses and had the professional expertise needed to evaluate subjective responses to licensing exam questions.
In 1953, the USCG conducted a major study of the examination process. In addition to the unevenness and obsolescence previously noted, the agency found that the essay system had been compromised. Wohlfarth (1978) found that:
This compromise occurred with the compilation of "ponies" which accurately reproduced license examination questions. Allegedly some were exact duplicates of the card files maintained by the examiners.
A part of the compromise problem was the tendency of some schools to teach from such ponies, and the inclination of some license applicants to learn solely from ponies by rote memorization.
The U.S. Coast Guard was partially responsible for the existence of such ponies through its failure to conduct timely revisions to eliminate obsolete, ambiguous, and incorrect questions. Some license applicants felt compelled to study such ponies in order to learn "the answers required by the U.S. Coast Guard," thus enhancing their chances of passing the examination.
Although the 1953 study did not lead to direct changes, its findings indirectly resulted in the inclusion of multiple-choice questions for the first time in license examinations. Although there are several exceptions, the USCG ultimately changed its license program to rely almost exclusively on multiple-choice examinations to ascertain an individual's competence. One exception is a requirement that candidates for pilot licenses or endorsements draw a chart from memory of the route for which pilotage is sought. Candidates for master, mate, and pilot licenses must also complete USCG-approved training leading to issuance of a radar observer certificate. With these exceptions, there are no federal marine license requirements that incorporate a demonstration of competency.
The shift to the multiple-choice format began in 1969. Education Testing Services (ETS) in Princeton, New Jersey, was contracted to study all aspects of the marine licensing program. The study resulted in recommendations for interim and long-term improvements. In response to the study, the USCG:
- developed job descriptions for masters and mates, and
- eliminated essay questions from the license examinations, substituting multiple-choice questions.
The ETS was contracted to prepare the job descriptions and to develop multiple-choice examinations.
The job descriptions were developed in broad consultation with the marine community. Consultants were subcontracted to write the multiple-choice questions, which were protested at some of the maritime academies, union schools, and USCG marine inspection offices. Acceptable questions became part of printed
examinations and were provided in early 1973 to a newly established division at the USCG Institute (Wohlfarth, 1978). The Institute itself had been relocated to Oklahoma City, partly to take advantage of mainframe computer resources that could be used for scoring the license examinations.
There were significant problems in the revised licensing program during implementation, which were corrected as experience was gained with the new format. The multiple-choice format was generally accepted within the marine community. This initial success was attributed to the extensive coordination and consultations that were conducted (Wohlfarth, 1978). Dissatisfaction was limited principally to specific questions rather than to the format. Mariners and operating companies also benefited from the substantially reduced time needed to complete the examination. It is unclear how much this contributed to acceptance of the new format, although it was probably a significant factor.
In recent years, the USCG's multiple-choice examination approach has been criticized widely within the marine and towing industries and by marine pilots. One concern is that the examination does not ensure that an individual is professionally competent. Because all the examination questions were released by the USCG to the public in response to Freedom of Information Act requests, the integrity of the examination process has been compromised. This public release and the advent of computer-aided teaming techniques has enabled the creation of "electronic ponies," which are in wide use at many license-preparation schools. The rote memorization of the essay examination of the past has been replaced by the "programming" of license candidates to successfully pass license examinations.
Dissatisfaction with the multiple-choice examination format has caused some marine groups to call for a return to the essay format. Such an approach would be confronted by the same types of problems that the earlier essay system experienced. One alternative is to involve marine simulation in the process, either as required training, as an evaluation platform, or as a combination of these two options.
Nautical Credentials of USCG Marine Licensing Officials
The nautical expertise of marine licensing officials is an important consideration in applying marine simulation to professional regulation. The marine licensing authority must either maintain adequate resident expertise to fully administer the program or find alternative sources of expertise for the same purpose. It would be difficult for the USCG to revert to the earlier system of licensing, because the agency has not maintained a cadre of licensing examiners with the requisite qualifications. As discussed in the main body of this report, a move by the USCG to use simulators could also necessitate the maintenance of a staff with extensive marine qualifications to effectively oversee the validation and use of marine simulation.
The merchant marine and towing industry operational experience and expertise possessed by USCG personnel involved in marine licensing have
been criticized by the marine industry and the piloting profession as inadequate. Members of the industry have also criticized the agency's capability to establish professional standards for commercial mariners and to credibly evaluate mariner knowledge, skills, and abilities. This latter complaint was not a problem in 1972, when the multiple-choice examination format was established. Indeed, the individuals assigned to the USCG Institute to administer and update the examination questions all had considerable prior seagoing service, and all but one held merchant marine licenses (Wohlfarth, 1978).
Over the past two decades, the USCG's basis for establishing seagoing professional expertise among its officer and enlisted corps has progressively deteriorated (NRC, 1994). Generally, the seagoing expertise of most USCG members is derived from USCG cutters, most of which are designed for naval service. The operations and shipboard organization aboard cutters are substantially different from those aboard commercial ships and towing vessels. Thus, a substantial portion of nautical experience of some Coastguardsmen is of limited relevance to commercial operations.
Determining the actual status of USCG nautical expertise is not easily accomplished because the agency does not maintain automated databases that would permit the agency to determine the nautical qualifications of the merchant marine and sea service of individuals who conduct marine licensing at regional examination centers. An informal survey of 36 military personnel and civilian employees that administer the marine licensing program at USCG Headquarters in Washington, D.C., determined that 23 percent had prior sea-service experience in either the USCG or the merchant marine and 17 held merchant marine licenses. The number of individuals holding marine licenses at the agency's regional licensing examination centers is substantially less (NRC, 1994).
USCG Assessment of Licensing Alternatives
The USCG is acutely aware of the criticism of its marine licensing program and the nautical credentials of marine licensing personnel. The agency is interested in ways to incorporate competency determinations in the professional regulation of mariners and has been active in advancing the concept of using marine simulation for this purpose nationally and through its international representation of U.S. interests before the International Maritime Organization and its committees and working groups.
The USCG conducted an internal assessment of its marine licensing program as a partial response to public and congressional criticism of the professional qualification of mariners. The resulting report, Licensing 2000 and Beyond (Anderson et al., 1993), covers a broad array of licensing issues.
The report was prepared by a focus group consisting entirely of USCG uniformed armed service and civilian personnel and reflects their perspective. Representatives of user groups were not included in the focus group membership.
The focus group recognized its composition and urged that its findings and recommendations be referred to and validated by the USCG's new Merchant Personnel Advisory Committee (MERPAC) and USCG's Merchant Vessel Personnel Division at USCG Headquarters in Washington, D.C. The report also contains recommendations within the areas of interest of the USCG's Navigation Safety Advisory Committee and Towing Industry Advisory Committee.
The fundamental nature of changes recommended by the report are far reaching with respect to their potential effects on the merchant marine. The report provides a conceptual basis for guiding improvements in marine licensing, but its supporting facts and analyses are limited. The focus group acknowledged that its resources were insufficient to adequately develop some important inquiries. The resulting facts and analyses provide only a partial basis for identifying and assessing the potential effects.
With respect to USCG-approved courses, the report states that ''proper utilization of the approved course concept would allow reductions in actual sea service experience, enhancement of professional proficiency and in many instances elimination of the U.S. Coast Guard Examination Process." The assertion that sea service can be reduced and some USCG examination requirements eliminated are not substantiated by facts and analyses. In particular, no substantive basis is provided to justify the levels of substitution of sea time.
The requirement for radar observer certification (which uses radar simulation) is cited as a working example of the use of simulation, but no evidence is presented, nor does there appear to be any evidence that the training is actually improving the licensing process, or—more important—improving the quality of professional development for effective use of radar. Radar observer training, as it is presently used, has been criticized by many within the marine community and piloting profession because the requirement has not kept pace with the change in radar technology, especially with respect to automated radar plotting aids. Furthermore, the effectiveness of radar observer certification has not been evaluated empirically with respect to knowledge and skill requirements, the degree to which these skills are reinforced or refreshed during actual operations, and the frequency and nature of part-task training that is necessary to restore degraded knowledge and skills.
The report states that improvements are needed in the qualification of mariners and instructors and in the preparation of USCG personnel for administrative oversight of agency-approved courses. The need for instructor qualifications is a view that is widely shared within the marine community and the piloting profession. The report also strongly encourages that USCG personnel involved in marine licensing receive more complete and rigorous training in the licensing process. The report is silent, however, with regard to the nautical expertise and recency that would be needed for effective performance by USCG personnel who serve as licensing examiners, who set marine licensing standards, or who
validate USCG-approved courses. It implies that expertise can be developed through assignments of personnel in the vessel inspection program.
The report states that use of marine simulation for testing purposes is controversial and that "the wide-spread use of simulation as a test of more definitive subjective knowledge has yet to be fully demonstrated" (Anderson et al., 1993). It characterizes the use of marine simulation by operating companies as successful, based on use of the technology rather than on the practical results of the training in actual operations. Although the companies and marine pilot associations that use simulation believe that there is value to their operations, and thus continue to use simulation, the value added has not been determined empirically.
The report recommends that the potential for applying simulation in training and licensing be assessed and recommends the adoption of simulation for demonstrations of competency. In particular, the report recommends development of "performance standards for a high current/tight quarters maneuvering simulator training program" and a requirement to complete such a "simulator training and testing program as a prerequisite for issuance of a Western Rivers OUTV." The state of practice in computer-based simulation of inland towboat operations, especially for large barge flotillas, is not identified. The training resources that are currently available to support the training of inland towing vessel operators is the same infrastructure that is used for masters, mates, and pilots. For these reasons, it is not clear that the focus group's recommendations could be implemented in the near term or mid-term.
The report defined competency as:
… the total set of skills, knowledge and judgments necessary for the proper performance of one's duties in a specific position on a specific vessel. Competency, thus defined, can be broken down into two subsets. The first is the base level of skills, knowledge and judgments necessary to perform the duties of generic positions, e.g., Chief Mate, on a wide range of vessels of similar size and type.…The second competency subset consists of the skills, knowledge and judgments peculiar to a specific vessel or trade (Anderson et al., 1993).
The focus group in its report further stated that:
Ensuring that mariners possess the base level, also known as minimum competence, is a proper role of government. The consensus of the Focus Group is that public safety and the environment will be better protected by improving the methods by which mariners obtain this minimum competence. The second competency subset consists of the skills, knowledge and judgments peculiar to a specific vessel or trade. It is the responsibility of the owners, operators and individual mariners to ensure that these are obtained in a timely manner (Anderson et.al., 1993).
The report found that the level of improvement varied among the elements of the marine licensing program. The focus group recommended that the USCG move
forward in assessing computer-based training and testing systems and in using these methods to verify professional competency.
The USCG has not publicly stated its official position on the report's findings, having recently referred it to MERPAC for validation. Nevertheless, the USCG has proceeded on its own initiative to advance the application of marine simulation in professional development and marine licensing. Many courses had previously been approved for sea-time credit (USCG, 1994). In November 1994, the agency published its course-approval policy guidance, which strongly encourages simulation training and provides general criteria for the granting of sea-time equivalency. (Subpart C of 46 CFR 10 contains the general criteria used by the USCG to approve courses of instruction.) Also in November 1994, the agency established a precedent by accepting a combined simulation-based training and evaluation course as a nonmandatory substitute for the entire master (unlimited oceans) license examination (see Appendix F).
Anderson, D.B., T.L. Rice, R.G. Ross, J.D. Pendergraft, C.D. Kakuska, D.F. Meyers, S.J. Szczepaniak, and P.A. Stutman. 1993. Licensing 2000 and Beyond. Washington, D.C.: Office of Marine Safety, Security and Environmental Protection, U.S. Coast Guard.
NRC (National Research Council). 1994. Minding the Helm: Marine Navigation and Piloting. Committee on Advances in Navigation and Piloting, Marine Board. Washington, D.C.: National Academy Press.
USCG (U.S. Coast Guard). 1994. U.S. Coast Guard approved course listing as of October 1, 1994. Unpublished. Merchant Personnel Division, Office of Marine Safety, Security, and Environmental Protection, U.S. Coast Guard, Washington, D.C.
Wohlfarth, W.G. 1978. Licensing examination modernization. Pp. 4–9 in Proceedings of the Marine Safety Council, January. Washington, D.C.: U.S. Coast Guard.