Strengthening U.S. Coast Guard Oversight and Support of Recognized Organizations: The Case of the Alternate Compliance Program (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

67 The review of the findings and recommendations from the El Faro investi- gations and the actions agreed to by the U.S. Coast Guard (Coast Guard) in Chapter 3 revealed both strengths and shortcomings in their implementa- tion. Significantly, the Coast Guard has established a new Flag State Control Division (CVC-4) responsible for third-party oversight and has added staff trained in and dedicated to third-party oversight (third-party organi zation [TPO] coordinators). CVC-4 has helped develop and implement a series of new and updated policies, guidance, and procedures for marine inspec- tors and marine safety personnel that are indicative of the Coast Guard’s commitment to strengthening its support for and oversight of the Alternate Compliance Program (ACP) and the recognized organizations (ROs) that participate. These important developments address the Coast Guard’s own acknowledgment that it had not developed and sustained the needed policy framework for oversight. They also respond to a number of recommenda- tions by El Faro investigators and requirements by Congress to create an administrative unit dedicated to third-party oversight and to revise and establish new oversight policies and procedures. The Coast Guard recognized the importance of taking early actions to establish a stronger organizational and policy foundation for an effec- tive flag-state control regime that has become increasingly dependent on ROs and other TPOs to verify that vessels meet all applicable safety and environmental protection requirements. Having succeeded in building this foundation, one of the Coast Guard’s remaining challenges is to support the work of the new office and implementation of the new policies through more tangible and sustained progress in furthering a number of other 4 Data, Metrics, and Risk-Informed Tools for Compliance Verification and Oversight of Recognized Organizations

68 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs actions agreed to by the Commandant and recommended by the El Faro investigators. They include initiatives to enhance the knowledge and profi- ciencies of Coast Guard inspectors and other prevention personnel and to introduce new data systems, metrics, and tools to support their work and that of the ROs. In agreeing to actions in response to the El Faro investigations, the Commandant stressed that vessel compliance verification and RO over- sight activities should be data driven and risk based. However, the review in Chapter 3 revealed few concrete accomplishments in this regard. Im- provements to data metrics and risk-based tools hinge on longer-term and broader-based Coast Guard initiatives such as the eventual replacement of the Marine Information for Safety and Law Enforcement (MISLE) data system. In some other cases, such as for more cross-organizational sharing and integration of vessel compliance records, Chapter 3’s review suggests that progress has been marginal at best. This chapter takes a closer look at the Coast Guard’s efforts to be- come more data driven and risk informed to ensure effective oversight of the ACP and ROs. The first section considers the kinds of data and data systems needed by marine inspectors and RO surveyors to perform their duties and by the CVC-4 to support, monitor, and oversee performance. In the case of inspectors and surveyors, they need access to complete, current, and accurate records of the condition and compliance history of vessels. CVC-4 has the same data needs as well as data systems and tools that can be used to monitor the compliance of vessels in the ACP, design and target compliance verification activities more effectively, and monitor the perfor- mance of inspectors and ROs to better support them. CVC-4 has developed a set of 10 statistics that it refers to as key performance indicators (KPIs) of vessel compliance and RO performance. These statistics are discussed, their strengths and weaknesses are noted, and opportunities for replacing or augmenting them are considered. CVC-4’s use of risk-based methods for prioritizing and targeting compliance verification activities and RO oversight is discussed, but progress in this area has been limited by many of the same shortcomings in data and lack of data reporting and analytic tools that have hindered the development of KPIs. The Coast Guard’s plans to use KPIs and risk-based methods to guide its oversight of the ACP and ROs remain largely aspirational, with limited prospects for progress in the absence of better data and reporting and analytic tools. That progress, in the committee’s view, must start with the development of more complete, accurate, and accessible records of vessels in the ACP fleet, as needed foremost by field inspectors and ROs to perform their duties but also by CVC-4 for more effective program and RO oversight. The ACP fleet is modest in size, and therefore does not present formidable data collection, integration, and validation challenges.

DATA, METRICS, AND RISK-INFORMED TOOLS 69 In recognizing that a full replacement for MISLE may take several years, the committee recommends steps that can be taken in the interim to meet nearer-term ACP oversight needs while also informing plans for MISLE’s replacement. DATA AND DATA SYSTEMS In response to Marine Board of Investigation recommendations (Safety Recommendation 21 and Administrative Recommendation 4), the Com- mandant agreed to enhance the Coast Guard’s data systems, including upgrading MISLE to include fields for documenting vessel deficiencies iden- tified during ACP oversight examinations by marine inspectors that are otherwise referred to ROs for resolution. The Commandant also agreed to ensure that the data system enhancements could be used to support trend analyses across the inspected fleet as well as to monitor the performance of ROs. Except for adding the Management System Oversight (MSO) module to MISLE (noted in Chapter 3), the study committee did not find evidence that the Coast Guard has made the agreed-to improvements to data systems and their utility or has taken other measures to ensure that the compliance history of ACP-enrolled vessels is readily available to Coast Guard field inspectors and RO surveyors alike. Suffice to say that even if the MISLE fields had been added, their addition would not address the concern raised by the National Transportation Safety Board that the El Faro’s RO, the American Bureau of Shipping (ABS), did not have access to MISLE and its full record of the vessel’s deficiencies, including the narratives logged in by Coast Guard inspectors.1 The Coast Guard restricts access to MISLE. Although its public-facing version, Port State Information Exchange (PSIX), is accessible to ROs, it lacks details on the nature or types of deficiencies in a vessel’s history. ROs can obtain more details on a vessel’s compliance history through e-mails and meetings with marine inspectors, but such exchanges can be difficult to coordinate. Hence, even as the restrictions on MISLE access remain, the Coast Guard has not created a means for ensur- ing that a vessel’s full compliance history can be shared and consulted on by RO personnel. Furthermore, the committee is not aware that MISLE records have been updated for increased accuracy. The data provided by the Coast Guard for the ACP and the Maritime Security Program (MSP) Select– enrolled fleet profiles in Chapter 2 proved cumbersome to manipulate and 1 NTSB. 2017. Sinking of US Cargo Vessel SS El Faro, Atlantic Ocean, Northeast of Acklins and Crooked Island, Bahamas, October 1, 2015, Accident Report, NTSB/MAR-17/01, December 12, p. 71.

70 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs were rife with missing, incorrect, and inconsistent data.2 In accessing PSIX, the committee also found many omissions, inconsistencies, and errors in vessel status and inspection records. There are many potential explanations for the shortcomings of MISLE, many of which have been documented by the U.S. Government Account- ability Office in its July 2020 report Coast Guard: Actions Needed to Ensure Investments in Key Data System Meet Mission and User Needs (see Box 4-1).3 One significant issue is that the data system was created to serve multiple Coast Guard missions, with Marine Safety and Pollution Prevention being only 2 of 11 missions. MISLE, therefore, is not optimized for reporting on and informing compliance verification. As discussed in Chapter 3, the Coast Guard indicated that plans are being made to replace MISLE, but they are in their early stages with no firm timelines and subject to resource availability. Hence, the prospects of MISLE being significantly upgraded appears to be low in the near future, and a replacement data sys- tem is likely to take many years. Moreover, if the upgraded or new system continues to have restrictions on external access, the problems associated with poor data integration and sharing will persist. Inasmuch as having up-to-date and accurate data on vessels and their status is essential for compliance verification and effective RO oversight, the Coast Guard’s continued reliance on MISLE is incongruent with the concerted efforts made over the past 5 years to upgrade organizational structures and policies for supporting and overseeing the ACP. Yet, there are fewer than 500 vessels in the ACP enrolled fleet (and only about 350 that are active); hence, a fleet-specific database that allows for greater re- cord sharing and integration, ease of use, transparency, and customization for the purposes of compliance verification and RO oversight should not be time-consuming, costly, or otherwise complex to build and maintain. Such a custom-built database, in the committee’s view, would have an array of features and capabilities that could serve the needs of the Coast Guard, ROs, vessel companies, and the public, as shown in Table 4-1. The table is intended to illustrate desirable features of a database that could provide integrated RO and Coast Guard data for an RO’s vessel. The data would need to derive from multiple sources, internal and external to the Coast Guard, and be kept current and regularly verified for accuracy. However, for a small fleet that does not change much from year to year, it is reasonable to assume that only a modest amount of effort would be 2 The MISLE data were provided in spreadsheets, making extractions and manipulations time-consuming, burdensome, and not well suited to automated report creation. A large num- ber of fields were empty (NULL) and the committee needed to consult many other sources to identify the required data. PSIX data are searchable by vessel name but not by the RO authorized for the vessel. 3 GAO-20-562, July 16, 2020, https://www.gao.gov/products/gao-20-562.

DATA, METRICS, AND RISK-INFORMED TOOLS 71 BOX 4-1 Summary of Relevant Findings from the U.S. Government Accountability Office’s Review of the Coast Guard’s Marine Information for Safety and Law Enforcement System The Coast Guard’s Marine Information for Safety and Law Enforcement (MISLE) system was created in 2001 for the purpose of capturing and reporting informa- tion on vessels, cargoes, facilities, waterways, and maritime organizations related to the Coast Guard’s many statutory missions. Examples of information stored in the data system include the results from vessel boardings for law enforce ment, marine inspections, marine safety investigations, pollution response actions, search and rescue operations, and port security activities. Specifically, marine inspectors use MISLE to schedule and record the results of marine safety inspec- tions, generate official documents, and help determine the scope of a vessel examination based on risk factors. Data stored in the system are also used to generate system-wide safety reports. In 2020, the U.S. Government Accountability Office (GAO) reported on its review of the extent to which MISLE supports Coast Guard operations and deci- sion making. It also reviewed efforts by the Coast Guard to enhance and upgrade the system. While GAO concluded that MISLE generally supports Coast Guard operations, it found numerous instances of incomplete, inaccurate, and missing records and limited use capabilities stemming from the system’s design and legacy technology. For instance, after interviewing Coast Guard personnel who enter and use the data, GAO reported how users encountered repeated chal- lenges entering and tabulating the MISLE data. The investigators concluded that the system’s design contributed to these problems by requiring users to record information in multiple places, often in narrative formats that were not only time- consuming to create but led to variability in the terms and nomenclature used. The marine inspectors interviewed reported that duplicate vessel and operator records made it challenging to search for and plan inspection activities because pertinent inspection histories could be located under duplicate or multiple vessel records. The inspectors expressed concerned that if they do not find and open each duplicate vessel record, they might not have full or accu rate information to review on prior safety deficiencies before conducting vessel inspections. The inspectors also complained that they could only access MISLE records from a workstation. Their inability to enter data from the field contributed to lags in the transfer of information to the data system during periods when a large number of inspections and exams were being conducted. The GAO report characterized the Coast Guard’s efforts to develop and improve MISLE as slow and challenging. In 2015, the Coast Guard implemented the current version of MISLE—MISLE 5.0—which included a replacement of the underlying system architecture. However, according to GAO, MISLE system managers did not follow key redesign and development processes to ensure that the end-state system met user requirements. GAO reported how MISLE managers have near-term plans to make MISLE enhancements but that these plans address only a portion of the performance and functionality issues that continued

72 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs have been identified. The Coast Guard’s characterization of MISLE as a non- major acquisition program has also reduced its prioritization for funding, making it difficult for system managers to respond to all outstanding user needs and system requirements. SOURCE: U.S. Government Accountability Office. 2020. Coast Guard Actions Needed to Ensure Investments in Key Data System Meet Mission and User Needs, GAO-20-562 July. BOX 4-1 Continued required for data gathering, formatting, and checking—not out of line with the effort already expended by the Coast Guard to strengthen its procedures and policies. For instance, the core data could be derived from MISLE and circulated to each vessel’s and vessel company’s ROs for verification and additional input before being made available to Coast Guard marine safety personnel and RO surveyors and auditors. Such a custom database would need to have an architecture that facili- tates data sharing, integration, and analysis. It would need to be usable for vessel documentation searches by field inspectors and RO surveyors and auditors. It would also have to serve the needs of the CVC-4 to calculate KPIs, perform risk-based targeting of vessels, support trend analysis, and perform other analytics for oversight purposes. While the Coast Guard’s limited progress in developing these capabilities is discussed next, a funda- mental reason for this circumstance is the continued reliance on MISLE and its inadequacies as detailed above. PERFORMANCE STATISTICS AND METRICS The use of metrics for monitoring the performance of vessels in complying with the regulations and standards is commonplace. Indeed the International Maritime Organization’s (IMO’s) RO Code (discussed in Chapter 2) requires that each RO develop KPIs. Interestingly, however, the RO Code does not provide guidance on the metrics that the maritime authority (in this case, the Coast Guard) should use to inform its monitoring and oversight of the ROs in carrying out their delegated functions. The RO Code places the onus for developing metrics on the RO, but in some jurisdictions, it is the flag state that specifies the metrics that the ROs must provide to the flag state.4 4 For example, the European Maritime Safety Agency sends spreadsheets to the ROs recog- nized by European flag-state administrations seeking the metrics it requires.

73 T A B L E 4 -1 D es ir ab le F ea tu re s an d C ap ab ili ti es o f a V es se l D at ab as e fo r th e A lt er na te C om pl ia nc e Pr og ra m a nd t he M ar it im e Se cu ri ty P ro gr am S el ec t Fl ee ts D at ab as e C oa st G ua rd A cc es s R O A cc es s V es se l C om pa ny A cc es s Pu bl ic A cc es s V es se l pa rt ic ul ar s, i nc lu di ng ty pe , si ze , to nn ag e, d at e of b ui ld , A C P or M SP , R O , ve ss el n am e, v es se l co m pa ny • C on du ct s ea rc he s ba se d on v es se l pa rt ic ul ar s • C on du ct s ea rc he s ba se d on v es se l pa rt ic ul ar s • C on du ct s ea rc he s ba se d on v es se l pa rt ic ul ar s • C on du ct s ea rc he s ba se d on v es se l pa rt ic ul ar s C oa st G ua rd a nd R O st at ut or y in sp ec ti on re po rt s, d efi ci en ci es , an d co rr ec ti ve a ct io ns w it h as so ci at ed n ar ra ti ve r ep or ts an d se ar ch ab le c od es • A cc es s to a ll in sp ec ti on da ta w it h se ar ch ca pa bi lit ie s • V al id at e C oa st G ua rd ve ss el i ns pe ct io n da ta an d cl as si fic at io n ce rt ifi ca ti on s in da ta ba se • D ev el op K PI s ba se d on co m bi ne d C oa st G ua rd an d R O i ns pe ct io n da ta • Pe rf or m a ut om at ed ri sk t ar ge ti ng o f ve ss el s ba se d on C oa st G ua rd pl us R O d at a • A cc es s to a ll in sp ec ti on da ta w it h se ar ch ca pa bi lit ie s • V al id at e R O v es se l in sp ec ti on d at a an d cl as si fic at io n ce rt ifi ca ti on s in da ta ba se • de ve lo p K PI s ba se d on co m bi ne d C oa st G ua rd pl us R O i ns pe ct io n da ta • Pe rf or m a ut om at ed ri sk t ar ge ti ng o f ve ss el s ba se d on c om bi ne d C oa st G ua rd p lu s R O da ta • V al id at e co m pa ny ve ss el d at a an d ce rt ifi ca te s in d at ab as e • Id en ti fy e rr or s fo r ev en tu al M IS L E re vi si on • A cc es s to C oa st G ua rd – an d R O -i ss ue d st at ut or y in sp ec ti on d efi ci en ci es w it h sh or t de sc ri pt io n of de fic ie nc ie s, w it h se ar ch ca pa bi lit ie s (d es cr ip ti on s si m ila r to d efi ci en cy in fo rm at io n av ai la bl e in PS IX a nd i n E qu as is ) co nt in ue d

74 D at ab as e C oa st G ua rd A cc es s R O A cc es s V es se l C om pa ny A cc es s Pu bl ic A cc es s C oa st G ua rd a nd R O Sa fe ty M an ag em en t C er ti fic at e (S M C ) an d D oc um en t of C om pl ia nc e (D O C ) au di t re po rt s an d au di t fin di ng s (o bs er va ti on s, no nc on fo rm it ie s, m aj or no nc on fo rm it ie s) a nd co rr ec ti ve a ct io ns w it h as so ci at ed n ar ra ti ve r ep or ts an d se ar ch ab le c od es • A cc es s to a ll C oa st G ua rd a nd R O S M C an d D O C d at a w it h se ar ch c ap ab ili ti es • V al id at e C oa st G ua rd ve ss el a ud it d at a an d IS M c er ti fic at io ns i n da ta ba se • D ev el op K PI s ba se d on C oa st G ua rd v es se l an d co m pa ny a ud it fin di ng s, a nd c om bi ne d C oa st G ua rd p lu s R O au di t fin di ng s • Pe rf or m t re nd a na ly si s us in g SM S an d D O C fin di ng s • A cc es s to a ll C oa st G ua rd a nd R O S M C an d D O C d at a w it h se ar ch c ap ab ili ti es • V al id at e R O v es se l au di t da ta a nd I SM ce rt ifi ca ti on s in da ta ba se • D ev el op K PI s ba se d on R O v es se l an d co m pa ny a ud it fin di ng s, a nd c om bi ne d C oa st G ua rd p lu s R O au di t fin di ng s • Pe rf or m t re nd a na ly si s us in g SM S an d D O C fin di ng s • A cc es s to C oa st G ua rd – an d R O -i ss ue d SM C a nd D O C a ud it fin di ng s (o bs er va ti on s, no nc on fo rm it ie s, m aj or no nc on fo rm it ie s) f or co m pa ny ’s v es se l w it h de sc ri pt io n of fi nd in gs w it h se ar ch c ap ab ili ti es (e xc lu de s na rr at iv e re po rt s) • A cc es s to C oa st G ua rd – an d R O -i ss ue d SM C o bs er va ti on s, no nc on fo rm it ie s, m aj or n on co nf or m it ie s in t he f or m o f sh or t de sc ri pt io n of fi nd in gs , w it h se ar ch c ap ab ili ti es ; D O C a ud it i nf or m at io n ex cl ud ed D et en ti on s an d de fic ie nc ie s is su ed b y Po rt S ta te C on tr ol ag en ci es ( e. g. , C oa st G ua rd , Pa ri s M em or an du m o f U nd er st an di ng [ M O U ], To ky o M O U ) • A cc es s to R O d at a of R O v es se l de te nt io ns • A cc es s to C oa st G ua rd da ta o f R O v es se l de te nt io ns • A cc es s to R O a nd C oa st G ua rd d at a of c om pa ny v es se l de te nt io ns • A cc es s to v es se l da ta de te nt io ns i n th e fo rm at pr ov id ed i n C oa st G ua rd A nn ua l Fl ag S ta te a nd Po rt S ta te C on tr ol R ep or ts N O T E : A C P = A lt er na te C om pl ia nc e Pr og ra m ; D O C = D oc um en t of C om pl ia nc e; I SM = I nt er na ti on al S af et y M an ag em en t; K PI = k ey in di ca to r of pe rf or m an ce ; M IS L E = M ar in e In fo rm at io n fo r Sa fe ty a nd L aw E nf or ce m en t da ta ba se ; M O U = M em or an du m o f U nd er st an di ng ; M SP = M ar it im e Se cu ri ty P ro gr am ; PS IX = P or t St at e In fo rm at io n eX ch an ge ; R O = r ec og ni ze d or ga ni za ti on ; SM C = S af et y M an ag em en t C er ti fic at e; S M S = sa fe ty m an ag em en t sy st em . T A B L E 4 -1 C on ti nu ed

DATA, METRICS, AND RISK-INFORMED TOOLS 75 Following its El Faro investigation, the Marine Board of Investigation recommended that the Coast Guard publish an annual report on U.S. fleet compliance that also contains statistics and metrics on the performance of TPOs, including the ROs authorized for the ACP. The Commandant agreed that enhancements to data fields in MISLE (as discussed above) would be used to develop these performance metrics. Since 2018 the Coast Guard has annually published the report Flag State Control in the United States. In the 2019 and 2020 reports, five statis- tics are provided and characterized as KPIs of compliance by vessel owners and operators, as well as the performance of ROs. For reasons explained below, the committee is reluctant to refer to these statistics as KPIs. For instance, one reported statistic is simply the number of commercial vessel casualties divided by the number of commercial vessels in the flag fleet. While such a basic statistic merits reporting, its value for monitoring and measuring vessel compliance and RO oversight is minimal given its lack of specificity. The four other reported statistics, which are discussed next, have greater relevance but also some important limitations. In responding to the committee’s queries, the Coast Guard also stated that it maintains additional, unreported KPIs for evaluating the perfor- mance of owners, operators, and ROs. The committee therefore asked the Coast Guard to identify these KPIs, especially those pertaining to the performance of ROs. The Coast Guard pointed to five additional statistics. They too are discussed, their limitations cited, and their opportunities for improvement noted. Following the discussion of these publicly reported and internal statis- tics, consideration is given to additional types of metrics the Coast Guard might consider using to monitor and oversee ACP compliance and the performance of ROs. These metrics are offered to illustrate how the Coast Guard could potentially expand the array of information used for oversight purposes. Publicly Reported Statistics Statistic 1: Commercial Vessel Casualties per Commercial Vessel in the U.S.-Flagged Fleet While a basic indicator of trends in the overall safety performance of the U.S.-flagged fleet, the value of this metric for monitoring and measuring vessel regulatory compliance and RO performance is negligible. Neverthe- less, basic statistics of this type warrant tracking and reporting.

76 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs Statistic 2: Statutory Findings of Deficiencies per Recognized Organization Vessel Survey The Coast Guard reports that ROs visited 4,377 U.S. flag vessels in 2020 for the purpose of conducting statutory surveys. In doing so, they issued 3,368 statutory findings of deficiencies, averaging 0.8 per vessel. Most of the surveys involved vessels that are not in the ACP or MSP Select fleets that use ROs. This particular metric, therefore, provides background pertinent to overall U.S.-flagged fleet compliance, but it does not provide an indica- tion of RO performance in carrying out delegated functions in any direct or obvious way. As a background metric of the compliance performance of the fleet, this metric can be used for benchmarking the relative performance of vessels in the ACP and the MSP Select fleets. The reporting of “findings” per survey, however, has limited value inasmuch as it does not indicate the nature of the findings, such as whether they were issued for serious deficien- cies or resulted in detentions. The committee understands that survey and inspection findings are often recorded in narratives, which can complicate the monitoring and reporting of findings by type and severity. Therefore, improvements to the utility of this statistic, such as for more effective targeting of compliance enforcement on the basis of risk, may require changes to the reporting procedures themselves to enable the Coast Guard to monitor trends and patterns involving specific types of deficiencies. Statistic 3: Deficiencies and Detentions per Foreign Port-State Vessel Inspection In its annual Flag State Control report, the Coast Guard provides the results of port-state inspections of U.S. flag vessels conducted under the Paris and Tokyo Memoranda of Understanding (MOUs) Port State Control Regimes. When combining data from the Paris and Tokyo regimes, U.S. flag vessels had deficiencies in 66 of 133 inspections (49.6 percent) in 2019 and 71 of 135 inspections (52.6 percent) in 2018. The detention rate was 1.5 percent of inspections in 2019 and 2.9 percent of inspections in 2018. In the case of the Paris regime, the U.S.-flagged ship detention rate was low enough to move the fleet from the regime’s “Grey List” to its “White List” which represents fleets with a consistently low detention record. As with the previous metric, this statistic provides limited insight into the performance of ROs for the ACP and the MSP Select fleets. Because the vast majority of ACP and MSP Select vessel inspections are completed by ABS, the port-state inspection data cannot be used effectively to compare the performance of different ROs (ABS, Lloyd’s Register, Nippon Kaiji Kyokai [ClassNK] and Det Norske Veritas [DNV]) in these two programs.

DATA, METRICS, AND RISK-INFORMED TOOLS 77 Nevertheless, comparisons of U.S. fleet performance internationally can offer insight into the general performance record of the ROs active in the United States. It is notable, for instance, that the four classification societies authorized for ACP and MSP Select are all top ranked by both the Paris and Tokyo MOUs. ABS ranked first on the Paris MOU, considered best in classification for 2017-2019 RO Performance List.5 ABS and the other three ROs were all also listed in the top 10 of 32 classification societies by the Tokyo MOU.6 There may be opportunities for the Coast Guard to refine this interna- tional performance metric. The types of deficiencies noted in the Paris MOU and Tokyo MOU follow the particular reporting format of the Equasis database.7 For more granularity on deficiency type, the Coast Guard could potentially capture data from Equasis for performance evaluation and re- porting. It could also compare the U.S. flag fleet’s performance to that of other White-List flag fleets. Statistic 4: Findings per Recognized Organization Audit Related to Safety Management System Certificates The Coast Guard reports the number of findings from RO audits of U.S.- flagged vessels for compliance with requirements under the Convention for the International Safety of Life at Sea and its International Safety Manage- ment (ISM) Code. Companies are expected to have a safety management system (SMS) and the vessel must be operated in accordance with the SMS. An “audit” implies a systematic and independent verification of whether the SMS complies with the requirements of the ISM Code and whether the SMS is implemented effectively to achieve the code’s objectives. Ships compliant with the ISM Code are issued a Safety Management Certificate (SMC). Ac- cording to the Flag State Control annual report, in 2020, ROs visited 965 U.S.-flagged vessels to conduct SMC-related audits on behalf of the Coast Guard and issued 522 findings, resulting in 0.5 finding per audit. Audit findings include observations, nonconformities, and major nonconformities. As with the previous statistics, this metric does not provide a direct indication of RO performance, but its utility is further limited by not differentiating findings on the basis of the nature of the observation and severity and type of nonconformity. The committee notes that such data are available because they are reported under the International Association of 5 Paris Memorandum of Understanding. 2020. 2019 RO Performance List (2020-7 2021-6). https://www.parismou.org/2019-performance-lists-paris-mou. 6 Tokyo Memorandum of Understanding. 2021. Annual Report 2020, Table 13, p. 41. http://www.tokyo-mou.org/publications/annual_report.php. 7 See www.equasis.org.

78 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs Classification Societies’ (IACS’s) reporting structure (IACS PR-17). These more detailed data, for instance, may be used to further refine the metric by accounting for findings of technical, operational, documentation, and other deficiencies. Statistic 5: Findings from Recognized Organization Audits of Companies Related to Document of Compliance A Document of Compliance (DOC) is issued to a company that complies with the requirements of the ISM Code. The Coast Guard reports that ROs visited 185 ship management companies to conduct DOC audits on behalf of the Coast Guard and issued 532 findings, or an average of 2.9 findings per audit. As is the case for the metric on SMC audits, the utility of this DOC audit statistic could be improved if differentiated by each finding’s severity and type. The unreported statistics discussed next indicate that the Coast Guard has records (documented in IACS PR-17 reports) of individual findings per survey and by type and severity. Unreported Statistics Statistic 6: International Association of Classification Societies PR-17s Issued per Recognized Organization Survey Each RO-issued IACS PR-17 report contains a detailed description of the RO surveyor’s statutory findings that are considered to directly affect the imple- mentation of the ship’s ability to comply with the ISM Code. As noted above, the IACS PR-17 differentiates vessel survey findings according to whether they are technical, operational, documentation, or other deficiencies. This added detail is important, not only for assessing vessel compliance with indi- vidual statutory requirements but also for providing an indication of the RO’s observance of the requirements concerning the ship’s condition and opera- tion, for instance, in assessing the RO’s understanding of the relationship between the ISM Code’s regulation 7 (shipboard operations) and regulation 10 (maintenance of the ship and equipment). Statistic 7: Nonconformities Identified by the Coast Guard per Recognized Organization Statutory Survey and International Safety Management Audit Coast Guard marine inspectors who identify nonconformities during vessel inspections follow the guidance in Mission Management System Work Instruction 005 (CVC-WI-005) if there is objective evidence of an RO’s lapse in the execution of one or more delegated functions. ROs are required

DATA, METRICS, AND RISK-INFORMED TOOLS 79 to implement and maintain a quality management system (QMS). Fol- lowing guidelines in CVC-WI-005, the Coast Guard may request that the RO conduct an internal investigation, or root-cause analysis, known as a quality case. This analysis generally involves the RO evaluating its QMS and reporting findings and corrective actions to the Coast Guard. The committee believes that this a useful metric by providing insight into the effectiveness of the RO’s QMS and its implementation. Statistic 8: Total Coast Guard–Issued Deficiencies Related to Statutory Certificates per Coast Guard Inspection Conducted Depending on the nature of the deficiency, this metric may be indicative of the performance of the RO, particularly for ACP- and MSP Select–enrolled vessels that are surveyed by an RO and undergo Coast Guard annual exam- inations. The number of deficiencies identified by the Coast Guard and RO can be compared for patterns that may be indicative of RO performance. Statistic 9: Recognized Organization–Associated Flag-State Detentions per Statutory Surveys and Audits Performed The Coast Guard currently performs oversight examinations of ACP- and MSP Select–enrolled vessels after the RO completes its statutory survey. However, the Coast Guard only performs audits of vessels for which there have been findings related to noncompliance with the ISM Code. Detention- rate patterns may inform the need for such oversight actions. Statistic 10: Coast Guard–Issued International Safety Management– Related Deficiencies per Coast Guard Inspection Although potentially useful for monitoring RO audit performance gener- ally, this statistic can also provide value for monitoring and ascertaining Coast Guard inspector performance in detecting SMS-related deficiencies. Summary Assessment of Existing Metrics and Other Possibilities Although some of the Coast Guard’s statistics, both external and internal, might be generally indicative of RO performance, most have insufficient granularity and are too indirect to serve as useful indicators of trends and patterns for identifying needed improvements by ROs. Metrics based on specific types of findings, nonconformities, and deficiencies would be more informative for these purposes, albeit creation of such metrics could be challenging when the relevant information in MISLE is reported in a

80 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs manner that is difficult to search and analyze, such as entries that are made in narrative form. In thinking about expanding and improving on these 10 statistics, the Coast Guard might consider adding metrics that are more outcome-based, for instance, by making use of the Coast Guard’s shipboard safety and pollution records. In the committee’s view, outcome-based measures are generally more desirable than metrics that focus on inputs or activity levels. By way of example, Table 4-2 is taken from the United States Coast Guard Annual Performance Report for Fiscal Year 2019. It contains statistics for serious marine incidents, such as pollution incidents per tons shipped. While the reported metrics are for the entire fleet, they could potentially be differentiated and tracked for subgroups of vessels, including those in the ACP and the MSP Select fleets. Likewise, in seeking other useful performance metrics for RO oversight, the Coast Guard could potentially make greater use of the results of its quality cases, as discussed in Chapter 3. Indeed, the committee notes that CVC-WI-005 states that the Coast Guard “will use the data and records derived from QCs to inform key performance indicators (KPIs) for ROs and TPOs, in accordance with the RO Code and as specified in the agreement between the RO/TPO and the Coast Guard.”8 From 2018 to 2020, the Coast Guard conducted 16 quality cases. However, the set of statistics dis- cussed above does not include any metrics informed by these assessments. Another possible approach for developing metrics on the performance of ROs is to examine vessel inspection results to determine how ACP- and MSP Select–enrolled vessels perform relative to other vessels with respect to regulatory compliance. To explore this approach, the committee set up suitable U.S. flag comparison groups of potential vessels that are “ACP- eligible, but not enrolled.” The committee filtered or removed those vessel types, routes, and other features, such as year of build and vessel tonnage, that would not be appropriate for the ACP comparison groups. Of the 521 vessels eligible, but not participating in either the ACP program or the MSP program, the committee determined that 39 non-offshore vessels and 44 offshore vessels were eligible but not enrolled in ACP. For the MSP program, the committee determined that 39 vessels remained. The filtration process suggested that there is little merit in including either passenger ves- sels or mobile offshore drilling units in the comparison groups because of their small numbers. Nevertheless, the fleet does contain sufficient numbers of vessels, both offshore and non-offshore (see Chapter 2), that are not in the ACP and the MSP Select fleets but that are otherwise eligible for the 8 USCG Office of Commercial Vessel Compliance. 2020. Request for Recognized Organi- zation (RO) Internal Quality Management System (QMS) Review—“Quality Case.” CVC- WI-005(2). Mission Management System Work Instruction, March 31.

DATA, METRICS, AND RISK-INFORMED TOOLS 81 programs. These unenrolled vessels, therefore, could serve as comparison fleets, relating their inspection experience to that of the enrolled vessels when appropriately differentiated by vessel type. Table 4-3 shows the results of comparisons of deficiency rates by vessel type categorized by the various subchapters under which they are regu- lated. Separate comparisons are made for non-offshore, offshore, and MSP vessels. The committee also examined detention rates, but the incidence of detentions was too small for meaningful comparisons. As noted in the introduction to this chapter, the comparisons in Table 4-3 are intended to be illustrative only, providing insight into the kinds of analyses the Coast Guard could perform with further refinement and supplementation. The comparisons suggest that more sophisticated analyses could shed light on the rate of regulatory compliance of the ves- sels enrolled in ACP and MSP Select fleets. To be sure, there are many confounding factors that would need to be controlled for, including differ- ences in the mix of vessels in the enrolled and unenrolled fleets and among vessel operators who elect to enroll their vessels in the ACP and the MSP Select. For instance, it is clear from the comparisons that offshore vessels have much lower deficiency rates than non-offshore vessels; however, the relative youth of the offshore fleet (as reported in Chapter 2) could be one of the factors contributing to this outcome. It was noted above that more performance metrics should be outcome based, and the Coast Guard’s marine safety and pollution incident data- bases could be tapped to provide some additional metrics of this type. To TABLE 4-2 Examples of Outcome-Based Metrics for All U.S. Flag Vessels, FY 2018–2020 Metric FY 2018 FY 2019 FY 2020 Three-year average number of serious marine incidents 734 748 612 Three-year average number of commercial mariner deaths and critical, serious, and severe injuries 126 113 120 Three-year average of chemical discharge incidents per 100 million short tons shipped 8.0 8.1 7.9 Three-year average of oil spills in the marine environment per 100 million short tons shipped 9.2 8.8 8.2 NOTE: FY = fiscal year. SOURCES: The 2018 and 2019 data are from USCG. 2020. United States Coast Guard Annual Performance Report for Fiscal Year 2019, p. 3; 2020 data are from USCG. 2021. United States Coast Guard Annual Performance Report for Fiscal Year 2020, pp. 11, 13. The Coast Guard notes that many of the results in 2020 are as a result of COVID, where marine survey and inspection activity was less than usual.

82 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs illustrate how such data could be used, Table 4-4 shows casualty and in- cident rates for the same groupings of vessels used in Table 4-3. Casualty and incident rates are found to be higher for the ACP and the MSP Select fleets than the comparison fleets. However, as was the case for that previ- ous comparison, a more thorough review of the records would be needed to ascribe the factors that may be contributing to this difference, including factors that may be unrelated to RO involvement and performance. For instance, a closer examination is likely to reveal differences in rates by vessel age and intensity of use (e.g., whether a vessel is laid up), which are not considered in Table 4-4. It is also important to recognize that because of the small sizes of the comparison fleets, and the relatively rare occurrence of consequential incidents, the rates calculated in these tables may not be reflective of any underlying risk differentials among different types of vessels. TABLE 4-3 Deficiencies per 100 Vessels for Comparable Fleets Inspected by Recognized Organizations and by the Coast Guard Exclusively, Calendar Year 2019 Non-Offshore Subchapter Enrolled in ACP (N = 230) Eligible But Not Enrolled (i.e., Fully Inspected by Coast Guard) (N = 39) D, I, O-D, U 3.8 (N = 229) I, O-D, U 5.9 (N = 39) I 4.4 (N = 154) 5.8 (N = 27) Offshore Subchapter Enrolled in ACP (N = 120) Eligible but Not Enrolled (N = 44) I, L 1.0 (N = 119) 2.3 (N = 44) I 0.9 (N = 34) 4.8 (N = 4) L 1.1 (N = 85) 2.1 (N = 40) MSP Subchapter Select (N = 37) Regular (N = 39) D, I, O-D 9.2 (N = 39) I, O-D 3.2 (N = 37) I 3.4 (N = 35) 9.4 (N = 34) NOTES: In each grouping, the subchapter results are disaggregated from the total. Discrepancies in tables are due to inconsistencies in the raw MISLE data, such as missing or incorrect entries. SOURCE: Analysis of MISLE data records as of April 23, 2021, as supplied by the Coast Guard.

DATA, METRICS, AND RISK-INFORMED TOOLS 83 RISK-INFORMED OVERSIGHT The Statement of Task calls on the committee to consider “the prospects for introducing a more automated risk-based program for the RO oversight program and associated vessels.” Indeed, in responding to recommenda- tions of the Marine Board of Investigation, the Commandant stated that “the Coast Guard must, and will, establish a risk-based and enduring policy framework that is simpler to execute and enables more robust oversight of delegated functions.” The Commandant further agreed to refine internal risk models to inform oversight of vessel compliance and RO performance, such is in determining when and how often marine inspectors should attend SMS audits; the frequency, timing, and scope of ACP oversight examina- tions; and any additional oversight that is needed to ensure that ROs are properly executing their delegated functions. The progress made by the Coast Guard in furthering the actions agreed to by the Commandant has proved difficult for the committee to assess, particularly regarding the development of risk-based models and their ease and simplicity of use. As discussed in Chapter 3, the Coast Guard’s CVC-4 is responsible for developing an “ACP Fleet Risk TABLE 4-4 Casualty and Serious Incident Rates for Fleets Inspected by Recognized Organizations (Alternate Compliance Program and Maritime Security Program Select Fleets) and Fleets Inspected by the Coast Guard Exclusively, 2019 Vessel Category Serious– Critical Casualtiesa Deaths Vesselsb Serious–Fatal Casualty Rate per 100 Vessels Non-offshore vessels in ACP/ MSP Select Fleets 20 8 267 10.5 Non-offshore vessels eligible but not enrolled in ACP/MSP Select 1 1 78 2.6 Offshore vessels enrolled in ACP 1 0 120 0.8 Offshore vessels eligible but not enrolled in ACP 0 0 44 NC NOTES: NC = not calculable as zero. Discrepancies in tables are due to inconsistencies in the raw MISLE data, such as missing or incorrect entries. a Serious to critical injuries are those requiring significant medical/surgical management and some hospitalization. b Because the committee was not provided with incident data, the “by vessel” data are reported for 2019 on a per vessel basis, with the casualty rate per 100 vessels. SOURCE: Analysis of MISLE data records as of April 23, 2021, as supplied by the Coast Guard.

84 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs Assessment List” that contains 10 percent of the ACP fleet. The list is developed through a prioritization process that identifies vessels consid- ered to pose a greater risk warranting more scrutiny for inspections and evaluations of RO performance. The MISLE records of ACP vessels on the list contain a “lookout” marker for marine inspectors to know that the vessels are considered higher risk. The risk criteria and weightings used by CVC-4 to develop the Fleet Risk Assessment List are not publicly available. While the criteria are believed to include vessel age and records of deficiencies, marine casual- ties, and port-state control detentions, the committee could not assess the methodology used in the absence of more details on its design and elements. What the committee did learn, however, is that CVC-4’s development of the list is complicated by the need to obtain data from multiple sources, including MISLE and major nonconformities and detentions recorded in foreign port-state control data systems. These complications stem in large part from the inadequacies of MISLE that have already been described. The committee was informed that because of the manual effort required for data collection and analysis, the list is updated only annually, rather than on a more frequent or continual basis. Apart from this Fleet Risk Assessment List there is little evidence the Coast Guard uses other risk-based information to guide its compliance verification and RO oversight, including inferences about risk derived quali- tatively from knowledge and experience. For instance, it is unclear whether the Coast Guard consults risk factors when making choices about how to design all of its ACP oversight examinations, not only for targeting certain vessels for more frequent or comprehensive inspections but also for scoping inspections within subcategories of vessels. For example, in the case of older vessels an inspection might focus more on structural steel corrosion and the condition of equipment and closures, while for newer vessels greater scrutiny may be given to crew capabilities. Given the data limits that have been discussed in this chapter, it is dif- ficult to see how the Coast Guard can fulfill the Commandant’s agreement to be more risk based and data driven. Even the CVC-4’s limited use of risk factors for targeting ACP vessels requires a great deal of manual data gathering and formatting that deters the pursuit of more efforts along these lines. In short, the realistic answer to the Statement of Task’s question about introducing a more automated, risk-based program is that the prospects for doing so will be dim until the Coast Guard takes more concerted steps to build and maintain the data systems that are needed to bring about and support this essential capability.

DATA, METRICS, AND RISK-INFORMED TOOLS 85 SUMMARY ASSESSMENT AND RECOMMENDATIONS Having reviewed the data systems, performance metrics, and risk-based processes employed by the Coast Guard to support and oversee the ACP and ROs, the committee concludes and recommends the following: • The Coast Guard’s progress toward a more data-driven and risk- based approach to verifying compliance and overseeing the perfor- mance of ROs has not kept pace with the strides made in establishing needed policies, organizational capabilities, and procedures. More- over, there are few promising signs that the Coast Guard is on the verge of marked progress in becoming more data driven and risk based as needed for the effective and efficient implementation of these policies, organizational capabilities, and procedures. • The Coast Guard’s MISLE data system is poorly suited to support- ing vessel compliance verifications by marine inspectors and RO surveyors and Coast Guard oversight of ROs. The U.S. Govern- ment Accountability Office has recently reported on the history, uses, and shortcomings of MISLE and its legacy features.9 As that report makes clear, the data system’s intended uses are too varied, its design and structure are too outmoded, its records are too in- complete and unreliable, and its access is too restricted. Plans to upgrade or replace the data system are in their infancy and cannot be relied on as either near- or medium-term solutions. Even if a re- placement system is created, there is no assurance—due to security concerns—that it will permit the needed integration of RO data and access by ROs and other external parties. • For a data system to support ACP compliance verification and RO oversight, it would need an array of features that would serve the purposes of the Coast Guard, ROs, vessel companies, and the public (as shown previously in Table 4-1). The data would need to be gathered from multiple sources, both internal and external to the Coast Guard, and it would need be kept current and regularly validated for accuracy. The database’s architecture would need to facilitate data sharing, integration, and analysis and be readily searchable by field inspectors and RO surveyors and auditors. It would also need to meet the needs of CVC-4 for calculating speci- fied KPIs, performing risk-based targeting of vessels, performing trend analysis, and enabling other needed data analytics for over- sight purposes. 9 GAO-20-562 Report-COAST GUARD Actions Needed to Ensure Investments in Key Data System Meet Mission and User Needs, July 2020. https://www.gao.gov/products/gao-20-562.

86 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs • There are relatively few U.S. flag vessels that use ROs, which brings into question the Coast Guard’s lack of progress in developing a data set customized to these programs for meeting the needs of compliance verification and RO oversight. For a relatively small fleet that does not change much from year to year, one would expect that only a modest amount of effort would be required for data gathering, recording, and checking to develop such a data set that is external to MISLE and focused exclusively on vessels that use ROs, including the ACP and the MSP Select fleets. • Coast Guard’s KPIs consist of statistics that have limited relevance to the oversight of the ACP or ROs. The Coast Guard considers risk-related factors when profiling ACP vessels to inform the fre- quency and scope of inspections, but lacks sophisticated models for this purpose and is hindered by the need for manual data gathering and input. Absent better data systems and tools for data extraction and analysis, the Coast Guard’s agreement to be more risk based and data driven through the use of risk models and KPIs is not likely to be achieved to any meaningful degree. In the near term, the Coast Guard has little choice but to use MISLE along with any other relevant data it can obtain from ROs and other sources (such as foreign port-state control data) for the purposes of ACP compliance verification and RO oversight. However, looking further out— say, over the next 5 years—there is greater opportunity for the Coast Guard to collaborate with ROs to create a database specific to vessels using ROs, including the ACP and the MSP Select fleets. While a data system to fully replace MISLE is on the distant horizon, it holds the greatest promise to support the Coast Guard’s interest in more data-driven and risk-based ap- proaches for vessel compliance verification and RO oversight. The follow- ing four recommendations are therefore offered with these time dimensions in mind: Recommendation 1: In the near term, the Coast Guard should collaborate with ROs and vessel owners to develop and execute a plan to share and validate information for the purpose of ensuring that records derived from MISLE of vessels and vessel owners who use RO services are complete, accurate, and current with respect to compliance history. The augmented records should be scrubbed of sensitive information and extracted from MISLE so they can be made available to RO surveyors and auditors in addi tion to marine inspectors.

DATA, METRICS, AND RISK-INFORMED TOOLS 87 The data set should also be structured to meet CVC-4’s need for calculat- ing specified KPIs, performing risk-based profiling of vessels and ROs, and enabling other data analyses for oversight purposes. Additional data set filtering could enable access and use by vessel owners and the public. Because of the relatively small number of ACP and MSP Select vessels, the manual effort required to correct errors and omissions in their MISLE records and to add supplemental fields should not be formidable. However, it would be desirable for the Coast Guard to ensure that information from the Marine Safety Center’s plan reviews is incorporated—as this informa- tion does not currently reside in MISLE. Recommendation 2: While collaborating with ROs and vessel owners on means of supplementing, validating, and enabling greater access to relevant MISLE records, the Coast Guard and ROs should work together on the development of KPIs that are most relevant to monitoring and overseeing the performance of ROs. Although persistent shortcomings in MISLE-derived data will undoubtedly limit the quality of the indicators that can be developed, such collabora- tion will offer insight into the kinds of data and data system capabilities that will be needed for the design and functioning of a new stand-alone oversight database. One would expect that measures of outcomes, such as from casualty and pollution records, would be among the suite of KPIs considered, and that some metrics of this type could potentially be developed right away using information already available in Coast Guard and RO databases. Collaborative thinking about KPIs that serve as leading indicators would also be desirable. Even if current data do not allow for their development, this thinking would help inform the planning of a new data system. Recommendation 3: Congress should resource the development and imple- mentation of a stand-alone data system, fully external to MISLE, that is exclusive to the purposes of supporting compliance verification and RO performance and oversight. ROs should be engaged during the database’s planning—such as through a Coast Guard and RO database/information technology working group—to ensure that the database’s design, elements, and functional capabilities align with the needs of Coast Guard inspectors, other marine safety personnel, and RO surveyors and auditors. Because the data set recommended above—consisting of mostly MISLE augmentations—is a short-term fix, it will be important for efforts to get under way to introduce this more functional, stand-alone data system as soon as possible. For this external data system to support compliance verification and RO oversight, it would need an array of features that would serve the

88 STRENGTHENING U.S. COAST GUARD OVERSIGHT AND SUPPORT OF ROs purposes of the Coast Guard, ROs, vessel companies, and the public, as discussed earlier in this chapter. The data would need to be gathered from multiple sources, both internal and external to the Coast Guard, and would need to be kept current and regularly validated for accuracy. The database should be designed to facilitate data exchange, ensure data quality, and reduce the need for manual data entry. It would need access features that enable the Coast Guard, ROs, and vessel companies to push or pull data to or from the database between their respective data systems and sources. The database’s architecture, therefore, would need to facilitate data shar- ing, integration, and analysis and be readily searchable by field inspectors and RO surveyors and auditors. Ensuring that the types of inspection and audit findings, deficiencies, and nonconformities are recorded in a search- able format would help meet CVC-4’s need to calculate more insightful KPIs, perform risk-based targeting of vessels and companies, and develop data tools for oversight purposes. By making these data available, the Coast Guard could encourage external analyses, including scholarly research, that can inform KPIs and risk-based methods. To support such external analyses and assessments, the recommended database should also have a public- facing version with appropriate search capabilities and filters. Recommendation 4: Although it may take many years to bring about, the replacement of MISLE with a new, modernized data system should be a high priority for the Coast Guard to more fully support the work of marine inspectors and ROs and to monitor and oversee their performance. Whether the new system replaces, incorporates, or feeds into and from the external data system recommended above, it will be important that the compliance verification, performance monitoring, and analytic needs of marine inspectors, ROs, TPO coordinators, and CVC-4 are met. Public access, with appropriate protections and filters, should be another specified feature to enable external analysis and ensure more transparency about vessel compliance and RO performance levels. During planning and development of MISLE’s replacement data system, high priority should be given to ensuring that its architecture facilitates data sharing, integration, and analysis among the Coast Guard, ROs, and vessel companies. The efforts of the IMO to develop an international standard for data formats, exchanges, and harmonization in international shipping, as well as IACS’s development of a unified data exchange format for surveys and audits, should be consulted as models for the system’s electronic data interchange and sharing.