Airport Safety Risk Management Panel Activities and Outcomes (2016)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

9 An important step in this study was the review of methods currently used by Part 139 airports, for- eign airports, and military organizations. The purpose of this review was to identify most effective practices and tools that these organizations currently use. The effort relied on two key tasks: a lit- erature review, and a survey with Part 139 airports, foreign airports, military bases, and consultants. In addition to the survey, interviews were carried out with Part 139 staff involved with SRAs at the surveyed airports. LITERATURE REVIEW The objective of the literature review was to seek most effective practices and tools used by the industries to conduct and support safety assessments. References from various industries and differ- ent countries were reviewed for helpful information. Guidance information presented in this report was gathered from various sources and adapted or summarized. Tables 2, 3, and 4 present a summary of the literature review conducted to identify common SRM techniques used by different industries and particularly by the aviation industry. Table 2 contains spe- cific FAA references used internally to support FAA-led SAs. An important point is that the NPRM defines the need to establish an SRM process that incorporates five steps; however, the rule does not provide any detail on how safety risk assessments should be conducted by the airport. Table 3 presents the references applicable to airport-led SRAs, and Table 4 presents other references used in this study. Table 5 presents some of the available tools used in risk assessments. SURVEY AND INTERVIEWS This section describes the survey process used in this Synthesis study. The survey questionnaire is available in Appendix A, and the list of airports, consultants, and other stakeholders participating in the survey or interviews is presented in Appendix B. The survey was carried out with airport staff and consultants who have been involved with airport SRM, to collect information on their experience with the SRA process, procedures, and tools. The objective was to identify most effective practices, gaps, and solutions that have been used by the industry. Particular attention was focused on the application of SRM techniques involving multi- disciplinary teams to evaluate safety issues associated with airports. The majority of surveys were carried out with airport staff from Part 139 airports. A limited num- ber of responses were obtained from foreign airports, consultants, and military bases. The questions submitted to the airports addressed both SAs and SRAs with no distinction on the type of assessment associated with the response; therefore the conclusions from the survey are not related to the type of facilitation, whether led by the FAA or done internally by the airport. Table 6 summarizes the profile of survey respondents. Responses were obtained from 36 of 41 organizations and consultants to which the questionnaire was submitted, representing a response rate of 87.8%. To complement the survey information, 15 telephone interviews were conducted with Part 139 and foreign airports that responded to the survey. The purpose of the interviews was to clarify some responses and attempt to obtain materials and tools developed by the airports. The materials obtained are included in the appendices. chapter two STUDY TASKS

10 Source Related Content FAA, Order 8000.369A, Safety Management System, May 2013 To meet ICAO State Safety Program (SSP) framework, this Order explains the SMS principles and requirements; establishes the FAA SMS Executive Council and FAA SMS Committee; standardizes terminology for SMS; requires FAA organizations to implement SMS; and provides guidance to FAA organizations and their industry segments to implement SMS. FAA, Order 8040.4A, Safety Risk Management Policy, Apr. 2012 This order supports FAA Order 8000.369, Safety Management System Guidance, and establishes requirements for how to conduct Safety Risk Management (SRM) in the FAA. This order establishes the Safety Risk Management (SRM) policy for the FAA and also establishes common terms and processes used to analyze, assess, and accept safety risk. FAA Order 5200.11, FAA Airports (ARP) Safety Management System, Aug. 2010 Order 5200.11 is the basis for implementing SMS within the FAA Office of Airports (ARP). It defines Safety Management System (SMS) requirements and describes the roles and responsibilities of ARP management and staff as well as other FAA lines of business that contribute to the ARP SMS. FAA Office of Airports—Safety Management System (SMS)—Desk Reference, V 1.0, June 2012 Describes FAA ARP internal SRM and Safety Assessment process, including Safety Assessment Screening (SAS), using SRM panels, SRM triggering actions and safety assessment tools [Preliminary Hazard Analysis (PHA)], Comparative Safety Assessment (CSA), and Operational Safety Assessment (OSA). FAA, ARP SOP 4.00—Safety Risk Management (SRM) Under the FAA Office of Airports Safety Management System (SMS), Standard Operating Procedure (SOP), Sep. 2014 This SOP establishes uniform procedures for conducting the Safety Risk Management (SRM) component of the ARP Safety Management System (SMS) for certain ARP approval actions. ARP approval for these actions can only be given after completing the Safety Assessment (SA) in accordance with the SRM component. FAA JO 1000.37A, Air Traffic Organization Safety Management System, May 2014 Order JO 1000.37A establishes the SMS policies for ATO, defining the scope, requirements, and applications, and establishes the responsibility for owning and executing the SMS to all employees at all levels of the ATO. FAA, Air Traffic Organization, Safety Management System (SMS) Manual version 4.0, May 2014 This SMS Manual describes the objective of the ATO SMS and the interrelationship among the four components of the SMS, and instructs readers on the process of identifying safety hazards and mitigating risk in the National Airspace System (NAS). The Manual is complemented by the Safety Risk Management Guidance for System Acquisitions (SRMGSA), ATO Safety Guidance (ATO-SG) documents, and other FAA safety documents, to carry out the safety mission of the FAA and requirements of the SMS. Source: ASM Consultants. TABLE 2 FAA-LED SAFETY ASSESSMENTS (SAs)

11 Source Related Content FAA, Notice of Proposed Rulemaking (NPRM), Federal Register Volume 75, Issue 194 (Oct. 7, 2010) Proposed amendment to Part 139 to require certificate holders to establish an SMS for the airfield environment. The FAA proposal specifies implementation of a systematic process to analyze hazards and associated risks using a 5-step process, with no further specifics on the SRM process. FAA, AC 150/5200-37A (Draft), Safety Management Systems for Airports, 2012 This AC, released in draft to supplement the NPRM, presents the concepts of a Safety Management System (SMS) and provides detailed guidance about developing and implementing SMS on an airport. It applies to all civil airports when adapted to the size, activity level, staff level, and resources of each facility. Also, it establishes guidelines for SMS implementation for and by airport operators at Part 139 airports. ACRP Report 1: Safety Management Systems for Airports, Volume 2: Guidebook, 2009 Reference to help users understand what constitutes an airport SMS; describes its components and their interactions; and offers guidance in the planning, implementation, and operation of an airport SMS. ACRP Report 131: A Guidebook for Safety Risk Management for Airports, 2015 Guidebook organized to help airport staff understand where the SRM process falls within an overall SMS. The guidebook provides information on conducting a safety risk assessment (SRA), explaining how small airports with fewer resources can scale it down. Tools and templates are provided as appendices. ACRP Synthesis 37: Lessons Learned from Airport Safety Management Systems Pilot Studies—A Synthesis of Airport Practice, 2012 Describes data and information from the FAA SMS Pilot studies initiated in 2007 and completed in 2011. Source: ASM Consultants. TABLE 3 AIRPORT-LED SAFETY RISK ASSESSMENTS (SRAs) Source Related Content Aviation Risk Management Solutions (ARMS) Working Group, The ARMS Methodology for Operational Risk Assessment in Aviation Organizations, 2010 Presents a methodology for Operational Risk Assessment (ORA). The primary target for the methodology is the airlines, but it is also applicable to other aviation organizations, such as airports. The working group consisted mainly of safety practitioners from airlines. FAA/EUROCONTROL ATM Safety Techniques and Toolbox, V2.0, Oct. 2007 Document contains some of the best safety assessment techniques available in 2007 for Air Traffic Management applications, based on the joint experience of the FAA and EUROCONTROL, and based on a review of more than 500 safety techniques used in nine industries. The result is a set of 27 techniques that can be used by safety practitioners and managers to evaluate and improve safety in Air Traffic Management. Bircham, B., A Simple Guide to Hazard Identification, Presentation, Amey Seco, 1999 PowerPoint Presentation of practical guidance for identifying hazards by individuals or a panel approach. Naval Safety Center, Time Critical Risk Management (TCRM) Multimedia Training Module, Facilitator Guide, Version 2, Aug. 2009 Training module introduces a tool to help improve performance in applying the four principles of risk management: Accept risk when benefits outweigh the cost. Accept no unnecessary risks. Anticipate and manage risk by planning. Make risk decisions at the right level. TABLE 4 ADDITIONAL SRA REFERENCES (continued on next page)

12 Source Related Content Air Force Pamphlet 90-803, Feb. 2013 This pamphlet provides the definitions, guidelines, procedures, and tools for integration and execution of risk management as a risk reduction process to assist personnel in identifying and controlling safety and health hazards in making informed decisions. The term “Operational Risk Management” (ORM) has been replaced with the term “Risk Management” (RM) to emphasize the importance of hazard and risk mitigation and management in all aspects of the Air Force, not just Operations. UK Civil Aviation Authority, CAP 760— Guidance on the Conduct of Hazard Identification, Risk Assessment and the Production of Safety Cases for Aerodrome Operators and Air Traffic Service Providers, Dec. 2010 This document is a consolidated reference addressing the development of Safety Cases for the purposes of assuring the safety of ATS and airport operations. The guidance is based on a seven- step safety assessment process to airport operators and ANSPs on the development of a Safety Case and, in particular, on hazard identification, risk assessment and the mitigation techniques that may be used. A Safety Case presents adequate evidence and argument to demonstrate that the new system or change is tolerably safe. EUROCONTROL, Guidelines for the identification of hazards—How to make unimaginable hazards imaginable?, NLR- CR-2004-094, Mar. 2004 Presents guidelines on how to perform hazard identification brainstorms. These guidelines are based on experience at NLR and supplemented with other knowledge judged valuable. Combinations of functional and brainstorming approaches to hazard identification are expected to be valuable because of the different subsets of hazards these methods yield. It is recommended and motivated to perform brainstorms first. Source: ASM Consultants. Pullan, P., Murray-Webster, R., A Short Guide to Facilitating Risk Management, 2011 Describes how skilled facilitators can help groups who need to manage risk. The focus is the brainstorming workshop. European Commercial Aviation Safety Team (ECAST)—SMS WG, Guidance on Hazards Identification, Mar. 2009 This document develops the concept of “the hazard” within a safety risk management framework, which also defines risk, safety events, undesirable events, outcomes, consequences, and risk controls (barriers or mitigations). It describes the basic concepts behind hazard identification methodologies (data-driven and qualitative). European Commercial Aviation Safety Team (ECAST)—SMS WG, Planning and Conducting FHA Sessions, Jan. 2003 The purpose of this Guidance Material is to provide recommendations to conduct sessions to identify a hazard and its worst credible effect. It describes the role of the panel and the psychology of the brainstorming session, in addition to practical guidance for planning the workshop. National Aerospace Laboratory (NLR), Safety Methods Database, V1.0, Mar. 2013 Living document that presents an overview of Techniques, Methods, Databases, and Models that can be used during a Safety Assessment. Divided into three parts: Part 1—Overview of Safety Methods; Part 2—Statistics; and Part 3—References. Includes description of 807 methods. This version includes the methods identified in a project on safety methods conducted by NLR for the FAA in 2011–2012. TABLE 4 (continued)

13 Source Related Content FAA Air Traffic Organization—SRM Panel Orientation Briefing, July 2015 Model briefing for SRM panels. EUROCONTROL, Air Navigation System Safety Assessment Methodology (SAM), Ed. 2.1, Oct. 2006 Guidance material for conducting safety assessment of air navigation systems. This toolkit/manual was developed by the EATMP Safety Assessment Methodology Task Force (SAMTF). The manual contains best practices for safety assessment of Air Navigation Systems and provides guidance for their application. ICAO Toolkit, 2013 The toolkit is downloadable and includes the latest editable appendices of the ICAO Safety Management Manual (SMM) 3rd Ed., 2013. Civil Aviation of New Zealand, Aviation Risk Management, an Introduction, Booklet 4, June 2013 Quick reference guide for practicing risk management. Source: ASM Consultants. TABLE 5 SAFETY ASSESSMENT TOOLS Organization Category SMS Implementation Study Participant Yes No Total Part 139 Airport Large Hub 3 9 12 Medium Hub 4 4 8 Small Hub — 1 1 Non-Hub 2 — 2 Reliever 1 — 1 GA 1 — 1 Foreign International — 5 5 Military Bases Air Force — 3 3 SMS Consultants N/A 3 — 3 Total 14 22 36 Note: N/A = not applicable. Source: ASM Consultants. TABLE 6 SUMMARY OF SURVEY PARTICIPANTS

14 It is important to note that only a small percentage of Part 139 airports have conducted SRAs or had SAs led by FAA, and that most airports currently using SRAs and SAs in a routine basis are medium and large hub airports. Therefore it was difficult to obtain survey responses from smaller airports, as shown in Table 6. SURVEY RESPONSES Each airport participating in the survey received a questionnaire with 22 questions about their experi- ence conducting safety assessments. A summary of the responses to each question in the survey follows. Question 1: Who were the SRA or SA facilitator(s), and how many SRAs or SAs were completed at your airport? At least 12 Part 139 airports that responded to the survey indicated they had SAs led by the FAA, in most cases by FAA ATO facilitators. Of the 25 Part 139 airports responding, 14 had SRAs led by airport staff. The foreign airports surveyed use SRAs on a regular basis. Most risk assessments were led by air- port staff, with an average of 40 SRAs per year reported by those five foreign airports. Most of those SRAs were developed by a small group of airport staff, and the large majority of discussions with stakeholders were not facilitated in a brainstorming session to identify hazards and assess risks. One of the foreign airports reported the availability of staff specifically for the development of SRAs. The military organizations surveyed indicated they have specific trained staff who develop preliminary assessments to discuss with the group affected by the issue; however, this procedure is normally used in situations in which time is not a critical factor. Question 2: What criteria do you use for selecting a facilitator? A summary of responses is presented in Figure 1. A few Part 139 airports identified FAA facilitators as leading SA brainstorming sessions. One airport pointed out that engineers from construction/ design firms have acquired skills to facilitate airport-led SRAs, particularly for safety issues related to airfield improvement formulation, design, and construction. Another Part 139 airport indicated that cost was the basis for selecting and hiring consultants to facilitate the SRAs. Foreign airports use their own staff for most risk assessments; however, the process is slightly different, and most hazards are identified and risks assessed by a small team. In the FIGURE 1 Selection of SRA facilitators (Source: ASM Consultants).

15 military organizations surveyed, risk assessments were conducted by their own staff, military or civilian. Appendix F of the FAA ARP SMS Desk Reference (2012) provides a list of qualifications that can be used for hiring SRM consultant services, including SRA facilitators. Question 3: During SRM panel meetings the facilitator uses________: Responses to question 3 are summarized in Figure 2. The majority of the respondents use a risk matrix, visual aids, and risk assessment worksheets. Foreign airports rely less on a briefing to stake- holders; preliminary risk assessments are prepared by a small team, and information is circulated among stakeholders, rather than having a brainstorming session managed by a facilitator. Military staff using the Operational Risk Management (ORM) approach use checklists of common hazards (however, the small sample of military organizations does not allow general inferences). Few air- ports have defined meeting rules for SRA brainstorming sessions. Some Part 139 airports have developed quick reference guides containing basic information required during the brainstorming sessions, including the risk matrix, likelihood and severity defini- tions, and guidance on the SRM process. Sometimes a “one-page” guide worksheet is used as an aid to the SRA exercise. For most SAs and SRAs related to formulation, design, and construction of airfield improvements, the use of concept drawings, plans, maps, illustrations, and other information to help understand the context and the changes are common. Military organizations use “electronic staff packages” contain- ing references, forms, templates, and other tools to support Risk Management. The FAA and many Part 139 airports use a PowerPoint briefing template. The briefing is modified to address the safety issue to be assessed, and the content includes the scope of the project, panel introductions, administrative issues, and an explanation of the SRA process. The briefing is considered helpful, particularly for participants with little experience with safety assessments. Question 4: How do you identify panel members, including subject matter expertise? Responses for question 4 are summarized in Figure 3. Airports rely on stakeholders affected by the safety issue to compose the SRM panel. A number of observers and airport staff have participated FIGURE 2 Tools used in SAs and SRAs (Source: ASM Consultants).

16 in SRAs to gain practical experience. A few airports have brought observers from other airports to acquire and share their experiences. Most foreign airports indicated the importance of including airline pilots and the regulator to review safety assessments, and in the majority of the cases air traffic control staff is part of the panel review- ing the analysis. Few airports indicated that specific subject matter experts (SMEs) were brought in to participate in the SRM panel. Many Part 139 airports have hired consultants to facilitate the sessions. Question 5: Which are your major obstacles to convening a panel to conduct an SRA? This was a key question in the survey, particularly to identify gaps that Part 139 airports have expe- rienced when conducting SRAs. Response results are summarized in Figure 4. The major difficulty reported by all categories of organizations surveyed was the scheduling of SRA sessions to ensure participation of key stakeholders. Foreign airports also reported that the lack FIGURE 4 Major difficulties in conducting SRAs (Source: ASM Consultants). FIGURE 3 Panel participants (Source: ASM Consultants).

17 of specific training for stakeholders participating in risk assessments has been a major issue. One probable reason is that the brainstorming session is not facilitated and there is no briefing to help participants understand and get familiar with the process. The time required to complete an SRA involving several airport staff is another major issue identified in the survey. Most Part 139 airports responding to the survey have experienced a limited number of airport-led SRAs and indicated some difficulty responding to this question. Some airports reported that they had limited experience with safety assessments and therefore had to rely completely on the facilitation team of FAA staff or consultants. Question 6: What are your most common triggers for convening an SRM panel? The majority of safety issues that led Part 139 and foreign airports to trigger SAs and SRAs were related to formulation, design, and construction associated with CIPs, as shown in Figure 5. Most SAs, particularly at airports that had not participated in the Pilot Study, were initiated after receiving notification from the FAA that they would convene a panel. SRAs generated from accidents and incidents at the airports were another frequent trigger for SRAs. Few Part 139 airports have used SRAs to evaluate new or revised standard operating proce- dures (SOPs). In the past few years it has become more common to have SRAs prior to special events (e.g., Super Bowl, air shows). Question 7: Which airport area was impacted by the project/change that triggered the SRA? As expected, most Part 139 and foreign airports use SRAs for safety issues associated with the air- side, in both the movement and nonmovement areas. There have been few experiences using SRAs for landside issues, particularly by Part 139 airports. However, some airports with more mature SMS, including Part 139 airports, have found it beneficial to use the SRA process to evaluate landside safety issues. Question 8: How does the airport document panel results? Full SA reports have been developed to document those SAs led by the FAA. Some Part 139 airports have developed simple templates to document SRAs triggered by internal processes and involving few stakeholders. FIGURE 5 SRA triggers (Source: ASM Consultants).

18 Some foreign airports prepare comprehensive reports, depending on the complexity of the safety issue. A couple of foreign airports surveyed use only a risk assessment template with sum- mary of risk mitigation actions and approvals. Some foreign airports use a preliminary hazard analysis to determine the need for further in-depth analysis of a safety issue. These studies are called Safety Study, Aeronautical Study, or Safety Case, depending on the convention used by the specific country. For Part 139 airports, a preliminary hazard analysis may be performed and lead to convening a panel for the safety assessment, similar to the safety assessment screening used by FAA ARP. Question 9: What is the method for mitigation follow-up? Based on the responses obtained, there is still a major deficiency in ensuring that risk mitigation actions defined in an SRA are actually implemented and monitored. A few Part 139 airports keep track of risk controls defined in the SRA; however, the procedures are still not systematic. Chapter seven presents techniques to improve implementation and monitoring of risk mitigation actions, gathered from the literature review and foreign airports. Most Part 139 airports rely on the assigned responsibilities described in the SRA Report, or a summary of control actions is passed to contractors for inclusion in the Construction Safety and Phasing Plan (CSPP). It is important to note that a CSPP contains actions under the responsibility of the airport contractor and that some of the actions may fall under the responsibility of another airport stakeholder. A couple of Part 139 airports have reported difficulties obtaining the support of stakeholders when the risk control action can affect their operations and cause delays. The NPRM issued by the FAA in 2010 requires that Part 139 airports “establish and main- tain records that document the certificate holder’s Safety Risk Management processes” and that these records “shall provide a means for airport management’s acceptance of assessed risks and mitigations.” FAA Order 8040.4A, which is only applicable to FAA-led SRM panels, points out the need to include a methodology for monitoring and tracking the predicted residual risk. FAA typically identi- fies parties responsible for mitigations within the SA documentation. ATO’s SMS Manual provides guidance to ATO personnel on developing monitoring plans and tracking of mitigations. Every foreign airport surveyed indicated that it tracks completion of those actions. A few Part 139 airports have risk registers to monitor completion of risk control actions; however, it was not possible to assess how effective these systems are. Some foreign airports from Europe and Asia record the actions in a risk register, however these airports think there is still room for improvement. One foreign airport reported that an airport staff is appointed to manage risk mitigation actions, and a summarized plan is developed to ensure the actions are implemented. Question 10: Do you have predefined procedures to handle conflicts during meetings and, if so, what are they? Other than attempting to obtain consensus and possibly voting when conflicting assessments are presented during brainstorming sessions, there are few predefined procedures to handle conflicts. Possibly this has to do with the lack of specific guidance and training on facilitation of brainstorming sessions for safety assessments. Some guidance on resolving these conflicts is presented in chapter six. A summary of responses to question 10 appears in Figure 6. One of the consultants indicated use of FAA Order 8040.4A—Safety Risk Management Policy as general guidance. Most Part 139 airports prefer to document the issue or opinion when a conflict arises during the sessions. Many airports reported the importance of a skilled facilitator to handle undesirable situations and control the duration of the session. When a consensus cannot be achieved about the severity or likelihood of a risk, many airports responded that they tend to use the highest level of

19 risk in the assessment. One of the techniques that has been used during conflicts is to have a ses- sion break. Question 11: How are media and public inquiries and participation handled? Few airports provided any response to this question. Most Part 139 airports have not faced this type of inquiry, and none of the SAs or SRAs conducted had participation of media or general public. However, many Part 139 airports pointed out that safety assessment related information requested would be provided if supported by local sunshine laws. Question 12: What are lessons learned and advice to share with other airports conducting SRM panels? A summary of responses to this question appears in Figure 7. Many airports find that preliminary information passed to the panel before the meeting and during the brainstorming is beneficial to a FIGURE 6 Techniques for resolving conflicts during SRAs (Source: ASM Consultants). FIGURE 7 Summary of SRA techniques used (Source: ASM Consultants).

20 successful SRA. In most cases, SMS consultants make extensive use of material prepared for the specific SRA. Some airports prefer not to develop a preliminary list of hazards because it may cause bias and limit creative thinking during the brainstorming process. Some airports have developed Quick Reference Guides to assist the SRA process. A couple of for- eign airports surveyed have been promoting the importance of SRAs by organizing safety workshops with the participation of stakeholders and other airports. A few airports have used smaller groups associated with specific airport functions for preliminary risk assessments, particularly when more than 20 stakeholders are involved in the SRA. A few Part 139 airports reported using the information from past projects, particularly when involving construction activities on the airside. During the SRA briefing, some airports have reminded the panel to use a compassionate attitude toward safety and a collaborative approach during the brainstorming session. To demonstrate the importance of the assessment, some air- ports have the director or other high level management representative attend the beginning of the session. The survey indicated that airport operators believed it was beneficial to have observers from other airports involved in the process. This allows the visiting airport representatives the opportunity to gain practical experience and knowledge to use at their airports, which may be in the early stages of SMS implementation. One consultant reported having preliminary meetings with the FAA, airport operations, and engi- neering to identify key hazards and pre-populate the SA and SRA worksheet before bringing the airlines and other stakeholders to discussions. During the initial briefing in the second meeting, the facilitator makes it clear that the list is only preliminary, to help initiate the brainstorming process. When using the preliminary hazard assessment, the total duration of the second session involving the complete panel can be better managed because the information is better organized and many hazards and risks have been identified. The preliminary material is normally circulated at least 1 week before the meeting. Some consul- tants reported limiting panels to a maximum of 20 participants to make the session more manageable. One of the consultants noted that the FAA ATO has a template for the SRA briefing that may be used by the airports. Having an experienced facilitator with expertise in the area being discussed is key to an effective SRA, according to responses to question 12. A lunch break sponsored by the airport helps bringing the group together and serves to recharge energy for the second half. Question 13: What kind of tools does your airport use to complete a risk assessment? As illustrated in Figure 8, templates, checklists, worksheets, and preliminary lists of hazards have been used by many organizations surveyed. A few airports indicated the use of one-page templates, and most Part 139 airports have used the available FAA guidance. PowerPoint presentations, projec- tors, flip charts, and white boards have been used extensively to identify and highlight hazardous conditions and to keep track of the assessments and SRA worksheets. Question 14: What additional tools would be beneficial to the maturing of SRA process? Many airports pointed out the need to have specific SRA training and guidance to airport staff. Some tools, including lists of typical hazards and industry baseline risk levels, are available in the recently published ACRP Report 131 and were not considered in the survey. Figure 9 summarizes the responses to question 14. Some of the tools and templates identified by the respondents appear in the appendices of this report. One airport indicated the need to develop an electronic risk register to help the airport keep track of hazards and risk control actions. Many of the airports were interested in developing electronic

21 tools (e.g., electronic worksheet) that could be helpful in running the SRA exercise. One of the con- sultants reported using the FAA internal SMS documentation (ARP SMS Desk Reference 2012) to develop similar tools that can be applied by Part 139 airports. Question 15: How do you verify that existing and new risk controls are implemented? Figure 10 summarizes the responses to question 15. It is clear that this is a major deficiency that Part 139 airports need to overcome. Many respondents cited the CSPP as the main tool to ensure that risk controls will be in place; however, it is important to note that the CSPP describes only the contractor responsibilities, and other stakeholders may be in charge of other measures. Also, even if the CSPP documents those risk controls, the airport can oversee those measures to make sure they are imple- mented and that the outcome is effective to reduce risks. FIGURE 8 Basic SRA tools (Source: ASM Consultants). FIGURE 9 Additional SRA tools (Source: ASM Consultants).

22 The most common response for verifying implementation was Part 139 airport inspections by operations staff. One of the airports developed a special form to track risk mitigation actions; it appears in Appendix C. Question 16: What was the total duration of the SRM panel meeting(s)? The time required to complete an SA and SRA and the additional workload by the stakeholders involved has been a major concern by Part 139 airports. Comparison between Part 139 and foreign airports indicate similar time spent with SRAs. As expected, as the complexity of the safety issue increases, the time it takes to perform the risk assessment increases proportionally. Foreign airports tend to involve fewer people to prepare a preliminary risk assessment that is discussed during a regular safety meeting for the given project or change. Some foreign airports commonly establish safety working groups for projects and have regular periodic meetings. These meetings provide the opportunity to discuss new hazards and track the status of risk mitigation actions in a continuous process that appears to be quite effective. Based on the small sample of military organizations, risk assessments take from 2 to 4 hours to complete; however, the sample of military organizations is too small for a strong inference. The military organizations use ORM standard procedures and have gained strong experience with this approach. Figure 11 summarizes responses to question 16. Question 17: Based on the SRAs and SAs completed at your airport, please indicate the approximate number of participants in the panel meetings. As expected, there is a wide range of panel sizes conducting SAs and SRAs, and results were similar for both Part 139 and foreign airports. Panels ranged from five to 25 people for most assessments. However, for the foreign airports surveyed, only one or two people are responsible for developing a preliminary risk assessment before it is discussed with the panel. The maximum number of people convened was 50, at a foreign airport. For military organizations, an average of nine people compose the SRM panels—once again, this inference is based on a small sample size. Limiting the number of panel participants was also a challenge, as most airport stakeholders can be impacted by changes in the airfield and there has been significant interest in participating in SRM FIGURE 10 Safety assurance of risk mitigation actions (Source: ASM Consultants).

23 panels to gain experience. Some airports highlighted the importance of balancing the need to identify as many hazards as possible with the need to keep the exercise as practical and fast as possible. Some consultants indicated the need to split a large group into smaller groups to keep facilitation under control and make the brainstorming more effective. Question 18: How did you avoid long SRM panel meetings? A summary of responses to this question appears in Figure 12. Airports with mature SMS and the military organizations make extensive use of preliminary risk assessments and premeeting informa- tion circulated among stakeholders and panel members. Preliminary information has proved to be effective, particularly when enough preparation time allows panel members to have internal discussions within the subgroups and may prevent longer discussions during the brainstorming meeting. The military organizations use standard procedures FIGURE 11 Duration of SRAs (Source: ASM Consultants). FIGURE 12 Techniques to control SRA duration (Source: ASM Consultants).

24 and are able to complete risk assessments within 2 to 4 hours, based on the small sample of air force bases evaluated. It is important to remember the response by a consultant to question 12, who reported having a preliminary brainstorming with a smaller group in preparation for the SRA with all stakeholders involved. The preliminary assessment of hazards saves time and allows structuring the brainstorm- ing session for more effective results. However, another airport reported avoiding using preliminary lists of hazards or even preliminary assessments because the outcome of the SRA may be biased and some important contributions to the brainstorming process may be lost. Another important help to limit the duration of an SRA is to have a skilled and experienced facilitator. Splitting the attendees into meetings of smaller groups before a final meeting is convened with repre- sentatives of all stakeholders also was believed to be an effective way of managing an SRA’s duration. Question 19: Some risk mitigation actions have not been implemented because_______: Responses to this question are summarized in Figure 13. Most organizations, particularly the mil- itary, will implement every risk control established in the SRA. Some foreign airports have not implemented actions because of the high cost involved. Also, some Part 139 airports have found it challenging to coordinate and assign responsibilities when external organizations are involved, particularly when the actions may impact that organization’s operations. One airport indicated that a CIP had to be postponed because risk control actions would affect the airport capacity. In addition, with limited experience in the United States, many stakeholders assume that the SRA exercise is complete upon approval of the report. In reality, it is necessary to monitor hazards and risks continuously, during the entire duration of the project or change. Some Part 139 and foreign airports have developed special checklists for tracking and oversight of risk mitigation actions. When there are significant changes in conditions of the system evaluated in the SRA, the impact of those changes can be evaluated and a corrective action plan can be devel- oped, if necessary. Foreign airports reported that it is harder to ensure implementation of risk mitigation actions assigned to external airport stakeholders, simply because these actions are not under the control of the airport or may impact the stakeholder’s operations. FIGURE 13 Reasons for not implementing risk control actions (Source: ASM Consultants).

25 Question 20: The Safety Risk Management (SRM) process is composed of steps that are followed according to the methodology applied (e.g., FAA-led SAs use five steps). Which of the steps used in your SRA was the most challenging? As shown in Figure 14, the greatest challenges are associated with the analysis and assessment of risks. Many airport staff find the process subjective and apt to lead to a broad range of opinions. However, the main challenge is making sure that risk mitigations will be implemented and monitored. Some effective techniques to track risk control actions are discussed in chapter seven. A few Part 139 airports have faced difficulties with risk identification; however, improvement is expected as the SRM process becomes part of the routine work and airport staff becomes familiar with risk assessments. There is still some misunderstanding on the differences between hazard, cause, risk, and effect. ACRP Report 131 clarifies these differences, and it is suggested to include these definitions with examples in a Quick Reference Guide for SRAs, as presented in Appendix D. Another challenge that was reported is the delay or difficulty of getting approval by upper man- agement for implementation of certain risk control actions involving higher costs or workload. Question 21: The SRA outcome was________: A large majority (88%) of organizations and consultants surveyed found beneficial results from SRAs. Risk assessments have helped those organizations safely implement projects and changes. Some for- eign airports with mature SMS have reported monetary savings from avoiding accidents and reducing insurance costs. Foreign airports that have implemented SMS in the last decade use safety assessments on a routine basis because they find value in systematically controlling safety risks. Question 22: Please describe any lessons learned when convening SRM panels that may be helpful to other airports. A wide range of answers were obtained from this question and are summarized in the following lists: Challenges • The logistics of the meetings was difficult and time consuming to handle. • Effective reporting and approval of report FIGURE 14 Main challenges in the SRM process (Source: ASM Consultants).

26 • Workload required to complete SA and SRAs • Difficulty to facilitate brainstorming with large number of participants • Unable to train staff in facilitation techniques • There is little information available about typical industry risk levels. • How should airports deal with construction changes? Is a new SRA required? Lessons Learned—Planning SRAs • Circulate relevant information prior to SRA meetings. • Provide background information and briefing to panel participants on the SRA process. • Get decision makers to participate in meetings. • Having facilitators with expertise and skills is key for successful SRAs. • Establish working groups for CIPs. • Gain experience sending observers for SRAs at other airports. • Ensure participation of construction management in the SRM panel and working group to dis- cuss potential changes to the project. • Bring pilots and controllers because they can bring valuable contribution to the SRA. • Avoid gaps in key experience; operational or risk analysis and facilitation experience alone provides ineffective assessments; combination of experiences is essential. • Involve competent people who are familiar with the activities involved. Lessons Learned—Brainstorming • Encourage participation of stakeholders impacted by safety issue. • Offer lunch as an SRA break. • Restate key definitions like hazards, risks, and consequences. • Use as many illustrations as possible during the briefing to characterize the safety issue. • Prepare illustrations to explain complex hazards. • Have the airport manager welcome panel participants during the briefing to enforce the level of importance of the SRA. • Do not completely assign responsibility for risk control actions to manager level; tasks should be assigned to supervisor or team leader levels. • Find the correct level of detail; keep it as simple as possible but do not over simplify. Lessons Learned—SRA Reporting • Keep reports concise and meaningful to management and decision makers. • Make risk mitigation guidelines available to the industry. • Write SRA reports to “non-specialists.” • Complete the SRA report as soon as possible and circulate while the panelists have recent memory of discussions. • Develop an objective plan for implementation and monitoring of risk mitigation actions. CONCLUSIONS OF SURVEY Based on the analysis and descriptive statistics of responses, it is possible to infer these key conclu- sions from the survey. • It is evident that those Part 139 airports that have taken the initiative to lead their own SRAs are having some difficulties finding airport-specific training programs on SRA planning, facilita- tion, and reporting. For the time being, consultants and FAA personnel primarily provide plan- ning, facilitation, and reporting services for SAs and SRAs. • A large majority of airports found that SRAs are beneficial; those Part 139 airports with more mature SMS and foreign airports that have implemented SMS for more than 10 years use SRAs on a routine and voluntary basis, and as a requirement under their internal SMS program.

27 • Many of the airports surveyed indicated the benefits of having an experienced facilitator guid- ing the risk assessment process, supported by SRM panels; hazard identification and risk assessments can be more effective. • Two of the main concerns reported by airports were SRA scheduling and the duration of the brainstorming session. Sometimes it is difficult to schedule participation of key people, and the SRA process demands a significant amount of time. • Some airports have confidence issues using subjective risk assessments with the help of a risk matrix; these airports are seeking additional tools, guidance, and training to conduct effective SRAs. • A major deficiency of the SRA process, reported by Part 139 and foreign airports, is the safety assurance of risk control actions; airports are still seeking more effective means to ensure that risk controls are implemented and monitored upon completion and approval of the SRA report.