Cover Image

Not for Sale



View/Hide Left Panel
Click for next page ( 129


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 128
128 Safety Management Systems for Airports agreement between the airport and the stakeholder to permit a safety assessment of the stake- holder unit at the airport. Some questions to consider might include the following: Are there provisions in leases/operational agreements to ensure the management of safety issues? Do stakeholders follow airport rules, regulations, and procedures? Do they keep safety records? Are they complete, accurate, and used to track progress? How well are they performing? Do they participate in airport safety meetings? Do they have their own safety meetings? Do their service providers have specific safety training? Do they report accidents and incidents? Are their employees aware of how to report safety issues? Do they have appropriate resources to operate? Is their equipment well maintained? Interfaces In this context, an interface is the boundary between two (or more) of the functional areas where they interact. Some of these interfaces are readily apparent in some of the previous ques- tions, for example: Can your staff use your assets while wearing PPE? This is an example of the interface between occupational safety and health, assets, and people. Do the procedures that they use apply to all weather conditions? Is their training program aligned with the department needs? Do their current schedules provide for supervision during all shift work? 6.7 Measuring SMS Performance--Trend Analysis Importance of Trend Analysis The causal factors of accidents and incidents, and the effectiveness of corrective actions at an airport can be fully understood and evaluated only if their frequency and distribution are exam- ined in terms of type of event, location, and conditions. Trend analysis is one leg of this analytic triangle. It is used for hazard surveillance and monitoring, forecasting, program evaluation, pol- icy analysis, and the investigation of potential causal relationships between risk factors and out- comes. Trend analysis is used for the following objectives: To identify the overall pattern of change in a safety performance indicator over time (increase or decrease, rate of change) To compare one time period with another (effectiveness of operations before versus after a risk control action or the implementation of new regulations) To compare different airport areas or seasonal differences (level of safety for Apron A and Apron B; accident rates in summer versus winter) To compare two or more groups (trained versus untrained, different service providers) To make future projections (monitor progress toward a safety objective; provide an estimate of the rate of future occurrence) What Is Trend Analysis? Trend analysis looks for changes in safety levels over a given time period (e.g., the last 12 months). The safety level is usually measured using key performance indicators selected by the airport.

OCR for page 128
SMS Operation 129 If the airport wants to measure the level of safety at the ramp, a simple safety performance indicator in this case may be the number of accidents that have occurred on the ramp in any given month. Checking if the number of accidents is increasing or decreasing over the months is trend analysis. The objective of trend analysis is to Determine the "safety health" state of the airport, Analyze trends to determine if things are Identify trends in safety levels, and getting better, getting worse, or staying Identify needs and determine actions required the same. to maintain and/or improve safety. Management could have set a goal to reduce ramp incidents by 20%. This statement is appropriate at the organizational level, but before ramp oper- ations can act on it in a meaningful way, it needs to identify at least two things: (1) the baseline (i.e., how many incidents are occurring now) and (2) which activities are having an effect on apron incidents (passenger bridge, service equipment-aircraft, vehicle-vehicle, etc.). This knowledge comes from trend analysis. If you do not have the necessary data, set up a method (as part of your reporting system) that will allow you to collect data relevant to those activities. Once you start collecting the data, you will know how many incidents are occurring. You can then set a baseline against which you can compare future performance. You should be careful when using the raw number of accidents or incidents to verify trends. The number of undesirable events is related to the number of operations and a drop in the num- ber of incidents may be a consequence of a decrease in activity level. You may prefer to use acci- dent rates instead of the number of accidents. An accident rate is the number of accidents divided by a fixed number of operations over a period (e.g., 10,000 operations in 1 year). A decrease in the rate of accidents is a clear and reliable picture of safety improvements at your airport. Analysis of the data collected should allow you to identify which activities have the most impact in the incident count. This information should help you focus on the major problems and develop action plans to reduce them. At the same time, it will allow you to track whether these plans are effective. Assume that the data collected show that, after 6 months, bridge-aircraft incidents add up to eight events. After analysis and investigation, you decide that implementation of a stricter train- ing program and spot checks should improve airline employees' skills and compliance with the rules, thus reducing the number of bridge-aircraft incidents. Six months later, when you do your next review, you should be able to see if the number of incidents has gone up or down. If the number of incidents has not decreased, additional actions will be required (e.g., intensifying supervision and enforcement of SOPs) to mitigate the likelihood of more incidents. Measurements must focus on factors that are related to overall safety objectives. Most impor- tant, measurements must be related to the most significant risk contributors. Although per- formance measurements include incident statistics, they are primarily intended to be proactive (i.e., identify problems before an accident occurs).

OCR for page 128
130 Safety Management Systems for Airports Some representative performance indicators can be as follows: Number of airside vehicle operation infractions per month Number of FOD reports Number of bird strikes Number of runway incursions Percentage of employees with basic SMS training Collecting information on the number of violations should be done carefully to avoid bias. If you do not have enforcement officers on the ramp, the numbers may actually go down, even when more incidents occur. For larger airports, performance indicators may be related to specific areas of the airport, for example: Number of airside vehicle operations infractions per month on Apron C Number of runway incursions at a specific "hot spot" Data Considerations For SMS, most trend analysis data are related to the key safety performance indicators selected for monitoring. The airport needs to make sure the data quality objectives are met. Trend analy- sis requires strict monitoring protocols. If the airport wants to determine the trends for apron accidents, it should make sure reliable data are gathered every month over a significant period (e.g., 3 to 5 years). It is essential to identify the audience and the type of analysis or presentation appropriate for that audience. For example, if it is necessary to report the trends to raise safety awareness at the ramp, the audience will be every person working on the ramp. In this case, the message should be very simple. Determining the cause of a trend is more difficult than determining the trend, but it is the crit- ical element for defining effective corrective actions. To identify causal factors for certain trends, it is necessary to consider all the exogenous parameters that can influence those trends (volume of traffic, environmental conditions, time of day, etc.). The following are some important issues to consider when developing a trend analysis: Consistent data quality--Trend analysis assumes that the same or equivalent methods and protocols are used for all the monitoring. Time frame and number of samples--5 years of monthly data for accident and incident trend analysis; for step trends, at least 2 years of monthly data before and after a major change (e.g., new SOP, change in personnel, application of a corrective action, change in organization). Seasonality--Parameters that vary naturally in different seasons of the year may require spe- cial statistics (e.g., certain types of birds can be more frequent in specific months of the year; apron accidents are more prone to occur under low visibility conditions).

OCR for page 128
SMS Operation 131 Statistical significance of a trend--It is common to observe upward or downward trends that simply occur by chance (see an example in Figure 22). This is particularly frequent for per- formance measurements over small periods or associated with low incidence events (e.g., number of runway incursions). The best way to check if the trend is real or is simply random variation is to set longer periods for evaluation to collect more data and use statistical tech- niques to check if the differences are statistically significant. Gathering Data After the airport safety objectives and goals are set, it is necessary to define the safety perfor- mance indicators that will be used to measure performance and check if the objectives are being met. Trend analysis is based on quantitative information collected for these performance indi- cators (e.g., number of incidents at the ramp, number of bird strikes). The airport operator who records each of these accidents will have a number for each month over a period of a few years. Other sources of data available for trend analysis are conclusions from accident/incident reports. The root causes identified in the investigations can provide valuable information on spe- cific safety issues that should be addressed by the airport. Data collection is a critical activity for trend analysis. It is recommended that a written proce- dure be established to define who will collect the data, the means for collection, specific proce- dures, and who will receive the information. Table 25 depicts an example of information on safety performance indicators over a 6-year period, and Table 26 shows an example of data gathered on the number of bird strikes in each month. Table 27 contains the number of job-related injuries at the ramp and the number of operations at the airport during 9 months. Very small operators may not generate enough data to allow for the monitoring of any trends. In these cases, a possible alternative is to look at the industry as a whole to help identify possible trends. Reviews of accident reports, articles in trade magazines, discussions with industry groups, participation in industry associations, and other operators can provide useful data. Table 25. Number of ramp accidents (example). Year Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total 2003 3 0 5 4 1 2 7 1 2 0 3 1 29 2004 1 4 2 2 1 5 6 4 0 2 1 4 32 2005 2 0 1 1 0 2 4 1 2 0 2 1 16 2006 0 1 2 1 0 0 3 1 0 2 0 1 11 2007 1 0 0 1 0 1 2 0 0 1 1 1 8 2008 0 0 1 0 1 0 2 0 0 0 1 0 5 Table 26. Number of bird strikes (example). Year Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total 2003 0 1 2 0 3 0 2 2 1 3 1 2 17 2004 1 0 3 0 0 1 1 2 0 2 1 1 12 2005 0 0 1 0 2 0 0 0 1 0 1 0 5 2006 1 0 0 2 1 1 0 1 0 1 0 0 7 2007 0 2 0 1 2 0 2 0 1 0 1 1 10 2008 1 0 2 0 1 1 0 0 3 1 2 0 11

OCR for page 128
132 Safety Management Systems for Airports Table 27. Number of job-related injuries at the ramp. Month Total Number of Total Number of Rate of Injuries per People Injured Aircraft Operations 10,000 operations Jan 6 6648 9.03 Feb 5 5180 9.65 Mar 4 4895 8.17 Apr 3 3650 8.22 May 4 5055 7.91 Jun 5 6230 8.03 Jul 6 8350 7.19 Aug 7 10640 6.58 Sep 8 10910 7.33 Measuring Trends There are two methods to determine trends: visual and statistical. As shown in Figure 20, graphing or mapping data for people to see is the easiest way to communicate trends, especially to a non-technical crowd; however, this method does not allow one to "measure" that there is a trend or how big it is. Even when a statistical analysis is performed, it should start with the visual assessment of trends. Another example is depicted in Figure 21. In this example, the root causes of runway incur- sions by type of vehicle over a period of 1 year are counted. It can be noted that the operation of emergency/snow removal vehicles was the main activity associated with runway incursions, and the airport should take actions to provide additional training to those drivers. In addition, based on the high frequency of incidents associated with communication procedures, the airport may decide to provide additional training on radio communication, replace existing equipment, and establish procedures for escorting contractors. The statistical method, on the other hand, can identify hard-to-see trends and can give a num- ber that is defensible and repeatable; however, such methods can be difficult and challenging to apply. A statistical analysis is best performed by a qualified person. Trends for Safety Performance Indicators 35 32 30 29 Number of Occurrences Ramp Accidents 25 Bird Strikes 20 17 16 15 12 11 11 10 10 7 8 5 5 5 0 2003 2004 2005 2006 2007 2008 Year Figure 20. Plotting for visual analysis (example).

OCR for page 128
SMS Operation 133 Root Causes of Runway Incursions by Vehicle Drivers in 2007 9 8 Number of Occurrences 7 6 5 4 3 2 1 0 Operation of Unfamiliarity with Lack of proper Communication Driver was emergency/snow airport phraseology or equipment failure distracted removal vehicles (contractor) equipment Figure 21. Runway incursions in 2007 (example). For most cases, a visual analysis with some very simple statistical measuring will be sufficient for evaluating most safety performance indicators used in airport SMS. Plotting and Smoothing Spreadsheet software like Microsoft Excel can be used to plot information on safety perform- ance gathered by the airport. Usually, the information is plotted over the period that the data were collected, sometimes called time series. When evaluating plots, care must be exercised regarding the following issues: People tend to focus on the extreme values and not on more subtle changes Gradual trends are hard to detect by visual analysis Seasonal variation and exogenous variables can mask trends in a parameter Viewers can "see what they want to see" sometimes Finding no trend may only mean the data were insufficient or the type of plot was inappropriate Using the example data for bird strikes in Table 26, the plot in Figure 22 can be generated. Obviously, it is very difficult to state any conclusions from this plot. However, by using the average number of bird strikes for the last 12 months instead of using the raw number of bird strikes in any given month, a trend may become more evident. Moving averages can be used to smooth out short-term fluctuations, thus highlighting longer- term trends or cycles. The threshold between short term and long term depends on the factor being analyzed, and the parameters of the moving average should be set accordingly. For the previous example, the average number of bird strikes for Jan 2003 is computed by averaging the results from Feb 2002 to Jan 2003. For Feb 2003, the value is the average from Mar 2002 to Feb 2003, and so on. Note that there are no data for the first 11 months simply because there is no information available to compute the average for the past 12 months. The resulting

OCR for page 128
134 Safety Management Systems for Airports Number of Bird Strikes per Month 3.5 3 Number of Accidents 2.5 2 1.5 1 0.5 0 Se 3 Se 4 Se 5 Se 6 Se 7 Se 8 M 3 M 4 M 5 M 6 M 7 M 8 03 Ja 4 Ja 5 Ja 6 Ja 7 08 -0 -0 -0 -0 -0 -0 0 0 0 0 0 0 0 0 0 0 n- n- n- n- n- n- p- p- p- p- p- p- ay ay ay ay ay ay Ja Ja Month Figure 22. Number of bird strikes per month. trend for bird strike data is depicted in Figure 23. In this case, the frequency of bird strikes was decreasing until the beginning of 2006, when it started to increase. Using Accident and Incident Rates The examples presented so far have used raw data only. However raw data can sometimes be misleading because the number of accidents and incidents tends to have a relationship with the volume of operations or the intensity of activities. Average Number of Bird Strikes for Previous 12 Months Average Number of Bird Strikes 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 Se 3 Se 4 Se 5 Se 6 Se 7 Se 8 M 3 M 4 M 5 M 6 M 7 M 8 Ja 3 Ja 4 Ja 5 Ja 6 Ja 7 08 -0 -0 -0 -0 -0 -0 0 0 0 0 0 0 0 0 0 0 0 n- n- n- n- n- n- p- p- p- p- p- p- ay ay ay ay ay ay Ja Month Figure 23. Bird strikes trend using a 12-month moving average.

OCR for page 128
SMS Operation 135 The example data given in Table 27 puts the problem in perspective and helps your under- standing. As noted in that table, the number of job-related injuries at the ramp dropped from January to May; and after June the numbers started to rise. The trend is shown in Figure 24. However, the volume of operations changed significantly during this period. One can expect to have a higher number of accidents if more operations are carried out at the airport. If, instead of raw numbers, the rate of accidents per 10,000 operations is used, a different trend can be observed. The rate can be calculated using the number of accidents in a given month and the number of aircraft operations for that same period, using the following equation: AccRate = TNAcc 10,000/TNOps Where: AccRate is the rate of accidents per 10,000 operations TNAcc is the total number of accidents in a given period TNOps is the total number of aircraft operations during the same period For example, using the data from Table 27, in January: AccRate = 6 10,000/6,648 = 9.03 injuries per 10,000 operations The fourth column in Table 27 depicts the accidents rates for the example data provided. When plotting the accident rates, instead of the raw data, the trend shown in Figure 25 is observed; the rate is clearly decreasing. Trend Analysis Reports As a minimum, the following information should be reported for each trend analysis: Display plots of the observed data over time Comments in narrative form on the stability of the rates and approaches used to improve it Report average percent change for periods when the rate is fairly constant Interpret in narrative form the trend and how it relates to achieving the safety objectives of the airport Number of Job-Related Injuries 9 8 Number Of Accidents 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 Month Figure 24. Trend for job-related injuries at the ramp.