Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
32 The improved risk models were validated by comparing the results of the analysis for a sample of airports to their as- sociated historical accident rates. The eight airports listed in Table 7 were selected using random stratified sampling tech- niques to run the analysis with the new models and analysis software; the results are compared to the actual rate of acci- dents at the selected airports. None of the selected airports were part of the NOD used to develop the risk models. The analysis runs with the eight airports also served to test the software. To run the risk analysis, one year of historical operations data were obtained for each airport. Data for air- ports were collected and consolidated. Operations data were retrieved from the FAA Operations & Performance Data and Aeronautical Information Management Laboratory. The weather data were obtained from the National Oceanic and Atmospheric Administration (NOAA) database for the mete- orological stations serving each airport. Historical accident and incident information for the airport was obtained from the NTSB, AIDS, and ASRS databases. Although the analysis was conducted to obtain risk assess- ment estimates, information on frequency calculated in the analysis also was used to compare expected and actual fre- quency rates for each type of incident. Similarly, actual and estimated accident rates were compared to evaluate the need to make adjustments to the models. Table 8 presents the rel- evant accidents and incidents identified for the eight airports selected. RSAâs and obstacles were characterized using satel- lite pictures from Google Earth, and RSA files were created for each runway. Relevant traffic volume information from 1981 to 2009 was estimated from information available in the FAA Air Traffic Activity Data System (ATADS). Part of the annual air traffic volume was extrapolated to estimate the total volume for the sample period. An average annual growth rate of 5% was assumed for air traffic in the period between 1981 and 1999 when air traffic information was not available online. The volumes were adjusted to remove operations of aircraft with less than 5600 lb and other movements non-relevant to this study. The volume and the number of accidents and incidents were used to estimate the frequency rates and accident rates for each airport and type of event. The analysis software proved to work well for each case study. There were no bugs identified during the software runs, and the results looked rational and within expected ranges for the individual airports. Table 8 contains the events for each airport occurring dur- ing the analysis period. Figure 40 summarizes the total num- ber of accidents and incidents occurring at the eight airports since 1981. The majority of the cases were landing veer-offs, and, for most types of events, the number of incidents was larger than the number of accidents. One notable exception was the case for TOORs. It is true that the number of cases is quite small for a sample of eight airports; however it is notable that there were fewer TOOR incidents compared to accidents. Approximately 50% of TOOR in the accident/incident data- base developed for this study were incidents, and it may be an indication of higher severity for TOOR. When comparing the location models for TOOR and LDOR, the percentage of air- craft stopping at any given distance is larger during the takeoff, compared to landing overruns. A summary of analysis results is presented in Table 9. More details on the analysis and additional results are presented in Appendix H. It is important to note that the RSA configurations used for the analysis at Yeager Airport were representative of conditions prior to the recent improvements that included the extension of RSAâs and implementing EMAS. The main reason for using these data for Yeager is that the plan was to compare the analysis results with historical accident/incident rates. As expected, risk for Yeager was the highest because its RSAs before the recent improvements were considerably smaller than current FAA standards. For simplicity, all analyses were conducted using the av- erage annual operations during the past 10 years. The ex- pected number of years between critical events is based C H A P T E R 6 Model Validation
33 State Airport Name Location ID City Hub FL Miami International MIA Miami L AK Ted Stevens Anchorage International ANC Anchorage M MO Lambert-St Louis International STL St Louis M WA Spokane International GEG Spokane S SD Joe Foss Field FSD Sioux Falls N WV Yeager CRW Charleston N AZ Deer Valley International DVT Phoenix GA FL Ft Lauderdale Executive FXE Ft Lauderdale GA Table 7. List of airports for model/software validation. Date Country City/State Source Event Type Event Class AircraftICAO Code Airport Code 07/01/1981 US Saint Louis, MO NTSB LDVO Incident DC6 STL 07/24/1981 US Charleston NTSB LDUS Accident BE60 CRW 10/15/1981 US Saint Louis, MO AIDS LDVO Incident DC6 STL 2/24/1983 US Anchorage, AK AIDS LDVO Incident LJ24 ANC 10/26/1983 US Saint Louis, MO NTSB LDUS Accident CV3 STL 12/23/1983 US Anchorage, AK NTSB TOOR Accident DC10 ANC 9/28/1987 US Saint Louis, MO AIDS LDUS Incident MD80 STL 12/26/1987 US Fort Lauderdale, FL AIDS LDVO Incident AC11 FXE 10/14/1988 US Anchorage, AK AIDS LDVO Incident YS11 ANC 10/23/1989 US Anchorage, AK MITRE LDOR Incident B741 ANC 1/6/1990 US Miami, FL 4-01NTSB TOOR Accident L29A MIA 02/17/1991 US Spokane, WA NTSB LDVO Accident MU2B GEG 03/11/1993 US Saint Louis, MO NTSB LDVO Accident DC93 STL 8/28/1993 US Fort Lauderdale, FL AIDS LDVO Incident LJ23 FXE 08/29/1993 US Charleston NTSB LDOR Accident MU2B CRW 7/27/1994 US Sioux Falls, SD AIDS LDVO Incident T18 FSD 11/29/1994 US Spokane, WA AIDS TOVO Incident B731 GEG 06/23/1995 US Miami, FL NTSB LDVO Accident C402 MIA 11/19/1995 US Anchorage, AK AIDS LDUS Incident C441 ANC 12/19/1995 US Saint Louis, MO AIDS LDVO Incident DC91 STL 9/17/1996 US Miami, FL AIDS TOVO Incident BE18 MIA 11/15/1996 US Sioux Falls, SD MITRE LDOR Incident DC91 FSD 8/7/1997 US Miami, FL 4-01NTSB TOOR Accident DC85 MIA 2/19/1999 US Miami, FL 4-01ASRS LDUS Incident A30B MIA 10/15/2000 US Anchorage, AK NTSB TOOR Incident B741 ANC 10/16/2000 US Saint Louis, MO AIDS LDVO Incident MD80 STL 10/20/2000 US Saint Louis, MO ASRS LDOR Incident MD82 STL 04/07/2001 US Anchorage, AK AIDS TOVO Incident B190 ANC 01/01/2002 US Miami, FL 4-01NTSB LDOR Incident MD83 MIA 06/15/2002 US Fort Lauderdale, FL AIDS LDVO Incident SW3 FXE 12/01/2002 US Spokane, WA AIDS LDOR Incident DH8A GEG 12/20/2002 US Spokane, WA 4-01ASRS LDOR Incident DH8A GEG 8/16/1999 US Fort Lauderdale, FL MITRE LDVO Accident CL60 FXE 4/17/2003 US Fort Lauderdale, FL AIDS LDVO Incident SBR1 FXE 6/12/2003 US Fort Lauderdale, FL AIDS TOOR Incident LJ24 FXE 8/9/2003 US Fort Lauderdale, FL AIDS LDVO Incident SBR1 FXE 2/20/2004 US Fort Lauderdale, FL 4-01NTSB LDOR Accident LJ25 FXE 3/31/2004 US Fort Lauderdale, FL NTSB LDVO Accident C402 FXE 7/19/2004 US Fort Lauderdale, FL 4-01NTSB LDOR Accident LJ55 FXE 09/08/2004 US Charleston NTSB TOOR Accident C402 CRW 12/1/2005 US Sioux Falls, SD AIDS LDVO Incident SW4 FSD 6/6/2006 US Fort Lauderdale, FL AIDS TOVO Incident SW3 FXE 2/4/2007 US Miami, FL NTSB LDVO Incident DC87 MIA 11/1/2007 US Fort Lauderdale, FL AIDS LDOR Incident GLF2 FXE 1/27/2008 US Spokane, WA AIDS LDOR Incident B731 GEG 5/23/2008 US Fort Lauderdale, FL AIDS LDVO Incident SBR1 FXE 01/19/2010 US Charleston NTSB TOOR Accident CRJ2 CRW Table 8. Accidents and incidents at selected airports.
34 on the average annual volume of operations during the past 10 years and the average level of risk for the entire airport, as shown in column 5 of Table 9. A âcritical eventâ is the focus of the analysis, and it may be an incident or an acci- dent. When running the analysis for risk, a critical event is considered a single accident or an event in which substan- tial damage to the aircraft and/or injury to passengers is the consequence. The most critical runway end is identified based on the risk of overruns and undershoots only. This risk is associated with the operations challenging the RSA adjacent to the run- way end. The runway end is identified based on the approach end of the runway. The last two columns of Table 9 contain the incident type with the highest chance of occurring and the associated runway. The validation effort was divided in two steps. The first step was to determine that the eight airports selected were representative of conditions across the United States. Al- though this is not an analysis required for validating the approach, the comparison helped gain confidence of the applicability of the risk assessment to other airports. Also, the estimated frequency rates of the airport sample were compared to the actual frequency for the eight airports. The second step was to compare the risk levels estimated from the analysis with the actual risk rates for the sample of airports. Validation of Frequency Models Figure 41 presents incident frequency rates for each type of incident and for three different estimates: the historical frequency rate in the United States, the actual incident rate for the sample of eight airports, and the estimated frequency rate for the sample of airports. The rates for the sample were calculated based on the weighted average for the eight airports. The actual rate represents the total number of in- cidents from 1981 to 2009 divided by the total volume of operations during the same period. The figure shows these results in both graphical and tabular format. Some differ- ences were expected given the small sample size of eight airports surveyed. The results presented in Figure 41 demonstrate excellent agreement between the accident rates for the sample of air- ports and the historical rate for all the airports in the United States. The results concur that the sample of airports is rep- resentative of conditions for the population of airports in the United States. The largest difference was found for landing veer-offs; however, the incident rate, particularly for Fort Lauderdale Executive Airport, was unusually high during the analysis period. The second conclusion drawn from these results is that the actual frequency rates for the eight airports agreed with the es- timated frequency rates for this sample. It is important to note that frequency rates involve both accidents and incidents, with no distinction for the level of severity. The plot in Figure 41 also shows excellent agreement between actual and estimated frequency values for each type of incident. Therefore, there is no need to recalibrate the frequency models or to apply adjustment factors. Airport Most Frequent Incident and Associated Runway *** Airport State Average Annual Volume of Ops for Past 10 yrs (in 2009) No. of Runways Avrg Airport Risk Avrg # of Years for One Accident to Occur Highest Risk Runway End** Incident Type Rwy ANC AK 293K (290K) 3 2.1E-07 16 14 LDVO 14/32 CRW* WV 50K (71K) 2 5.5E-06 17 15 LDOR 15 FSD SD 69K (86K) 2 3.1E-07 38 15 LDOR 15 FXE FL 169K (261K) 2 8.3E-07 13 31 LDOR 31 GEG WA 82K (81K) 2 4.1E-07 33 21 LDVO 03/21 STL MP 209K (226K) 4 1.8E-07 28 24 LDVO 06/24 DVT AZ 153K (376K) 2 3.7E-07 15 07L TOVO 07L/25R MIA FL 380K (384K) 4 1.4E-07 19 30 LDVO 12/30 * Risk estimated for condition before RSA improvements completed in 2007. ** Runway end with highest probability of overruns and undershoots only. *** Incident with highest probability of occurrence. Table 9. Summary of analysis results for airports selected for validation. 0 5 10 15 20 25 LDOR LDUS LDVO TOOR TOVO Type of Incident/Accident N um be r o f E ve nt s Accidents Incidents Total Figure 40. Summary of accidents and incidents at surveyed airports since 1981.
Validation of Risk Model The second part of the validation effort consisted of the com- parison of actual risk rates with those estimated for the sam- ple of eight airports. The estimated risk is associated with the likelihood of an accident, rather than an incident. According to NTSB, accident is defined as an occurrence associated with the operation of an aircraft where as a result of the operation, any person receives fatal or serious injury or any aircraft re- ceives substantial damage. This is also the definition used in this report to characterize an aircraft accident. Data presented in Table 8 contain the accidents that took place at the eight sample airports from 1981 to 2009. The ratio between the actual number of accidents in that period divided by the volume of landings or takeoffs at the airport provides the actual rate for each type of event. The total number of accidents of any type divided by the total number of operations in the pe- riod evaluated is the actual accident rate for the airport. Comparison of the actual rate for each type of accident at each airport is not very helpful because the number of events is relatively low, given the sample size of airports used in the validation. Therefore, the analysis consisted of comparing the rates for the whole sample of eight airports. The comparison is made for each type of accident and for the total accident rate. The first analysis compared the proportion between acci- dents and the total number of incidents. This was an impor- tant analysis to validate the consequence approach developed in this study. Three types of ratios were calculated for each type of accident: the estimated ratio for the sample of eight airports, the actual ratio for the sample, and the historical ratio in the United States. The results are shown in Figure 42 in both graphical and tabular form. 0.0E+00 2.0E-07 4.0E-07 6.0E-07 8.0E-07 1.0E-06 1.2E-06 1.4E-06 Type of Incident In ci de nt R at e (in c/o ps ) U.S. Historical Actual for Sample Estimated for Sample U.S. Historical 9.5E-07 2.4E-07 1.2E-06 2.4E-07 2.6E-07 Actual for Sample 5.8E-07 2.6E-07 1.1E-06 3.7E-07 2.1E-07 Estimated for Sample 1.2E-06 2.9E-07 7.0E-07 3.1E-07 2.6E-07 LDOR LDUS LDVO TOOR TOVO Figure 41. Actual frequency of incidents for sample of air- ports compared to historical rates. 35 0% 10% 20% 30% 40% 50% 60% 70% 80% Type of Accident R is k vs F re qu en cy R at io U.S. Historical Actual for Sample Estimated for Sample U.S. Historical 24.7% 48.5% 18.6% 50.0% 18.0% 25.7% Actual for Sample 27.3% 40.0% 25.0% 71.4% 0.0% 31.9% Estimated for Sample 31.6% 22.4% 25.6% 27.9% 30.1% 28.5% LDOR LDUS LDVO TOOR TOVO Total Figure 42. Percentage of accidents to the total number of incidents.
Again, the results are in excellent agreement with the excep- tion of the ratio for TOVO since none of the airports included in the sample had this type of incident. This can be attributed to chance, since the estimate is in good agreement with the his- torical level in the United States. The number of accidents is very low when using only eight airports, and larger variations were expected when comparing the parameters based on the number of accidents for the sample. The last analysis for valida- tion was the comparison of actual and estimated risk levels for the sample of airports. The results are presented in Figure 43. Again, the results between estimated and actual values are in excellent agreement. The validation study demonstrates the applicability of the approach and the models developed in this study. 36 0.0E+00 5.0E-08 1.0E-07 1.5E-07 2.0E-07 2.5E-07 3.0E-07 3.5E-07 4.0E-07 4.5E-07 Type of Accident A ve ra ge R is k (ac cid en ts/ op er ati on s) U.S Historical Actual for Sample Estimated for Sample U.S Historical 2.3E-07 1.2E-07 2.2E-07 1.2E-07 4.7E-08 3.7E-07 Actual for Sample 1.6E-07 1.1E-07 2.6E-07 2.6E-07 0.0E+00 4.0E-07 Estimated for Sample 3.7E-07 6.4E-08 1.8E-07 8.6E-08 7.9E-08 3.9E-07 LDOR LDUS LDVO TOOR TOVO Total Figure 43. Percentage of accidents to the total number of incidents.
