The current safety management process, which has evolved over the history of the aviation industry (see Figure 2-1), involves operators, manufacturers, and the FAA. Despite an outstanding safety record for the commercial aviation industry, the accident rate, as a function of departures, has improved only slightly in the last 20 years. At the same time, aviation travel (revenue passenger miles) has been increasing steadily (around 5 percent per year) thus increasing risk exposure. Public expectations regarding safety have also been increasing. It has become apparent that a significant, continuing reduction in the already low accident rate will require improvements in all aspects of the aviation industry, including the aircraft certification safety management process.
The Major Finding. The recommended safety management process should improve the ability of the FAA/AIR, manufacturers, and operators to take corrective action based on incident data—before an accident takes place—and to set priorities based on assessments of current and future risk. However, the current process is already highly effective—as indicated by the small contribution of aircraft system malfunctions to the overall accident rate—and changes to the current system must be carefully structured to avoid unintended consequences that might reduce safety in some situations.
The vast majority of aircraft that will operate during the next 10 years either have already been manufactured or will be manufactured to design specifications that have already been certificated. Therefore, monitoring the safety of operating aircraft is essential to obtaining an accurate understanding of safety levels, to detecting and resolving problems as soon as possible, and to validating airworthiness standards. Historically, improvements in standards for initial type certification have frequently been based on lessons learned from the continued airworthiness process. Consequently, making the continued airworthiness process more effective is essential to improving safety in the near term and to providing a foundation for improvements in the long term. Even though the primacy of continued airworthiness is reflected in AIR' s mission priorities, AIR's budget priorities, which are listed below, do not reflect this:
certification (which accounted for 53 percent of AIR's expenditures during fiscal year 1997)
continued airworthiness (35 percent)
rulemaking and policy development (12 percent)
Major Recommendation 1. It is critically important that the FAA and AIR conduct business in a new fashion with regard to aircraft certification and continued airworthiness. As an essential first step, AIR should revise its budget and manpower allocations to better reflect its mission priorities, which are as follows (FAA, 1998):
continued airworthiness and other activities related to continued operational safety
rulemaking and policy development
The committee examined interrelationships between incidents and accidents and the current safety management process and identified areas that could be improved. Some of the basic elements of the current process either are not fully coordinated and integrated or are duplicated elsewhere. The committee, therefore, recommends an improved top level safety management process, which is illustrated in Figure 4-1 and described below.
Major Recommendation 2. It is essential that the FAA improve its safety management process. The FAA should work with the operators and manufacturers of large transport
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service 4 Recommended Safety Management Process OVERVIEW The current safety management process, which has evolved over the history of the aviation industry (see Figure 2-1), involves operators, manufacturers, and the FAA. Despite an outstanding safety record for the commercial aviation industry, the accident rate, as a function of departures, has improved only slightly in the last 20 years. At the same time, aviation travel (revenue passenger miles) has been increasing steadily (around 5 percent per year) thus increasing risk exposure. Public expectations regarding safety have also been increasing. It has become apparent that a significant, continuing reduction in the already low accident rate will require improvements in all aspects of the aviation industry, including the aircraft certification safety management process. The Major Finding. The recommended safety management process should improve the ability of the FAA/AIR, manufacturers, and operators to take corrective action based on incident data—before an accident takes place—and to set priorities based on assessments of current and future risk. However, the current process is already highly effective—as indicated by the small contribution of aircraft system malfunctions to the overall accident rate—and changes to the current system must be carefully structured to avoid unintended consequences that might reduce safety in some situations. Priorities The vast majority of aircraft that will operate during the next 10 years either have already been manufactured or will be manufactured to design specifications that have already been certificated. Therefore, monitoring the safety of operating aircraft is essential to obtaining an accurate understanding of safety levels, to detecting and resolving problems as soon as possible, and to validating airworthiness standards. Historically, improvements in standards for initial type certification have frequently been based on lessons learned from the continued airworthiness process. Consequently, making the continued airworthiness process more effective is essential to improving safety in the near term and to providing a foundation for improvements in the long term. Even though the primacy of continued airworthiness is reflected in AIR' s mission priorities, AIR's budget priorities, which are listed below, do not reflect this: certification (which accounted for 53 percent of AIR's expenditures during fiscal year 1997) continued airworthiness (35 percent) rulemaking and policy development (12 percent) Major Recommendation 1. It is critically important that the FAA and AIR conduct business in a new fashion with regard to aircraft certification and continued airworthiness. As an essential first step, AIR should revise its budget and manpower allocations to better reflect its mission priorities, which are as follows (FAA, 1998): continued airworthiness and other activities related to continued operational safety rulemaking and policy development certification Recommended Safety Management Process The committee examined interrelationships between incidents and accidents and the current safety management process and identified areas that could be improved. Some of the basic elements of the current process either are not fully coordinated and integrated or are duplicated elsewhere. The committee, therefore, recommends an improved top level safety management process, which is illustrated in Figure 4-1 and described below. Major Recommendation 2. It is essential that the FAA improve its safety management process. The FAA should work with the operators and manufacturers of large transport
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service airplanes and engines to define and implement a proactive process that includes the following elements and tasks: Key Elements data collection database management risk analysis risk management/action monitoring effectiveness Specific Tasks Manufacturers, with the advice and consent of operators and the FAA, should define data requirements and processes for sharing data. Comprehensive FOQA (flight operations quality assurance) systems similar to BASIS (British Airways Safety Information System) should be used as a starting point. Operators should provide required data, as agreed upon. Manufacturers should solicit data from additional sources, such as the NTSB, ICAO, and National Aeronautics and Space Administration, to augment the operational database. Manufacturers, with oversight from the FAA and the assistance of operators, as required, should collect, organize, and analyze data to identify potential safety problems. Manufacturers should recommend corrective action for potential safety problems and seek consensus by operators. The FAA should make sure that actions proposed by manufacturers and operators will be effective, making regulatory changes and mandating compliance, as appropriate. Manufacturers and operators, with oversight from the FAA, should monitor the effectiveness and timeliness of corrective action and the safety management process (see Figure 4-1). The thrust of this recommendation is that industry should collect, organize, and analyze safety data and take appropriate corrective action to protect the safety of the fleet. The FAA should not independently collect, organize, or analyze safety data for large transport aircraft. Instead, the FAA should oversee the entire process, providing direction, assessing the accuracy and objectivity of industry's risk analyses, and mandating corrective action, as appropriate. The overall objective is a more effective safety management process that routinely monitors operations and maintenance, uses data on incidents and other abnormalities to identify potential hazards proactively, and takes corrective action before those hazards cause an accident. Many systems are currently used by industry and the FAA for generating and collecting data. Because most of these systems are not coordinated, however, there is a good deal of duplication, and much of the data cannot be used effectively. These systems consume scarce resources that could and should be applied to other safety efforts. The recommended safety management process would greatly reduce the number of systems and improve coordination among those that remain. FIGURE 4-1 The recommended process for aircraft certification safety management. The value of some databases is limited because of poor data quality and difficulties in interpretation. For example, although accidents and incidents are caused by a chain of events involving many cause factors (see Chapter 3), most data collection and monitoring systems are not formulated to identify hazards that may arise from unusual combinations of factors that may not individually present a significant hazard. Operators and manufacturers have much greater access to, knowledge of, and experience with their aircraft than the FAA. In addition, they may already have systems in place to collect the data needed for comprehensive safety management. A missing element in the current safety management process is a widely accepted risk analysis system or methodology, which is necessary to allocate resources appropriately and to define timely and effective corrective action. The process recommended by the committee would correct this problem. A credible and effective safety management process must accurately monitor, measure, and communicate the effectiveness of corrective action. This is especially important for
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service overcoming industry doubts about proposed actions that would disrupt airline operations or reduce the profitability and competitive standing of operators or manufacturers. Accurate information on the cost and effectiveness of remedial action would also put the FAA in a better position to justify its own priorities and allocate more resources to activities with the most potential for improving aviation safety. The recommended process could be implemented solely through regulatory changes that would require industry compliance. However, this approach would delay implementation for years because of the time it takes to make regulatory changes (see Chapter 6) and because of industry concerns that such changes would impose upon them an unproven system that would increase their costs and regulatory requirements. Therefore, the committee recommends that the process be implemented voluntarily, as much as practical. This would allow detailed procedures to be tested and improved while regulatory proposals are being formulated. A voluntary process would also help build the trust necessary for increased sharing of safety information, which will be essential to maximize the effectiveness of the recommended process. In developing its recommendations, the committee considered several possible approaches to improving aviation safety. For example, the current safety management process, which has achieved an excellent safety record, could be continued with only minor changes. However, based on recent trends in the accident rate, this seems unlikely to achieve the desired safety improvements. Another possibility would be to have a single organization collect and analyze safety data for all civil aircraft instead of sharing this responsibility among airlines and aircraft manufacturers, as recommended by the committee. Assigning this task to one organization, however, would separate the analysis function from the manufacturers, who have the most detailed insight into the design of their aircraft. In addition, establishing and maintaining a new organization to take on this massive task seems less practical than enhancing manufacturers' current capabilities. Manufacturers and operators already share a great deal of information derived from safety analyses, and comprehensive, widely accepted FOQA systems, such as BASIS, are increasing the flow of safety data and information. However, it seems unlikely that manufacturers would agree to share their proprietary design data, which an outside organization would need to conduct detailed safety analyses. The committee also concluded that mandatory implementation of the recommended safety management process is a good long-term goal, but voluntary participation is essential for near-term impact. Recommendation 4-1. In parallel with efforts to make appropriate regulatory changes, the FAA should expeditiously negotiate binding letters of agreement with manufacturers and operators to implement as much of the recommended safety management process as possible. DATA COLLECTION High quality data are essential to the effectiveness of the entire safety management process. Safety-related data are the foundation of the analytical processes used to prevent aircraft incidents or accidents. The type of data collected should be tailored to these analytical processes to ensure that enough of the right kind of data is collected and that the database management system is not overwhelmed with unnecessary data (see Appendix D). Currently, data are generated and/or collected by many organizations (airlines, manufacturers, regulatory agencies, pilots, repair facilities, investigative agencies, independent agencies, and others). More than 80 large databases of aviation safety data are being used worldwide, some of which are mandated by regulatory authorities. Most of the data, however, are collected voluntarily by industry or government organizations because of their interest in aviation safety. For example, various FAA offices maintain a variety of aviation safety databases. Maintaining the large number of current databases requires significant personnel and fiscal resources. Despite this investment, however, no existing database or collection of databases is fully satisfactory. In general, data collection efforts are fragmented, with individual efforts focused on different goals and objectives. For these reasons, current databases are unlikely to provide the comprehensive, high-quality data necessary to prevent incidents and accidents. For example, considerable data are collected by the FAA in the form of service difficulty reports from operators; pilot reports; and confidential safety reporting systems operated by the FAA and the National Aeronautics and Space Administration. However, much of this data is reported voluntarily or inconsistently, so a set of reports that seems to indicate that a particular aircraft system has developed a new problem may simply reflect a decision by one operator to start reporting a problem that has been present for some time. In this environment, developing a comprehensive and accurate understanding of system malfunctions is difficult. Yet such an understanding is essential for the development of a proactive safety management process that accurately predicts risks and identifies corrective action before an accident takes place. Government and industry are now exploring ways to construct advanced databases, such as the Global Analysis and Information Network (GAIN), which would serve as a single global database. The development of GAIN has been hampered, however, by disagreements in the global community on which data should be collected, how the data should be standardized, how they should be shared and disseminated, and so on. At best, reaching consensus on these issues will probably take several years, and implementing the agreed-upon course of action will probably take several more years. Fortunately, other options exist for making significant near-term improvements in data collection. The committee believes that the best source of most safety data is aircraft operators (including their maintenance
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service organizations, whether they are part of the operators' organizations or outside contractors). Operational (and maintenance) experience generates the most important data for safety analyses. To be complete, however, these data should be supplemented with data from regulatory inspectors, accident and incident investigators, and manufacturers' design and test engineers. The data should be made available to manufacturers and the FAA, as appropriate. The use of standardized formats for reporting data would facilitate data collection, database management, and the other elements of the safety management process. Thus, standardization is an appropriate topic for discussion between the FAA and industry. However, difficulties caused by nonstandardized data can be overcome, and implementation of the overall process should not be delayed by lack of consensus on standardized data formats. The manufacturers of large transport airplanes and engines seem most likely to be able to manage comprehensive aviation safety databases. Unlike operators or regulatory authorities, manufacturers collect data globally, which gives them a larger set of data than national regulatory agencies or individual operators can collect. Also, manufacturers have the equipment to store and analyze these data and have the most detailed understanding of their own products. Many individual aircraft manufacturers and operators are voluntarily installing QARs (quick access recorders) in new and existing aircraft. Unlike flight data recorders, QARs are usually not crashworthy. However, they can record up to 400 aircraft parameters and store data for many flight hours. The data from QARs are also easily accessible. QARs are generally used to monitor the performance of aircraft and engine systems. for operational and maintenance purposes. They can also be used to evaluate crew actions and performance. FOQA (flight operations quality assurance) programs use ground-based computers for routine analysis of operational data from QARs or digital flight data recorders (DFDRs). Like traditional flight data recorders, DFDRs provide data on aircraft and flight conditions for accident analyses, but DFDRs record much more information. FOQA programs enhance flight safety by providing more information about, and greater insight into, the total flight operations environment through automated recording and analysis of flight data. In 1991, before any U.S. airlines had established FOQA programs, the FAA sponsored a study of FOQA programs used by foreign airlines. This study determined that ''the appropriate use of FOQA data by airlines, pilot associations, and aircraft and equipment manufacturers would result in a significant improvement of flight safety by identifying operational irregularities that can foreshadow accidents and incidents'' (FSF, 1992). The FAA subsequently established voluntary pilot programs with several U.S. airlines to document the safety and cost benefits of FOQA programs, assess technology alternatives, develop guidelines for FOQA programs in the United States, and address organizational strategies for the use, protection, and management of FOQA data and information derived from that data. BASIS (the British Airways Safety Information System) is a comprehensive FOQA program. In addition to automated flight data, BASIS collects data from engineering reports, incident and accident reports, maintenance human factors reports, and flight crew human factors reports. The large pool of data submitted by participating airlines improves the capability of analyses to correlate data and identify rare phenomena that may have gone undetected in the past until they caused a serious incident or accident. Because the data collected by BASIS comes from many different types of aircraft, BASIS and systems like BASIS can also facilitate the exchange of lessons learned among manufacturers. This is important when a problem is relevant to aircraft produced by more than one manufacturer. Analysis of data by BASIS considers cost and risk factors and produces targeted reports for flight crews, engineering organizations, maintenance organizations, regulators, and others. Although manufacturers are not directly involved in the analysis of data by BASIS, having access to a comprehensive data collection system like BASIS helps manufacturers improve the efficiency of their own data gathering efforts. Airbus and more than 100 airlines worldwide participate in BASIS to varying degrees, making it the largest such program currently in use. Boeing is also interested in obtaining data from BASIS. In addition to BASIS and the pilot programs sponsored by the FAA, some airlines have implemented their own FOQA programs. The committee's recommendation to implement BASIS-like FOQA systems acknowledges BASIS's record as a widely accepted and comprehensive program. However, this recommendation is not an endorsement of BASIS over other similar systems, none of which the committee examined in detail. Recommendation 4-2. As the recommended safety management process is implemented, the FAA should eliminate internal efforts to collect and store data for aircraft manufactured by companies with whom agreements have been reached in accordance with Recommendation 4-1. Resources currently used for those purposes should be redirected to AIR's other safety-related functions. DATABASE MANAGEMENT Database management systems (DBMS) are computer-based systems used to store, manage, retrieve, and update data that is stored in a database. DBMS ensure the integrity of databases while allowing simultaneous access by many users. Relational, object, and object-relational databases are described below. Relational Database Management Systems Relational DBMS, which are the most widely used, are available from many vendors. In a relational DBMS, information is stored in a series of simple tables, much like a
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service series of spreadsheets. The rows in each table represent objects, such as airplanes, airports, or people. The columns in each table describe one facet of the object, for example, name, serial number, or number of engines. A relational database is a collection of one or more of these tables. Relational DBMS are relatively simple to build and are very good for traditional business applications. Standard Query Language (a computer language) is widely used for accessing information in relational DBMS. These systems work well with simple types of data and predefined operations and queries. However relational DBMS are limited and difficult to work with when the information in the tables is interrelated and when stored data includes video images, documents, pictures, and other complex data. Object Database Management Systems Object DBMS take a real-world approach to the definition and storage of data. These systems emulate the environments in which stored objects exist. For example, objects that could be part of an object DBMS for accident investigations include accidents, incidents, airplanes, and crews. Within the system, it is very easy to create and maintain connections between related objects, and most object DBMS can accommodate complex data types, such as video images, pictures, and documents. Although object DBMS are very good for storing complex objects, they are more difficult to use than relational DBMS because they usually cannot be accessed through a simple query language. In addition, these systems are not well suited to environments that have to support large numbers of users and large numbers of queries. Object-Relational Database Management Systems Relational and object DBMS both have strengths and weaknesses. Object-relational DBMS attempt to combine the best features of both. Object-relational systems can support the definition and maintenance of complex objects and complex data types, and they can provide easy access to the information using a variation of the Standard Query Language. An object-relational DBMS also has fewer performance problems than object DBMS because it can support both user and system-defined indexes to speed up transaction processing. The object-relational DBMS is the newest of the three systems described here, and only a limited number of vendors offer systems of this type. Choosing the Right Database Model It is easy to build a database that does nothing to enhance an organization. Some organizations seem to believe that simply putting information on line will solve information problems. Unless users have a clear understanding of the objectives, however, electronic information is destined to sit on a computer—unseen and unused. DBMS are tools. The first step in choosing a DBMS is to identify the most serious problems, determine what data are needed to address these problems, and determine what information will be stored. The next step is to understand the nature and capabilities of the people who will build, use, and maintain the system. Only then is it appropriate to select a DBMS. For instance, aviation safety analysis includes evaluations of a large number of specific events in various combinations and permutations to determine the conditional probability of a serious accident. Thus, a safety DBMS should have the capability to store and manipulate complex objects and data types efficiently and effectively. This is a difficult challenge. Recommendation 4-3. Manufacturers should establish aviation safety DBMS using the state-of-the-art data management technologies that are best suited to continued air-worthiness applications. The most suitable type of DBMS currently available is the object-relational DBMS. RISK ANALYSIS An effective safety management process should include risk analysis to provide a sound basis for risk management, which involves making decisions and taking action to reduce risks and, in the context of this study, to improve safety. Risk analysis involves three steps: listing possible outcomes (favorable and otherwise) estimating the consequences associated with each outcome estimating the probabilities of each outcome In this discussion, a consequence is defined as a numerical measure of the loss or harm associated with an adverse outcome. The scale chosen to measure loss or harm must allow for meaningful addition and multiplication over different events. Examples of appropriate scales are costs in dollars, losses in productivity, and reduced life expectancy. A more universal scale is utility (or disutility), which is a unitless parameter with a value between 0 and 1. The term risk means the probability that a particular adverse event (or outcome) will occur during a stated period of time or will result from a particular challenge (The Royal Society, 1992). As a probability, risk must obey all the formal rules of combining probabilities and is also subject to the vagaries of interpretation. (See Appendix E for a brief discussion of these interpretations.) Because the probability that systems will perform as expected is closely linked to their reliability and maintainability, these concepts lie at the heart of risk analysis and risk management. Failure data analysis is also important—in order to correctly interpret data on system performance.
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service Paradigm for Risk Analysis and Risk Management Risk analysis and risk management are closely linked, as illustrated in Figure 4-2. In many cases, risk management decisions are based solely on the personal judgment and expertise of responsible personnel. In other cases, the complexity of the issues and the magnitude of the potential consequences may warrant a more rigorous approach. In these situations, the decision maker may commission a risk analysis before selecting a particular course of action. Referring to Figure 4-2, suppose a management decision is needed to resolve a safety-related issue. In this case, management must select one of two possible decisions, D1 or D2. Decision D1 leads to one of three outcomes, O1, O2, or O3, whereas D2 leads to the outcomes O1, O3, or O4. The product of the probability of occurrence of each outcome and the severity of the consequences of the outcome is known as the expected utility of the outcome. The best method for maximizing total expected utility involves the use of decision trees to calculate consequences. As a hypothetical example, D1 might be a decision to install a smoke detector in the hold of all cargo planes, and D2 might be a decision not to install smoke detectors. At a top level, the possible outcomes include the following: O1—no fire in the hold O2—fire in the hold and the smoke detector functions properly O3—fire in the hold and the smoke detector fails O4—no inflammable material in the hold After the outcomes have been identified, risk analysis is used to evaluate the consequences of each outcome by calculating its utility or cost. U(D1, O3) would be the consequence of installing a detector that fails when there is a fire in the hold. In this case, the consequences of decision D1 would include the costs of acquiring, maintaining, and operating the smoke detectors, as well as the costs generated by a fire that is not quickly detected. Once the outcomes and their consequences have been defined, the next step is to determine the probability (P) that each outcome will occur. P(D1, O2) is the probability that outcome O2, a fire in the hold of an aircraft with a properly functioning smoke detector, will occur after a decision is FIGURE 4-2 A perspective on safety risk management.
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service made to install a smoke detector (D1). The probabilities are obtained via fault trees (see Appendix E) and reliability data, which are themselves based on failure data analysis tools, expert judgment, operational data, and maintenance records. Once the probabilities have been determined, expected utilities E[U(D1)] and E[U(D2)] can be computed. For example, E[U(D1)] would be the sum of the expected utilities for each outcome associated with decision D1. Based on this analysis, an informed risk management decision can be made. For the case shown in Figure 4-2, decision makers would logically choose decision D1 if the expected utility E[U(D1)] is greater than the expected utility of E[U(D2)]; otherwise they would choose D2. If E[U(D1)] = E[U(D2)], then either decision could be chosen. Risk analyses are based on quantitative procedures. However, because of the difficulty of accurately assessing probabilities and expected utilities, risk assessments today are imprecise and require some subjective inputs. Risk analyses consider how combinations of unusual circumstances could impact expected outcomes and consequences, but for a given problem, it may be difficult to define an appropriate set of abnormal conditions. In addition, judging the severity of the consequences for some outcomes may be subjective, and it may be difficult to obtain a consensus among the FAA, industry, and the general public about the severity of consequences or how to compare consequences. For example, professional pilots may believe that airlines have overestimated the costs of taking corrective action, which would lower its estimated cost-effectiveness, or decision makers and the public at large may not agree on the relative importance of expected utilities. Risk Analysis and the Air Transport Industry The committee met with aircraft manufacturers (Boeing and Airbus), airline operators (Alaska Airlines and Delta Airlines), and engine manufacturers (Pratt and Whitney and General Electric) to discuss industry approaches to risk analysis and risk management. Some committee members also have extensive industry experience. The committee concluded that industry generally does a good job of using risk analyses to identify and understand risks, although different companies use different methods. For example, manufacturers pay careful attention to engineering and operational details that affect safety and reliability. Fault tree analysis— which is widely used for reliability analyses in many other industries—was conceived in 1961 at Bell Laboratories and refined by Boeing so that the quantitative portions could be done by computer (Roland and Moriarty, 1983). Ongoing efforts by the aviation industry to improve the safety of existing and new aircraft include many quantitative techniques. The committee was unable to determine, however, the extent to which manufacturers use dependency models in their reliability assessments. The committee did not assess the effectiveness of industry processes either for demonstrating software reliability, which is a growing challenge, or for integrating individual risk analyses into a comprehensive package for risk management. The BASIS system mentioned earlier groups expected utilities into blocks, places the blocks into one of several categories, and proposes actions for each category. This approach comes closest to implementing the paradigm for risk analysis and risk management described at the beginning of this section. BASIS also attempts to describe the human factors aspects of airline operations using fault-tree-like approaches. Risk analysis and risk management activities by major engine manufacturers seem to be focused on engine reliability, which is extremely high. The Continued Airworthiness Assessment Methodologies (CAAM) is an engine reliability and failure data analysis tool used to identify and prioritize unsafe conditions. CAAM is reactive in the sense that it depends on data from incidents and other reported problems, and it cannot react to situations for which operational data are not available. More importantly, however, CAAM is proactive in the sense that it uses data from minor abnormalities to predict more serious problems. In other words, systems like CAAM that are reactive to incidents may be proactive to accidents. Overall, the committee believes that industry's risk analysis efforts, including BASIS, other systems similar to BASIS, and CAAM, have been laudable. Safety boards in individual companies formalize the safety management process, add discipline, and generate safety decisions in close cooperation with the FAA. Aircraft manufacturers have established a detailed and generally thorough process for quantifying risk, with a strong emphasis on collecting large volumes of data and heavy reliance on fault tree analysis and failure data analysis. Additional use of advanced analytical tools may improve the effectiveness of industry's analyses. For example, the committee saw little evidence that expert opinion or scientific judgments are formally incorporated into reliability assessments, and it may be possible to improve failure data analysis by relying more on scientific and engineering information to supplement operational and maintenance data. Risk Analysis and the Federal Aviation Administration Risk analysis and risk management are important tools for understanding risks, defining acceptable levels of risks, and reducing risks. Establishing consensus about the purpose, role, and capabilities of risk analysis is an important prerequisite for an effective risk analysis program. For example, some FAA officials believe that, because of its subjective nature, risk analysis should not be used to determine if a condition is unsafe and warrants mandatory action. These officials would use risk analysis only to establish a time frame for correcting an unsafe condition—not to determine
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service if the condition itself is unsafe. The committee, however, believes that proactively minimizing and avoiding the risk of serious incidents or accidents requires analysis tools, such as macrolevel fault trees, that integrate information from manufacturers and operators. Suggestions have been made both inside and outside the FAA that the FAA should adopt a more formal approach to risk management, and the FAA is currently improving its risk analysis capabilities. The committee believes that the FAA should focus its efforts on improving its ability to oversee industry's risk analysis, while encouraging industry to continue developing its capabilities. RISK MANAGEMENT/ACTION Risk management involves choosing the best combination of advantages and disadvantages from several alternatives in the presence of uncertainty. This is analogous to cost-benefit analysis in the presence of uncertain outcomes. In short, risk management boils down to choosing the best available alternative, without regard to alternatives that are not available (Derby and Keeney, 1981). The risk management/action function of the recommended safety management process is directly dependent on the risk analysis function for determining probabilities and costs. Risk management actions can be initiated by the FAA through modified regulations, new regulations, and ADs; by manufacturers through service bulletins; and by operators through operator-initiated engineering actions. AIR can initiate regulatory actions related to modifications of existing aircraft, future aircraft designs, and/or requirements for aircraft maintenance programs. The committee believes that manufacturers should use the results of risk analyses to recommend corrective action and seek consensus by operators. The FAA should make sure that actions proposed by manufacturers and operators will be effective and mandate compliance, as appropriate. The following discussion summarizes the current environment relative to ADs, service bulletins, and rulemaking and offers specific recommendations for improvement. Airworthiness Directives The AD review process is frequently and simultaneously criticized by external parties with diametrically opposed views. The media, Congress, NTSB, and other government officials often criticize the process for being too time consuming. Airlines and manufacturers often complain that the time allotted for logistical, engineering, scheduling, and maintenance actions is too short. The FAA is compelled by law and regulation to provide public notice of proposed actions, generally by publication in the Federal Register. Unless immediate action is needed to address an urgent safety issue, the FAA must provide sufficient time for public comment on the appropriateness and effects of proposed ADs and other rulemaking actions, and the FAA must complete a written review and analysis of public comments in terms of effectiveness, cost, and time. The existing process has been criticized for not being more thorough in responding to public comments, for limiting the degree to which alternative actions are considered, and for not being more accurate in estimating the time and cost for accomplishing proposed actions. Current procedures subject proposed regulatory actions to peer review, but the committee believes this peer review process should sometimes be more thorough. In addition, the accuracy of the FAA's time and cost estimates could be improved by developing models consistent with industry data and experience. 1 Recommendation 4-4. Consistent with regulatory procedures, the FAA should develop a more accurate methodology for assessing the costs and benefits of potential ADs and other rulemaking actions, as appropriate. In particular, the FAA should work with industry to develop more realistic and more reliable models for estimating time and cost. Manufacturers' Service Bulletins and Regulatory Actions by Foreign Airworthiness Authorities Manufacturer's service bulletins include procedures, lists of materials, and specifications for technical modifications or inspections to aircraft and aircraft systems. Service bulletins are developed by aircraft manufacturers and equipment manufacturers for use by airlines, repair stations, and other organizations authorized by the FAA to make the modification. Service bulletins are developed by manufacturers to improve aircraft characteristics in terms of safety, reliability, operating costs, etc. Often, the recommended timing and method of implementing a service bulletin are established with inputs from operators of the affected equipment based on their relevant service experience. If the FAA determines that implementation of a safety-related service bulletin should be mandatory, it generally publishes an AD that defines implementation of the service bulletin as the means of compliance. The FAA generally allows operators to request approval to comply with ADs by alternate means. To be approved, these requests must demonstrate equivalent levels of safety, integrity, and airworthiness. Industry requests for alternate means of compliance are most commonly generated for the following reasons: The operator may want to use equivalent materials that are not specified in a particular service bulletin and/or AD. Such a request could be driven by economic concerns or by the lack of availability of the specific materials delineated in the service bulletin. 1 Rulemaking processes and problems axe also discussed in Chapter 2, Chapter 6, and later in this chapter.
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service The operator defines an equivalent engineering fix or alternate means of compliance that provides an economic, maintenance, or timing advantage over the action specified by the AD. All operators are required to maintain records that define the configuration of their aircraft relative to regulatory, manufacturer, and operator-initiated engineering actions. However, operators are not required to use a standard format for verifying the configuration of individual aircraft. Consequently, an extensive records search is sometimes required to determine the actual configuration, particularly for aircraft that have had multiple changes in ownership or changes in ownership between foreign and domestic operators. Defining the configuration of an aircraft may be particularly difficult if the aircraft is not available for inspection (e.g., when it has been destroyed in an accident). In this case, determining the aircraft configuration could be essential to determining corrective action that would prevent future accidents. When the FAA initiates an AD applicable to aircraft that were originally certificated by the FAA (i.e., U.S.-manufactured aircraft), the regulatory agencies of many other nations adopt the AD immediately. The reverse is not true, however. Because of legislative, administrative, and regulatory requirements, the FAA must provide the same level of internal and public review for airworthiness actions issued by other nations as it does for its own regulatory actions. These requirements have helped create a backlog of hundreds of regulatory initiatives within the FAA. This situation generates a dilemma for U.S. operators of aircraft that were originally certificated by other nations (e.g., a U.S. airline operating Airbus aircraft). If U.S. operators do not implement the air-worthiness action as specified by the foreign regulatory agency, they could be operating their aircraft with a lower level of airworthiness than aircraft operated by foreign operators. However, if U.S. operators implement the airworthiness action as specified, the FAA may later approve other compliance requirements that would invalidate their actions. Recommendation 4-5. To eliminate the regulatory backlog and the ambiguities about implementing airworthiness actions of foreign regulatory authorities, the FAA should expeditiously determine what regulatory action, if any, it will propose in response to foreign airworthiness actions· The FAA should initiate its regulatory response no later than two weeks after receiving notice of a foreign airworthiness action. Rulemaking The FAA can react immediately to critical safety and security problems by issuing "immediately adopted" rules. The normal process for issuing a new regulation or modifying an existing regulation, however, is laborious and time consuming—quite often taking 5 to 10 years from start to finish. This is caused partly by the limited availability of personnel for rulemaking activities, the large volume of pending actions, and the procedural requirements imposed by the executive and legislative branches of the government (see Chapter 2). Several years ago, in an attempt to reduce delays, the FAA established the ARAC (Aviation Rulemaking Advisory Committee). The purpose of the ARAC is to allow FAA staff, industry experts, and other interested parties to reach an early and fully informed consensus on the need for and content of proposed rulemaking actions· However, this approach has not yielded the intended time savings, and long delays in the rulemaking process remain a critical barrier to improving the safety management process (see Chapters 6). Corrective Action vs. Blame After an incident or accident occurs, cause factors should be identified and effective corrective action should be implemented in a timely fashion. The process of assigning blame, on the other hand, often does not reduce the risk of future incidents and accidents. It is easy, but unprofitable, to associate "causes" with "blame." According to John K. Lauber, former member of the NTSB, current efforts to improve aviation safety are hampered by . . . a blurring of the distinction between incident and accident causes on the one hand, and legal, economic, and moral responsibility on the other. In our culture, we seem to be unable to deal with problems of any importance without assessing blame, and perversely will happily march over the cliff if we are certain that we know who to blame for our imminent demise. All too frequently, our search for someone to blame takes real priority over our search for solutions, and this seems especially true in matters of aviation safety (Lauber, 1989). The committee believes that the safety management process recommended in this report would help shift the focus of incident and accident investigations away from the question of who is to blame. Instead, the process would focus on identifying corrective action to prevent similar problems and their consequences. MONITORING EFFECTIVENESS One of the critical elements of any effective control system is a feedback loop. The feedback loop measures the effects of system variables on the outcome of the system and indicates how the system can be improved. Feedback enables the system operator to maximize the performance of the system and evaluate the effectiveness of previous attempts to improve system performance. Considerable feedback data relevant to continuing airworthiness are available to the FAA. The committee believes that evaluating the effectiveness of ADs and other required actions would be greatly facilitated if industry, with FAA oversight, placed a higher priority on monitoring the effectiveness of corrective action.
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service Changes in accident rates can provide a sense of the overall effectiveness of ADs but cannot be used to measure the effectiveness of individual ADs. However, it is relatively easy to assess the effectiveness of most ADs. For example, if a safety-related part repeatedly fails prior to regularly scheduled inspections, one or more ADs may be issued to increase the inspection frequency and, later, to install an improved part after one has been designed, tested, and approved for use. The effectiveness of such ADs can be verified by continuing to track unexpected failures. By monitoring the effectiveness of ADs, the FAA could enhance risk reduction by placing its highest priority on areas that are shown to be most effective. Industry could facilitate the FAA's efforts to monitor the effectiveness of airworthiness actions by agreeing upon and maintaining a standardized summary of aircraft configuration that records the implementation of voluntary service bulletins and air-worthiness actions issued by foreign regulatory authorities, as well as ADs. An important aspect of the recommended safety management process is a comprehensive FOQA system, such as BASIS, for collecting data from automated flight data recorders, maintenance reports, incident and accident reports, etc. APPROVED DESIGN ORGANIZATIONS With few exceptions, existing legislation and regulations do not require applicants for type certificates, amended type certificates, STCs (supplemental type certificates), TSOAs (technical standard order authorizations), PMAs (parts manufacturing approvals), and other approvals to show that they have the technical qualifications to develop a safe design or to conduct the engineering evaluations and certification tests necessary to show compliance with applicable FAA airworthiness standards. In addition, applicants for type certificates and STCs are not required to establish or maintain technical organizations to monitor, evaluate, and propose corrective action in response to operator reports of safety problems for which they are responsible. Of course, major aircraft manufacturers have skilled and experienced engineering organizations, and they do not need regulatory encouragement to closely monitor the safety performance of their products. However, the lack of statutory authority and implementing regulations has two negative effects: it limits the FAA's ability to take advantage of the capabilities of certificate holders' design organizations, and it requires the FAA to spend considerable resources each year on "false starts" by applicants that do not have the technical qualifications to complete the application process. The latter situation arises most frequently with applicants for STCs and PMAs. The process by which the FAA regulates the production of aircraft, engines, and propellers is a model that could be applied to other aspects of the type certification process. Before granting a production certificate, the FAA evaluates the applicant's production quality control system. After a production certificate is issued, the FAA conducts periodic audits to make sure products are being manufactured in accordance with the approved quality control system. However, the FAA does not routinely make detailed inspections to determine if individual aircraft, engines, or propellers conform to the approved design and are eligible for airworthiness certification. In other words, the FAA promotes the safety of individual products by verifying that a safe and effective production system—that includes its own internal checks—has been established and is being maintained. A similar approach should be used to promote the safety of product designs. The FAA should assess and approve the capabilities and procedures of an applicant's design organization rather than the current process, which requires FAA engineers to analyze independently the safety implications of new and modified designs. The design organizations of aeronautical engineering consulting firms, airlines, repair stations, and other organizations could also be approved by the FAA to enable applicants that do not have their own qualified engineering organizations to apply for STCs, TSOAs, and PMAs. When the current type certificate process was developed, aircraft and engines were much smaller and less complex than they are today. In those days, it was feasible for the FAA to verify independently safety-related aspects of manufacturers' designs. Today, however, a major airframe manufacturer may employ as many as 8,000 engineers, flight test pilots, and inspectors to design, develop, and certificate a new wide-body passenger jet. These large staffs are necessary to investigate the design complexities of modem aircraft. The number of labor hours invested by a manufacturer in designing a large new jet may be several hundred times greater than the number of labor hours the FAA has available to verify the safety of the aircraft design. This huge discrepancy raises a question about the FAA's ability to analyze independently new aircraft designs and locate safety-related design flaws that are subtle enough to have escaped the attention of the manufacturer's much larger design team. In fact, as designs have become more and more complex, the FAA has had to rely more and more on spot checks of new designs (instead of comprehensive reviews). The committee believes that design safety would be enhanced if the FAA devoted its engineering resources to promoting the safety and efficacy of manufacturer's design teams and processes, rather than trying to identify problems in specific designs. The FAA should examine the technical qualifications and integrity of design organizations, including their understanding of regulations and policies and their ability to properly implement them. Qualified organizations should then be certificated as approved design organizations (ADOs), allowing them to make detailed findings of compliance in accordance with published policies. FAA audits would verify continued compliance, in part by ensuring that
OCR for page 29
Improving the Continued Airworthiness of Civil Aircraft: A Strategy for the FAA's Aircraft Certification Service ADOs' level of involvement in specific projects is appropriate in light of the technical issues involved. Each ADO would be rated with limitations consistent with its technical capabilities and needs. For example, an applicant for an ADO to approve the designs of small interior parts such as tray tables or galley drawers would only need to demonstrate that it has the technical capabilities to determine that these products meet applicable airworthiness standards, and its authority would be limited accordingly. Establishing a system of ADOs would expand the current system under which the FAA already delegates specified certification functions to individual designees, such as designated engineering representatives, as described in Chapter 2. In addition, FAR Part 21 already authorizes the FAA to designate qualified companies to perform selected type certification functions. However, except in cases where special exemptions are sought and approved, current regulations prohibit extending this authorization to manufacturers of transport category aircraft, turbojet engines with more than 1,000 pounds of thrust, propeller engines with more than 500 brake horsepower, and propellers for these engines. Recommendations for removing this prohibition date back to 1966 but have not yet been implemented (FAA, 1966). A working group of the ARAC (Aviation Rulemaking Advisory Committee) is currently developing a draft rule to establish an "Organization Designated Authorization," which would extend the current delegation authorizations to manufacturers of large aircraft, large engines, and propellers for large aircraft (Federal Register, 1993). Adopting such a rule would improve the efficiency of the current certification processes. A more comprehensive restructuring of the process, which would include establishing ADOs, requires legislative authorization.2 AIR is also supporting nonregulatory approaches to improving the certification process by trying to define modified processes to achieve the following goals: early definition of applicable airworthiness standards, including special requirements for novel and unusual design features and exemptions where safety would not be compromised early agreement on what constitutes acceptable means of compliance and on findings of equivalent safety early completion of basic safety assessments to identify areas that require more detailed FAA involvement early agreement by the FAA and the applicant on a plan for completing the application process Major Recommendation 3. AIR should promote aircraft safety by certifying the competency of applicants' design organizations rather than relying on the FAA's ability to detect design deficiencies through spot checks. The FAA should work with industry and Congress to obtain legislative and regulatory authority in a timely fashion to do the following: Certificate and rate ADOs and invest them with the responsibility for ensuring that applications for type certificates, type certificate amendments, STCs, TSOAs (technical standard order authorizations), and PMAs (parts manufacturer approvals) comply with applicable airworthiness standards. ADOs would be required to have the technical capabilities necessary for competently approving designs only within the limitations of their rating. Require ADOs and holders of production certificates to collect and analyze relevant safety data received from operators and to define corrective action in the event unsafe conditions are detected. Require applicants for design approvals to either hold an ADO certificate or employ the services of an ADO. As an interim step, give higher priority to the ongoing rulemaking action that would increase organizational delegation to manufacturers of large aircraft and engines under the FAA's current legislative authority. The FAA already uses this authority to grant organizational delegation to manufacturers of small aircraft and engines. REFERENCES Derby, S. L., and R. Keeney. 1981. How safe is safe enough. Risk Analysis 1(3): 217-224. FAA (Federal Aviation Administration). 1966. Airworthiness Standards Evaluation Committee Report to the Administrator. Washington, D.C.: FAA. FAA. 1998. Aircraft Certification Mission Statement. Federal Aviation Administration. Online. Aircraft Certification Service (AIR) Headquarters Office Home Page. Available: http://www.faa.gov/avr/air/hq/mission.htm. February 10, 1998. Federal Register. 1993. Aviation Rulemaking Advisory Committee. Delegation System Working Group Tasking. 58: 1673. FSF (Flight Safety Foundation). 1992. Flight Operations Quality Assurance Programs . Washington, D.C.: FSF. Lauber, J.K. 1989. Human performance and aviation safety: some issues and some solutions. Accident Prevention Bulletin 46(4): 10-13. Roland, H., and B. Moriarty. 1983. System Safety Engineering and Management. New York: John Wiley & Sons. The Royal Society. 1992. Risk Analysis, Perception and Management. Report of a Royal Society Study Group. London: The Royal Society. 2 Chapter 447 of Title 49 of the U.S. Code would have to be amended to authorize ADOs. Appendix F contains the draft of a sample legislative amendment that could make this change.