Cross-Industry Applications of a Confidential Reporting Model
LINDA J. CONNELL
Human Factors and Research Technology Division
NASA Ames Research Center
A strong emphasis on public safety in the United States is apparent in many arenas of public life. The efforts toward preventing accidents is especially prominent in critical-outcome environments, where if a mistake is made, there can be tragic results. The loss of life and substantial injury that may result from accidents is especially tragic if it is discovered in the process of an investigation that the event could have been prevented. In large, complex, and dynamic environments like aviation, nuclear power, medicine, and other industries where sometimes minor errors or flaws in systems can lead to serious incidents or accidents, the challenge of maintaining safety is significant. Therefore, effective risk management, which includes risk assessment and risk mitigation, is crucial to ensuring safety.
THE AVIATION SAFETY REPORTING SYSTEM
The U.S. aviation community and the public have benefited from a historic Interagency Agreement that was signed in 1976 between the Federal Aviation Administration (FAA) and the National Aeronautics and Space Administration (NASA). This cooperative agreement was in part a response to an aircraft accident in 1974 that was the result of an ambiguous and misunderstood communication between air traffic control (ATC) and a flight crew. The flight crew descended too soon and hit a mountain in what is called a controlled-flight-into-terrain accident. In the accident investigation by the National Transportation Safety Board (NTSB), it was discovered that another airline, six weeks prior to the accident under investigation, had also misunderstood the ATC
instruction, begun their descent, and barely missed the mountain. Although that airline had quickly warned its flight crews of the problem, other airlines were not informed. It was “an accident waiting to happen.” During this investigation, the aviation industry and the government agreed that the country required a reporting system for near-misses. The FAA and NASA established the voluntary, confidential, and non-punitive reporting program entitled the Aviation Safety Reporting System (ASRS) (Reynard et al., 1986). The FAA provided immunity to aviation personnel who agreed to report to NASA under the new program (FAA, AC 00-46D).
Since that time, ASRS has accepted almost 580,000 reports from pilots, air traffic controllers, flight attendants, maintenance technicians, and others describing aviation safety events that they experienced or witnessed (Figure 1). ASRS has processed this information and contributed to the improvement of aviation safety throughout the United States and abroad (Reynard, 1991). In aviation,
ASRS has been recognized, both domestically and internationally, as a model for collecting unique safety data from frontline personnel.
Currently, seven countries besides the United States are operating aviation safety reporting systems modeled after the original ASRS, and many other countries are working to establish systems. The value of confidentiality, contributions to aviation safety, and the ability to gather information often not reported through other avenues, was quickly recognized by the United Kingdom and soon after by Canada and Australia. ASRS meets annually with these countries to coordinate and compare information concerning worldwide aviation safety through the International Confidential Aviation Safety Systems (ICASS), a group formed in 1988 that has since been recognized by the International Civil Aviation Organization (ICAO). In the ICAO Annex 13 documents, member countries throughout the world are encouraged to initiate and operate systems similar to those used by ICASS countries. New countries are referred to ICASS for assistance in designing and implementing new systems.
The confidential reporting model has developed and matured for more than 25 years through collaboration between NASA Ames Research Center, ASRS, and the FAA Office of System Safety. The system has been recognized for providing unique safety information not available through any other means (Connell, 2000, 2002). Other disciplines and industries that have recognized the advantage of ASRS have consulted with ASRS to assess its relevance and potential contributions to their own safety efforts. The nuclear power industry has adopted a similar approach to gathering safety information to complement its traditional data-collection methods. The maritime industry is currently considering the best application of the confidential reporting model to its environment (Connell and Mellone, 1999).
The medical community has begun a strong initiative to adopt the ASRS model to collect safety information from frontline medical personnel. In To Err Is Human: Building a Safer Healthcare System, the Institute of Medicine (IOM) directly addresses the ASRS model (IOM, 2000).
In 1997, prior to the release of the IOM report, the Department of Veterans Affair (VA) asked NASA ASRS to join an expert advisory panel being convened in Washington, D.C., to advise the VA as they began a new focus on patient safety. The VA invited numerous cross-industry participants to describe how their industries addressed safety and which methods had been successful. At those meetings, the VA asked NASA’s ASRS director if assistance could be provided to the VA to create a medical reporting system modeled after ASRS. The offer was enthusiastically accepted, and NASA entered into an interagency
agreement with the VA in May 2000 to establish a collaboration between ASRS at NASA Ames Research Center at Moffett Field, California, and the VA National Center for Patient Safety (NCPS) in Ann Arbor, Michigan. The new system, the Patient Safety Reporting System (PSRS), which replicates ASRS, is the proof-of-concept for medicine; the model is expected to evolve to meet the safety needs of this complex environment (McDonald and Connell, 2001). The PSRS is in operation and is receiving reports that are providing constructive safety information. The VA and NCPS have introduced numerous safety innovations in recent years, and the PSRS is expected to be complementary to those efforts (Weeks and Bagian, 2000). The PSRS is expected to provide benefits to healthcare delivery similar to ASRS’s benefits to aviation.
The resources of both NASA and the VA, and strong VA protections of data from legal discovery under 38 U.S.C. 5705, have enabled the confidential reporting model to flourish and grow in medicine. The NASA Ames Research Center, the Center of Excellence in Information Technology development for NASA, has a world-renowned group of researchers in human factors. All of the technology development and human factors research that have supported ASRS are available to the NASA/VA PSRS project.
In addition, substantial developments in automated report processing, data mining, textual analysis, and data visualization tools have been made. These software and hardware tools are human-centered; that is, they support the human analysts who are essential to the success of the ASRS model. These developments and the adaptation of the aviation model to the medical environment have already contributed to patient safety and a knowledge base for proactive change.
A new project is being initiated to create a separate avenue of reporting for security events. Since September 11, ASRS has received increasing numbers of reports describing aviation security incidents. But a gap analysis and study of these reports revealed that these reports were extremely sensitive and would require different methods of analysis and evaluation. In addition, although ASRS hears from pilots, air traffic controllers, flight attendants, and mechanics, other groups of personnel involved directly with the security processes have not been exposed to or educated about the confidential reporting model of ASRS. Therefore, NASA is proposing that a security incident reporting system (SIRS) be part of a new NASA program, the Aviation Safety and Security Program (AvSSP). The proposed SIRS project would be a replication of the ASRS model with all of the essential success criteria of the original model. However, because of the unique nature of this type of reporting, SIRS will probably provide alternative processing features that include more extensive protections. A consortium of industry and government stakeholders will be created to advise NASA during the development of SIRS.
The importance of risk management in high-reliability systems and industries cannot be overstated. Many concepts and methods have been proposed for effective risk management. Risk management can be defined as “the organized process of identifying and assessing risks, then establishing a comprehensive plan to prevent or minimize harmful effects from those risks being asserted” (NASA, NPG 2810.1). One method is to perform risk management during all of the life-cycle phases in the development of a new technology. In NASA guidance for research and development, risk management encompasses risk assessment, risk mitigation, evaluation of residual risk, and risk acceptance. The definition of risk used in this guidance is “a function of the probability of a given threat source exercising a particular vulnerability and the resulting impact of that adverse event on the organization” (NASA, NPG 2810.1). In high-reliability industries, where the impact of an incident can have catastrophic results, risk must be considered in relation to “threat sources” that capitalize on system “vulnerabilities.”
The voluntary, confidential, non-punitive model for the reporting of safety events is a significant tool in risk management. One NASA approach to total risk management includes nine steps in the risk assessment process (Table 1).
The confidential reporting model is most useful for threat identification (Step 2) and vulnerability identification (Step 3). The stated objectives of ASRS are: (1) to identify deficiencies and discrepancies within the aviation system; and (2) to provide data and information for system planning and improvement (Connell, 2002; Reynard, 1991). In addition, ASRS is a national resource that provides three types of information: (1) identification of problems and issues in aviation systems; (2) the generation of hypotheses for further research; and (3) information about unique human and operational factors. Thus, ASRS is well situated to provide information on risk in terms of both threats and vulnerabilities.
TABLE 1 Nine Steps of Risk Assessment Model in NASA NPG 2810.1
1. System characterization
2. Threat identification
3. Vulnerability identification
4. Control analysis
5. Probability determination
6. Impact analysis
7. Risk determination
8. Control recommendations
9. Results documentation
Because ASRS is independently administered, reporter confidentiality is protected, and the system is non-punitive, people on the front line of aviation can report in a protected environment. People who work in the system every day freely provide candid and introspective reports about their performance, whether they performed well, or not so well, in the complex aviation system. The information they volunteer describes activities and events that precede sometimes serious incidents. In reading and analyzing these reports, specialists in aviation transform the report data into information that can be used to assess risk in the system.
Because of the conditions of reporting and limited immunity established between NASA and the FAA, ASRS is a robust source of information for both threat and vulnerability identification, its main contribution to risk assessment and, thus, to risk management. By using de-identifying policies and procedures, ASRS has preserved the confidentiality of reporters for more than 25 years of successful operation. ASRS has established a reputation of trustworthiness that encourages honest, open reporting. Currently, ASRS receives approximately 38,000 reports per year (Figure 2).
Based on trust and confidence, frontline personnel have provided high-volume, high-quality, candid reports that have identified many threats and
vulnerabilities. But these reports have not only revealed system weaknesses that could, combined with other factors, lead to serious incidents or accidents. They have also provided clues to some strengths in the system. ASRS reports have enabled analysts to discover how people detect anomalies in the system and how they recover from potentially dangerous events and avert fatal accidents. The people involved can then discuss thoroughly the event from the beginning. In the trusted and protected environment of ASRS, these individuals are willing to explain their roles in the occurrence. Their insights, the “human factors content,” have made ASRS data valuable for improving aviation safety.
ASRS attempts to maintain a neutral, unbiased position between the numerous factions in aviation. The information generated by the system and distributed through a variety of products is provided to the government and industry aviation safety community, which develops and acts on safety solutions. ASRS often states that “it works through the good offices of others.”
The contribution of ASRS information to risk management is largely through threat and vulnerability identification and descriptions of the context in which incidents occur. ASRS does not monitor or demand corrective action in the aviation system in response to the information it provides. To preserve its role as an independent, external, and neutral contributor to safety improvement, ASRS remains outside the ongoing process. Perceptions of bias, however subtle, can adversely affect people’s willingness to report. The trust and the voluntary nature of ASRS are unequivocally protected.
Responses to the threats and vulnerabilities identified by ASRS and risk management are developed through mechanisms outside of ASRS, although ASRS can provide a neutral forum for continuing discussions to reduce threats and vulnerabilities.
ASRS is a proven, effective system for confidential reporting and an exemplary system for application in other industries interested in safety improvements. This model, where the “devil is in the details,” can be replicated, adapted, and evolved to be an intuitive, productive, information-collection mechanism for safety improvement in any system. ASRS’s biggest contribution is in the identification of threats and vulnerabilities.
ASRS’s characteristics and features are unique among other information-gathering systems. But its success requires constant nurturing, support, and advocacy. For people to feel that they can safely report what actually happened and happens in a system, trust and confidence can never be sacrificed to other interests. When frontline personnel in a system believe and trust that they are protected, even if they are the bearers of bad news about system flaws or they expose their own errors in the interest of system integrity, they will provide truly rich and illuminating data that can lead to safety improvements.
Connell, L.J. 2000. Aviation Incident Reporting: Valuable Information for Safety. Presented at National Symposium for Building Systems That Do No Harm: Advancing Patient Safety Through Partnership and Shared Knowledge, June 29, 2000, Dallas, Texas.
Connell, L.J. 2002. Aviation Safety Reporting System: Contribution of Confidential Reporting to Aviation Safety. Presented to Institute of Medicine, Committee on Patient Safety Data Standards, September 23, 2002, Washington, D.C.
Connell, L.J., and V.J. Mellone. 1999. Aviation Safety Reporting System: A Blueprint for Maritime Safety. Presented to Society of Naval Architects and Marine Engineers, Human Factors Panel, February 4, 1999, San Francisco, California.
Federal Aviation Administration. 1997. Aviation Safety Reporting Program. Advisory Circular 00-46D. Washington, D.C.: Federal Aviation Administration.
IOM (Institute of Medicine). 2000. To Err Is Human: Building a Safer Healthcare System, L.T. Kohn, J.M. Corrigan, and M.S. Donaldson, eds. Washington, D.C.: National Academy Press.
McDonald, H., and L.J. Connell. 2001. Patient Safety Reporting System: The Who, What, Where, and Why. Presentation to 4th Annual Meeting, National Association of Inpatient Physicians, March 27, 2001, Atlanta, Georgia.
Reynard, W.D. 1991. The Acquisition and Use of Incident Data: Investigating Accidents Before They Happen. ASRS Internal Publication. Washington, D.C.: National Aeronautics and Space Administration.
Reynard, W.D., C.E. Billings, E.S. Cheaney, and R. Hardy. 1986. The Development of the NASA Aviation Safety Reporting System. NASA Reference Publication 1114. Washington, D.C.: National Aeronautics and Space Administration.
Weeks, W.B., and J.P. Bagian. 2000. Developing a culture of safety in the Veterans Health Administration. Effective Clinical Practice 3(6): 270–277.