The Future of Air Traffic Control: Human Operators and Automation (1998)

As a result of the panel's studies of various existing and proposed systems in air traffic control, human factors lessons learned from other systems (e.g., the flight deck), and our understanding of good human factors practices in other fields, we have identified several key issues that are critical to the successful development and integration of new technology. We describe these issues below and note, where relevant, both positive and negative ways in which certain aspects of air traffic control automation represent them.

One common theme underlying all issues is the need for close harmony between research and development. That is, research is necessary for good product development, and many issues encountered during development directly contribute to the research base or at least feedback to define important research questions. As a consequence, these two concepts are closely linked in this chapter.

Human factors activities should be integrated comprehensively across the development cycle. Too often, program managers perceive human factors as a marginal design function constrained to the details of the human-computer interface (e.g., color-coding, character height, workstation anthropometry). This perception ignores the importance of human factors throughout the analysis, functional specification, detail design, test and evaluation, and implementation phases of development (Booher, 1990).

As noted in the Phase I report and as recommended by the Federal Aviation Administration's Subcommittee on Human Factors of the agency's Research, Engineering, and Development Council (Federal Aviation Administration, 1996e), integration of effort would be enhanced if human factors research were

treated as a unified program. To do so will require the establishment of a single locus within the FAA organization that has, at minimum, an oversight function for all such research and development. Ideally, such a unit would also have the authority to establish research goals and standards of rigor in the conduct of the program of studies. This arrangement would not reallocate roles and responsibilities now assigned to the FAA's Civil Aeromedical Institute and the FAA Technical Center, but it would impose central management of the human factors support for the integrated product teams. General recommendations for integrating human factors research and development activities into a comprehensive, iterative program are echoed in the earlier reports of three technical meetings, sponsored in part by the FAA (Wise, Hopkin, and Smith, 1991; Wise, Hopkin, and Stager, 1993; Wise, Hopkin, and Garland, 1994).

Our emphasis here is on the integration of research and development activities into a systematic and comprehensive human factors program . Such a program involves the coordinated actions of many contributors (including managers, systems engineers, a variety of users, and human factors specialists) at the FAA, cooperating government agencies, and subcontractors. These activities occur across all phases of acquisition and implementation (including analysis, definition of requirements, design, test and evaluation, and training and selection of personnel) and involve a mix of research and evaluation methodologies (including analysis, modeling, simulation, prototyping, laboratory studies, and field studies). This comprehensive approach must be applied to individual subsystems and to the wider national airspace system into which the subsystems must be integrated. We note here that the maintenance of such a comprehensive program requires the commitment and support of FAA management, as well as of those constituents with whom the agency interacts (e.g., legislators, regulators, commercial users of the national airspace system, and representatives of employee unions). This commitment must be demonstrated by continuous allocation of resources (e.g., budget, staffing, facilities, and educational materials) sufficient to accomplish both the breadth and depth of research and development activities we discuss below. In systems with direct safety implications, such as those that the FAA is developing, the resource commitment to human factors during development and testing should be at least as large as that associated with hardware and software reliability.

In the Phase I report, we identified a wide range of research methodologies that could contribute to understanding the information needs and task requirements of controllers, as well as to the appropriate design of equipment and interfaces to serve those needs. These methodologies include literature searches, incident analysis, modeling, laboratory studies, simulation studies, and field tests.

Each is briefly discussed below as it relates to human factors work on the current and proposed systems presented in Part II of this report.

There is an extensive body of literature that describes fundamental principles, methodology, data, and lessons learned relevant to the general activity of human factors development and testing. However, despite valuable sets of technical reports produced through such organizations as the FAA's Civil Aeromedical Institute and the FAA Technical Center, the body of literature has been largely derived from domains other than air traffic control. Caution must be exercised before lessons, data, and recommendations derived from other domains are applied to air traffic control; in many cases such information is best used to formulate and refine hypotheses that must be tested in the air traffic control setting. This represents the fundamental principle of replication, important in scientific settings, whereby the external validity of findings is demonstrated across different domains.

Within the domain of air traffic control, the human factors literature evinces uneven coverage. For example, although Kerns (1994) seems to have done an exemplary job of amassing and organizing literature specifically relevant to the development of data link, there is a paucity of relevant literature for such developments as the user request evaluation tool (URET), the precision runway monitor (PRM), and the converging runway display aid (CRDA). The operational impacts of these latter developments may be smaller than those of data link, yet efforts to document the human factors issues, findings, and decisions during their development and evaluation would contribute useful data and lessons learned to the human factors database.

Incident analysis is a useful methodology that has been used heavily and productively, for example, in the refinement of the traffic alerting and collision avoidance system (TCAS) (Mellone and Frank, 1993). Incident analysis can serve as an important trigger for developing a particular automation functionality by revealing the existence of a problem. For example, reported problems in the cleared for visual approach situation have triggered exploratory investigations of means to extend TCAS to support visual clearances (Mundra et al., 1997). A limitation of the incident analysis methodology, however, is that the post hoc analysis process identifies a problem only (which automation may address); incident analysis can be used as a tool for evaluating the proposed solution only if it is applied early in implementation. Another limitation is that post hoc analysis using reports of incidents often relies on data that have been filtered through a conceptual system that is reflected in the classification structure of the database

itself. That is, the elicitation of the data may be constrained by the structure of the database in a manner that does not garner needed human factors data or that combines data in ways that obscure meaningful parameters.

The continuing challenges for the improvement of incident analysis databases are to centralize and connect relevant data; encourage data input by users; structure data inputs and analyses to permit addressing specific questions pertinent to the automation of air traffic control functions; and develop user-friendly interfaces for analysts.

Progress has been made in recent years and new work is currently under way to increase the usefulness of computable models in the analysis and evaluation of complex systems. A major center for such work at a basic, generic level is located at Carnegie Mellon University, where the SOAR (state, operator, and results) model has been developed and used as the computational framework for studies in artificial intelligence as an aid to human performance. Such work is currently being carried forward in support of the development of operational systems (Tambe et al., 1995). The expansion of such models to incorporate many of the details of human behavior is currently being supported by the Defense Modeling and Simulation Office (Pew and Mavor, 1997). Advanced modeling of specific air traffic control operations is under way at the Massachusetts Institute of Technology and elsewhere in the FAA and NASA (see Odoni et al., 1997, for a review). The FAA has also used SIMMOD and other human performance models for feasibility and certification analyses of the precision runway monitor and the converging runway display aid. In addition, NASA is sponsoring work to develop modeling capabilities specifically suited to examination of the human performance component of air traffic management systems (Corker et al., 1997). Air traffic control workload models are being developed in the United Kingdom at the National Air Traffic Control Simulation facility (Evans et al., 1997).

Valid models of air traffic control error generation and situation awareness, however, remain insufficient. The uses of such models in human-centered systems research are multiple. One use is the forward testing of major changes in system composition. The large advantage is that radical innovations can be given a preliminary evaluation at low cost and zero risk. An even more attractive use is the ranking of alternative design solutions to a particular, generic operational problem. An example of a generic problem is the variation in performance profiles between aircraft landing on a common runway. Should aircraft having higher speed and rate of descent be automatically slotted in ahead of slower planes? Such a situation is inherently computable because the phenomena in question are quantitative. Many different mixtures of aircraft types could be evaluated through modeling in a short time. For example, spacing between pairs of aircraft is computable in four-dimensions. Automatic adjustments in separation

could be engendered by situations such as the presence of a slow aircraft in-trail behind a faster aircraft (taking into account such variables as wake turbulence). In such cases, the separation will increase over time with no intervention by the controller.

It is increasingly possible to include some dimensions of human capabilities in such models. Consequently, although modeling cannot directly answer many human factors questions, it can serve as the means to carry out exploratory studies. Such studies can point to specific issues that need resolution through the use of other methods. For example, if human abilities to discriminate speed and rates of closure are included in the model, it should be possible to form highly focused hypotheses about the relationship between aircraft heterogeneity and subjective workload.

Another area of usefulness for operational models is the simultaneous testing of more than one innovation. This use gets directly at the problem of systems integration. It could be very cumbersome if not dangerous to evaluate the levels of compatibility between data link, the center TRACON automation system (CTAS), and free flight by other methods. For example, under free flight it can be assumed that arrivals to the terminal area could appear at any spot on the boundary rather than at the intersection with one of the standard air routes. In effect, this transition will take the arriving aircraft out of the free flight mode and into a nominal instrument flight rules mode. CTAS will designate a landing slot. Data link will transmit a standard greeting and local meteorological data (i.e., information from the automatic terminal information service). In short, many events will be under way at the moment of entry to the terminal area. Will there be some way to assure both pilot and controller that situation awareness is complete on both sides? Will cognitive load effects become excessive when arrival rates peak at more than one per minute? Will vectoring of the aircraft to adjust orientation to the final approach line disrupt the slot assignment provided by CTAS? If there is clear-air turbulence at the threshold to the runway, will the controller need to override the data link to ensure that the pilots take this hazard into account in choosing the ideal altitude at the point of threshold crossing?

Examples of areas in which computational modeling is being productively applied include the terminal area productivity (TAP), CTAS, and TCAS projects. Examples of areas in which modeling would be helpful, although modeling applications have not been extensively applied or planned for, are the CRDA, airport movement area safety system (AMASS), airport surface traffic automation (ASTA), and the airway facilities operations control center (OCC) projects.

Laboratory studies, which permit rigorous control over experimental conditions, can be very valuable for investigating issues that arise early during the development of automated systems and that may not have been predicted prior to

system specification or preliminary design. For example, within the domain of air traffic control, laboratory studies at the FAA's Civil Aeromedical Institute contributed important understanding pertinent to the design and use of flight strips (see Chapter 4); laboratory studies at the University of Illinois resulted in useful guidance for the design of displays to present three-dimensional information; and laboratory studies were effectively employed to address specific design questions during the improvement of TCAS (Chappel, 1990). Examples of good candidates for additional laboratory study address workload issues associated with data link and vigilance issues associated with PRM and TCAS. In addition, laboratory studies can be very valuable for examining variables that are too complex or numerous to study in more costly, full-mission simulations. Often such laboratory studies serve to identify the most critical variables for incorporation into the full-mission simulations.

There is concern that researchers conducting applied laboratory studies retain understanding of the need to transition experiments into a contextually real environment, and that they also ask operationally important questions. Still, because laboratory studies generally represent low-cost, high-power (i.e., experimental power) methodologies, their value should not be undersold.

Simulation provides a more relevant context for study. Simulation has been applied successfully to study human factors aspects of data link, of the PRM, and of European conflict probe designs. Simulation can be especially useful in support of investigating how multiple systems do or do not harmonize and of the impacts of new systems on teamwork.

Simulation has been used effectively to study the combinatorial effects of CTAS with existing tools and tasks, and the Canadian automated air traffic system (CAATS) program (see Stager, 1991) used simulation effectively to study electronic flight information presentation, considering how the relevant subtasks of flight information processing fit into the overall air traffic control task set. Simulation can also be usefully applied to the study of combining effects, for example, of CTAS and URET predictive capabilities, of PRM with CRDA for landing coordination, of multiple components of ASTA and the taxi navigational situation awareness (T-NASA) system for surface conflict avoidance, and of all the components of the surface movement advisor (SMA).

Simulation has also been, and should continue to be, productively applied to evaluate data link innovations. An example of such simulation is described in the Airborne Data Link Program Plan for fiscal 1995 (Rehmann, 1995), which identifies projects that use the air traffic control simulator at the FAA Technical Center in combination with the reconfigurable cockpit simulator. This simulation requires the valid representation of the whole avionics subsystem that would probably be in use in the cockpit alongside the data link-equipment. The early

research is aimed at determining whether there are gross incompatibilities between data link and existing instrumentation.

The team setting is a key context for introducing technologies and is amenable to study through simulation. The teamwork aspects of TCAS, including shifts of authority between controllers and pilots, have increasingly been studied through simulation, with productive results (Hoffman et al., 1995). Other candidates for simulation study of teamwork are the maintenance OCC (considering teamwork within maintenance organizations and between controllers and maintainers), and airport surface automation projects such as ASTA, T-NASA, and the SMA. The manner in which controllers perform complementary strategic and tactical tasks during the use of conflict probes such as URET merits study to determine whether specific design (or procedural) features affect the cooperative task performance.

One concern with respect to simulation studies is that statistical power is preserved by the introduction of as much experimental control as feasible and by the use of large numbers and representative types of subjects. Another concern is that, because of expense and complexity, all of the relevant real-world variables may not be included in a given simulation study. Some of these variables (e.g., those associated with low-frequency, unexpected events, or error recovery) can have significant implications for operational safety and should be included. Others, however, while contributing to the operational realism of the simulation, may be superfluous to the task performance characteristics that are to be measured and hence impose unnecessary expense. Objective guidelines exist to help identify the latter classes of variables (Sheridan and Hennessy, 1984; Druckman and Bjork, 1994).

When an advanced prototype or preproduction model of a new subsystem is available, field investigations can be conducted. Field tests are almost always a formal stage preceding approval and certification. They provide the opportunity for operational evaluations of elements of a system to be conducted on integral equipment that is being used online in the control process. In addition, field studies are useful when the interactions of a comprehensive set of variables (including the interaction of new equipment with one another and with existing equipment) can be observed dependably only in the actual operational environment.

One of the advantages of collecting human factors information in a field setting rather than in real-time simulation is that the live operational environment provides a means of capturing the subtleties in operational practices and work habits that may not carry over into a simulation environment. The ability to capture controllers' experience with new technology, for example, is especially

important for complex automation in which the implications of the interactions between system components are largely unknown prior to implementation.

Field studies, which are inherently face-valid, can serve as a comprehensive source of information. Although they are retrospective in the sense that some design decisions will have been made, the resultant information can also tell the designers whether the system retains major flaws. Examples of programs that are harvesting useful data through field studies are CRDA, CTAS, URET, runway status lights (RWSL), PRM, ASTA, and T-NASA. The HOST/ARTS field experience, resulting in implementation of the implied functions capability that reduces the requirements for manual data entry, is a representative example of the use of field study data to improve design. We wish to emphasize that the concept of field study must also include the ongoing efforts to collect operational data from new systems well after they have been installed, as well as a proceduralized willingness to consider design modifications if field lessons reveal nontrivial human factors concerns. The TCAS system with its newsletter and continued refinement of algorithms provides a positive example of this.

As with simulation studies, one concern with respect to field studies is that, depending on the design of the study, the full range of relevant operating conditions may not necessarily be assessed. Another concern is that the validity of the study can be jeopardized if a representative sample (i.e., representing the range of relevant characteristics of intended users) of properly trained operators are not observed.

In parallel with system development, starting with early conceptual development and continuing through installation, a system-specific analysis of the interactions between system attributes and operators' capabilities should be undertaken—even though the procedures for a given controller position may be well-known. In doing so, opportunities to compensate for operator limitations and to take advantage of operator strengths, as well as specific compatibilities and incompatibilities, should be identified.

It is always important—and especially so when the design includes automation features—that human factors participation begin at the earliest phases of analysis, when decisions are usually made about which functions to automate. Human factors activities should not be confined to the implementation of specifications; they should precede and contribute to the development of the specifications. System specifications typically include both functional requirements (what the system shall do) and performance requirements (how well the system shall perform its functions). These requirements are usually quite detailed in specifications for hardware and software elements of systems. However, functional requirements for the human user are often less clearly specified, and human performance requirements are often not specified at all. A significant reason for

the absence of useful human functional and performance requirements is the lack of supporting analyses that link human performance to system performance. Analyses must also be supported by a database of human performance metrics derived from research and from tests and evaluations. Stager (1993, 1994) provides a comprehensive treatment of methodological issues pertaining to the validation and certification of complex systems, including air traffic control systems. Applying research to fill existing gaps in the body of applicable performance metrics is a challenge that requires commitment and investment by program management (Stein and Wagner, 1994).

The discussion above leads to the suggestion that all methodologies have strengths and weaknesses (Chapter 10 in the Phase I report provides a detailed discussion of the strengths and weaknesses). The best approach to successfully fielding a system is to integrate lessons learned across methodologies. Kerns (1994) provides a good example of this integration across laboratory studies, simulations, and field tests for data link. The TCAS, OCC, ASTA, T-NASA, and SMA projects have applied, or plan to apply, a mix of research methodologies that includes modeling, simulation, and field tests; laboratory studies are conspicuously scarce for these programs. Other projects, such as URET, PRM, CRDA, and AMASS, are examples of an imbalance of methodologies, focusing on field tests with little or no evidence of incremental laboratory, modeling, and experimental simulation study. Such imbalance can reflect fractionation of research and development activities. For PRM, CRDA, and AMASS, in particular, the reports describing the functionality do not provide evidence that a comprehensive human factors methodology has been applied.

The advantages of involving users include a better identification of operational needs that drive specifications, garnering useful operational perspectives as the design develops, identifying procedural and organizational implications of design features, and enhancing user acceptance in advance of fielding the system. Recognizing the importance of user involvement, the FAA has established an integrated product team approach to the development of new systems. This approach brings together representatives from systems engineering, users (including air traffic controllers and airway facilities specialists), human factors, and others for the duration of the development cycle (the team works together from earliest analysis of needs and requirements, through post-fielding evaluations). However, as Small (1994) points out, such teams must have a clear understanding of the roles of each member.

Integrated product teams constitute a useful vehicle for cooperative work, but they are not always successful in practice if user inputs are not appropriately structured and filtered by the guidance of the human factors practitioner, and if the advice of the human factors practitioner is not provided or heeded. It is helpful if the users are familiar with fundamental human factors principles and methodology, and if the human factors practitioner has received adequate training in both human factors and air traffic control operations.

User involvement should not be accepted as a substitute for engineering or human factors expertise. A significant risk associated with overreliance on user inputs is the common divergence of user-preference and actual performance. Users may prefer one design option but actually perform better with a less preferred option (Andre and Wickens, 1995; see also Chapter 9 in the Phase I report). Small (1994) recommends that users be expected and encouraged to express their findings in terms of operational needs and tasks, rather than in contractual system specification language. However, Small also stipulates (e.g., on the basis of lessons learned from the AAS program) that users not be allowed to specify a system's detailed design (e.g., the human-computer interface).

Human factors practitioners must be involved in defining the functionality (including the allocation of functions to human or to automation), not just the human-computer interface. The most valuable contributions of users occur during the definition of requirements, including functional requirements and performance requirements. These requirements are typically defined in system-level specifications. User inputs are less valuable, and may be counterproductive, if not filtered by human factors professionals, for specifying the detailed design of the system, including the human-computer interface characteristics.

Human factors guidance is properly provided at all phases of development: recommending task analysis techniques, filtering and integrating user input and subjective opinion, and designing assessment tools (including appropriate use of rapid prototypes, experiments, simulations, and literature searches). The CTAS program, for example, illustrates early and continued involvement of human factors practitioners applying a variety of methods in concert with user involvement (Harwood et al., in press). The CTAS project represents exemplary cooperation between users, human factors practitioners, and engineers during the definition of system functionality. The AAS and the AMASS systems, in contrast, illustrate that user control of detailed design decisions can result in ''requirements creep" that delays system development and can engender costly reengineering.

Rapid prototyping is a technique for gathering the comments and impressions of future users and others regarding the capabilities and limitations of a simulated air traffic control workstation or workspace while it can still be changed without major cost or schedule impacts. Prototyping is not a substitute for testing, but it can be a useful technique to discard fundamentally flawed options quickly, to identify crucial combinations of circumstances that require testing,

and to discover main topics of agreement and of disagreement among evaluators. Prototyping has been usefully applied to such projects as CTAS, TCAS enhancements, data link, and URET, and is planned for application to such projects as SMA and the maintenance OCC. European developers make much use of effective prototyping, as did Canadian developers of electronic flight data presentation for CAATS (see Stager, 1996, for a discussion of automation in CAATS).

To be most effective, user inputs must be structured. The question "Is this design option usable or operationally suitable?" often engenders costly and time-consuming debate that cannot be resolved solely on the basis of the preference of users or of design engineers. The standard human factors task analysis (e.g., see MIL-H-46855B) is a powerful tool that can constitute a common ground for deliberation of operational suitability, as well as for the development of specifications, training requirements, and testing procedures (Small, 1994). Recent efforts have expanded traditional task analysis into the domain of cognitive task analysis, targeted specifically at air- and ground-based aviation systems (Seamster et al., 1997). The task analysis formally defines the tasks that the system and its users are expected to perform to accomplish required functions. When users contribute to the definition of operational scenarios that drive the analysis, when systems engineers contribute to the definition of data flows and machine functions, and when human factors specialists contribute to the definition of operator tasks—and when all team members formalize performance requirements (e.g., time, error) for functions and tasks—then operational suitability decisions can be cooperatively made by asking "Applying a given design option, will the tasks and functions be accomplished within the performance constraints?"

By structuring team roles and inputs, the formal task analysis assists the team in addressing two corollary questions. One is "Where do we need more information to make a decision?" When the team is unable to specify the human performance data associated with a given task, clear direction is implied for supporting research (laboratory, modeling, or prototyping). A second question is "When is the design suitable enough?" This question introduces the issue of cost-effectiveness of development efforts. Too often, users and designers identify design options and ask "Which option is more suitable?" This, however, is not necessarily the appropriate design question. The search for the most suitable (or the most preferred) design can lead the team to unnecessarily expensive options. In fact, the decision to automate a task or function can sometimes result in unnecessary expense. To determine operational suitability, it may not be productive to compare options against one another (e.g., comparing automating a function against performing it manually, or comparing one human-computer interface feature against another) without also comparing both options against some specification of acceptable suitability (e.g., the option must permit the performance of a given function within specified time and error constraints). If neither option meets requirements, then neither is suitable. If both options meet requirements, then the relatively "better'' (or preferred) option is not necessarily more suitable, and the

decision should include considerations of cost-effectiveness (Small, 1994), as well as harmonization with existing equipment and procedures.

In the Phase I report we advanced an incremental approach to testing and validation that reflects a general "build a little, test a little" development and test philosophy from the Phase I report (Del Balzo, 1995). U.S. developers of air traffic control systems have experienced difficulties when they have followed a philosophy that relies on building a complete system according to complex specifications that must be amended as designs develop unforeseen problems, as new technologies appear, and as more is discovered about human performance. The AAS represents a recent example of the difficulties attendant to reworking specifications for a complex air traffic control system (Del Balzo, 1995). In contrast, European developers have adopted an approach that encourages more exploratory research prior to specification (so that problems are discovered early). The CAATS program (see, for example, Stager, 1991) in Canada fell somewhere between these two approaches to system development. For CAATS it was recognized that many design specifications for the controller workstations and displays could not be stated in the absence of additional human engineering data, and that the specifications would need to be resolved through the human engineering program plan.

Automation will be associated with larger conceptual and technological leaps than have characterized past developments. Particularly for large, complex systems (e.g., AAS) this argues all the more for earlier concept exploration activity prior to the writing of system specifications, as well as for the incremental development of system specifications, rather than a traditional approach that specifies all requirements at detailed levels (see Stager, 1993, for an in-depth discussion of the interplay between validation methodologies and system development). The "build a little, test a little" philosophy also relies less on specification of a large system (which takes a long time to develop and to discover whether it is successful) than on the piecemeal development of subsystems, which are flexibly specified and implemented as they are validated (Del Balzo, 1995). In any case, operational testing, evaluation, and validation activities should begin early in the development cycle (Grossberg, 1994) and can be significantly facilitated by cost-effective prototyping and simulation (Small, 1994).

Examples of good incremental development appear to include the CTAS, data link, T-NASA, ASTA, and SMA—all of which remain under development. An example of poor incremental development is the AAS program, whose overall failure was due in part to ambitious attempts to specify in advance a very large and complicated system with many development risks (Del Balzo, 1995).

The introduction of automated tools into the workplace should proceed gradually, with adequate attention given to user training, to facility differences, and to user requirements. The operational experience from initial introduction should be very carefully monitored, with mechanisms in place to respond rapidly to both positive and negative lessons learned from those experiences. Adequate funding should be provided at all stages of system development and field evaluations both before and after systems acquisition.

Operators chosen to work with new systems or subsystems should be given an understanding of the principles of system operation. They should be educated with respect to the logic and algorithms underlying the system as well as the practice of system operations. The panel has previously discussed the features of air traffic control training programs and the importance of human factors support of training in detail (see Chapters 3 and 9 in the Phase I report). It remains important that users of systems with automation be trained to perform any manual tasks that will be required in the event of the failure of automation, that they be trained in recovery procedures (which may represent totally new tasks), and that this training be reinforced periodically to maintain appropriate manual skill levels.

Training should include familiarization and part-task instruction on automated systems, but such training should progress quickly to the level of real-time exercises in the setting of interactive simulations. Embedded training should be considered as a useful approach to help controllers maintain skills with automated systems.

The goal of training is to bring each candidate to a high standard of proficiency. Therefore, valid and reliable performance measures should be defined, and proficiency should be determined in regard to the specific measures. The training activity also represents an opportunity to collect data on user performance and to elicit comments and criticisms of the new system.

The TCAS program stands as an example of inadequate initial training (see Chapter 5), whereas training for transition to the ARTS was effective (see Chapter 4). In both cases, as with CTAS, training was enhanced by vendors or by such organizations as the MITRE Corporation and NASA, which, as sponsors of the tools, contributed understanding of the facility climate.

Several aspects of human factors involvement encourage user transition to and acceptance of new systems. The interaction between human factors specialists

and users during early analyses, detail development, and testing forms a working relationship that inspires confidence in prospective users, as well as a structure for training and for field evaluations. This confidence can be reinforced by continued data collection that promises responsive review by program management. Trust is also reinforced by a transition strategy, generally followed by the FAA, that maintains previous equipment and procedures as backup until controllers develop confidence in the new system.

Training that provides to controllers an adequate mental model of how the system works, why it can be expected to be reliable, and how the controller can fulfill assigned responsibilities in the event of system degradation also contributes to trust—whereas "dumbed down" training that does not provide such information can detract from trust. It is important that users possess a clear understanding of their expected tasks in the event of system degradation or failure. These expectations will vary depending on system design and procedures. If the system design includes the expectation that, in the event of system failure, users will perform manually some of the same tasks performed by automation, then users must be trained (and the training must be periodically reinforced) to perform the manual tasks. However, if the system design includes the expectation that users will not be able to perform certain automated tasks manually, then this expectation must be made clear, and users must be trained to perform other recovery tasks (which may be quite distinct from performing the automated tasks manually). In addition, the general "build a little, test a little" philosophy introduces new systems in a manner that allows for an easier absorption rate by users; this may help smooth transition.

Continued human factors involvement throughout training, testing, and implementation contributed to user acceptance of ARTS and is contributing to user acceptance of CTAS and URET. In contrast, AMASS had poor user acceptance due to improper structuring of user involvement (National Transportation Safety Board, 1995b).

It is a hallmark of the integrated product team process that team members serve a program throughout the life cycle of its product. This has also been a hallmark of effective human factors programs, which include activities during implementation of systems. MIL-H-46855B is one example of a well-known guide that specifies a comprehensive human factors program to be applied throughout the life cycle of military systems. Continued human factors activity after system installation has three key advantages: development and administration of effective training programs, collection of data for long-term validation of systems, and facilitation of user transition and acceptance.

Performance data and user feedback should be collected throughout the early field use of the system. All data should be applied, when appropriate, to validation

(or contradiction) of prior system analyses (which include assumptions about how tasks will be performed and how well they will be performed) and, if necessary, to system upgrades. An additional area of investigation should be the interaction effects between the procedures associated with use of the new system and the organizational context (including formal structure and informal culture) into which the system is introduced.

It is very important, then, to establish a method for ongoing data collection, error and incident analysis, and sharing of lessons learned. TCAS, CTAS, and URET, for example, share lessons learned through newsletters. The ASTA and SMA projects update findings through the Internet. Technical and peer review reports are also important means of sharing findings (see the discussion of literature searches, above). It is not clear that the PRM project has established a method for performing and sharing the results of continued evaluation.

Currently in the developmental pipeline are a series of substantial air traffic control subsystems that are prospective inclusions in the national airspace system. These systems are described in Chapters 3 through 7 of this report. More research is needed to determine if these new subsystems can perform as well as expected and whether they fit together to make an effective total system. So far, subsystems have been developed in relative isolation from one another and from the overall modernization program. For example, the STARS and DSR specifications require that developers provide an architecture that will allow future plug-in of preplanned product enhancements; however, no human factors analysis of how those enhancements will be integrated with one another or with the standard terminal automation replacement system (STARS) and display system replacement (DSR) baselines is evident. As examples of products being developed in isolation from one another, URET is being tested in a different facility from CTAS, and there does not appear to be research considering the consequences of using both tools simultaneously. There is also little coordination evident between PRM and CTAS and between ASTA and SMA. As we discussed in detail in the Phase I report, the airway facilities monitoring and control workstation is a long-standing example of the negative effects (e.g., inconsistent human-computer interface, haphazard workstation configurations) of failing to consider in advance the integration of multiple components.

The lack of evidence of a unifying human factors analysis for advanced automation products, in order to guide their integration into complementary workstation designs or procedures, is also exemplified by CTAS. Although NASA's in-house scientists and their supporting contractors are also working on projects such as cockpit automation and data link in air traffic control, the role of data link with respect to CTAS, and the potential constraints of data link on CTAS, have not received significant attention of researchers.

In general, tests that determine intersubsystem compatibility should follow the tests that demonstrate subsystem performance. When the compatibility between pairs of subsystems has been established, and possible sources of confusion resulting from conflicting sensors, databases, and algorithms have been identified, the assembly should be enlarged to include other innovations until the subsystems that must be used together have been included in an overall test. At each stage, the evaluation should include a comparison against the base case represented by the current operational system.

Typically, despite careful analysis and validation efforts, not all human errors can be predicted. The human-computer interface is not the only source of error; new systems can introduce new sources of error (e.g., mode, logic, and procedural errors). This may be especially true when a given system will be integrated within a set of existing systems, or when systems in parallel development will be implemented together, because such integration can produce unexpected and unintended consequences. Reason and Zapf (1994) note that testing components in isolation and then putting them together opens the opportunity for previously unidentified "resident pathogens" to strike. Systems designed without consideration of the implementation context risk incorporating such error-inducing features as computer-interface logic that conflicts with that of other systems, information that unnecessarily duplicates (and possibly conflicts with) that provided by other systems, information whose interpretation or use requires data from other remotely located systems, information that confuses what is offered by other systems, alarms that distract the user from those of other systems, and disruption of team work flow (Miller et al., 1996). It should be noted that even field testing can miss unanticipated errors, caused by combining new systems, if systems planned for simultaneous implementation are not field tested together.

Such cumulative or interactive effects should be taken into account throughout a system development process that anticipates the integration of system elements, as discussed in detail above. In addition, since controller training, sector staffing, operational procedures, control room conditions, and equipment maintenance affect system effectiveness, system development and testing should include attention to how these context factors affect controller tasks, workload, and performance during use of the system under development and test (Grossberg, 1994). Modeling and analytical techniques, as well as prototyping and simulation, are all important methodologies for examining possible interactions between new technologies and the equipment and procedural contexts into which they are introduced. These techniques do not, however, obviate the need for operational validation in the actual air traffic control context.

The acquisition of commercial off-the-shelf (COTS) and nondevelopmental items (NDIs) does not warrant exemption from human factors analytical, design, or evaluation scrutiny. In the Phase I report, we suggested that human factors analysis, test, and evaluation activities should be applied, for a given application, to these items to ensure that they are compatible with the capabilities and limitations of users. In fact, there are aspects of such acquisitions that require human factors support beyond that provided for developmental acquisitions. For example, COTS items and NDIs may require modification of designs; such modifications can require design and evaluation activities that take into account the constraints imposed by the designs and procedures that come part and parcel with the items—while developmental acquisitions may proceed without these constraints. In addition, integrating the COTS items or NDIs into existing equipment, human-computer interfaces, and procedures may require additional study.

The FAA recognizes that human factors activities must be applied to COTS items and NDIs. The primary human factors policy document within the agency (Order 9550.8, Federal Aviation Administration, 1993c) emphasizes that human factors considerations must be applied to all acquisitions that involve human users. The agency's Human Factors Acquisition Requirements and Planning (HARP) document emphasizes the importance of human factors support of the following early phases of acquisition, which apply equally to developmental and nondevelopmental items: (1) mission analysis (e.g., conducting function analysis, identifying human performance and safety shortfalls, and developing the mission need statement); (2) investment analysis (e.g., performing trade-off studies, identifying staffing and training concerns); (3) solution implementation (e.g., conducting prototyping and simulation studies, conducting test and evaluation, and performing risk assessments); and (4) in-service management (e.g., supporting equipment installation and transition, conducting follow-on tests, monitoring staffing and training programs, and identifying requirements for modifications or redesign).

The FAA's human factors guidelines documents (Human Factors in the Design and Evaluation of Air Traffic Control Systems [Federal Aviation Administration, 1995f] and Human Factors Design Guide for Acquisition of Commercial Off-the-shelf Subsystems, Nondevelopmental Items, and Developmental Systems [Federal Aviation Administration, 1996f]) provide comprehensive descriptions of procedures for developing a program of human factors activities.

A special concern for COTS items and NDIs is the evaluation of system performance data and operational lessons learned. From the point of view of system validation, the acquisition process should include garnering of available system data relevant to human factors, including: specifications to which the system was built; modifications to those specifications, as well as rationale for

those modifications; the human factors program plan applied to the development of the system; documentation of approaches applied to define the functionality of the system (including the allocation of functions to human or to machine) and the human-computer interface (including options considered and rationale for selecting the options implemented and for rejecting others); test procedures and results; field evaluation data; program trouble reports; and training experiences. Ideally, human factors specialists and representative users from the FAA should interview past and current users of the system proposed for acquisition, as well as human factors and engineering personnel involved in its design, if feasible.

Currently, the STARS program represents a significant COTS/NDI acquisition. Since the STARS acquisition is quite recent, the program should be closely monitored for lessons learned.

A description of the current distribution of human factors research responsibilities by the FAA appears in the panel's Phase I report. In general terms, the headquarters unit is responsible for the coordination of information at high management levels. CAMI at Oklahoma City is responsible for research and development in support of controller selection and training and for short-term applied research requested by program management offices. The human factors unit at the Technical Center in Atlantic City is responsible for supporting the evaluation of operational alternatives, including new subsystems. The Volpe Center in Cambridge, Massachusetts, is not an interior part of the Federal Aviation Administration but has some responsibility for supporting the human engineering aspect of system design.

Much of the advanced research and technical development is delegated to contractors. In this regard, the NASA research facility at the Ames Center functions as a contract research and development unit. Here, advanced systems such as CTAS are developed with human factors participation from the resident staff of human factors specialists. Some exploratory research on human factors issues (e.g., advanced data link usage procedures) is also performed in this setting. Similar responsibilities have been allocated to the MITRE Corporation in McLean, Virginia, and the Lincoln Laboratory in Bedford, Massachusetts. Other commercial organizations and university research units are engaged in relevant research under contract from time to time.

The general picture presented of the human factors research and development effort within the FAA is that of a patchwork arrangement (Federal Aviation Administration, 1996e). This circumstance is not necessarily adverse except for the evident lack of coordination between units and the absence of a common set of conceptual objectives. These negative attributes are partially balanced by

positive features such as the stability of established role structures and a degree of local autonomy. For example, the relationships between researchers and operational people in the various FAA regions are generally mutually beneficial (and are reinforced by recent memoranda of understanding under which NASA researchers are supporting developments in such areas as the terminal area productivity program and the AATT/free flight initiative). Consequently, change in the present delegation of human factors research responsibilities for the sake of centralization of functions is not recommended. However, the need for serious reforms in the scope of activities should be considered. For example, the present provision for long-range feasibility studies is not adequate given the conditions of rapid evolution in computer technology. Such work should probably be done by organizations outside the Federal Aviation Administration and, indeed, outside the government. The in-house laboratories and those other facilities with close linkages to specific systems or subsystems need to continue in their current modes of operation. However, there is need for much more investment in generic studies and what might be called basic research.

Reliable mechanisms for the management of forward-looking research activities exist in settings such as the National Institutes of Health. In that setting, mission-oriented research and development activities are undertaken by the researchers employed within each institute. This arrangement corresponds to the present one within the FAA. However, each institute also sponsors extramural programs and projects by means of grants and contracts with organizations such as independent research centers and university departments. Proposals for research from such outside organizations are subject to rigorous evaluation by expert panels of peer scientists who are not employees of the institutes. Consequently, the record of scientific quality of the research sponsored by the National Institutes of Health has been exemplary. Applying this model to the management of extramural human factors research for the FAA would require special attention to the prevention of such potential problems as accepting research proposals that do not demonstrate expertise in air traffic control operations, that are not predicted to contribute knowledge useful to the development of air traffic control systems, and that do not support an integrated research and development program.

The budgetary allotment for such research by the Federal Aviation Administration would be many times smaller than that established by the National Institutes of Health. However, there is no reason why the quality of the science should be any lower.

One conclusion of the panel's Phase I report is that effectiveness of human factors activities also requires coordination and oversight by a central human factors management within the Federal Aviation Administration. In reaching that

conclusion, we considered requirements for an effective human factors program (as opposed to fragmented human factors research and development activities) identified in the following FAA documents: (1) the Human Factors Policy (FAA Order 9550.8, Federal Aviation Administration, 1993c), (2) the report of the Human Factors Subcommittee of the Research, Engineering, and Development Council (Report of the Committee on the Status and Organization of Human Factors Within the FAA, Federal Aviation Administration, 1996e), and (3) the National Plan for Civil Aviation Human Factors (Federal Aviation Administration, 1995g). Although our conclusion applies to all human factors activities within the agency, we reemphasize here, with respect to the development and acquisition of automation systems for air traffic control, that a centralized human factors program management for the Federal Aviation Administration should:

• Coordinate communication of human factors performance data across integrated product teams and between researchers, developers, users, and testers in the United States as well as in other countries;

• Develop and monitor human factors program plans;

• Monitor and guide the activities of contractors' human factors representatives;

• Develop policies and procedures for the application of human factors to the development, testing, and implementation of automated systems;

• Evaluate the qualifications and performance of human factors specialists; and

• Guide trade-offs pertaining to cost and schedule of human factors activities.

Human factors management should play a key role in identifying the appropriate mix of research and test methods that support system development. Human factors management should interface with engineering and program management personnel at the FAA and at support contractors to ensure that human performance requirements drive specifications and that hardware and software developers are responsive to the human performance requirements. Poor alternatives are the unfortunate situations in which human factors specialists become documenters of previously written computer code for the human-computer interface, or in which training is expected to compensate for poor design.

It is also the role of human factors management to remind program managers, as necessary, that good human factors is a "pay now or pay more later" proposition. By the time a system reaches late stages of development or testing, major design commitments have been made, resources have been spent, and there is reduced motivation to discover design flaws that threaten deployment schedules. It is not unusual for system designers or program managers to request that human factors specialists devise improved training programs to compensate for discovered design problems, after system designs are frozen. Training, however,

should not be considered a substitute for effective design (reliance on training will not prevent errors if the design itself is inadequate), and flawed systems often require redesign despite improved training methods. Systems in which human factors are not properly addressed may require costly redesign after inadequacies are discovered (Grossberg, 1994; Stein and Wagner, 1994).

Of course, an effective human factors program presumes the activity of knowledgeable human factors specialists. In addition, it is important that researchers, system developers, and developers of policies, procedures, and regulations share appreciation of the importance of human factors activities and understanding of fundamental human factors principles. There are several avenues by which the FAA can pursue development of human factors understanding throughout the agency, as well as the enhancement of human factors expertise:

• The human factors management function, as stated above, should include coordination of information sharing between researchers and system developers. One appropriate vehicle would be a human factors (in air traffic control or in aviation) newsletter, broadly disseminated within and beyond the agency to summarize studies, lessons learned, and issues raised by fielded systems, analogous in some respects to ASRS CALL BACK.

• The existing Human Factors Coordinating Committee, established by FAA Policy Order 9550.8 (Federal Aviation Administration, 1993c), represents a vehicle whereby FAA program managers can identify human factors needs and human factors specialists can provide (or link managers with sources of) information. To be effective, the committee must meet frequently.

• Widespread appreciation of fundamental human factors principles requires education of those within the agency who perform research, support system development and testing, and establish regulations and procedures. The FAA has developed at least four useful educational tools: (1) the National Plan for Civil Aviation Human Factors (Federal Aviation Administration, 1995g), which identifies key general human factors goals and principles for agency activities; (2) the guidebook Human Factors in the Design and Evaluation of Air Traffic Control Systems (Federal Aviation Administration, 1995f), which provides a comprehensive discussion of human factors principles, is written specifically to educate agency personnel, and includes a software tool for assessing systems under development; (3) the Human Factors Design Guide for Acquisition of Commercial Off-the-shelf Subsystems, Nondevelopmental Items, and Developmental Systems (Federal Aviation Administration, 1996f), which provides reference information for the selection, analysis, design, development, and evaluation of new and modified air traffic control systems and equipment; and (4) FAA human factors web sites (e.g., for the AAR-100 organization and for the FAA Technical Center) that contain useful human factors resources and identify human factors contacts within the agency. We trust that this panel's two reports will constitute a fifth key educational tool. We note here that such educational materials are expected to

convey an appreciation for fundamental human factors issues and methodologies, not expertise in human factors. We also note that there are many other documents that provide useful guidance for those interested in human factors (see Chapter 10 in the Phase I report for a detailed discussion of sources of human factors guidance and information).

• FAA acquisition programs have generally relied on development contractors and subcontractors to perform human factors activities. Qualifications of good human factors specialists, however, are not often made clear during the hiring of personnel by contractors, and the FAA has not generally reviewed the qualifications of human factors specialists hired by contractors. One function of FAA human factors management should be to do so.

• Training within the agency and hiring from without the agency remain alternative means of enhancing the human data base of human factors principles and criteria. Two additional resources are available. (1) Other government or government-funded agencies are repositories of human factors expertise and data. For example, the FAA shares a memorandum of agreement with NASA, whereby NASA human factors specialists support FAA research and development activities. (2) Academic programs throughout the country offer programs of study in human factors, at both the graduate and undergraduate levels. The FAA (e.g., through AAR-100, the Technical Center, and the Civil Aeromedical Research Institute, as well as through NASA) seeks the support of faculty from these institutions. In addition, the students at these institutions represent valuable sources of actual and potential human factors knowledge. The training they receive in human factors principles, methods, and criteria could be usefully augmented by increased opportunities for apprenticeships and internships within the FAA—whereby the agency would be increasing the very restricted pool of individuals whose expertise includes both human factors and air traffic control knowledge.

• The FAA should continue to work toward an agency infrastructure in which some human factors training is provided to personnel and program managers at all levels of the organization (and to contract teams).

• The FAA should continue to support integrated product teams with well-trained human factors specialists assigned to the team. These specialists should be responsible to human factors management within the FAA as well as to project managers.