3
Performance Assessment, Selection, and Training

The loss of half of the controller workforce from the 1981 strike placed significant pressure on the FAA to maintain continued high levels of performance with the influx of large numbers of new trainees who had little or no experience in performing air traffic control tasks. During the 1980s, in order to meet staffing demands, the FAA hired between 1,800 and 3,400 applicants a year as air traffic control specialist trainees. Although personnel researchers and managers had been working on performance appraisal, selection, and training programs over the years, the need for focused and efficient efforts in these areas became critical as applications from inexperienced individuals flooded into the government's Office of Personnel Management. For example, after the strike, approximately 67 percent of those hired as air traffic controllers had no prior experience in aviation, compared with 30 percent prior to the strike. Other differences are that the post-strike groups had slightly more formal education beyond high school and included slightly fewer minorities. As stated by Manning et al. (1988:1):

The continued safety of the NAS [national airspace] requires that ATCSs [air traffic control specialists] be carefully selected and trained. Each candidate for the occupation is continually evaluated, from an initial aptitude selection test battery through grueling performance-based screening at the FAA Academy, and finally in on-the-job training, conducted at the assigned facility. Because of the safety-related, critical aspects of the job, identifying and screening for characteristics in individuals that will predict success in air traffic control is especially important. In fact, research has demonstrated that not all individuals have aptitudes required to perform the duties required of an ATCS.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control 3 Performance Assessment, Selection, and Training The loss of half of the controller workforce from the 1981 strike placed significant pressure on the FAA to maintain continued high levels of performance with the influx of large numbers of new trainees who had little or no experience in performing air traffic control tasks. During the 1980s, in order to meet staffing demands, the FAA hired between 1,800 and 3,400 applicants a year as air traffic control specialist trainees. Although personnel researchers and managers had been working on performance appraisal, selection, and training programs over the years, the need for focused and efficient efforts in these areas became critical as applications from inexperienced individuals flooded into the government's Office of Personnel Management. For example, after the strike, approximately 67 percent of those hired as air traffic controllers had no prior experience in aviation, compared with 30 percent prior to the strike. Other differences are that the post-strike groups had slightly more formal education beyond high school and included slightly fewer minorities. As stated by Manning et al. (1988:1): The continued safety of the NAS [national airspace] requires that ATCSs [air traffic control specialists] be carefully selected and trained. Each candidate for the occupation is continually evaluated, from an initial aptitude selection test battery through grueling performance-based screening at the FAA Academy, and finally in on-the-job training, conducted at the assigned facility. Because of the safety-related, critical aspects of the job, identifying and screening for characteristics in individuals that will predict success in air traffic control is especially important. In fact, research has demonstrated that not all individuals have aptitudes required to perform the duties required of an ATCS.

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control All entry-level employees attend the FAA Air Traffic Control Academy in Oklahoma City for initial training in their respective type of facility (tower, TRACON, en route, field service station).1 Air traffic control specialists are certified at one of two different levels of training. Developmental controllers are in apprenticeship positions and are learning skills both on the job, under the observation of more skilled controllers, and in periodic training sessions within the facility. Full-performance-level controllers are fully certified and well trained for their tasks. Many of them engage in the training of developmental controllers. The objective of personnel selection and training is to produce successful controllers in the most cost-effective manner, cost being the expenditure of both money and time. Several methodological questions bear on reaching this objective. The key questions include: What criteria is the selection system designed to predict—performance on the job? Performance in training? What measures are used to define performance and how are they obtained? What cognitive and perceptual factors are related to effective performance on the job? How well do selection instruments measure the appropriate cognitive, perceptual, and personality factors? How well do selection instruments predict performance in training and on the job? How effective are various training programs both at the Air Traffic Control Academy and in the field in producing full-performance-level controllers? What are the attrition rates at various phases? Performance criteria are used by researchers to define the level of effectiveness to be achieved by personnel. Selection is based on the definition of the abilities that candidates need to achieve good job performance, and training is the activity by which individuals are taught specific aspects of the job. Without good measures of job performance, it is difficult to develop effective selection and training programs. Although it appears that the basic abilities needed by controllers have not changed over the years, they have had to undergo new training in response to the introduction of new equipment and procedures. With additional automation on the way, it is possible that changes will take place in both the array of capabilities and the attitudes needed by candidate controllers, in the ways in 1   Exceptions to this policy are made when an employee has had other training or experience that satisfies the requirements, such as attending certain private schools that specialize in air traffic control training or certain military air traffic control training or working as a controller prior to the 1981 strike.

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control which these capabilities and attributes are refined by training, and perhaps in the ways in which controller performance is assessed. This chapter discusses the historical development and current status of performance assessment, selection, and training of individual air traffic controllers and how these programs might change as more control functions are automated. Team training is discussed in Chapter 7. PERFORMANCE ASSESSMENT Our discussion of performance assessment is divided into two sections. The first section deals with the practices and issues surrounding the evaluation of controllers for the purposes of determining success in on-the-job training, effectiveness in job task performance, and eligibility for salary increases and promotion. The second section concerns the development of job performance measures for use as criteria in personnel selection research studies. Each section presents measurement issues and complexities. Performance Assessment for Management Decision Making Full-performance-level air traffic control specialists are responsible for the safe and efficient flow of aircraft through the airspace and the ground space they control. A variety of methods are used to evaluate the performance of these specialists, including real-time monitoring on the job, specially designed simulation exercises, checklists, and annual written performance appraisals. Real-time monitoring on the job provides immediate detection of violations in standards for aircraft separation and other errors. These data can be used to make immediate decisions about the controller's work status and the need for additional training. Checklists and written reviews are used once or twice a year to provide general technical assessments. In en route centers, dynamic simulations are employed extensively for training and skill assessment of developmental controllers. The most widely used method of technical performance assessment has historically involved immediate supervisors observing controllers at work and completing a checklist (Table 3.1 is an example) indicating the level of performance in each general task area (e.g., maintaining separation, communication). Until 1993, supervisors conducted 40-minute, over-the-shoulder evaluations of this type twice a year. In 1993 the over-the-shoulder program was deemed inadequate and, in 1994, work began on a new assessment program called the operational assessment program. The proposed new program, still in draft form, is planned to assess each controller's technical performance on an ongoing basis in the areas of separation standards application, communications, position/sector management, equipment operation, and customer service delivery. The proposed program calls for quarterly assessments and includes a summary of the supervisor's evaluation of the controller's strengths and weaknesses. Stein and

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control TABLE 3.1 Performance Checklist Job Function Category Job Function Separation 1. Separation is ensured.   2. Safety alerts are provided. Control Judgment 3. Awareness is maintained   4. Good control judgment is applied.   5. Control actions are correctly planned.   6. Positive control is provided Methods and Procedures 7. Prompt action to correct errors is taken.   8. Effective traffic flow is maintained.   9. Aircraft identity is maintained.   10. Strip posting is complete/correct.   11. Clearance delivery is complete/correct/timely.   12. LOA's/Directives are adhered to.   13. Provides general control information.   14. Rapidly recovers from equipment failures and emergencies.   15. Visual scanning is accomplished.   16. Effective working speed is maintained.   17. Traffic advisories are provided. Equipment 18. Equipment status information is maintained.   19. Computer entries are complete/correct.   20. Equipment capabilities utilized/understood. Communication/Coordination 21. Required coordinations are performed.   22. Cooperative, professional manner is maintained.   23. Communication is clear and concise.   24. Uses prescribed phraseology.   25. Makes only necessary transmissions.   26. Uses appropriate communications method.   27. Relief briefings are complete and accurate. NOTE: A 3-point scale is used to rate the job functions: (1) satisfactory, (2) needs improvement, and (3) unsatisfactory. SOURCE: Wing and Manning (1991). Sollenberger (1996) have been working on a performance measurement checklist based on psychometric measurement principles. Checklists are also used to evaluate controller candidates who are undergoing on-the-job training to become full-performance-level controllers. These evaluations are completed daily by the assigned instructors and monthly by their supervisor. The results are used to diagnose areas in which additional training is needed and to determine when the developmental controller is ready for certification. Further detail on training and the assessment of trainees as they move through the various stages of the program appears later in this chapter. In addition to these checklist evaluations, supervisor-prepared annual written

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control performance appraisals are given to both full-performance-level controllers and those still in training, the developmentals. According to an FAA memorandum of understanding on performance assessment (April 1994), each controller receives a mid-appraisal progress review as part of the process. Annual ratings have three levels: exceeds standards, fully successful, unacceptable; they are assigned to various performance elements, including operating methods and procedures, communications, and training. Although the ratings are not anchored with specific behavioral examples at the three category levels, some examples are provided for performance that exceeds the standard. Final ratings are based on weighting the elements (different weights are used for full-performance-level and developmental controllers). The results of the annual performance assessment are used in making decisions about promotions and salary increases. Relying on performance appraisal systems based on supervisor judgment has been a long-time concern of many private and public organizations, particularly with regard to the validity and reliability of the assessment instruments. However, according to the Research Council's Committee on Performance Appraisal for Merit Pay (Milkovich and Wigdor, 1991), the extensive research literature in this area does not provide strong guidance in choosing a performance appraisal system. The committee found mixed results regarding the advantage of job-specific ratings over global ratings. Moreover, although some researchers believe that scales based on job analyses and behavioral examples are advantageous in providing employees with constructive feedback, the committee found no clear evidence that behaviorally anchored scales are superior to other scale formats in informing the decision-making process. The use of objective measures to supplement supervisor judgment is most effective in assessing performance for jobs in which the tasks can be quantified. To some extent, the job of the air traffic control specialist provides such an opportunity. However, the problem has been that job analyses have shown wide variation in the specific content of the job depending on the type and level of facility to which the controller is assigned (Hedge et al., 1993). Not only is it true that controllers working in terminals perform different tasks from controllers working in en route centers, and that controllers in facilities with low-volume traffic have different job requirements from controllers in facilities with high-volume traffic, but also each controller's job is tied to a specific air or ground area or sector that has a unique set of features with which the controller must be extremely conversant in order to perform effectively. Indeed, there is evidence that knowledge of the specific airspace features around a facility is one of the most critical aspects of controller expertise (Redding et al., 1992). Because of these task variations and constraints, it has not been possible to develop a uniform performance test or set of tests that would fairly measure controllers' performance across the board. A more detailed discussion of the controllers' cognitive tasks and mental models can be found in Chapter 5. Over the years, however, some attempts have been made to address these

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control issues by creating simulated scenarios in a generic airspace. The use of simulation for assessment of air traffic controllers was first proposed in a study by Buckley and his colleagues in 1969. In this study, high-fidelity simulations of different traffic configurations in a generic airspace were used to measure individual differences is en route air traffic controller performance. In 1983, Buckley et al. extended this work and identified four important and independent categories for scoring controller performance across different sector geometries and traffic densities: Confliction: number and duration of conflicts, Occupancy: time and distance flown under control, Communication: number and duration of ground-to-air communications, and Delay: number and duration of delays. Although there is a great deal of merit in the approach proposed by Buckley et al. (1983), the argument against using generic exercises has been twofold (Borman et al., 1992; Hedge et al., 1993). First, becoming proficient in a new sector takes time, and, second, being tested has required going to a central location. Although it is possible that such an approach may be useful in developing performance criteria for selection, given the current technology, it does not appear workable as a performance measure to be used in making decisions about the future job responsibilities or salary levels of full-performance-level controllers. A further complication in using generic simulated exercises for the evaluation of full-performance-level controllers is the difficulty in obtaining reliable measures unless traffic densities are higher than the busiest live traffic in any sector (Buckley et al., 1969). As a result, the generic simulation is not an accurate test of performance requirements in the workplace. In this regard, it is of interest to note that, with current live traffic loads, the base rate of operational errors reported in one year is extremely low—there are approximately 800 errors spread across more than 15,000 active controllers generating about 3 billion opportunities for error (Hedge et al., 1993). It should be noted that reported errors represent the lower bound, since many of the minor errors go unreported. According to reports from the field, simulation exercises incorporating local features are currently used in en route centers and TRACONS to measure controller technical performance; however, most of these simulations are designed for training and testing developmental controllers or for upgrading the skills of full-performance-level controllers, using sector features with which the controllers are familiar, rather than for evaluation of job performance.

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control Performance Measures as Criteria for Selection Performance in Training Criterion measures used to date for the selection of air traffic control specialists have not been based on data collected on representative work samples of full-performance-level controllers; instead, they have been derived from the performance of developmental controllers during different stages of training (Hedge et al., 1993). Researchers at the Civil Aeromedical Institute have used a variety of measures that are based on weighted combinations of scores from written and simulated training exercises and from the time required in hours, days, and months for a developmental controller to move through each stage of on-the-job training and reach certification as a full-performance-level controller (Manning et al., 1988, 1989). The measures for training time for various stages of on-the-job training have been calculated separately for tower, TRACON, and en route facilities. Success rates at different training stages have also been calculated. For purposes of analysis, those who do not succeed are classified in the following categories: remaining in the same type of facility as a developmental but transferring to a lower-level facility, switching options (e.g., en route to tower), and separating from the air traffic control specialist occupation (Manning et al., 1989). All of these data are available through the training tracking system established by the researchers at the Civil Aeromedical Institute. In an article discussing progress toward developing criterion measures for air traffic control specialist performance, Hedge et al. (1993) show the various criterion measures that have been used and their intercorrelations. These measures are classified in the three areas of field training performance that include training time and subjective performance ratings by instructors; experimental measures of job performance such as high- and low-fidelity task simulation studies (Buckley et al. 1969, 1983) and job performance ratings; and operational job performance ratings used for decisions about salary increases and promotions. Figure 3.1, taken from Hedge et al. (1993), summarizes the results of studies examining the relationships among these criterion measures. Among the experimental measures, the most highly correlated are supervisor general and specific ratings (.86), performance on low-fidelity videotape simulations and peer nominations (.70), performance on high-fidelity simulations and over-the-shoulder ratings (.60), and specific supervisor ratings and specific peer ratings (.59). The strongest relationship between experimental and operational performance measures was between peer nominations and supervisor annual ratings (.56). Performance on Job Tasks Because good job performance is the ultimate goal of selection, it is generally acknowledged (Wigdor and Green, 1991) that selection variables should be linked to measures of operational job performance rather than to measures of

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control FIGURE 3.1 Summary of empirical evidence for interrelationships among criterion measures. Source: Hedge et al. (1993).

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control training performance. As noted above, in the domain of air traffic control, the most frequently used measures of job performance are supervisor ratings. What is recommended (Wing and Manning, 1991) is collecting data on full-performance-level controllers who are performing a representative set of work tasks. The precedent for such an effort is the Job Performance Measurement project, a Department of Defense project begun in 1980 to develop robust measures of performance in entry-level military jobs so that, for the first time, military enlistment standards could be linked to performance on the job (Wigdor and Green, 1991). The impetus for the project was the need to establish the credibility of military selection procedures after technical errors in computing test scores were discovered. The goal was to determine how well the Armed Services Vocational Aptitude Test Battery was able to predict performance on high-fidelity measures of job performance. Prior to this 10-year study, training performance had been used as the primary criterion measure. Throughout its development, the Job Performance Measurement project addressed many important methodological issues that apply directly to the development of hands-on job performance measures for air traffic control. Among these are: (1) identifying and selecting representative tasks as work samples; (2) developing performance measures and establishing criteria for what is to be considered as effective performance (what the full-performance-level controller does on the job and what he or she can do as demonstrated by simulation exercises); and (3) creating a comprehensive data collection plan. The separation and control hiring assessment (SACHA) program is an effort currently being undertaken at the FAA to develop a selection system that predicts performance of air traffic control specialists at work. It is anticipated that performance will be measured though work samples, behavioral ratings, and time required to achieve proficiency on different aspects of the controller's job. A critical part of this work, which draws directly on the experience gained in the Job Performance Measurement project (Wigdor and Green, 1991), is the development of job performance criteria based on hands-on tests. As noted above and discussed in Chapter 5 on controller cognitive tasks, there are several complexities associated with developing a representative set of work samples for test purposes. Key among these are the variability in controller jobs and the differences in the sectors being controlled. However, based on a task analysis of controllers in towers, TRACONS, en route centers, and flight service stations, there appears, at the broad level of job duties and worker requirements, to be some commonality among the first three positions (Nickels et al., 1995). Job duties and responsibilities are grouped in the following categories: Perform situation monitoring, Resolve aircraft conflicts, Control aircraft or vehicle ground movement, Manage air traffic sequences,

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control Route/plan flights—manage airspace, Assess weather impact, Respond to emergencies and conduct emergency communications, Manage sector or position resources, Respond to system or equipment degradation, and Multitasking. At the present time, FAA researchers and contractors are working from the results of the task analysis to develop simulation scenarios that can be used to test the core technical skills of full-performance-level controllers. These scenarios will probably use generic sectors that must be learned by controllers in the criterion sample. It is anticipated that training on the generic sectors may require three or four days. Performance testing is expected to require two to four hours. The development of both high- and low-fidelity simulations is under consideration. To supplement the data from work samples, FAA staff members working on the program are also developing a series of behaviorally anchored rating scales to be used as criteria in the selection system. Initial effort in preparing written examples of different levels of behavior in various categories was accomplished in workshops attended by subject matter experts in air traffic control. To date definitions and performance examples have been developed for the following categories: coordinating, communicating and informing, maintaining attention and vigilance, managing multiple tasks, prioritizing, technical knowledge, maintaining safe and efficient air traffic flow, reacting to stress, teamwork, and adaptability/flexibility. When complete, these scales will be used to assess the performance of a representative set of full-performance-level controllers. SELECTION The goal of any personnel selection system is to accurately identify applicants who will be successful in performing the job. For over 50 years, researchers have been working on developing effective selection tests for air traffic control specialists (Sells et al., 1984). As stated in the previous section, the criterion used in determining the validity of these tests for predicting success in the workplace has been performance in training (Sells et al., 1984; Manning et al., 1989). What follows is a brief history of selection research in the FAA. Underlying all this research is the use of task analysis techniques to identify the critical characteristics of the air traffic control specialist's job and the abilities needed to perform the job effectively. It is important to note that all selection tools and procedures have been tested to determine if they have an adverse impact on minorities, as defined by the Uniform Guidelines for Employee Selection Procedures established in 1978. In cases in which an adverse impact was found, the necessary adjustment were made (Manning et al., 1988).

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control Screening for Cognitive Abilities Selection research in the 1950s involved a number of studies using commercial aptitude tests to predict performance in controller training programs. The results of this research led in 1962 to the first Civil Service Commission test battery for selecting air traffic control specialists. This battery, which contained a series of tests on arithmetic reasoning, spatial relations, abstract reasoning, and air traffic control problems, was used for 20 years. In 1981, approximately 2 months after the strike, a new selection battery developed by the Office of Personnel Management (OPM) was implemented as a first-stage selection screen. The OPM battery consists of three tests: the multiplex controller aptitude test (MCAT), the abstract reasoning test (ABSR), and the occupational knowledge test (OKT). The current version of the MCAT includes paper-and-pencil simulations of activities required for controlling air traffic; several of the items portray situations that may result in aircraft conflicts, whereas others require time distance computations and manipulations of spatial relationships. An air route map showing allowable flights paths is provided (see Figure 3.2 for example). The ABSR is a 50-item test assessing the ability of applicants to infer relationships between symbols. The OKT contains items in seven knowledge areas related to controlling air traffic. Based on early experimental administrations of the OKT, Lewis (1978) found that the test was a better predictor of success in second-stage screening than self-reports of prior experience. A weighted average of the MCAT (80 percent) and the ABSR (20 percent) is used for the initial qualifying score; and applicant who receives a score of less than 70 is eliminated from the candidate pool. Those with scores of 70 and above can improve their total by the results of the OKT and by points assigned for veteran preference. The combined total score is referred to as the rating. Because of the historically high percentage of candidates failing to complete training and become full-performance-level controllers (approximately 44 percent in en route centers), in 1976 Congress recommended that a standardized, centralized program be put in place at the Air Traffic Control Academy. The goal was to put in a second-stage screen that would weed out the candidates who were less likely to succeed in field training. As a result, two nine-week programs were developed: one to screen candidates initially selected for the en route option and the other for candidates in training for tower positions (including TRACONS). In 1985 the two programs were combined into a single screen, and assignment of candidates to options occurred after the screen was completed (i.e., all candidates were screened and then assigned to positions). This second-stage screen, which combines selection and training for candidates with no prior experience, contains a set of nonradar-based air traffic control principles and rules and presents a series of laboratory simulation exercises to test the application of the principles. The laboratory exercises are standardized, timed scenarios that are graded. These exercise grades are combined with written knowledge and skills tests to calculate

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control FIGURE 3.2 Sample information from the Multiplex Controller Aptitude Test. Source: Federal Aviation Administration (1995). a composite performance score that determines whether the candidate passes or fails the screen. In 1989, Manning et al, examined the degree to which performance on these first-and second-stage selection and screening tests predicted success in training. The data for this study included: (1) field training performance measures for 3,185 en route and 1,740 terminal developmentals and (2) baseline data on 125,000 applicants who took the OPM selection battery as well as over 9,000 entrants to Academy programs. All correlations reported in this study were corrected for restriction in range. The mean score of Academy entrants on the MCAT was 90, and the mean overall selection rating was 91.6. Candidate performance scores in the Academy nine-week screen were consistently higher for the academic portions of the course than for simulated laboratory exercises; scores on these two parts were combined to obtain a composite performance score. The first analysis in this study was designed to determine how well the scores on the OPM tests predicted success in the nine-week Academy screen. The results show corrected correlations of .55 between the MCAT and the composite performance score of en route trainees in the nine-week Academy screen and .58 between overall selection rating (MCAT + ABSR + OKT + veterans points) and the composite performance of en route trainees in the Academy screen. For terminal trainees the correlations were slightly lower (.48 for the MCAT and composite performance score in the Academy screen and .52 for overall selection rating and composite performance). The second analysis examined the strength of the relationship between each

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control of the two screens (OPM and Academy) and the time required for a controller to reach full-performance level. Again, all correlations were corrected for restriction in range. Among the OPM tests, the MCAT is the best predictor of time to complete the phases of field training and reach full-performance level for en route controllers (-.26), whereas the OKT is the best predictor of time for VFR tower and TRACON controllers to reach full-performance level (-.21 and -.16, respectively). For the Academy screen, the best predictor of time to reach full-performance level for all three positions was the composite performance score (en route, -.36; tower, -.42; TRACON -.24). In general, the results suggest that both the OPM test and the nine-week Academy screen are useful predictors of training performance as measured by time required to reach the full-performance level. In another study, Manning et al. (1988) compared the attrition rates in field training before and after the introduction of the common Academy screen. They reported that, prior to the introduction of the screen, 38 percent of those in field training left the agency; after introduction of the screen, the field loss rate was cut to 10 percent. Even with this level of effectiveness in refining the population of candidates entering field training, it was felt that the nine-week screening program was too expensive and time-consuming. The FAA's policy was to hire candidates and pay them as employees while they attended the nine-week screen regardless of whether they passed the screen and continued on to field training. Another consideration in reducing the screen time was a desire to minimize the disruption in the lives of those who failed to complete the screen, particularly with regard to pursuing other employment opportunities. The approach to this problem was to ask researchers at the Civil Aeromedical Institute to develop a shorter, more efficient screen to replace the nine-week course. The result was a one-week computer-based pretraining, preemployment screen that includes tests designed around the abilities and aptitudes identified for effective performance of air traffic control specialist task (Weltin et al., 1992; Broach and Brecht-Clark, 1994). The major categories of aptitudes examined include sensory/perceptual, spatial working memory, verbal working memory, long-term memory, and attention allocation. The tests designed to assess the attributes are described in detail by Weltin et al. (1992). One test, the air traffic scenario test, provides a low-fidelity dynamic simulated work sample; the other two tests measure various cognitive abilities. Essentially, the candidates practice with the computer tests for 3.5 days and then are tested with a series of exercises. In 1991 a concurrent validation study was conducted to compare the power of the one-week pretraining screen with the nine-week screen to predict training performance (Wetlin et al., 1992; Broach and Brecht-Clark, 1994). In this study, training performance was defined as a combination of field training times and scores in the radar course. Although the relationship appears weak, the results showed that the new one-week screen was slightly better in predicting training performance (according to the above definition) than the nine-week screen. The

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control corrected correlations were .25 and .21, respectively. In 1992 the one-week screen replaced the nine-week screen. Currently, however, the one-week screen is not being used because it was not validated against performance on the job. As the new selection system proposed under the FAA SACHA program is developed, it is planned to incorporate new versions of these computer-based tests. Biographical and Personality Characteristics Other variables involved in selection research include (1) biographical data, such as high school performance, age, and prior air traffic experience (Collins et al., 1990) and (2) personality characteristics, such as openness to experience, extroversion, agreeableness, conscientiousness, anxiety, curiosity, and tendency to anger (Schroeder et al., 1993; Nye and Collins, 1991). Personality characteristics of air traffic control specialists have been studied using the following types of tools: the sixteen personality factor questionnaire developed by Cattell (1970), the state-trait personality inventory developed by Spielberger (1979), the occupational personality questionnaire, and the NEO personality inventory, which contains scales for five major personality constructs (Barrick and Mount, 1991), and a variety of self-report surveys. The contributions of personality variables to the screening of air traffic control candidates were studied by Schroeder et al. (1993). They examined five personality factors and found that, collectively, they explained an additional 3 percent of the variance in training performance over that explained by the cognitive measures. Specifically, air traffic control trainees, when compared with normative samples on the NEO personality inventory, exhibited lower average scores in neuroticism and higher average scores on the dimensions of extroversion, openness to experience, and conscientiousness. Another study of personality variables conducted by Nye and Collins (1991) found that air traffic control trainees appeared to have slightly less anxiety and anger and more curiosity than individuals in the normative group. Overall these relationships appear to be weak at best. Biographical characteristics of air traffic control trainees have been studied by VanDeventer et al. (1983, 1984), Manning et al. (1988), and Collins et al. (1990). Since the 1981 strike, the population of air traffic controllers has changed significantly with regard to prior experience—more than two-thirds of those hired after the strike have no prior experience in aviation compared with less than one-third before the strike (VanDeventer et al., 1983). Other differences are that the post-strike group is slightly more educated and contains slightly fewer minorities and slightly more women (Manning et al., 1988). The demographic variables that appear to have the greatest relationship to success in training are high school math grades, age, self-expectation of performance as a controller, and prior military experience (VanDeventer et al., 1983; Collins et al., 1990). Among these, age at entrance is particularly interesting.

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control VanDeventer et al. (1983) report that the pass rates for the second-stage screen fall off significantly at age 29 to less than 50 percent. It appears from their analysis that the age group most likely to succeed is under 26. As a result, the FAA established a policy of not accepting applicants for air traffic controller positions who are over 30 years of age. Potential Influences of Automation on Selection Researchers at the Civil Aeromedical Institute have conducted two studies examining the potential influences of automation on the selection of air traffic control specialists (Della Rocca et al., 1990; Manning and Broach, 1992). Both studies were based on task analyses of advancements proposed in automated en route air traffic control and the advanced automation system sector suites. Della Rocca et al. (1990) report two analyses that identified task changes as a result of proposed automation; basically they suggest that the controller's task would become less tactical and more strategic; the controller would have more information and be provided with an array of computer aids for conflict avoidance. However, the underlying cognitive and sensory attributes would not change significantly over those required in the current system. In the other study (Manning and Broach, 1992), nine air traffic controllers who had analyzed future requirements of proposed automation were asked to describe how they would expect controllers to perform a selected set of tasks with this automation and to identify the underlying abilities needed for effective performance. The expert controllers suggested that automation would lead to less verbal coordination and less need for the controller to process detailed information. However, they also believed that the underlying abilities needed to perform the job would not change from those currently required. As new automated solutions are proposed, it will be necessary to continue the task analysis process. The conclusion that automation will not require a change in the underlying abilities to perform air traffic control tasks was derived by considering the specific forms of automation proposed for the advanced automation system. It should be noted that other forms of automation are possible, and the implications of automation for selection should be reconsidered, whenever new forms of automation are contemplated. For example, it is possible that different abilities—or different weightings of abilities—will be required depending on whether automation shifts controller tasks: Toward more decision making and away from calculation or spatial perception, Toward strategic or toward tactical emphasis, Toward supervisory monitoring and away from hands-on control, Toward more human-computer dialogue and away from human-human voice communication,

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control Toward increasing emphasis on efficiency while preserving safety,or Toward team behavior and away from individual behavior. Researchers with the FAA's SACHA program are continuing efforts to develop a new selection screen that will eliminate the need for the OPM screen. Since researchers have found that learning occurs by taking the MCAT test and that those who take it more than once achieve higher scores, it was decided to replace it with a more reliable measure of underlying cognitive ability. The new battery will be built on the results of the recently completed job and worker requirements analysis (Nickels et al., 1995) and will include measures of cognitive, perceptual/spatial, and interpersonal characteristics. Plans are to incorporate the computer-based selection tests developed for the one-week screen into this test battery. Currently, it is anticipated that a concurrent validation study of this new battery will be conducted within two years using proficiency measures from high-fidelity simulations of air traffic test problems. Full-scale implementation is expected by 2000. It is also expected that changes in the program will occur as additional information is gained concerning plans for automation. One useful source of information in developing the new selection system is the continuing research of Ackerman and his colleagues at the University of Minnesota. Most recently, Ackerman et al. (1995) examined the power of a broad set of ability and personality traits to predict skill acquisition during different stages of training in a TRACON simulator. Their results show that, whereas cognitive and perceptual ability scales provided the strongest predictions, overall predictive power could be enhanced by pooling ability measures with measures of personality and self-concept. One of the most powerful predictors was perceptual speed. Work in this area should be conducted on a regular basis to reflect the potential changes in the relationships among the air traffic controller tasks and the power of the predictor variables. TRAINING Air traffic control specialist training is accomplished in several phases. Air traffic control specialists are employed by the federal government under the general service (GS) pay system.2 As noted in the previous section, a nine-week course for applicants with no prior experience in air traffic control was introduced 2   The grade for an individual position is determined by several factors, including the level of service demand at a particular air traffic facility, which can change depending on the traffic count. Facilities range from level 1 (lowest activity) to level 5 (highest activity). Flight field stations have three levels, and the highest grade for those controllers is GS-12. The terminal facilities have five levels, and highest grade is GS-14. En route centers have three levels, and the highest grade is GS14. Supervisor grades for all three types of facilities range between GS-12 and GS-15. Entry-level positions are generally GS-7; however, individuals sometimes qualify for entry at the GS-9 level.

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control at the Air Traffic Control Academy in 1976. This course had two functions—to provide initial skill training in nonradar tasks and to act as a screen for the next stage of training. In the past, those who passed the course (approximately 60 percent) entered field training as developmentals. Those who completed the nonradar portion of field training returned to the Academy for radar training and then went back to the field for additional on-the-job training and eventual certification. At each stage, pass/fail points were built in. In recent years, several changes have been made. One change is that the philosophy throughout the program has become ''train for success." Instead of imposing several pass/fail screens in the training process, the idea is to provide a supportive learning environment for the developmental—one that incorporates deficiency diagnosis and skill enhancement training as part of the overall program. This philosophy has led to a redesign of training in the facilities to ensure continuity of on-the-job instruction throughout the process. The underlying rationale was to create a training system that is supportive and reinforcing rather than one that is punitive. A second change is that more training is conducted at the facilities. Specifically, for those in the en route option, all radar training now takes place at the facility; however, those assigned to towers and TRACONs still take the Academy radar course. Thus, except for the introductory course, all en route training occurs at the assigned facility. A third change is that new applicants are being accepted only if they have previous experience in air traffic control; as a result, the first introductory course at the Academy is not being taught at the present time. The current sources of these experienced trainees include former members of the Professional Air Traffic Controllers Organization (PATCO), the military, the collegiate training initiative (CTI) for air traffic controllers, those currently functioning as air traffic assistants, and those in special cooperative and predevelopmental programs in the FAA. At the present time, most controllers are former PATCO or CTI graduates. The CTI program plans to expand from its five original institutions to 10 or more, with the goal of producing approximately 700 graduates a year. It is anticipated that some of the new programs will be located at institutions that are in regions and areas in which staffing has been difficult. As the requirement for new controllers increases and as new equipment is introduced, it is anticipated that selection techniques will become increasingly important for identifying those applicants with appropriate abilities for the job. In addition, modifications in training are also expected. As a result, researchers are actively working on new programs to be put in place by 2000. An important aspect of this effort is the significant strides being made with microcomputer-based simulations for training. One example is the work currently being conducted using the TRACON simulation developed by Wesson International (Ackerman et al., 1995). According to the Air Traffic Control Technical Training Order 3120.4H

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control (1995), once a developmental reaches his or her assigned facility, a team is formed to manage the individual's training. The team includes the developmental, two assigned on-the-job training instructors, the developmental's supervisor, and perhaps the facility's training administrator. All technical training requirements and instructional program guides are provided by headquarters. Some training is standard for all developmentals; other training is facility-specific. As a result, training and its evaluation may vary substantially from one facility to another. For the most part, facility training begins with classroom instruction, followed by work on the simulators. The content of these activities is tied to the facility's areas of responsibility. Many of the simulation exercises are developed around live traffic scenarios that have occurred in the sectors covered by the facility. These DYSIM (dynamic simulation) exercises are used to assess the developmentals' performance on different aspects of the job. In the future, SATORI (situation assessment through re-creation of incidents), which provides a graphic display of data with synchronized tapes of verbal interaction, can be used to review the performance on DYSIM problems. That is, a trainee's performance can be replayed as a means of providing immediate, detailed, corrective feedback (Rodgers and Duke, 1994). In the en route centers, the first training is on the D-side (nonradar side) followed by the R-side (radar side) training. For both sides, the cycle begins with classroom and simulator training, followed by on-the-job training. The average and maximum time for each training activity is established by the facility. As developmentals move through the on-the-job training phases, they are observed and rated on a checklist each day by one of their on-the-job training instructors. These evaluations are used to determine the readiness for certification or the need to return to the classroom or simulator for skill enhancement training. Each month a supervisor also observes the developmental and completes the checklist. The purpose of continuous evaluation and the support of a training team is to provide the developmental with every opportunity to gain the necessary competence to achieve certification and move forward. If a developmental exceeds the maximum time allocated for training in a phase, then he or she may be given remedial instruction or be assigned to another option or lower-level facility. There are 13 training phases to be completed before a trainee reaches the full-performance level. At each phase there is a certification examination. Certification on equipment is provided by the Academy; position certification is provided by the facility. All examinations are developed by the Academy. The time required for a developmental to reach full-performance level may be in excess of three years. When new hardware and software are introduced, new training is required for all controllers who will be using the equipment. At the time of purchase, the FAA determines who will be responsible for the development of a training package to support the equipment. One choice is to have one contractor develop the

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control entire package (hardware, software, and training); other options include selecting a different contractor or developing the training package in-house. Training on new equipment may be conducted in three different ways. In some cases, the FAA will choose to train a special cadre of specialists, who in turn train the appropriate personnel on site. In other cases, a contractor is given the responsible for on-site training. A third approach is to give the training package directly to the facilities where it will be used and have the facilities conduct their own training. Larger facilities have training departments that manage and administer training for the facility, whereas smaller facilities may have only one person responsible for training. It is of interest to both the FAA and those enrolled in training that training program effectiveness be maximized. Academy and field training programs have attempted to reach this goal through the use of classroom instruction, specifically designed and controlled simulated exercises, and on-the-job apprenticeships. Although there have been no systematic comparative evaluations of these procedures as employed by the FAA, there is, however, some anecdotal evidence that training has a positive impact on controller performance. For example, the approach controller at Sioux City Gateway Airport who guided the flight crew of a severely crippled United DC-10 to landing on July 19, 1989, had recently benefitted from completing an excellent facility-based training program. This controller was praised by the FAA administrator for his professionalism, skill, training, and personal dedication. A review of the transcript of his radio communication with the flight crew reflected exceptional performance under extremely difficult and stressful conditions (personal communication, National Transportation Safety Board, 1996). Positive impact is, of course, demonstrated by the fact that full-performance-level controllers are produced by the training system. A critical question in designing effective training programs concerns how well training in a simulator or on the job transfers to actual performance on the job. On-the-job training as practiced in air traffic control facilities is essentially an apprenticeship program designed to systematically move an individual from the status of developmental to full-performance level. Thus, by the time the developmental reaches full performance, he or she has been performing the job for some time under the guidance of an on-the-job training instructor, thus making transition essentially seamless. The concept of apprenticeship learning emphasizes the idea that, if individuals are trained on elements that are identical to those in the job, the degree of transfer will be maximized. Recently this idea has been highlighted in the theory of situated learning (Lave and Wenger, 1991; Greeno et al., 1993), which further states that learning is a social activity that is facilitated by the context in which it occurs. Some general principles of situated learning are: (1) an individual's knowledge about an action is dependent on the situation, (2) learning occurs by doing, and (3) to understand learning and performance, it is important to understand the social situation in which the learning and performance occur. Although

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control the concepts of situated learning on the job have been well established, there may be concerns about safety, job stress, and the ability to provide the trainee with the full range of experiences in the real-world setting. Furthermore, in some settings, such as the flight deck, the stress of live performance and the difficulty of providing immediate and accurate feedback makes on the job training a less-than-ideal learning environment (O'Hare and Roscoe, 1990). As a result, it may be useful to use simulation techniques as a supplement (Druckman and Bjork, 1994). A key issue in the design of simulators is the degree of required fidelity and realism. According to Miller (1954), there are two types of fidelity—engineering and psychological. Engineering fidelity refers to the degree to which the physical features of the simulator represent the real-world equipment. It has been suggested by Patrick (1992) that psychological fidelity is far more important than engineering fidelity, particularly for training of cognitive and procedural tasks. The focus of this view is to determine the factors that are required to produce psychological fidelity in the simulation. A more complete discussion of situated learning, training transfer, and the features of effective simulations for training can be found in Druckman and Bjork (1994). Another consideration in designing simulated exercises is the potential effectiveness of decomposing the task and providing separate training for the subtasks. Naylor (1962) proposed that such part-task training would be most effective when the task is complex and the components are not highly integrated. Evidence for this approach is provided in a study of training for airplane flight skills (Knerr et al., 1987). By analogy, it is reasonable to assume that part-task training may be effective in helping air traffic controllers acquire skills for those tasks that are complex and not structurally integrated. The FAA is currently using the operational computers in the air traffic control facilities (HOST and ARTS) to provide simulation training. In the en route centers' simulation, it is called DYSIM, and in the terminal facilities, it is called ETG (enhanced target generator). Both systems generate simulated targets that can be maneuvered by a pseudo-pilot operating from a remote radar display with a keyboard. In these simulations, the radar display and keyboards are the same as those used in the actual control room, thereby creating what is called full-fidelity simulation. Scenario scripting is done by outlining the path of each aircraft in a specific simulation syntax that is difficult to master. This development process is extremely time-consuming. When the simulation is operating, it cannot be stopped and replayed for lesson reinforcement. In the ARTS ETG, in some instances the computer capacity of live traffic will limit the number of simulated targets available or even drop the simulated targets in the middle of a training session. To overcome the drawbacks of the DYSIM and ETG, the FAA is currently in the process of studying methods for providing simulations through the use of personal computers. The personal computer-based radar simulation will not be full fidelity, i.e., the simulators will not be the exact duplicate of what the controller

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control will use in the live control environment, but they will emulate the radar display and the keyboard entries used in the live environment. In addition, the FAA is conducting research and development tests at some terminal facilities using these simulators. State-of-the-art personal computers that can be purchased off the shelf are being used; they are user-friendly and scenarios can be generated within minutes, compared with the days required for ETG and DYSIM. Training scenarios can be stopped and rewound for lesson reinforcement and played back at a later time for review. Aircraft can be piloted by use of a pseudo-pilot as with the DYSIM and ETG, or they can be piloted through the use of voice recognition. In the initial stages of a controller's development, this can be a very useful tool, as it saves on the use of additional personnel needed to play the pseudo-pilot role. These systems are also less expensive than using the actual radar displays for simulation. Four personal computer-based control positions cost approximately $250,000—the equivalent cost of one radar display. In 1994, the FAA formally requested bids to provide personal computer-based radar simulators for the terminal environment. Contract award was expected sometime in 1996. SUMMARY Performance assessment for full-performance-level controllers has primarily involved checklist ratings by supervisors. Although some work has been put into designing simulated exercises, there appear to be too many complexities to consider standardized simulated exercises for purposes of performance assessment. According to the FAA, improvements are being planned for the over-the-shoulder method—an approach that offers direct assessment in the actual job environment, which is accomplished readily and can be repeated as often as needed. As automation is increasingly applied and system-level performance goals are established, additional work on controller performance evaluation will be required to parse out those aspects of system performance that can be attributed to the controller. This task becomes increasingly difficult with additional automation of functions, because human performance may be masked by machine/computer performance. The challenge is to develop precise definitions of system performance that permit the identification of the contributions of both controllers and machines/computers—and that allow for their assessment, both individually and collectively. There has been a significant amount of research on personnel selection within the FAA, and researchers have conducted numerous studies examining the relationships between predictors and criteria. The principal drawback in this work has been the lack of good performance criteria. Now, with the SACHA program, researchers are working toward the development of job samples that can be used to collect hands-on performance. The panel encourages development of these job-related criteria.

OCR for page 54
Flight to the Future: Human Factors in Air Traffic Control A comprehensive, integrated selection battery is needed that can be given to potential candidates in a short period of time. The content of such a battery should include tests of the skills and knowledge relevant to current and proposed job tasks as well as assessments of personality and demographic variables and their relationship to performance. As automation is introduced, it will be important to reevaluate the elements of the selection battery. A program is needed to provide formal evaluation of operational air traffic control training; most facility-based on-the-job training is idiosyncratic, with each facility making its own decision. There are currently evaluations of on-the-job training and simulator training for other jobs, but not for those of the air traffic control specialist. The panel encourages the use of simulation for training at each facility, particularly in light of the need for full-performance-level controllers to efficiently receive refresher training as well as training in the operation of new equipment. Reduction in staffing levels puts additional pressure on the development of such a capability.