4

FAA Approaches to Estimating Staffing of Airway Transportation Systems Specialists

Chapter 4 examines how well the existing staffing models for the Federal Aviation Administration (FAA) Air Traffic Organization—namely, the Windows Staffing Standards Analysis System (WSSAS), the Tech Ops District Model, and the new approach planned by the Grant Thornton study team—comport with the modeling philosophy and criteria developed in Chapter 3. The aim of all three models was, and is, to help the FAA define its needs for accurate and timely staffing. The committee reviews the existing two models and the Grant Thornton approach by comparing them against the criteria identified in Chapter 3 to evaluate how well they meet the needs of the FAA. Next, alternative approaches from other domains, including the U.S. Air Force, FAA Air Traffic Control, and other countries, are discussed as additional potential sources for alternative and perhaps better models.

This review provides information regarding the desirable and undesirable features of existing staffing models and lessons learned from them. The comparison between the two FAA Airway Transportation Systems Specialists (ATSS) staffing models and their evaluation against explicit, documented criteria together provide a logical basis for future approaches that are likely to lead to valid models with practical utility for both staffing level decisions and allocations of a predetermined staff level to sites and tasks. The committee has highlighted those aspects of existing models that provide useful data and techniques, to aid the FAA in building on existing capabilities.

The committee drew on a number of sources for its evaluation of the various models, including FAA reports of evaluations of the two existing models (Grant Thornton, 2011), users’ and technical guides to the WSSAS model (FAA, 2012g), and a final recommendations report on the existing models and proposed approach, written by Grant Thornton (2012). These written sources were supplemented by briefings from relevant personnel at FAA, Professional Aviation Safety Specialists (PASS), and Grant Thornton, interviews with FAA management and Grant Thornton, interviews with PASS leadership, and stakeholder input via a public invitation to comment on the project via an Internet website (see Chapter 2). Another major resource was the committee members’ own expertise in modeling, human-systems integration (HSI), industrial and systems engineering, organizational psychology, economics, and operations research and their experience in industry, aviation, the U.S. Air Force, and the Department of Defense. Two technical papers on queuing models of ATSS maintenance tasks and their staffing



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 64
4 FAA Approaches to Estimating Staffing of Airway Transportation Systems Specialists Chapter 4 examines how well the existing staffing models for the Federal Aviation Administration (FAA) Air Traffic Organization—namely, the Windows Staffing Standards Analysis System (WSSAS), the Tech Ops District Model, and the new approach planned by the Grant Thornton study team—comport with the modeling philosophy and criteria developed in Chapter 3. The aim of all three models was, and is, to help the FAA define its needs for accurate and timely staffing. The committee reviews the exist- ing two models and the Grant Thornton approach by comparing them against the criteria identified in Chapter 3 to evaluate how well they meet the needs of the FAA. Next, alternative approaches from other domains, including the U.S. Air Force, FAA Air Traffic Control, and other countries, are discussed as additional potential sources for alternative and perhaps better models. This review provides information regarding the desirable and undesirable features of existing staffing models and lessons learned from them. The comparison between the two FAA Airway Transportation Systems Specialists (ATSS) staffing models and their evaluation against explicit, documented criteria together provide a logical basis for future approaches that are likely to lead to valid models with practical utility for both staffing level decisions and allocations of a predetermined staff level to sites and tasks. The committee has highlighted those aspects of existing models that provide useful data and techniques, to aid the FAA in building on existing capabilities. The committee drew on a number of sources for its evaluation of the various models, including FAA reports of evaluations of the two existing models (Grant Thornton, 2011), users’ and technical guides to the WSSAS model (FAA, 2012g), and a final recommendations report on the existing models and proposed approach, written by Grant Thornton (2012). These written sources were supplemented by briefings from relevant personnel at FAA, Professional Aviation Safety Specialists (PASS), and Grant Thornton, interviews with FAA management and Grant Thornton, interviews with PASS leadership, and stakeholder input via a public invitation to comment on the project via an Internet website (see Chapter 2). Another major resource was the committee members’ own expertise in modeling, human- systems integration (HSI), industrial and systems engineering, organizational psychology, economics, and operations research and their experience in industry, aviation, the U.S. Air Force, and the Depart- ment of Defense. Two technical papers on queuing models of ATSS maintenance tasks and their staffing 64

OCR for page 64
FAA APPROACHES TO ESTIMATING STAFFING OF AIRWAY TRANSPORTATION SYSTEMS SPECIALISTS 65 (Hecht and Handal, 2001; Hecht et al., 1998) provided data on the feasibility of stochastic modeling and outcome prediction in this domain. The enabling legislation for the current study specified that the resulting report to Congress be completed within 1 year; this time constraint precluded the committee from gaining hands-on experience with the two existing systems, as neither of them is currently in use or projected for future use. HISTORY OF FAA MODELING EFFORTS FOR ATSS STAFFING The existing staffing models and the Grant Thornton approach are briefly described here as the basis for more detailed examination and assessment in subsequent sections. WSSAS Staffing Model The WSSAS staffing model, which was developed in the late 1980s and early 1990s, is an updated, Microsoft Windows–based version (using the Microsoft Access database management system) of an earlier MS-DOS-based program. The two factors driving staffing demand in the model are (1) the number and types of equipment maintained, and (2) the time required to maintain the equipment. The WSSAS model is based on a synthesis of the required maintenance times for each piece of equipment in the inventory (Grant Thornton, 2011). Original data on times required for both routine and nonroutine maintenance come from direct time studies conducted by the FAA or studies conducted by contractors as part of the development of each piece of equipment. Times for each task are totaled across each unit, with allowances added for travel time, nonavailable time, and indirect work. An overhead allowance is also added to cover “technical and program support.” Many reports from the model are available to managers, but the two most frequently discussed in committee meetings were the “Book 2A” and “Book 2B” reports. The Book 2A report gives overall staffing levels for each unit for current and previous years, with the added allowances. Book 2B provides a further staffing breakdown by ATSS specialties (FAA, 2012g). WSSAS is a complex model that requires considerable input data on a regular basis to provide managers with useful information regarding staffing levels. There were no data on reliability or validity of staffing predictions of this model, although users appeared to trust its outputs. Tech Ops District Staffing Model The Tech Ops District Model was developed in 2006 and updated in 2007 as an alternative to WSSAS.1 It is a regression model based on then-existing staffing levels and six high-level input variables, including a number of large Terminal Radar Approach Control (TRACON) facilities. Its basic premise is that the overall 6,100 staffing level (with an annual 2.5 percent reduction in head count based on “expected efficiencies”) is an exogenously determined (i.e., set by others without regard to the findings of a model) staffing level that is not subject to change through staffing model analyses. The model uses six predictors of district-level workload requirements, derived from a larger set of potential workload predictors. The regression was constructed based on 38 districts and accounted for more than 93 percent of workload variability in the dataset. All six coefficients were positive and statistically significant, which are necessary conditions for a structurally sound model as well as a valid one.2 Output from the Tech 1Tech Ops Models: WSSAS and District Model. Personal communication from the FAA to the Committee on Staffing Needs of Systems Specialists in Aviation, November 29, 2012. 2Rich McCormick, director, Labor Analysis, FAA, data from unpublished 2006 “Technical Workforce Staffing and Training Plan.”

OCR for page 64
66 ASSESSMENT OF STAFFING NEEDS OF SYSTEMS SPECIALISTS IN AVIATION Ops District Model gives the staffing levels for each district or unit. The Tech Ops District Model is simpler than the WSSAS model and requires fewer data inputs to produce outputs in each time period. Staffing Approach by Grant Thornton–Led Study Team In April 2011, Grant Thornton was commissioned by the FAA to bring together a team, including representatives from PASS and FAA management. This team assessed the WSSAS and Tech Ops models and began to develop a new approach. In a presentation to the committee, FAA representatives noted that “presently, staffing decisions are based on management’s assessment” rather than formal models. 3 The assessment of the two current models by Grant Thornton (2011) was designed to influence the design of new models, based on the best features of existing models, to better serve the staffing needs of ATSS. A design based on this assessment is now being studied (Grant Thornton, 2012), although bids for devel- opment have not been solicited and the new model is not intended to be completed before developers can take advantage of any recommendations in this National Research Council report. Based on their review of the FAA historical models, Grant Thornton has proposed an alternative approach, which is the subject of this section. Their methodology builds on the positive aspects of the WSSAS. Staffing demand is estimated from the amount and type of equipment supporting the National Airspace System (NAS) and the time necessary to maintain each component. The validity of the WSSAS (and consequently its predictive ability) was questioned by Grant Thornton because of questions regard- ing the data capture and the integrity of the data inputs used in the model. Because the main causes for concern with existing models were seen to be validity of data capture and lack of prediction ability, an improved approach should address these concerns (Grant Thornton, 2011). Note that “predictive abil- ity” as used by Grant Thornton means the ability to accurately estimate changes in staffing levels as new equipment enters the NAS. The WSSAS model did not do that very well because of weaknesses in the data. To improve the predictive ability of a new model, Grant Thornton made three suggestions: (1) Place greater emphasis on testing and using valid data with improved data collection processes, potentially from currently collected sources such as the Facility, Service, and Equipment Profile (FSEP), Remote Monitoring and Logging System (RMLS), and Labor Distribution Reporting (LDR). (2) Use WSSAS as a solid basis for testing a new, predictive model’s results in real situations, to evaluate the new model’s accuracy. (3) Produce reports in a format understandable to the different end users. Such an approach could use stakeholder analysis (i.e., the systematic collection of the needs of the various stakeholders for the model) to determine explicit model requirements, collect the necessary data, plan and develop the model, and evaluate the model results against data. Stakeholder analysis describes the HSI process of determining what those who have to interact with the model (input, analysis, and output) require to have the model meet their job needs. Neither WSSAS nor a model constructed by the proposed (to date) Grant Thornton methodology would be able to predict the performance consequences of staffing at various levels. 3Rich McCormick, director, Labor Analysis, FAA, presentation to the Committee on Staffing Needs of Systems Specialists in Aviation, October 19, 2012.

OCR for page 64
FAA APPROACHES TO ESTIMATING STAFFING OF AIRWAY TRANSPORTATION SYSTEMS SPECIALISTS 67 COMPARISON BETWEEN CURRENT AND PAST MODELS To provide a sound basis for recommendations on future ATSS staffing models, the committee com- pared in detail the characteristics of each of the above models and compared each model with the criteria for valid staffing models provided in Chapter 3. The main headings of Table 4-1 are the four criteria areas of Box 3-2: Logical Design Process, Structural Detail, Quality, and Performance. To provide a succinct comparison that can guide future modeling efforts, Table 4-1 shows the relevant characteristics of the compared models as rows, with the three models as columns. Committee findings and conclusions on each of the three models are presented in the subsections below. Findings and Conclusions on WSSAS The WSSAS model was based on valid structure and suitable types of data and data sources. Essen- tially, WSSAS incorporated the pieces of equipment at each site, the number of technician hours required for each corrective or preventive maintenance action on each piece of equipment, and the probability of failure for each type of equipment. These measures were combined to establish the maintenance workload for each piece of equipment, and workloads were summed across all equipment at the site. Allowances were added for technician unavailability (e.g., training, leave, travel, etc.), and a factor was added for other required activities (indirect productive activities, including time at meetings, recordkeep- ing, housekeeping, etc.). Time data came from time studies and consensus judgments of subject matter experts, supplemented by contractor-provided times for new equipment. Neither the reliability of input data to the model nor the validity of the output staffing levels was measured. WSSAS was a deterministic model; in other words, the effects of unpredictable events were not factored into the model. The reports generated were useful to supervisors and administrators, and the model was reasonably transparent to all users. The model has been updated continually since its develop- ment; however, it has not been used recently. There was never an intention to validate WSSAS against outcome measures such as NAS equipment availability. Finding 4-1: The WSSAS model does not appear to contain stochastic elements in places where these may have been appropriate. Conclusion 4-1: The approach represented by the original WSSAS model included many of the impor- tant variables (e.g., equipment counts and task durations together with failure rates and allowances) to determine the staffing required at each site. Finding 4-2: The WSSAS model made no prediction of outcomes such as the impact of staffing levels on NAS availability and safety. Findings and Conclusions on the Tech Ops District Model The Tech Ops District Model was an allocation model only (a model aimed at distributing available resources effectively, irrespective of their collective adequacy), rather than a sufficiency model (a model designed to predict the resources needed to sustain system performances at an acceptable level). Thus, it is not an appropriate basis for moving forward toward a valid model for determining staffing require- ments. It is a nonmeasurement regression model that only shows, at best, how a number of variables are related in a statistical manner to then-current staffing levels by district. The Tech Ops District Model was not validated against outcome measures and is restricted by the fixed number of employees (i.e., 6,100). It does not have the capability to predict numbers needed based on goals of safety and performance.

OCR for page 64
68 TABLE 4-1 Structure and Evaluation of Current and Proposed Staffing Models for ATSS WSSAS Tech Ops District Model Grant Thornton Approach Logical Design Process Comprehensive Design Uses number and types of equipment Uses a regression model where Plans to use number and types of Captures major with estimates of task duration dependent variable is workload equipment with estimates of task workload drivers for each, plus allowances using an staffing. Rather than the tasks duration for each, plus allowances. Key algorithm described in WSSAS User themselves, six high-level parameters will be estimated largely in Manual (Grant Thornton, 2012). determinants of workload are the the same way as the WSSAS model. independent variables. Stakeholder Input No data on the stakeholder input were Regression-based to give rapid Plans to gather input from stakeholders Based on needs of all available for this historic model. assessment and minimize input. as basis for structure, inputs, and users User input not specifically stated to outputs. committee. Structural Detail Model Type Deterministic, additive task-based Multi-variable regression of existing The proposed approach is deterministic Additive components vs. components based upon number workforce in baseline year against and additive, closely linked to the regression and types of systems maintained existing drivers. The regression model WSSAS model. It follows the basic and measured or estimated person- identifies the predictor variables driver of the WSSAS model: equipment hours required to maintain each. within a district that directly impact inventory and time for maintenance Model applies allowances to account staffing and uses these variables to of that equipment. It is deterministic for time spent on nonmaintenance estimate staffing at the district level. in that corrective maintenance is activities and generates staffing This is actually an allocation model, not considered stochastically but is requirements. because no true measurement was based on historical, deterministic time performed. requirements (Grant Thornton, 2012).

OCR for page 64
Input Variables Captures equipment types Not based on true measurement of Plans use of equipment types and task Uses all significant and task durations plus travel hours required to perform duties, durations, plus travel times. Intended inputs times; accommodates other FTE detailed equipment counts, or tasks. inputs: consumption through use of No coverage of allowances. Variables N •  ew categories of work. Change allowances. useda: from 3 (recurring, nonrecurring, D •  istrict needs = Number of allowances) to 7 (maintenance, technical workforce employees monitor and control, other duties, needed in the district. admin, nonrecurring, travel, •  aximum hours available. M allowances) (Grant Thornton, 2012). •  umber of commissioned and N A •  djustment factors. Nonstandard noncommissioned, nonreportable 2 person, environmental, work area facilities, services, and equipment. type. N •  umber of Air Route Traffic N •  ew workload assessment process Control Centers (ARTCCs), to determine level of effort (Grant Large TRACON facilities, national Thornton, 2012). network centers. •  mproved Precommission Facility I File (PFF) for estimating new staffing requirements as a function of equipment changes. Output Variables Variety of reports as WSSAS outputs, Gives District-level staffing based on Plans to use stakeholder input to guide Produces all required synchronized to the FAA’s budget 2005-2006 levels and planned 2.5% appropriate outputs. Will be Web- outputs cycle. Book 2A and Book 2B were per year overall staffing reduction. based and use WSSAS outputs as the most frequently cited as useful basis for required outputs. Plans to outputs. Book 2A report gives overall apply an organization-specific version staffing levels for each unit for of overhead within the model for current and previous years with the District, Service Area, and National added allowances. Book 2B provides reports. District personnel noted that a further staffing breakdown by their definition of overhead includes ATSS specialties. Outputs include administrative personnel, supervisors, estimations of needed overhead (FAA, and potentially program support. At 2012g:4). FAA headquarters, Technical Support and additional levels of management are included (Grant Thornton, 2012). continued 69

OCR for page 64
70 TABLE 4-1 Continued WSSAS Tech Ops District Model Grant Thornton Approach Sources of Data Relied on direct time study data for Data gathered from sources readily Data sources are improved versions Subject-matter expert durations, Supplemented by estimates available at higher management and of the following FAA systems (Grant estimates vs. current gathered by experienced analysts. staff levels. Specific data sources are Thornton, 2012): staffing vs. time study Other data collected from systems as above, namely: FSEP with improved PFF to better (see Chapter 3 for fed by a combination of technicians D •  istrict needs—Number of predict future demand. discussion of time who performed maintenance, technical workforce employees RMLS is used for logging and estimates) data personnel at facilities, and needed in the district. tracking maintenance and has several management. •  aximum hours available. M components: Uses the following data-specific •  umber of commissioned and N •  ogging/tracking tool l sourcesa: noncommissioned, nonreportable •  eporting tool r F •  SEP—identifies type of facilities, services, and equipment. •  cheduling tool s equipment/services, specific N •  umber of ARTCC, large •  og for outages and maintenance l configuration of the equipment TRACON facilities, and national events. and its location; predicts new network centers. LDR is FAA system for logging time equipment and decommission­ng i worked on specific activities. System of old equipment. has several known weaknesses, but it C •  ost centers (facility). can at least be used reliably to account •  taffing values file—workload S for leave allowances. level of effort for each eLearning Management System for configuration of equipment. tracking training; will be suitable as •  ategories of work—recurring, C input for training allowance. nonrecurring, and allowances. Quality Usability Output reports were assessed as Easy to use, but output of limited Usability not yet measured, but Easy to use and update meeting users’ needs. Updated value to users. good process should be followed through adding data on new for obtaining user input at design equipment and revising data on stage. Uses archival data sets already current equipment as physical being collected so should be easier equipment and processes changed. than WSSAS to update. A likely New time studies rarely performed. improvement over WSSAS in that System fell out of use largely because it will be centrally updated and of difficulty of updating. maintained.

OCR for page 64
Staffing Validity Was widely seen by users as giving Assumed then-current District Validity in the context of the plans for Predicts staffing sensible staffing levels until it became staffing levels. This model only this model includes tests to ensure that levels perceived to be outdated. redistributed resources based on staffing equation code is without error, appropriate •  SEP data should be used to F ratio of current workforce to various works as intended, and incorporates a identify current NAS systems, workload predictors. Also forced a priori validity of subject matter experts. subsystems, equip and services in 2.5% per year decrease in available There is no validation in the sense of a the new model.a resources per year on assumption that prediction that is compared to a realized •  SEP not as accurate in predicting F undocumented efficiencies would be value for staffing conse­ uences. The q out-year additions, deletions, achieved. plans indicate better prediction of new and modifications to systems, staffing demands with improvements in equipment, and services due to the PFF. cumbersome modifications to model required because of PFF data limitations. •  ome perception that the model S may have been generous or overstaffed, as levels predicted were rarely fully funded and yet mission objectives still appeared to be met. Transparency Widely accepted by administrators Not task-based, thus not transparent Aims for understandable formulation Ease of understanding and users as transparent basis for to final users. based on task durations for each workload planning because of logical equipment type, plus allowances. structure of equipment counts, Similar to WSSAS. performance times, and allowances. Scalability Can be scaled down to level of District-based only. Would require Designed to aggregate at necessary Can be aggregated at individual specialty and scaled up to new regression to aggregate at higher levels, similar to WSSAS. Plans to go various levels district and national levels. levels or to decompose to individual down to level of individual specialty specialty level. and be aggregated to district and national levels. Performance Consequence Validity None. No consequences on system None. Outputs will not include consequences Predicts safety and availability or maintenance backlogs of staffing such as maintenance throughput outcomes are predicted, and thus WSSAS was backlogs or expected system availability never tested for validity. rates. aTech Ops Models: WSSAS and District Model. Personal communication from the FAA to the Committee on Staffing Needs of Systems Specialists in Aviation, November 29, 2012. 71

OCR for page 64
72 ASSESSMENT OF STAFFING NEEDS OF SYSTEMS SPECIALISTS IN AVIATION Any work regression model, without measurement of actual hours required to perform the task, will necessarily preserve the status quo in terms of the staffing level that is input to the regression. Therefore, the performance consequences of the output distribution of staffing cannot be assessed with the model and may not meet current or future performance requirements. The committee advises against the Tech Ops District Model as a source of either framework or data for future work, and this report does not consider it further as a basis for FAA staffing models. Finding 4-3: The Tech Ops District Model was an allocation model and not a sufficiency staffing model. Conclusion 4-2: The Tech Ops District Model is not an adequate framework for future work. Findings and Conclusions on the Grant Thornton Approach The design proposed by Grant Thornton for a staffing model appears to be an extension of past efforts. The Grant Thornton approach builds on the WSSAS model but suggests improvements to the key data sources for the future model. It appears to capture the relationships between equipment and maintenance staffing, and it has provisions for idiosyncratic factors (variances) affecting staffing. Although the Grant Thornton design represents distinct improvements over WSSAS, the committee has two concerns with an approach built on the basis of the WSSAS model. First, corrective maintenance in the ATSS job often occurs in an unscheduled, stochastic manner, and the WSSAS model does not account for the intrinsically stochastic nature of these events. Even if some elements like mean time between component failures can be predicted, some randomness or unpredictability related to the timing and location of a specific failure requiring corrective maintenance remains. Before dismissing various probabilistic aspects of maintenance work and methods, these stochastic elements and their relationship to adequate and safe staffing levels need to be understood and thoroughly explored to determine if and how they should be included in the model design. Second, like the WSSAS and Tech Ops District Models, the proposed model does not predict the consequences or results of staffing at alternative levels. It would be advisable to gather data necessary to study stochastic properties of outages, required time to repair, and shift profile dynamics mentioned in Chapter 3. It would also be helpful to explore potential linkages of key internal Technical Operations performance metrics to various levels of staffing allocations. Finding 4-4: The Grant Thornton prospective model builds on the earlier WSSAS model, and thus inherits some of its strengths and limitations. Its strengths include the same elemental data structure, supplemented with more recent data partly derived from existing data bases. Conclusion 4-3: Based on the latest description of the proposed model in the Grant Thornton report, the limitation of the Grant Thornton approach is the plan for a deterministic model that does not consider the implications of stochastic elements. Further, it makes no predictions of outcome measures such as NAS equipment availability and safety. There are a number of criteria based on the committee’s analysis from Chapter 3 that both the WSSAS model and the proposed Grant Thornton approach meet. Both of these models are task-based, using counts of equipment to be maintained and times required to maintain each piece as key inputs for a task. They also include allowances and variances to take account of indirect work, such as meeting attendance, paperwork, and administrative time, as well as travel time variances. They are solid logical models with only two omissions from the criteria in Chapter 3: (1) Neither of the models has a stochas- tic component to capture the inherent variability of task times. (2) Neither model’s outputs are directly

OCR for page 64
FAA APPROACHES TO ESTIMATING STAFFING OF AIRWAY TRANSPORTATION SYSTEMS SPECIALISTS 73 related to risk factors or overall NAS performance and safety. If the FAA were not concerned with these two issues, then the proposed model would be adequate. POTENTIAL ALTERNATIVE MODELING APPROACHES Before starting a major effort in model development, organizations commonly look to models that are in use in analogous situations elsewhere, both internal and external to the organization. These working, analogous models can provide a reasonable basis for evaluating approaches to new model development, as well as insights and lessons learned from others’ experiences. In the case of staffing models, an exist- ing model that is both accepted and used within the organization can be an important starting point for the development and design of a new model. Within the FAA, staffing models are used for at least two other job series—en route air traffic con- trollers (see National Research Council, 2010) and aviation safety inspectors (see National Research Council, 2006). The committee examined these FAA models for insight into model development pro- cesses that have worked in the past to give FAA management and staff useful models for staffing the organization. The committee notes that Section 608 of the Federal Aviation Administration (FAA) Mod- ernization and Reform Act of 2012 mandates a study of air traffic control staffing, with an approximate completion date of mid-2014. However, that study is running in parallel with this study, and findings from it are not yet available. The committee sought models from organizations that have a workforce similar to the ATSS work- force. Specifically, the committee attempted to review models from the Canadian and German equivalents of the FAA, and from the U.S. Air Force. Other FAA Staffing Models An effort within the FAA’s Office of Aviation Safety (AVS) organization resulted in creation of model criteria for staffing standards for the aviation safety inspector workforce (National Research Council, 2006). The committee understands that the AVS Staffing Tool and Reporting System (ASTARS) includes a predictive model for both the safety technical specialist workforce and the operational support workforce. ASTARS provides specific modeling equations for position types (e.g., operations inspec- tors, maintenance inspectors). The model considers the complexity requirements of specific jobs (based on factors such as the size of the fleet being inspected, the variety of aircraft type for which a specific inspector is responsible, and the experience levels of the maintenance staff subject to inspection). The committee understands that the FAA is also expanding the use of LDR data in ASTARS. The use of LDR data was considered problematic for the AVS workforce in the past because of reporting inconsisten- cies among the aviation safety inspectors personnel and limited categories for reporting certain types of tasks (National Research Council, 2006), and this committee has heard similar concerns from the ATSS workforce, as discussed in Chapter 2. If the AVS organization has now overcome those concerns about LDR, understanding their process could help the Technical Operations organization as it develops its models. Regardless of the use of the LDR for data input, the parallels between the AVS organization and the Technical Operations organization make the AVS model criteria a useful source for modeling processes in Technical Operations.

OCR for page 64
74 ASSESSMENT OF STAFFING NEEDS OF SYSTEMS SPECIALISTS IN AVIATION U.S. Air Force Staffing Models The U.S. Air Force has employed teams of manpower analysts to create staffing models for many years. Currently, the Air Force Manpower Division of the Air Force Personnel Center (formerly the Air Force Manpower Agency) provides Air Force officials at all levels with the tools to identify essential personnel resources required for the effective and efficient accomplishment of the Air Force mission. These tools are used to program personnel resources and are used as the foundational target for access- ing, training, assigning, and utilizing Air Force personnel resources. Air Force representatives presented the committee with an overview briefing of their modeling and requirements determination process. Because the Air Force includes a number of occupations that are similar to the ATSS job series in the FAA, the committee spent substantial time reviewing the models that the Air Force uses to establish staffing standards for some of those occupations. These models included Air Force Manpower Standards (AFMS) for Ground Radio Maintenance (AFMS 38AC), Meteorological and Navigation Systems Equipment Maintenance (AFMS 38AA), and Air Traffic Control and Landing Systems Radar Maintenance (AFMS 38AB) (see U.S. Air Force, 1998, 1995b, 1994, respectively). Each of these models quantifies the manpower required to accomplish the tasks described in a detailed process-oriented description for varying levels of workload. The AFMS outline specific core team requirements and predict personnel resource needs based upon counts of spe- cific equipment sets or equipment equivalents, multiplied by a measurement of personnel-hours required to maintain the various end-items. Several of the models contain variances for factors such as extended travel distances, unique equipment, corrosion control, seasonal storm days, and snow and ice removal. The instructions outline application of various allowances and factors and refer the reader to tables that show the FTEs required by skill specialty and grade. Although the Air Force has used these specific models for an extended time, the committee under- stands that new process-oriented models are currently under development, some of which may combine existing methods with process modeling techniques. The models the committee reviewed are similar in process, structure, and output to the FAA’s WSSAS model. The Air Force employs the Logistics Composite Model (LCOM) program to identify the manpower positions required to perform many aircraft maintenance activities. It is now a SIMSCRIPT 4 model that can run on many current desktop PCs, with many tailored versions built for specific weapon systems and scenarios. The LCOM simulation model integrates stochastic features such as queuing, Markovian processes, and vast amounts of available maintenance data to support a wide range of investigations into resourcing, skills required, staffing levels, performance, and consequences. Several iterations of the Air Force staffing models are usually run to help establish suitable outputs for creating staffing scenarios, which are often tied to peacetime or wartime utilization of the aircraft fleet. Other Potentially Relevant Models Although the committee requested detailed modeling information from NavCanada and Deutsche Flugsicherung, the Canadian and German counterparts to the American FAA, these private organiza- tions were not able to release any formal documentation concerning their proprietary staffing models. 4According to the website dictionary.com, SIMSCRIPT is a free-form, English-like general-purpose simulation language produced by Harry Markowitz and colleagues at RAND Corporation in 1963. It was implemented as a FORTRAN preprocessor for the IBM 7090 mainframe computer and was designed for large discrete simulations. It influenced Simula. Later versions included SIMSCRIPT I.5 and SIMSCRIPT II.5.

OCR for page 64
FAA APPROACHES TO ESTIMATING STAFFING OF AIRWAY TRANSPORTATION SYSTEMS SPECIALISTS 75 Another potential source for modeling assistance could be from industry modelers such as those in the aircraft maintenance sector whose models are proprietary and could not be viewed by the committee. A LOGICAL APPROACH TO A NEW MODEL FOR ATSS The search for alternative approaches to staffing models from other enterprises also provided some insights that largely confirm that a detailed deterministic modeling approach in the near term, with per- haps a later, more dynamic (stochastic) modeling approach, would be worthwhile and useful if feasible. The FAA’s modeling efforts based on the 2006 report on staffing models for aviation safety inspectors (National Research Council, 2006) resulted in the ASTARS model, which has been used to help deter- mine 2014 and 2015 staffing requirements (FAA, 2012f). The en route air traffic controllers’ model developed by Mitre’s Center for Advanced Aviation System Development (National Research Council, 2010) has been used in staffing of the en route centers except for those in Anchorage, Guam, and San Juan and the Oceanic areas of New York and Oakland. Both of these FAA models and those used by various parts of the Air Force are largely in line with the committee’s criteria discussed in Chapter 3 and summarized in Box 3-2. The committee concluded from its analysis that the approach suggested by Grant Thornton was likely to be successful if implemented because it is based on the logically sound approach of the WSSAS model, rather than on the Tech Ops District allocation model. The WSSAS model was correctly based on the counts of each equipment type at each location and on the times required for the different main- tenance needs of those pieces of equipment. In addition, the proposed Grant Thornton approach would be easier to maintain because it would use data capture from a number of existing sources to supple- ment the proposed new time study data. Thus, its continuing data requirements would potentially be less onerous than those of WSSAS, decreasing the risk that the new model would fall into disuse from lack of updating as NAS equipment and maintenance requirements change over time. The committee notes, however, that direct time studies can only measure durations of observable tasks. As work becomes less physical and more cognitive, task durations become more difficult to measure because fewer observable actions delineate their boundaries. It is still possible to measure overall task durations using time study, but component elements may not be as clear. However, the committee has two potential concerns with the proposed model. First, as currently proposed, it is a deterministic rather than stochastic model, even though it is known that time between failures and times for maintenance do vary, often considerably. Second, it predicts only staffing outcomes rather than true performance outcomes. The committee believes it is worthwhile for the FAA to consider each of these issues in more depth because they represent potential weaknesses that should be addressed at an early stage of model design. The two issues are related in that the level of service provided by the ATSS is a function of stochastic factors. In Chapter 3 the committee noted that the workload demand at any one point in time is highly variable, combining relatively predictable scheduled tasks and relatively unpredictable unscheduled tasks. The level of workload demand, when matched against the workforce available, determines the level of ATSS service accomplishment and thus the functioning of the NAS. The NAS has great designed-in redundancy; it even has mitigations such as increased Miles-in-Trail5 to keep traffic moving when all the redundancy is consumed. At some point though, the sheer number of tasks instantaneously requiring attention, compared to the available number of ATSS personnel, could lead to a loss of service. While much of the focus is on the nationwide staffing level, the maintenance 5Miles-in-Trail (MIT) restrictions are one of the most commonly used traffic management initiatives. They are most of- ten used to manage arrival flows into airports. Traffic managers often use MIT restrictions to protect a destination airport, particularly when capacity has been reduced due to weather or during periods of high volume. They also use MIT restric-

OCR for page 64
76 ASSESSMENT OF STAFFING NEEDS OF SYSTEMS SPECIALISTS IN AVIATION Probability of Service Staffing Level FIGURE 4-1 Conceptual logical relationship between service and staffing level. Probability of service is defined as potential availability of the NAS. It incorporates both failures and time to restore. SOURCE: Adapted from multiple figures in Hecht et al. (1998). system can become overwhelmed locally, even if national staffing is adequate. A truly valid model of staffing for ATSS personnel should be able to address this issue directly so that decision makers can make intelligent decisions balancing level of service and taxpayer cost at both local and national levels. An early example relevant to ATSS is the paper by Hecht and colleagues (1998). Their model used a stochastic modeling approach with different technician skills, an exponential relationship between failure rate and duration of repair, and a valid model of redundancy to predict the consequences of dif- ferent staffing levels. In this way the impact of staffing decisions on overall system performance could be quantified. Although the specific equipment levels, staffing levels, and redundancies used by Hecht and colleagues (1998) may no longer be relevant, applying their methodology is likely to result in valid models. Better system design can increase the level of service for any given staffing level. For example, technology improvements, training improvements, or organizational improvements such as better deploy- ment of multi-skilled ATSS personnel can increase the safety and stability of the NAS. For any given state of system design, there will be some relationship between service and staffing, generally showing increased service with increased staffing. This logical relationship is shown in Figure 4-1. It is not the responsibility of the modelers, nor of this committee, to determine the correct tradeoff between staffing and service: defining the acceptable tradeoff is a policy role of the decision makers at the FAA and ultimately of the federal government acting in the public interest. However, the FAA should provide a model that defines the service/staffing curve with some transparency of its linkage to underlying, objective drivers of that relationship. Such a model allows for explicit decisions on appro- tions to smooth out flows to support merging streams. (Definition found at http://www.mitre.org/work/tech_papers/tech_ papers_07/06_0967 [June 2013].)

OCR for page 64
FAA APPROACHES TO ESTIMATING STAFFING OF AIRWAY TRANSPORTATION SYSTEMS SPECIALISTS 77 priate staffing level to meet the desired level of service and also allows for system improvements to be evaluated in terms of their impact on overall system performance. The committee has assumed in Figure 4-1 that all instances of service denial are equal, so that prob- ability of service is the only measure of performance or effectiveness. However, the definition of service also needs to include a dimension of severity of consequences. The same outage (e.g., of a Primary RADAR or Radio channel) has far greater consequences at a major airport or en route sector than at a regional airport. Severity measures could include the number of flights affected or even the number of near-miss or actual collisions. This finding of differential impact from the same event implies that two staffing sites requiring the same probability of service should not necessarily be equally staffed. Almost any safety management system will include the two dimensions of probability and severity in assessing risks (see Figure 3-3, as an example), so this concept is certainly not novel to the FAA. A model incorporating both stochastic demand and stochastic ATSS staffing levels would be much more realistic than a simpler deterministic model. Such a model would also enable those responsible for staffing ATSS to show the impact of the chosen staffing levels on the measure of most direct interest to the FAA’s air traffic management customers: the service level of the NAS. Whether a deterministic or a stochastic model best meets the needs of the FAA is a decision that needs to be made based on the inputs and modeling effort required in relationship to the outputs needed—and in particular the potential consequences of ignoring the stochastic relationships. That decision should not be based on cost and convenience alone. SUMMARY AND RECOMMENDATIONS This chapter has taken the modeling approaches and criteria developed in Chapter 3 from staffing model knowledge and has applied them to the WSSAS model, the Tech Ops District Model, and the proposed Grant Thornton approach to modeling. First, the factual basis of three models was tabulated and assessed in terms of the criteria. Next, other sources of successful modeling in similar situations were assessed for the insights they might add to the future models of ATSS staffing developed by the FAA. The committee can summarize the attributes and the evaluation of models based on these attributes as a set of statements about what are good criteria for the FAA’s future modeling efforts. The best sci- ence currently available supports the following recommendations for a valid and usable model. Recommendation 4-1: The FAA should develop a new ATSS staffing model based on the modeling framework and criteria developed in this report. The model should be developed using a model structure that is based on equipment inventory, failure rates, and time to perform each task, and should include any valid allowances and accommodations. The model structure should include both deterministic and stochastic estimates for variables such as task duration, as appropriate. The developed model structure should be based on the different specialties of ATSS technicians, rather than providing just an overall staffing level at each facility. Recommendation 4-2: The FAA should develop a model that captures stochastic elements, unless it can be demonstrated that stochastic aspects of the maintenance process have no material effect on the staffing. For example, some tasks may exhibit multiple deterministic durations of identifiable elements, rather than strictly stochastic durations. Recommendation 4-3: The FAA should incorporate data for the model that are appropriate to the du- ration and frequency of the tasks modeled and to its data collection capabilities. Specifically, the FAA

OCR for page 64
78 ASSESSMENT OF STAFFING NEEDS OF SYSTEMS SPECIALISTS IN AVIATION needs a process to systematically validate through direct observation both historical estimates of task durations and estimates by subject matter experts. Recommendation 4-4: The FAA should ensure that the ongoing data collection and input essential for model use do not place an unacceptable burden on data providers. Recommendation 4-5: The FAA should ensure that output reports from the system predict consequences such as overall NAS availability time, deferred preventive maintenance activities, and overtime required. Recommendation 4-6: The FAA should ensure that output reports are tailored closely to the needs of FAA’s internal users at multiple organizational levels, in order to increase transparency of, and user trust in, the model.