Assessment of Staffing Needs of Systems Specialists in Aviation (2013)

3

Considerations in Creating a Staffing Model

The primary challenge for the committee was to find ways to create effective staffing models to determine Airway Transportation Systems Specialists (ATSS) staffing level requirements in Technical Operations work sites and appropriately accommodate the various stakeholder concerns discussed in Chapter 2. To address this challenge, the committee reviewed the fundamentals of modeling in general as applied to developing staffing estimates, which is the subject of this chapter. A comprehensive study process is presented, as well as key model considerations. The chapter concludes with the quality factors against which a staffing model is evaluated. The review of these modeling fundamentals sets the stage for the evaluation of existing models and creation of recommendations for the use of modeling to successfully define and predict ATSS workforce requirements in future efforts.

WORKFORCE MODELING AS PART OF A LARGER CYCLE OF WORKFORCE PLANNING

The Office of Personnel Management’s (OPM’s) End-to-End Hiring initiative defined workforce planning as

the systematic process for identifying and addressing the gaps between the workforce of today and the human capital needs of tomorrow. Workplace planning is based upon a set of workforce analyses which provide insight into how agencies can align their workforce to meet human capital goals and objectives that link to the agency’s mission and strategic objectives. (Office of Personnel Management, 2008:12)

The workforce planning cycle as defined by OPM entails five basic steps as shown in Figure 3-1. Step two of this workforce planning cycle—analyze workforce, identify skill gaps, and conduct workforce analysis—includes much of the committee’s task, which is to provide the Federal Aviation Administration (FAA) with guidance in determining the current ATSS workforce size and allocate ATSS personnel to match predicted workload. This portion of the cycle can be a difficult and poorly executed step of an overall workforce planning process, as developing tools to ascertain the kinds, numbers, and location of workers needed to accomplish an enterprise’s goals and objectives can prove a tremendous

FIGURE 3-1 OPM’s workforce planning cycle.
SOURCE: Office of Personnel Management, undated.

challenge. For a large enterprise, skipping the workload assessment/workload analysis dimensions and only performing a gap analysis on existing position structure and supply, then creating a plan to fill open positions, will likely be inadequate for defining the number of employees with particular skills sets and credentials needed in a variety of facilities and geographic locations.

Because ATSS personnel maintain tens of thousands of pieces of equipment of different types and at various stages of the equipment lifespan across a broad geographic area, and at a high level of operational readiness, defining and measuring the workload is formidable. Different philosophies about maintenance—for example, a philosophy of preventive inspections and maintenance versus one of “repair when the system breaks”—create a wide spectrum of potential staffing outcomes.¹ The expected levels of performance and tolerances for time between failures of systems may drive the need for extra shifts or ATSS personnel assigned to a particular problem, facility, or geographic location. ATSS technicians may be assigned to a particular task or may be in standby status on a shift and available via telephone for call-outs.

All organizations base staffing decisions on a paradigm of the underlying production process [or the means by which work is accomplished], whether they do so explicitly or not. This conceptualization is often referred to as a staffing model. A staffing model is a formal representation of the mechanisms that drive the need for staffing resources. (National Research Council, 2006:4)

__________________

¹As the FAA adopted reliability centered maintenance practices, some tradeoffs have already been carefully weighed and incorporated into the agency’s guidance for technical operations.

Changes in the services provided and, in the case of Technical Operations, changes in the amount or type of equipment maintained should drive the types and numbers of ATSS personnel required. An effective staffing model should represent work done with existing processes, unless the processes modeled are deliberately modified to reflect anticipated changes in the work; significant change to existing processes require updating or refining the model to ensure its accuracy. If important factors that have an effect on staffing are identified and accurately measured, then the algorithms of a good model should provide useful standards or staffing projections.

VALUE OF PRACTICAL MODELS FOR ATSS PERSONNEL

In many public and private enterprises, human resources account for at least 50 percent of the cost of the operation and can consume a great deal more of the organization’s financial resources.² The FAA’s Technical Operations branch currently employs approximately 6,000 full-time ATSS personnel, representing an estimated investment of more than $450 million a year in salary and benefits, or approximately $2.25 billion over a 5-year budget cycle. The true fully burdened workforce cost (which includes 6,100 workers) is likely higher.³ Most large enterprises keep track of their human capital costs, and that effort typically includes using tools that depict the size and skills of their workforce accurately. The output of well-designed modeling tools can strengthen the argument for a correctly sized workforce in budget discussions by providing clearer and reasoned estimates of the risks if the workforce has fewer people than the organization’s missions demand. Although technology is essential to the FAA mission of keeping the national airspace safe for movement of passengers and freight, the right mix of talented human beings is also essential to achieve that mission.

There are many ways to acknowledge the value of the ATSS workforce. Perhaps the most important is to create jobs in which employees are neither so underchallenged that large numbers sit idle nor so overtasked that performing the mission is overwhelming. A properly sized workforce helps ensure achievement of the FAA’s mission of maintaining the safety of the National Airspace System (NAS) at reasonable costs and levels of efficiency. Moreover, appropriate levels of staffing are likely to reduce turnover that is due to morale issues stemming from placing a responsibility on workers that cannot reasonably be met. An ancillary benefit to careful workforce planning is personal life balance for ATSS personnel.⁴ In addition, the value of an accurate staffing model also has to take into account the value of a high functioning NAS that is both safe and efficient. Although these costs are generally recognized and can be estimated with varying degrees of precision, it would be impossible to calculate the exact cost of understaffing in terms of its impact on business and commerce. One outage may have far-reaching and costly impacts that are never fully identified. Despite the difficulty of specifying the exact dollar amount, a robust staffing model must consider the far-reaching effects of understaffing.

__________________

²For example, a 2011 report stated that 80 percent of U.S. Postal Service outlays were spent on labor costs (U.S. Postal Service, 2011). National Income and Product Accounts data from the Bureau of Economic Analysis indicate that approximately half of state and local government total expenditures are wage and benefit related (http://www.bea.gov/itable/index.cfm) [June 2013].

³This is an approximate and conservative estimate, given the previous negotiated floor/threshold of 6,100 workers. If 6,000 full-time workers were employed at a fully burdened cost of just $75,000 per year per worker (which is on the low side), the annual cost would be $450 million and the 5-year cost (assuming no inflation) would be $2.25 billion. The FAA employs more than 30,000 workers and for fiscal year (FY) 2013, the Technical Operations budget request was $1.7 billion, which included a request for authorization of 8,050 full-time equivalent (FTE) employees (Department of Transportation, 2012).

⁴Comments submitted to Staffing Needs of Systems Specialists in Aviation Stakeholder webpage, 2013.

FIGURE 3-2 Steps for successful modeling.

CONCEPTUAL APPROACH TO MODELING USED BY THE COMMITTEE

The next section of this chapter presents the fundamental aspects of modeling that lead to success. Figure 3-2 highlights these components: (1) a comprehensive study process, (2) key model considerations, and (3) quality factors that can enhance the likelihood of (4) desired model performance. Because these components provided the basis for the committee’s assessment of the current assumptions and methods used by the FAA to estimate ATSS staffing needs and for its recommendations regarding more appropriate approaches to staffing models, each component is described in more detail below. Experienced modelers rely more on logic than the rigid application of any given method for developing a staffing model (Law and Kelton, 2000).⁵ Thus, throughout any modeling effort, the developers must continually rely on logic and think critically about the task.

Comprehensive Study Design Process

To be accurate in its estimates, a staffing model should be designed using a comprehensive development process that captures major drivers of ATSS workload at the appropriate level of detail and properly links the workload to the number of person-hours required to achieve the defined tasks of the job incumbents. Although it was not the intent of the committee to prescribe a detailed staffing model methodology, this report does discuss the steps in a generic logical model development process so that essential actions are considered and not overlooked in creating a specific model for ATSS personnel.⁶

__________________

⁵For a simulation approach, Law and Kelton (2000) provide examples of essential steps and considerations.

⁶These steps are derived primarily from modeling staff requirements in Department of Defense entities. The Navy Total Force Manpower Requirements Handbook, April 2000, lists five steps: Planning, Data Gathering, Data Analysis, Documentation and Reporting, and Implementation (U.S. Navy, 2000). The Air Force has used similar 6- and 7-step approaches. The Army once used a 12-step modeling algorithm but allows for many more variations today. The committee’s point here is to focus not on the amount of steps used but on the employment of a logical, comprehensive, phased approach that includes certain deliberate activities to create viable manpower staffing models. A caveat is that the conditions under which the military services analyze staffing requirements are both similar to and decidedly different from those confronted by the FAA. These organizations all have ATSS-like positions whose incumbents are responsible for maintaining the equipment necessary to manage air traffic in the airspace. However, the work environment and work rules that may have an effect on staffing vary considerably. For example, the military services, unlike the FAA, all have a greater degree of control of their personnel. Furthermore, the FAA

FIGURE 3-3 Logical design process: model development phases for comprehensive study.

The comprehensive modeling process described here consists of six phases: feasibility, familiarization, measurement design, measurement, analysis/model development and selection, and implementation. Most seasoned workforce modeling experts use a similar approach. For example, the Air Force uses its Management Engineering Program, which describes similar steps that have been refined over decades to produce generations of effective workforce staffing tools.⁷Figure 3-3 outlines the major phases of the comprehensive study process.

Phase 1. Feasibility

The objective of the feasibility phase is to determine if a modeling study effort should proceed or should be canceled or delayed due to problems such as nonstandardization, operational or organization instability, higher or conflicting priorities, and so on. The decision to continue development efforts is based on initial data- and fact-gathering concerning the responsibilities of job incumbents, the environment in which work is performed, and the resources available to the modeler such as time records, equipment lifespan data, etc. Two approaches to viewing work are commonly used: a work site approach and a work process approach. The work site approach focuses on a location and describes all the work that is done at that location. The work process approach focuses on a particular line of work and describes how it is performed in multiple locations. The environment to be studied includes dimensions such as the fiscal/budget situation, technological complexity of the work and equipment and anticipated changes, and the rate of change associated with the work performed. Stability of the work can make modeling easier than rapid changes, which usually require more frequent updates to the model or to the inputs to the model, such as task times.

__________________

must comply with work rules and contend with a labor union. Although these differences may not affect what steps need to be taken to develop a staffing model, they may affect how each step is implemented and what factors are taken into account.

⁷More information on the Air Force approaches can be found at AFI 38-201 and AFMAN38-208 Volumes I and Volume II; Army Guidance includes Army Regulation 570-4 and publications from the U.S. Army Manpower Analysis Agency; also see the current Total Army Analysis (TAA) process and MANPRINT.

A critical question to ask is “Do stakeholders agree it is the right time to conduct a study?” A major consideration in answering it should be the level of effort, time, and resources required to complete a study in light of the outputs desired. If the decision is to proceed with the study effort, a study scope and framework are established to guide collection of information needed to plan and conduct model development. Stakeholders should take the time to carefully define and review study goals, scope, and milestones, as well as consider the potential limitations of the results.

A number of multi-level considerations are relevant to feasibility. Specifically, there appear to be a number of potential cross-level interactions among individual training and higher-order factors such as location and distance. The committee believes that the model would probably benefit from exploring such nonlinear and cross-level interactions.

A memorandum of agreement should be established between the modelers and the primary stakeholders that documents what the modelers plan to do, what time frame they will do it in, what will be required of various stakeholders and the organization, and what output should be expected. A study announcement should be developed and shared with appropriate stakeholders, who may include management representatives from various functions and locations, job incumbents, union representatives, and others, to inform them of the effort and their responsibilities and to engage their support.

Phase 2. Familiarization

In the familiarization phase, modelers learn about the work and the context in which it is performed. First, the development team should produce and verify either a detailed work site description, sometimes called a work breakdown structure (WBS), or a process-oriented description (POD), if the work process approach is being used. A WBS is usually created by taking major work categories or components and breaking them down into smaller and smaller subcomponents. A POD is designed to document functions by inputs, process, and outputs, and then identify the subcomponents or details of these process elements.⁸ The WBS or POD should contain a fairly complete description of the various tasks performed in the work sites because it forms the basis for the staffing model. Creating a useful and accurate WBS or POD at the right level of detail is one of the most important outputs of this phase, as an accurate accounting of the activities that drive most of the worker effort is directly related to the accuracy and value of the staffing model output. It is also helpful to create a statement of conditions—that is, a description of the normal work environment and the operating challenges and unusual conditions faced by workers performing particular types of work in various locations.⁹ Modelers also find it useful, if not essential, to capture an initial set of key baseline data, such as the present structure of the organization, the number of full-time equivalents (FTEs) funded and currently employed, the allocation of workers across the organization, and staffing information such as details about personnel accessions by location and specialty.

Another component of this familiarization phase is the identification of potential workload factors (i.e., those factors that affect the hours of available labor for a work site) and the data sources in which information about each factor reside. Historical data related to the number of employees, time to completion for various tasks, failures and outages, etc., can be useful for trend or regression analyses whose results can inform the model of staffing needs. If the organization does not maintain or does not usually collect data on some of these potential factors, it must consider how best to gather the information

__________________

⁸Unlike a classic WBS, the Air Force shift to a POD allows ready process mapping and modeling, and may tie more directly with specified outputs of the activities, which in turn may potentially be more directly linked with performance measures.

⁹It should be noted that U.S. Air Force manpower studies, and their classic end product, an Air Force Manpower Standard (AFMS) include a statement of conditions. For more information, see AF/A1MR (2011).

required for accurate modeling and continue to maintain the data bases related to these factors. Examination of documents related to the job under study, such as official guidance, directives, standards, policies, performance measures, and transformation plans related to the organization, the work, or the equipment can also increase the modeler’s understanding of the job and of anticipated changes in the work.

Phase 3. Measurement Design

In the measurement design¹⁰ phase, the information gleaned from the feasibility and familiarization phases is used to determine a comprehensive study approach that includes the modeling tools, data inputs to the model, and the means to gather the data for the model. A critical question for the study team is how the data relevant to the modeling task will be gathered. For example, the team might use interviews of subject matter experts, a time study based on samples of work, or a by-location, by-work-element shift profile analysis.¹¹ Any combination of methods for collecting data may be used, but the approaches to collecting data need to be clearly defined, preferably tested on a sample of the overall workforce, and then refined based upon the lessons learned in the test, prior to conducting full-scale data collection. In addition, the team must be aware of the potential impact on the people who provide the data of any intervention necessary for data collection, especially direct observations. Otherwise, the team may receive distorted information about the nature of ATSS work.

The measurement plan should define samples that are representative of the population and sufficient to make the types of statistical inferences necessary for modeling. When the job under study involves a large number of workers across many work locations, the modeler does not necessarily have to measure in detail the work of every person at every location; instead, a subset of workers and locations that represent adequately the overall population will be sufficient in most cases. The modeling effort needs to include enough data points to be representative of the various types of facilities, types of equipment and tasks, and work centers to be statistically significant. If the organization is investing in a model covering thousands of FTEs, a 90 percent confidence level or higher may be desired. At one extreme, a belief that 100 percent of a large population should be measured can incur excess costs and time that consume study resources. At the other extreme, only measuring a few locations or relying on a small input sample, in an effort to expedite the effort or save resources, is even more likely to produce an unrealistic model output—with disastrous results if applied to the whole workforce.

The quality of the job information gathered must be considered, and when data are deficient, steps must be taken to improve the accuracy of the information to be used in the model. Several approaches to data verification exist. For example, multiple data collection teams can gather data and measure work performance at different sites. If multiple teams are used to conduct measurement and data collection, the study team must provide guidance to standardize the methods for data harvesting. If instead a “same eyes” approach is used, one team travels to all locations to collect data. Some modelers use a workshop or series of workshops facilitated by the study leads to collect data from subject matter experts. Often, the best approach is a combination of methods for systematically studying the tasks required in the job, the frequency with which the tasks are performed, and the corresponding number of person-hours needed to accomplish the tasks properly.

__________________

¹⁰For more information, see U.S. Air Force (1995a, 2003).

¹¹Time studies, work sampling, elemental time, and other methods are not discussed in depth here but may be researched more in Introduction to Human Factors and Ergonomics for Engineers, Lehto and Landry (2013), and other industrial engineering texts.

A measurement approach must be consistent with the later phases of analysis and model selection. Consequently, the study team must anticipate the types of models to be explored, designed, and ultimately implemented. At times, the phases of the study process are iterative, so that the modelers must reconsider earlier decisions based on decisions made in later phases. Often, the limitations of available data and the feasibility of collecting more robust data constrain the type of modeling possible; when accurate models are required, such constraints must be remedied with extensive data collection plans. However, costs are a relevant factor in most modeling projects, and the team, in conjunction with the organization’s management, must decide what approaches to data collection are affordable. They must also consider the financial implications of understaffing and overstaffing if the model’s validity is weakened by insufficient data collection.

Phase 4. Measurement

This phase involves the execution of the measurement plan created in phase 3 and refined through testing. Data collected in this phase contribute to model selection and serve as the input to the selected model. The quality of the data collected significantly affects the value of the model; thoughtful measurement can make or break the study. All data obtained in this phase must be consistent with the plans for the study and should be validated by examining the data for completeness, accuracy, and logical consistency.

Phase 5. Analysis and Model Selection

The purpose of model development is to depict accurately the worker to workload relationship in order to derive the number of workers required by function or discipline and by facility type. Often, correlation and regression techniques are employed to examine the data and determine the worker-workload relationship. Study experts frequently discover several subpopulations that should be grouped and then develop separate tools to portray the worker-workload relationships for each group. In the case of the ATSS modeling challenge, the five different types of facilities may be suitable for individual groupings. Alternatively, the five disciplines might form the basis for grouping. For example, a model for depicting Core/Large Airport requirements and a separate model for GNAS, or a model for Environmental and another for Communications, may produce more accurate results. Statistical tests of the relationship may be used to determine the sufficiency or best fit of the model equation.

The end result should be an equation with a related set of statistical tools that the organization can use to determine the staffing requirements. These tools are often used at the location level and then aggregated to obtain total system requirements. Ideally, algorithms produce a set of tables that indicate the correct number and skill mix for a work site to accomplish its mission sets. For example, the Air Force manpower management engineering process has built-in “Smart Sheets” that take workload factor data as input and produce electronic reports showing the number, skills, and grades of full-time workers required to match the workload.

The more transparent and accessible for use the model is to different levels of the organization, the more likely the model is to be embraced by personnel at those levels. However, access to the model output may be limited, based on the user’s needs so as not to overwhelm the user with massive amounts of information that are not relevant to that user.

Model design should also take into account the model’s adaptability to changing conditions. An effective model must accurately depict the current worker-workload relationships, but the better model also offers “scalability” so that the model can be easily updated as the work or the work environment change. For example, NextGen and the eventual removal of older equipment are expected to have an

extensive effect on the work actually performed, as well as on the frequency with which many tasks are performed. Often a modular approach is used so that the relevant components of the model can be updated without overhauling the entire model. As with any other modeling effort, a sensitivity analysis would also be appropriate to assess how sensitive the model outputs are to changes in various input variables.

Another component of model design is adherence to good human-systems integration (HSI) practices, including safety and human factors concerns. The model and its output should ensure that the staffing levels reflect what is necessary to protect workers’ health (U.S. Air Force, undated).

Often, the model design phase includes validation and verification¹² elements, especially for highly complex models. A rigorous verification process in which predictions are compared to real-world situations of the past for which outcomes are known is begun at the same time the model is being developed. Data from the validation and verification effort can be used to refine the model or enhance the data inputs to it.

The verification, validation, and acceptance (VV&A) processes used to improve the model and increase the organization’s confidence in it can take other forms. For example, qualified operations research or staffing model experts, teamed together with functional experts, can evaluate the assumptions, data quality, and modeling algorithms and point out opportunities for improvement. VV&A activities should be practiced throughout the modeling effort, both before and after implementation.

Other, more robust, formal VV&A approaches may be useful and worthy of careful consideration; however, as Carson (2002) noted, “a model developer intermixes debugging, verification and validation tasks and exercises with model development in a complex and iterative process … it should also be noted that no model is ever 100% verified or validated.” The final determinant of the value of the model will be the accuracy of the predictions regarding labor necessary to achieve the goals of the job class. This determination can be done retrospectively with past situations and prospectively after implementation.

Phase 6. Implementation

Implementation of a new staffing model can be complex and requires close attention to how the model is introduced. How the model is implemented will have an effect on the level of acceptance achieved throughout the organization. At some point, the recommended model must be presented to decision makers for approval for use. Revealing the “test” impact of the model to management and workers within a public-sector function with many diverse stakeholders can be challenging, as there will typically be perceived “gainers and losers” by location as the workforce is adjusted by the model results to match the work requirements. It will be useful to stakeholders to have an honest picture of the future staffing levels derived from the model, as well as an explanation of how they were derived.

Budgets also need to be adjusted before implementation, particularly when there are increases to the staffing levels. Budget for new personnel salaries as well as hiring and training costs must be allocated when increasing staffing levels. When staffing levels decrease, the costs of reductions in force and redeployments must be factored into budgets. The committee had several discussions regarding how an effective model would look and how it could predict an effective number of ATSS personnel necessary for maintaining the NAS in a safe, efficient, and effective manner. These discussions did not focus on a total number of ATSS personnel, but there was an understanding that, should the model generate a number greater than 6,100, the anticipated FAA budget would need to be quickly increased to avoid the

__________________

¹²Validation is the process of determining the degree to which a model or simulation and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model. Verification is the process of determining that a model or simulation implementation and its associated data accurately represent the developer’s conceptual description and specifications in DODI 5000.61 (Department of Defense, 2009).

negative implications for the safety and efficiency of the NAS. In that case, the FAA’s Air Traffic Organization would have to employ risk mitigation to compensate for the less than sufficient staffing while the budget was approved and additional personnel were hired, trained, and assigned to the appropriate sites.

Conclusion 3-1: Dedicated budget requirements for the ATSS are likely to result from application of any comprehensive manpower staffing model and will need to be addressed.

Public announcements related to increases or decreases in head count must be crafted to help outsiders understand the situation and win the public’s approval. Often, examples of locations where the work is radically different are persuasive in demonstrating how the local uniqueness was addressed. For example, higher levels of staffing in locations that experience special environmental conditions, such as corrosive salt water, extreme temperatures and weather, or extended travel to service remote equipment may serve to demonstrate the sensitivity of the model.¹³

Value of a Logical, Comprehensive Design Process

Following a comprehensive design process that contains the essence of the six phases described above should greatly enhance the likelihood of creating an effective staffing model that enables the workforce to achieve the organization’s goals. Some of the phases overlap, and ideally, study leaders intentionally look backward and forward in the process so that the end result is logical, valid, and compliant with the stated purpose. In particular, data examination during the measurement and analysis phases may necessitate revised data collection procedures or additional research. Moreover, those who develop the model need to plan for future improvements to it as modeling methodology and data collection procedures evolve and as the FAA obtains greater understanding of the causes of variability in the tasks performed by ATSS. Insights gained at any phase can redirect the study in unexpected ways; thus, the study team needs to be not only flexible but also seasoned in handling unexpected situations and understanding which approaches are most likely to be effective in a given situation.

Recommendation 3-1: The FAA should execute a modeling process that allows for future improvements in data modeling techniques and applicability.

Key Model Considerations

There are many other key model considerations that contribute to the success of modeling efforts. In this section, the committee reviews important considerations that will enhance the staffing model.

Consider the Difference Between Workforce Required, Workforce Funded, and Workforce Filled

Staffing standards produced by a given model are not identical to authorized or filled positions; required, funded, and filled are three different ways to describe the workforce and the numbers required. Staffing models may generate a fairly useful recommendation for defining a required workforce that is often refined by highly knowledgeable staff and managers, who then must compete for necessary funding in a resource-constrained public sector environment. Thus, the required workforce is not the same as the

__________________

¹³These conditions are termed as a “variance,” which is an adjustment to the model for FTE time added or subtracted from the core requirements and is used and described in Air Force manpower requirements determination literature such as Air Force Manual 38-208, Volumes 1 and 2 (U.S. Air Force, 1995a, 2003).

funded workforce. FAA headquarters can create policy and routines to apply standards on an ongoing, cyclical basis for the creation of staffing plans and projections, and can also provide authorization (i.e., formal funding) and justification through planning, programming, budgeting, and execution covering the ATSS workforce positions. Once positions are recognized and funded, they in turn must be filled. However, a filled workforce is not entirely dependent on hiring. Other tools of the human resource life cycle (recruiting, development, sustainment, retirement, and so on) affect the number of positions to be filled.¹⁴ For example, a model could predict a requirement for 18 FTEs to perform communication systems maintenance in one location, while the budget process may only authorize funds for 16 in that area, and there may be only 14 persons actually “on board” as filled positions. However, a good staffing model will provide decision makers with information regarding the expected consequences of under- and over-staffing. If fewer positions than are needed are authorized, for example, the model may indicate that a maintenance backlog is likely to emerge or that the risk of nonavailability of the NAS may increase. If more positions are authorized than are currently in the job, the hiring-training-development cycle needs to begin. Table 3-1 provides an example of staffing in one location across the five skill sets and management.

TABLE 3-1 Notional Example: “Location X” Staffing for a Particular Point in Budget Year

An advanced modeling system for a large enterprise may have both short term (1-2 years) and longer term (out to approximately 5 years) staffing requirements. The difference between the two estimates provides forewarning to the organization that changes in staffing levels are likely to occur. Proactive organizations build staffing-process ramps to increase or decrease the number of individuals with the skills to match the current staffing needs that are determined by changing equipment, services, and budget realities. Once the organization determines the extent of change that is likely to occur, it must adapt other systems (recruiting, hiring, training, etc.) and acquire the necessary funding.¹⁵

__________________

¹⁴These tools fall under the personnel and training domains of HSI and in themselves may require improved systems and their own respective modeling algorithms outside this discussion. Examples are maintenance training throughput calculations and attrition/retirement/hiring estimating tools, among others.

¹⁵Without straightforward and useful estimates of workload for the out-years, it is challenging to expect a model to predict staffing needs—with the exception of creating forecasts based on insightful trend data. Air Force manpower data systems usually show positions for multiple quarters/years out in the budget cycle, and future adjustments are often calculated based on known changes in quantity and type of weapon systems that will be present or approved as missions change.

Consider the Difference Between an Allocation and a Sufficiency Model

The National Research Council conducted a study for FAA aviation safety inspectors in 2006 that had a charge similar to that of the current study on ATSS staffing. Many of the former committee’s considerations at that time are applicable to the current task. Therefore, the present committee has quoted liberally from that prior report in this section. The following passage, for example, is directly applicable to the current study, with a few word changes to apply it to ATSS staffing:

__________________

¹⁶For example, the relationship to demand for meals served and the staffing of a food service operation may be modeled simply as a straight line approximation of a factor of additional workers per incremental change in how many meals are required per day.

[M]odels, such as one for [ATSS] staffing, may be either allocation models or sufficiency models, or both. An allocation model is one aimed at distributing available resources equitably and effectively irrespective of their collective adequacy, whereas a sufficiency model is designed to predict the resources needed to sustain system performance at what is deemed an acceptable level. A sufficiency model is more difficult to develop. It requires the organization to make decisions about, and set standards for, acceptable performance and to develop performance measures so that outcomes can be evaluated against those standards (i.e., it can be empirically validated).

[B]y providing an estimate of the resources necessary to meet policy and safety goals, a sufficiency model can be the most rigorous way to determine staffing needs and to support budget requests for [ATSS] positions.

[D]espite the distinction between sufficiency and allocation roles, we think that an [ATSS] staffing model should serve both functions. That is, it should be able to estimate aggregate staffing demand, provide predictions regarding the consequences of alternative levels of staffing, and help guide the allocation of resources across functions, regions, and offices. (National Research Council, 2006:32, 39, 40)

Consider the Essential Features of the Situation to Be Modeled

“[T]he more faithfully a model captures the essential features of its real-world counterpart, the better able it may be to fulfill its intended function” (National Research Council, 2006:30). A powerful example used elsewhere in government, by the U.S. Air Force, is the model employed to understand, predict, budget, and allocate aircraft maintenance manpower. Mathematical representations of aircraft maintenance processes, repair networks, and component failure rates allow analysts to study enormous amounts of data and apply that data to predict with reasonable accuracy, for example, the number of Air Force maintenance personnel required to service and support a wing of fighter aircraft in order to provide a specified desired sortie generation rate. This predictive power is bounded, of course, by access to the necessary data and by changes within the system that require updating within the model. Models typically represent the relationship of variables through algorithms and equations that may vary in complexity and formalization.¹⁶ Estimating the ATSS staffing needs will require computational power and methods that can capture the complexity of the job demands and generate system-wide estimates.

Consider Whether the Model Should Be Descriptive or Predictive

The 2006 report on Aviation Safety Inspector staffing contains a good explanation of whether a model should be descriptive or predictive. That account is quoted here with additional wording for clarification and application to ATSS staffing:

[M]odels are generally characterized as either descriptive or predictive. Descriptive models typically document the structure and processes of a system, but they do not add a computational component to enable predictions about system behavior as a function of system design. An information flow diagram for a business process is an example of a simple descriptive model. [It shows] the steps, decisions, and outputs of a process, but alone does not offer insight in terms of the capacity or throughput of the system. Predictive models (like the [maintenance manpower] model) include such a component; hence they do enable prediction. In this project we have focused on predictive models because our charge is to articulate methods for determining the appropriate numbers and types of [Airway Transportation Systems Specialists] as a function of the factors that drive the demand for their services. Unless a staffing model can predict with some level of precision how well the [maintenance and service providing] system will perform given the need structure, it would be impossible to estimate appropriate staffing levels [objectively]. (National Research Council, 2006:30)

Consider Whether the Model Should Contain Stochastic or Deterministic Elements

Models can also be stochastic or deterministic.¹⁷ The 2006 report includes an in-depth treatment of this subject.

Stochastic models, a prominent form of which is the Monte Carlo simulation model, attempt to take into account the unpredictable elements of system behavior, whereas deterministic ones do not. For example, [the repair frequencies and time required for each repair of various national airspace subsystems] cannot be predicted with 100 percent accuracy even under optimal circumstances because of unknown factors. (National Research Council, 2006:30-31)

Almost every system has some elements of uncertainty in it, so the question is not whether variability exists but rather how important it is to the system behavior that the model is designed to predict. If ignoring the variable nature of the system is likely to lead to inaccurate predictions or, equally important, a failure to recognize potential staffing risks, then stochastic modeling techniques should be [considered]. However, if the variability is not likely to [significantly] affect model predictions, or the variability is small and unimportant, a deterministic model—one that [minimizes treatment of] the stochastic properties of the system—should suffice. (National Research Council, 2006:31)

These unpredictable elements include an array of factors such as the reliability of individual components within each device, employee illness, weather, corrosion, and ease of access affecting the time required to complete tasks. Equipment failures are inherently stochastic. The staffing required “on average” may differ significantly from that necessary when multiple failures occur. A stochastic model provides insights on the risks associated with low probability events that may have significant consequences and potentially high costs.

A deterministic approach to modeling will typically produce the same results for FTEs needed when the same coefficients or work counts are used, because probability or variation is either not addressed or removed intentionally through the use of averaged process times. A deterministic approach may involve many input variables, but in simple terms can be represented as an equation where the FTE requirement y is a function of multiple inputs x₁, x₂, …, x_n. A potential risk of deterministic models is that they predict staffing demand based upon mean values (i.e., average conditions), rather than recognizing that staffing demand per shift may be significantly greater (or less) than average values due to stochastic factors such as multiple failures.

__________________

¹⁷A deterministic model is a function of multiple inputs such as the type of equipment and the nature of the task to be performed. Law and Kelton (2000) provides fundamentals on how to go about addressing many stochastic situations and incorporating them into practical model designs.

The committee that wrote the 2006 staffing report noted that a deterministic model can provide sufficient predictive power to yield “fairly straightforward answers to a number of key staffing questions.” While the complexity and cost of a stochastic model can be high, it is important to incorporate some stochastic elements that more accurately reflect reality. “It is important to recognize that both the stochastic model and the deterministic model can produce useful expected values for an outcome” (National Research Council, 2006:31, footnote). Thus, to address the immediate concerns of ATSS staffing, the creation of an initial series of deterministic algorithms should be followed by development of robust simulations and queuing models to fully assess stochastic elements of the ATSS job.

Stochastic models provide a better notion of the potential staffing risk associated with unusual events, such as multiple failures. In the ATSS staffing situation, stochastic modeling may incorporate, for example, the probability that the appropriate number of ATSS specialists will be available to meet the demands of required maintenance, because it will take into account queuing issues (e.g., surges and unscheduled multiple maintenance demands) and the stochastic nature of factors driving the need for services or repairs.¹⁸ Hecht and Handal previously developed and demonstrated a prototype model they called SMART 4, which explored relationships between maintenance staffing, mean time between outages, mean time to restore, facilities, and other key ratios (Hecht and Handal, 2001; Hecht et al., 1998, 2000). Hecht and Handal’s pioneering work may be a good starting point for bridging from standard deterministic models (such as ratio unit time equations per equipment or service and evolving support) and stand-alone powerful modeling and simulation tools for NAS resource allocation analysis. They also noted that these powerful simulation tools have been possible to create for some time, but their practical application has been bounded by the cost and complexity of implementation (Hecht et al., 2000). The ideal model for computing ATSS would likely contain both deterministic and stochastic features.

Consider the Critical Data Required for Analysis or That Drive Workforce Demand

The 2006 report on aviation safety inspector staffing explained the data required for analysis or for driving workforce demand; that explanation is reproduced here with added wording where applicable:

[T]he distinction between the underlying predictive model and the data needed to make predictions using the model is critical. A model is created on the basis of the inherent properties of the system that drive its behavior. In the case of the aviation safety inspection system [maintaining the integrity of the NAS systems and services,] this includes factors that drive demand for [ATSS] resources and how these [ATSS] resources are deployed in response to that demand. However, even if these relationships are understood and well represented in a quantitative model, the model is worthless without the data that enable meaningful and realistic predictions. (National Research Council, 2006:32)

Modeling the relationship between staffing levels and the number and type of equipment maintenance needs in the NAS by region requires collecting data by region. Without a reliable baseline count and ongoing accounting or data systems that contain this information, there is no practical way to create such a model. Because modeling and data collection are interdependent, the cost of developing and sustaining data collection systems to feed into modeling must be considered. This is not to say that only “easy to collect” data should be used in the model. Key data that have not been readily available in the past should not be ignored, and methods to gather these data in a straightforward, economical manner should be developed. Too often, analysts have created an interesting but impractical manpower determination tool because of difficulty in routinely gathering the data required for the model’s input

__________________

¹⁸For a primer on stochastic modeling and simulation concepts relevant to the ATSS maintenance environment, see Beichelt and Tittman (2012:Section III).

variables. As noted above, a critical step in the model development process is determining the source of input data and verifying its reliability.

Consider Task Duration and Data Validity Issues¹⁹

The utility of a model depends on the data used to populate the model as well as the structure of the model itself. Validity is defined as the extent to which a measure represents the construct being measured. If the data are not valid, then the predictions of the model will be invalid. For example, a valid measure of the time that Task X will take for completion, will be one that is accurate. If task duration estimates are based on workers who have not fully learned the task or conditions that do not represent the normal environment in which the task is performed, then the time estimates will not be valid measures of the time for completion under usual conditions. However, such measures may be valid measures of the time to complete a task under those unusual conditions.

Another confounding factor in making accurate estimates in task time completion is the bias in human judgments. Unless time estimations are made by a third party, the judgments may be consciously or unconsciously skewed. For example, a tedious task may be perceived by a subject matter expert to have taken longer than it actually did. Alternatively, a subject matter expert who likes to take his/her time in completing certain types of tasks may indicate a longer time than necessary to ensure the work can be completed at the preferred pace.

The validity of a measure is limited by its reliability, that is, the extent to which a measure is free from error and repeated measurements of the same entity (e.g., the same task, event, or object) yield similar values. If the reliability is low, then so is the validity. If the duration of task i is highly variable, a measure of duration based on the mean will have a large standard deviation. While the measure may be reliable, it may not be an accurate estimate of task duration in all situations. The actual time for task i could be almost constant, but the measurement process (such as relying on subject matter experts’ estimates) may be error-prone and thus unreliable. Measurement reliability can be improved by using better methods of data collection; the inherent variability of the task itself is best dealt with by using stochastic models rather than deterministic models.

The other issue in collecting and using duration data is that of granularity. For example, the duration of an off-site maintenance task will be composed of at least two components: travel time and core task completion time. These two components will be affected by different variables, that is, travel distance, terrain, weather, and traffic for travel time; task complexity and ATSS personnel competence level for core task time. If the two subtasks are separated, simpler models can be built for each. Times for completion of the component tasks can be added to obtain the total time for the off-site task. This task decomposition can be taken to lower and lower levels, eventually ending at the element level of predetermined motion-time systems. Synthetic time estimates for new tasks can be computed by aggregating the known times for each element in the task.

There are three means of collecting data on task duration, each with different levels of reliability, validity, usefulness, and simplicity of use.

1. Subject Matter Expert Estimates. Asking supervisors or incumbents to estimate durations of tasks that they have performed in the past is simple and low-cost but generally quite unreliable. Human memory and judgment suffer from well-known biases (Kahneman et al., 1982) that potentially affect

__________________

¹⁹For more information, see Kahneman et al. (1982) and Bisantz and Drury (2004).

both reliability and validity of such estimates. Emerging technologies such as mobile devices and other electronic tracking devices may provide ways of obtaining more accurate time estimates in the future.

2. Historical Data. There are existing data bases such as the Remote Monitoring and Logging System and the Labor Distribution Reporting system that capture actual times for task completion contemporaneously and appear to have some relevance for estimating task durations. The data in these data bases were collected for different purposes (Bisantz and Drury, 2004), and so applying them to duration estimation for staffing models may not be reliable. Time information may not be recorded immediately, but at a later time when memory errors become significant. Typically such data bases only record total task duration rather than more meaningful and useful components as noted above. To improve the confidence in historical data or in subject matter experts’ judgments, their accuracy may be verified by performing checks such as stopwatch measurement of task duration on a sample of the data.

3. Direct Time Study. A third approach is to observe directly the performance of the task and its components and record their durations. Time studies are an expensive proposition if a large amount of data is required, but the results generally have high reliability and validity. Best practices for conducting such time studies cover factors such as how many tasks to time, what degree of component decomposition to use, and how to select tasks and operators for observation and timing. If tasks change appreciably from time to time and situation to situation, then overall task durations will also change, requiring additional data collection and associated expense. If task components are measured and only a few components of the task change, costs can be reduced by only remeasuring that portion of the task that has changed.

Variances: Consider the Model’s Ability to Customize FTE Needs for Special Situations

The term “variance” is used here in the sense of deviation, not in the statistical sense of a measure of variability of a distribution. Variances adjust core requirements either by increasing (positive variance) or decreasing (negative variance) earned FTE increments at specified location(s) or situations due to unique mission or environmental differences. Many models are often created to predict the average FTE requirement per various levels of workload demanded. Such models thus seek to capture the manpower required to do work that is common to all applicable locations. However, in actual work settings, certain processes or work activities are frequently not performed at all locations. Moreover, operational conditions at a location are not always the “average” conditions depicted within the model. If relevant and significant, these conditions can be addressed through calculation of positive or negative variances for the given location and situation.

For instance, a mission variance can add or subtract hours for location-specific required work that is not addressed in the work site description (a positive variance) or work identified in the work site descriptions but not performed (a negative variance). These differences in work should be documented, evaluated, and approved at appropriate level(s) as a matter of policy, not just through a local team leader’s preference.

An environmental variance adjusts hours for required tasks that are addressed in a work site description but are affected by environmental differences among locations (i.e., mountainous territory or snow). The need for environmental variances in NAS maintenance may be based on challenges related to snow removal, need for de-icing of equipment, effects of a marine/salty environment on required corrosion control activities, effects of geographical separation on travel times, presence of remote versus on-site equipment or service monitoring capability, etc. For example, Technical Operations locations that maintain “inside the fence” systems may earn a normalized FTE requirement for the equipment maintenance

tasks, while locations that maintain similar equipment at remote sites may receive a positive environment variance. In such circumstances, a “calculator” may be built into the model to compute the variance for travel time based upon number of times the team must go to the remote site.

It is usually not cost-effective to pursue and document variances that drive small adjustments. When building staffing models for small organizations, a half-FTE change may be worth documenting; however, for large organizations, the study team may establish a larger FTE value threshold for including variances in the model.

Consider Various Components of Total Staff Time

Ideally, any advanced modeling effort needs to address various components of total staff time, including direct productive time, indirect productive time, nonallowed time, nonavailable time, and on-call time and other work situations (U.S. Air Force, 1995a:53-58).

Productive time is time workers spend doing work that is essential to achieve their mission. There are two categories of productive work activities: direct work and indirect work. Direct work activities are required by guidance, technical orders, or directives; are essential to and directly support the work site’s mission; and can be identified with a particular service or end product. Direct work activities are considered productive work that must be accomplished as part of the organization’s primary mission. The time required for direct work tasks should be documented by task, whereas indirect work may be quantified either through similar means or by applying a previously computed and agreed upon indirect allowance factor to the total direct hours.²⁰

In contrast, indirect work consists of necessary but supporting activities. Indirect work is performed in support of the function, does not add value to a particular end product, and may not be readily identifiable with a specific output or service. Common examples of indirect work include participating in human resource activities, giving management direction, preparing reports for higher level review, attending meetings, and housekeeping activities. If indirect work is not measured independently, then the analysts need to create an allowance factor and apply that factor as part of the model algorithm by crediting the measured hours of direct work with a percentage increase based on the indirect allowance factor.

The design of measures of critical data such as time to complete tasks and the careful collection of that must neither omit measurement of some kinds of data nor double count them. In particular, work dimensions such as travel, training, and supervisory tasks need to be carefully and consistently accounted for to ensure inclusion without double or triple counting. For example, if the enterprise-wide mean travel time associated with preventive maintenance of a specific type of navigational aid were credited to the task “maintains navigational aid type X” and included in the total average direct process time, then travel for such activities should not be re-counted as a separate indirect activity. As discussed above, substantial location-specific deviations from the mean travel time could potentially be computed as a positive or negative environmental variance. Similarly, the model should not credit travel at each subcomponent of the task “maintains navigational aid type X” because the travel occurred only one time. Nor should travel be counted for all tasks performed when a worker made only one trip but performed preventive maintenance checks on all of the equipment at the remote location.

__________________

²⁰U.S. Air Force requirements determination analysts sometimes choose not to perform indirect task measurement for every function, as the Air Force has created separate standard indirect allowance task descriptions and factors (U.S. Air Force, 2011).

Nonavailable time is time that is directed and approved by management. During this time, the worker is not available to perform direct or indirect productive tasks. Examples of nonavailable time include approved leave, medical appointments, additional, directed duties such as serving as a voting liaison or office security officer, and time spent in approved education and training. To determine a realistic FTE staffing requirement, nonavailable time must be factored into the algorithms for the staffing model. When accounting for nonmandatory leaves such as annual leave or medical leave in a staffing model, the actual documented use of leave and not the maximum days allowed should be used; otherwise, the model results may be artificially inflated.

Allowances for days off, holidays, leave, and medical absences need to be decided on and approved (U.S. Air Force, 1995a). The original FAA modeling efforts dating from the 1980s and 1990s took many of these into account, and Order 1380.40C documents the treatment of the different dimensions (FAA, 1992). Categories of allowable time, such as Labor Distribution Reporting (006), Watch Schedules (031), Working Hours (032-036), Holidays (038), Annual Leave (040), Sick Leave (041), Family and Medical Leave (043), Leave for Special Circumstances (044-045), and other such factors are specifically defined in the current labor agreement between the Professional Aviation Safety Specialists (PASS) and the FAA, signed on December 16, 2012 (Professional Aviation Safety Specialists, 2012). Other considerations such as allowed time for meals, breaks, and rest should be considered and carefully defined. It should be noted that some of the variances in allowances will be stochastic.

Personal, fatigue, and delay (PFD) factors may be incorporated into a staffing model to account for necessary breaks, trips to the restroom, or other realistic and valid activities.²¹ PFD factors may be considered in individual task measurement times or accommodated elsewhere.

On-call Time and Other Work Situations. On-call time is a period of time when an off-duty worker is available for work at a specified off-duty location and can be reached by telephone or other means. When authorized work is required and cannot be deferred to the next shift or work day, a work measurement effort should credit the work site with productive time expended and the travel time needed to get to the job site and return to the off-duty location. Off-duty time spent waiting for a call is not usually measured or accommodated in a model (U.S. Air Force, 1995a). However, because the FAA agreement with PASS regarding ATSS personnel contains a provision for Compensated Telephone Availability, such time should be considered in the model design process to optimize shift schedule requirements for meeting peak outages or demands (Professional Aviation Safety Specialists, 2012).

Often, the term “standby time” is used to explain the time a worker is awaiting work. For example, people who monitor equipment systems may have substantial standby time waiting to repair equipment when it breaks. A study team can measure and include standby time in a model when such duty is required and no other productive work (direct or indirect) can be accomplished. Workers who are loaned to perform another function’s tasks should generally not be accounted for or included in a model for their own function.

__________________

²¹For more information, see Lehto and Landry (2013).

Consider Incorporating Shift Profile Analysis into the Model Development Approach and Model Application Tools

Maintaining equipment and providing services for a robust and safe NAS can require ATSS workers to be at work, available, or on call for long periods of a day. Coverage of the NAS components 24 hours a day, 7 days a week is not uncommon in many locations. Normal work hours in a traditional week schedule are 8.5 hours per day, including a meal break, for 5 days each week, with exceptions for alternate work schedules such as 10 hours per day for 4 days a week, or for overtime and Compensated Telephone Availability. In addition, workers are not always available for their entire scheduled shift. Activities such as annual leave, sick leave, military leave, training, and travel to remote sites all result in compensated workers who are not available to perform the tasks assigned to them during their shift.

A careful analysis of the shift scheduling and the work performed during shift can help optimize the total system workforce requirements for an organization. For example, the Air Force uses shift profile analysis to determine the amount of time its maintenance workers are available for direct work.

Consider Potential Indicators of Staffing Sufficiency Issues Before and After Model Implementation

Seasoned work analysts often check three indicators of staffing sufficiency: (1) use of overtime, (2) level of work site backlog, and (3) use of shortcuts to accomplish work. Examination of data in each of these areas can help to reveal the extent of understaffing or overstaffing.

Use of Overtime. Overtime can be useful for addressing greater needs for personnel due to peak work demands or temporary loss of personnel. However, excess use of overtime (or of borrowed workers from other organizations and contract workers) may indicate that the threshold staffing levels or shift schedule designs are not optimal within a work site. Steady use of large amounts of overtime can signal shortfalls in available “regular shift” FTEs, either from inaccurate staff targets, poor allocation, or inadequate fill action. In addition, high overtime utilization may indicate less than ideal management practices, which may lead to fatigue or vigilance problems not conducive to high levels of job performance nor ultimately to effective maintenance of the safety of the NAS.²²

Work Backlogs. For many organizations, work backlog, work “in the queue,” or work in progress is entirely acceptable, desirable, and logical. For the ATSS enterprise, it appears logical that periodic maintenance inspections and preventive maintenance activities would be pending for some period of time, but not necessarily postponed for an extended time. Presumably, the longer the period of time these activities are deferred beyond some limit, the higher the inherent risk to the NAS. Some work associated with almost any activity can be deferred, but work deferred indefinitely can be a sign of insufficient resources to meet the work demands. An important question for modelers to explore is whether or not the backlogs are reasonable and growing or decreasing over time, and why (Ouvreloeil, 2001).

Backlogs of work are not always attributable to staffing deficits. Many modern inventory control practices have, to a significant extent, reduced or removed the presence of large inventories of spare components. Some maintenance delays may be attributed to the unavailability of parts, not the unavailability of skilled ATSS workers to perform the work. The Air Force, for example, tracks metrics associated with the nonavailability of parts in order to analyze and improve logistics for aircraft maintenance processes (U.S. Air Force, 2009).

__________________

²²For more information on overtime considerations, see Capshaw (2011).

Use of Shortcuts. The FAA operates within a formal Safety Management System that has available an array of tools and resources to prevent, identify, and analyze risk, including risk management through maintenance policy and procedure changes (FAA, 2007a). Nevertheless, on-scene work measurement or expert workshop review of tasks and times for many other maintenance-related fields have shown that a work site may occasionally employ shortcuts or deviations from standard practices that are not permitted or described in technical guidance in order to provide a service or maintain the required systems. Employing these shortcuts could be a valid innovation if properly studied and approved, but may unfortunately create a hidden level of risk within the system and long-term negative effects overall. In the modeling arena, if the time required for procedures officially documented in the Technical Operations guidance is accurately measured, but in practice the work site operates differently from the guidance for whatever reasons, the time estimates are likely to be inaccurate. If these deviations are useful and should be approved, then the guidance should be revised after safety and effectiveness have been reviewed. The underlying rationale for the use of shortcuts may, however, relate to perceptions of time pressure, poor training, or a lack of updated guidance. Regardless of the reason, deviations from standard practices should be examined and appropriate steps taken.

Quality Factors

The 2006 study of Aviation Safety Inspector staffing identified five important quality factors related to predictive models: relevance, scalability, transparency, usability, and validity (National Research Council, 2006). Box 3-1 defines these five factors, which were used by the current committee as criteria when reviewing the FAA’s current and planned modeling efforts for ATSS personnel.

The 2006 report emphasized the importance of validity in assessing the value of a model:

Validity is the final and, in many respects, the most critical feature. The extent to which the predictions of the model correspond to the actual, real-world outcomes constitutes its validity. Indeed, the most powerful means of evaluating a model’s worth—the ultimate proof of the pudding—is the direct comparison of predicted with observed outcome (criterion) measures when such measures are obtainable. It is often the case that the ultimate criterion (i.e., [NAS] safety) is not directly measurable in any practical sense, so the model’s predictive validity must be estimated against surrogate criterion measures. (National Research Council, 2006:33)

The prediction of outcomes associated with alternative levels of staffing is a necessary condition for the model to be testable and its validity assessed. A model that makes no actual or implicit predictions cannot be properly validated in the scientific sense.

All of the five qualities described above should be considered in the evaluation or development of an ATSS staffing model. They apply equally to models that the FAA has used or is using as well as to any future modeling effort it may undertake. (National Research Council, 2006:34)

SUMMARY AND CRITERIA FOR ASSESSING MODELING

A starting point for developing or assessing a staffing model is to review the nature of the work, the environment under which the work is conducted, and the manner in which the work is accomplished. The committee’s assessment of the work and the environment suggested that a viable staffing model for ATSS should

•  capture the full extent of the NAS, the geographic diversity across facilities, and the staffing implications of travel time for equipment in remote areas;

•  clearly distinguish and quantify domain disciplines required in achieving workload demand; and

•  include performance measures or outcomes, both final (ultimate) and intermediate, in order to provide predictions regarding the outcomes of a given staffing plan.

A staffing model for ATSS should be “sufficient” in that it estimates staffing necessary to accomplish a given workload and is not simply a model that allocates a predetermined level of staffing across work sites. Moreover, because the nature of the workload itself—repairing equipment that fails—has an inherent stochastic component, serious consideration should be given to developing a stochastic staffing model. Furthermore, the model should be able to predict consequences associated with staffing at various levels, and these predicted outcomes should closely relate to what is actually observed at actual staffing levels. Chapter 3 has discussed the potential criteria for model evaluation. Chapter 4 will review the past and present FAA models used for the ATSS workers and examine how these models compare to some of the modeling philosophies that this chapter has explained. It also provides further recommendations that incorporate considerations introduced in this chapter.

Drawing on the previous discussion of the ingredients for successful modeling (i.e., a logical design process, key model considerations, and vigilance in seeking to attain the five quality factors), the committee created a checklist or set of criteria, shown in Box 3-2, with which to evaluate past models, models in progress, and other potential modeling approaches for the ATSS workforce. This checklist is intended as an aid to a modeler considering the criteria to use when designing a model for ATSS employees of the FAA.