Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 28
Staffing Standards for Aviation Safety Inspectors 2 Modeling as Applied to Staffing What Is a Staffing Model? The term “staffing” is often used to refer to a range of processes, such as recruitment, selection, placement, and training, through which an organization applies human resources to the work needed for it to achieve its goals (Schneider, 1976). At the Federal Aviation Administration (FAA), staffing models and staffing standards are terms used to denote tools for the management of manpower only. As explained in Chapter 1, these are tools that enable the organization to determine the right number of people with the right skill sets in the right positions to accomplish the responsibilities of the job in a satisfactory manner. Staffing needs are determined mainly by how the organization has defined its goals and designed the jobs that make up its total workload. How effectively an organization is able to meet those needs, once they are defined, depends on both its staffing processes and the characteristics of the available human resource pool. While manpower planning and management, and the models used in these activities, are important parts of the overall staffing picture, many other factors enter into human resource management, as noted earlier. In this chapter, we concentrate on models for manpower management, that is, for deciding how many workers, of what general types, are needed to staff the organization without regard for specific job characteristics, worker qualifications, or performance standards. In Chapter 4 we discuss these and other considerations that are also important for effective deployment of human resources.
OCR for page 29
Staffing Standards for Aviation Safety Inspectors It is important to recognize that determining the appropriate number of aviation safety inspectors (ASIs), system-wide or locally, at any given time is by no means a simple matter, since it is dependent on how work is structured and what defines acceptable individual and system performance, as well as the characteristics of the current and projected workforce. For example, the FAA’s adoption of the system safety philosophy radically changes the nature of many ASI jobs and, as a consequence, the skills required to perform them. The expectation of the system safety philosophy is that high standards for safety in the aviation industry can be maintained with the same or fewer people. However, estimating how many people are needed is difficult at best. All organizations base staffing decisions on a paradigm of the underlying production process, whether they do so explicitly or not and whether it approaches reality or not. This conceptualization is often referred to as a staffing model. The Flight Standards (AFS) and Aircraft Certification (AIR) offices have made a commitment to using staffing models as a tool to develop staffing standards (the FAA’s term for the documents that detail the numbers of staff needed at its facilities). A staffing model is a formal representation of the mechanisms that drive the need for staffing and of the interactions among staffing needs and staffing resources. The operation of a good model should provide useful standards as an output if the proper data are input and the algorithms of the model accurately reflect the mechanisms driving staffing needs. Staffing standards are not identical to authorized or filled positions. They are one source of guidance used by AFS headquarters, regional offices, and facilities in the development of staffing plans and projections and in the authorization of safety inspector and other staff positions. The current Automated Staffing Allocation Model (ASAM) reflects many, but not all, of the factors that drive the need for ASI staff and the decision process must take these other factors into consideration. A staffing model that accounts for more of these factors could perhaps play a more central role in staffing decisions, leaving less of the decision making subjective and thus open to question. However, there will no doubt always be some subjectivity in the setting of staffing levels, some need to consider regional differences and local, short-term, or emergent demand factors that are not practical to include in a staffing model, as well as the changing strategies or priorities of AFS. For these and other reasons, managers will still always be responsible for the final staffing decisions. We were asked to examine and evaluate both the FAA’s model and possible alternative models for ASI staffing. The more faithfully a model represents reality, of course, the more likely it is that staffing processes based on it will satisfy staffing demands. In view of the complex and dynamic nature of ASI staffing demands, as well as the wide variety of
OCR for page 30
Staffing Standards for Aviation Safety Inspectors forms that models and modeling efforts can take, it is necessary at this point to consider some of the more salient aspects of modeling per se. Only then will we be in a position to apply modeling principles to the issues involved in ASI staffing. Distinguishing Features of Models “Model” is a widely used term with many meanings and implications. In this section, we define more specifically how the term is used in the present context and identify some of the issues we have considered as the term applies to ASI staffing decisions. First, a model is simply a representation of some actual process or system, typically created for the purpose of understanding it more completely and predicting the future state of affairs. The purpose of a model is to make inherently complex processes simpler, so that their essential elements can be better understood. A model is an abstraction of reality. However, the more faithfully a model captures the essential features of its real-world counterpart, the better able it is to fulfill its intended function. A familiar example is the modeling used by meteorologists to better understand and predict weather phenomena. Mathematical representations of atmospheric and oceanic processes allow meteorologists to analyze enormous amounts of data very quickly to predict, for example, the probable intensity and path of a hurricane with some accuracy. As the precision of these models has improved, so too has their practical utility. While not always quantitative or highly formalized, the most precise and useful models represent the phenomena of interest through sets of algorithms and equations. As with the meteorological example, the complexity of the ASI staffing need structure requires both formalization and the computational power of the modern computer. Second, models are generally characterized as either descriptive or predictive. Descriptive models typically document the structure and processes of a system, but they do not add a computational component to enable predictions about system behavior as a function of system design. An information flow diagram for a business process is an example of a simple descriptive model. Predictive models (like the hurricane model) include such a component; hence they do enable prediction. In this project we have focused on predictive models because our charge is to articulate methods for determining the appropriate numbers and types of aviation safety inspectors as a function of the factors that drive the demand for their services. Unless a staffing model can predict with some level of precision how well the inspection system will perform given the need structure, it would be impossible to estimate appropriate staffing levels. Third, models can be stochastic or deterministic. Stochastic models, a
OCR for page 31
Staffing Standards for Aviation Safety Inspectors prominent form of which is the Monte Carlo simulation model, attempt to take into account the unpredictable elements of system behavior, whereas deterministic ones do not. For example, the need for aviation safety inspections cannot be predicted with 100 percent accuracy even under optimal circumstances because of unknown factors, such as the increase or decrease in the general aviation population, random or unpredictable factors affecting the time required to complete tasks, and changes in the location of aviation maintenance facilities. Almost every system has some elements of uncertainty in it, so the question is not whether variability exists but rather how important it is to the system behavior that the model is designed to predict. If ignoring the variable nature of the system is likely to lead to inaccurate predictions or, equally important, a failure to recognize potential staffing risks, then stochastic modeling techniques should be used. However, if the variability is not likely to affect model predictions, or the variability is small and unimportant, a deterministic model—one that ignores the stochastic properties of the system—should suffice.1 For example, if one were developing models to estimate the time that airplanes spend waiting for safety inspections, the variability associated with the arrival rate of airplanes into the inspection process would need to be incorporated. However, if the goal instead were to estimate the total number of aviation safety inspection hours required, this variability would be far less important, and it would be sufficient simply to enter average arrival rates. The importance of the distinction between questions requiring stochastic model properties and those for which deterministic properties are sufficient cannot be overemphasized as it applies to the committee’s charge. While we think that a deterministic model can provide enough predictive power to yield fairly straightforward answers to a number of key staffing questions, we can envision issues for which the complexity of a stochastic model would be required. Consider the following contrasting examples. If the FAA needed only an estimate of the total demand for inspectors and their optimal geographic distribution, a deterministic model that simply tracked all of the important demand factors and translated the projected demand into hours of inspector time at locations around the world (considering both performance and cost consequences) would suffice. However, if the question concerns not only how many hours of inspector time were needed, but 1 It is important to recognize that both the stochastic model and the deterministic model can produce accurate expected values for an outcome. The stochastic model’s advantage is that it can also provide an estimate of the variation in realized values around the expected value. This allows one to better assess risk in the staffing decisions that are made.
OCR for page 32
Staffing Standards for Aviation Safety Inspectors also the likelihood that an ASI inspector will be available when an inspection is required, then it is necessary to invoke a stochastic model that will take into account queuing issues and the stochastic nature of factors driving the demand for inspections. Fourth, the distinction between the underlying predictive model and the data needed to make predictions using the model is critical. A model is created on the basis of the inherent properties of the system that drive its behavior. In the case of the aviation safety inspection system, this includes factors that drive demand for ASI resources and how these ASI resources are deployed in response to that demand. However, even if these relationships are understood and well represented in a quantitative model, the model is worthless without the data that enable meaningful and realistic predictions. For example, if a model hypothesized a relationship between the number of aircraft in the U.S. fleet by zip code where they are based and the need for inspectors in various regions, that model would be useful only if we could collect the data on aircraft by zip code. Therefore, as the FAA considers the choice of an underlying model to represent ASI processes and resources, it must also consider the availability and cost of collecting the data needed to implement the chosen model. Fifth, predictive models tend to be either decision support tools, designed to allow the user to explore alternative options for achieving desired results, or summative evaluation tools, which tell the user how well the proposed system is going to achieve the specified goals. While not mutually exclusive alternatives, the distinction represents primary emphases that drive model development. We think the present effort should emphasize the decision support role, since the complexity of the need structure and the difficulty of quantifying ultimate criteria render summative evaluation problematic at this time. Finally, models, such as one for ASI staffing, may be either allocation models or sufficiency models, or both. An allocation model is one aimed at distributing available resources equitably and effectively irrespective of their collective adequacy, whereas a sufficiency model is designed to predict the resources needed to sustain system performance at what is deemed an acceptable level. To date, apart from one aborted effort, the staffing models developed by AFS have been exclusively of the allocation variety—the goal being to achieve the most effective distribution of limited ASI resources across organizational units. A sufficiency model is more difficult to develop. It requires the organization to make decisions about, and set standards for, acceptable performance and to develop performance measures so that outcomes can be evaluated against those standards (i.e., it can be empirically validated). A validated sufficiency model has the advantage that it can be used to justify budget requests and other decisions by generating predictions of organizational performance with
OCR for page 33
Staffing Standards for Aviation Safety Inspectors and without additional resources, or with different distributions of current resources. Desirable Model Characteristics Predictive models developed as decision support tools may be characterized in terms of five important qualities: transparency, scalability, usability, relevance, and validity. Each of these is described briefly below. Transparency is the extent to which the model can be explained and understood by interested individuals other than the model developers— most importantly the users of the model and those affected by decisions based on model implementation. Models that are relatively transparent, those in which the critical relationships among variables can be seen and understood by stakeholders, are inherently more likely to be accepted. Scalability refers to the extent to which a model can be useful at different levels of systems analysis. For example, is it useful for predicting ASI staffing needs for regions as well as the entire nation? Can it provide guidance on staffing at the flight standards district office or other facility level? Usability refers to the ease with which the model can be implemented and enhanced to make the predictions for which it was designed. Does it have an interface that is sufficiently intuitive to enable the model users to enter data efficiently and accurately? Is it appropriate to the skills and preferences of the intended users? Are the results presented in ways that support decision making? Can the model easily be updated to reflect changes in the ASI work requirements and environment or changes in FAA policy? Relevance concerns the extent to which the model addresses the important portions of the issues for which it is designed and, equally important, the extent to which it excludes extraneous or marginally relevant issues or data. Does the model capture all of the important ASI workload drivers? Does it operate at the right level of detail? Validity is the final and, in many respects, the most critical feature. The extent to which the predictions of the model correspond to the actual, real-world outcomes constitutes its validity. Indeed, the most powerful means of evaluating a model’s worth—the ultimate proof of the pudding—is the direct comparison of predicted with observed outcome (criterion) measures when such measures are obtainable. It is often the case that the ultimate criterion (i.e., aviation safety) is not directly measurable in any practical sense, so the model’s predictive validity must be estimated against surrogate criterion measures (e.g., level of risk posed by various inspection scenarios). As becomes clear later in the report, estab-
OCR for page 34
Staffing Standards for Aviation Safety Inspectors lishing meaningful criteria is one of the main challenges facing the developers of any predictive ASI staffing model. All of the five qualities described above should be considered in the evaluation or development of an ASI staffing model. They apply equally to models that the FAA has used or is using as well as to any future modeling effort it may undertake. Alternative Approaches to Model Development In view of the fact that the questions posed to the committee imply models of the sufficiency variety that are capable of making and validating performance-based staffing predictions, and that none of the staffing models developed or in use by the FAA possesses those characteristics, it may well be that an entirely new approach is called for. If that turns out to be the case, there are two major alternatives representing substantially different modeling concepts that should be considered. Either one, if implemented appropriately, could satisfy the staffing model requirements we have set out. The first approach, process modeling, incorporates the key processes that drive the need for staff, while explicitly representing staffing resources and their use in those processes. The second approach, statistical modeling, does not focus on the explicit processes that drive the staffing need. Rather, it assumes that the relationship between future staffing requirements and the factors driving those needs—whatever the underlying processes may be—is relatively stable. If one has historical data on which to build statistical relationships between staffing demand factors and staffing requirements, then these statistical relationships can be projected onto new situations without understanding what accounts for them. Simply put, a statistical model seeks merely to describe a stable empirical relationship; a process model attempts to depict the mechanisms underlying that relationship. While either could serve in the present context, the advantage of a process approach is that it is more easily adapted to change, and change is rather prominent in today’s aviation landscape. The advantage of the statistical model is that it is empirically based, and is likely to be less costly to develop and implement. A detailed description of process and statistical modeling methodology appears in Appendix A. Below are brief descriptions of important distinguishing features of each, as well as of alternative approaches to the estimation of model parameters. Process Modeling Process modeling can be more or less complex and detailed in its representation of the relationships among system factors, but by defini-
OCR for page 35
Staffing Standards for Aviation Safety Inspectors tion it requires decomposition and analysis of work processes. To develop a process model, then, modelers must understand in some depth the operations of the system to be modeled. Usually this is accomplished by involving subject matter experts in the model analysis and design phases. One example of a process modeling method is called task network modeling. A graphic representation of a part of a task network model is shown in Figure 2-1. To build a task network model, the modelers must decompose the work represented in the model, that is, break it down into successively smaller units, usually ending at the level of the task. This process is commonly called a task analysis. Each task is then modeled, along with its relationships to other tasks—hence a task network—and the system representation is synthesized from these components. Key attributes of each task must be specified: how long it takes to perform, who must perform it, the task’s priority, what other tasks must be completed before it can begin, what its outputs are and how they are used, whether decisions are made based on the outputs, etc. The level of detail needed in the model will depend on the complexity of the system modeled and the outputs needed by the users. For the purposes of ASI staffing, a process model would probably not require a very detailed task analysis, although the level of detail required would have to be determined during the model requirements definition process. Outputs of process models can be as varied as the systems they represent. Typical outputs for staffing process models fall into just two general categories: measures of personnel utilization (e.g., how busy each type of ASI will be at each location) and estimates of delays or failures to complete work associated with the unavailability of staff to perform the work. Such outputs would allow users to estimate the effectiveness with which ASI staffing resources are used and their ability to meet work demand. There are many ways to express and quantify such information, and a model can be designed to provide the most appropriate and usable output to serve the needs of its users. Over the past decade, many tools have emerged that have made the job of building and maintaining process models easier and more transparent than older tools. A model designed using one of these tools should be able to: allow data entry in a manner consistent with user terminology and expectations; automatically import data from other digitized sources; and present the simulation results in a usable and understandable form. Some of the modeling tools now on the market include the Extend product line from Imagine That, Inc.; SimScript from CACI Products
OCR for page 36
Staffing Standards for Aviation Safety Inspectors FIGURE 2-1 Sample task network model of a process control operator responding to a warning. SOURCE: Laughery (2005). Reprinted with permission.
OCR for page 37
Staffing Standards for Aviation Safety Inspectors Company; Flexsim from Flexsim Software Products, Inc.; Micro Saint Sharp from Micro Analysis and Design; and Arena from Rockwell Software. These tools provide such useful features as interfaces to common database applications, optimization capabilities, and the ability to run “what-if” simulations. When requirements and a modeling approach have been defined for a specific project, appropriate tools can be selected, purchased or licensed, and employed. Statistical Modeling A statistical model relies on empirically defining mathematical relationships between system inputs and outputs. The mathematical formulas are estimated from the observed historical data on the system, such as records of work accomplished by known staffing resources over a given time under known environmental demands and other relevant conditions. Thus a statistical staffing model is dependent on accurate work recording systems to provide the data it uses to estimate critical relationships. A statistical model does not require a task analysis, but it does require that the modelers identify and represent in the model all factors that substantially affect the relationship between staff resource inputs and work outputs (system performance). Other factors, such as operating policies and procedures, are not modeled explicitly but are implicit in the input-output relationships. For this reason, if there are changes to work policies or procedures or other factors not modeled but material to the way the work is done, the model will no longer generate accurate predictions. The system will have to operate for some time under new conditions to produce the data needed to update the model. Finally, it is important to recognize that the dichotomy drawn here between a process simulation model and a statistical model may be overstated. As we discuss next, a process model may use statistical methods to estimate some, or all, of its parameters, and a statistical model may include relationships between resource inputs and overall or intermediate output at fine levels of detail, similar to the tasks and steps of the process model. Parameter Estimation For any model, values must be obtained for the parameters of the model. The parameters are the values that are used in a model to quantify key relationships among variables. Thus, if one is modeling the time taken by a train to travel between two points, one parameter would be the distance between the points. The time is a function of the distance, the train’s speed (another parameter), and other factors. Similarly, in a staff-
OCR for page 38
Staffing Standards for Aviation Safety Inspectors ing model, an important parameter might be the time it takes an ASI to perform inspection Activity x. There are several ways in which parameters for a model may be estimated, some more suitable than others for particular situations. The most reliable approach, when reliable data are available, involves statistical estimation based on documented relationships among important variables. An example of this would be to estimate the time needed to perform Activity x from work records showing time spent by a large sample of ASIs on many recorded performances of that activity. This empirical method works best for a staffing model when the model designers have access to data from work records that are known to be accurate and representative of the current state of the system to be modeled. An alternative, but generally less desirable, means of estimating parameters is through expert judgment. An example of this would be to gather a group of ASI subject matter experts and ask them to estimate how long it takes to perform Activity x, given their experience with the activity. Of course, expert judgment is subjective, so a model with parameters estimated in this way may not be considered credible unless it is empirically validated by testing predictions generated by the model against observed outcomes. Finally, parameters may be estimated by calibration or fitting. In this method, modelers generate preliminary parameter estimates based on judgment and then fine-tune them by running the model against known outcomes and adjusting the parameters until the model produces acceptably accurate predictions—provides a good fit to the known data. This method is a hybrid of the statistical and the expert judgment methods. It uses judgment first and then an ad hoc statistical and empirical procedure to adjust the parameter estimates, but it lacks the rigor of a formal statistical method. Practical Considerations in Model Development When considering the generic features of predictive models used as decision support tools, it is important to recognize that practical circumstances often dictate the relative weight that should be accorded various facets of the modeling approach. As explained above, models are abstractions of the reality they seek to represent and are created for a number of different purposes. The more clearly and precisely the model’s purpose can be specified, the more readily one can judge the relative importance of the various features, whether evaluating an existing model or developing a new one. Although we devoted considerable attention to understanding the goals of an ASI staffing model, gaining some insight into the aspirations
OCR for page 39
Staffing Standards for Aviation Safety Inspectors involved, the committee is not in a position to articulate for the FAA what its specific priorities should be. Rather, we simply cite the specification of the model’s purpose as a critical first step in any modeling initiative that the FAA may undertake. That is, actual model development should be guided by a set of clear, concrete statements about how the model will support, first, FAA decision making and, ultimately, the FAA mission. Another set of practical issues that should be given serious prior attention are the operational constraints that will be placed on the use of the model once it is developed. Who are the users? What are their expectations of the model? What skills and knowledge do they have? What data can be used to populate the model and how easily can those data be obtained? Will data need to be manually input or can they be captured from existing management information systems? What resources can be made available to implement and maintain the model and its data sources? What is the time frame for model use and, by inference, the amount of time between the assignment of a prediction task and the deadline for an answer to be provided? What level of precision is required of the answer? A successful ASI staffing model will be possible only if the team that is tasked with building it confronts these issues at the outset, head-on. Another critical consideration in building a staffing model is the need for measures of outcomes—performance measures—that can be applied to ASI system performance. The importance of performance measures to the design and utility of a staffing model cannot be overemphasized. Such measures are required to rigorously specify both the purpose of staffing and the consequences associated with staffing decisions. A discussion of some issues related to the development of performance measures for the ASI situation is found in Chapter 4. Value of a Model for ASI Staffing With the foregoing discussion of modeling characteristics, concepts, and principles as context, we return now to the current application—the use of models in ASI staffing. Before proceeding further, however, one fundamental question must be addressed: Is modeling a potentially useful approach for aviation safety inspector staffing? Although once again we must note that neither modeling per se nor any approach relying exclusively on manpower management tools can ensure optimal staffing, we think modeling does have potential in two areas that correspond, respectively, to the distinction between sufficiency and allocation models. First, by providing an estimate of the resources necessary to meet policy and safety goals, a sufficiency model can be the most rigorous way to determine staffing needs and to support budget requests for ASI posi-
OCR for page 40
Staffing Standards for Aviation Safety Inspectors tions. To do this, however, the model must be able to estimate aggregate staffing requirements, to justify the appropriateness of that estimate, and most importantly, to predict consequences of staffing below the prescribed level. It should be noted that none of these estimates is possible without a credible means of documenting performance; hence performance measurement is essential for any staffing model to realize its full potential. In the present case, performance measurement poses a number of daunting challenges. The second area in which modeling could prove useful is the distribution of ASI resources—an allocation model could help guide these decisions. Available ASI resources should be allocated to regions and offices in which they are needed the most, so that they provide the greatest possible benefit to flight safety. To do this, a model must reflect all elements of the ASI need structure, including the external drivers as well as internal policies, processes, and practices as applied to ASI functions and across regions and offices. Finally, despite the distinction between sufficiency and allocation roles, we think that an ASI staffing model should serve both functions. That is, it should be able to estimate aggregate staffing demand, provide predictions regarding the consequences of alternative levels of staffing, and help guide the allocation of resources across functions, regions, and offices. A single model would help ensure consistency across both aggregate and local staffing decisions. Requirements for a Staffing Model Having summarized salient model characteristics and the potential of modeling for ASI staffing applications, we conclude with a review of the model features that we think should characterize an ASI staffing model. First and foremost, as noted above, we think that a single model could and should serve both the sufficiency and allocation functions. To do this, a staffing model should have the structure depicted in Figure 2-2. In the following sections, we consider first the demand side of the model and the diagram, then the supply side. Finally, we discuss how the two components come together and the importance of this for model relevance and validation. The Model Should Be Driven by Demand for Work. The model must be demand driven. That is, it must represent the full array of factors that in combination determine the total amount of work required of the workforce. These demand factors, which in the FAA case derive primarily from two sources (the environment and FAA policy, procedures, and guidance), must be captured adequately in the model
OCR for page 41
Staffing Standards for Aviation Safety Inspectors FIGURE 2-2 Generic staffing model: Essential elements. (i.e., must “drive” it) in order for its output to yield a useful estimate of total workload. Actual staffing demand will thus vary over time with changes in the distribution and mix of workload drivers (e.g., for the FAA, numbers of certificates, public airports, and pilots requiring oversight; job specifications; technological innovations; regulations and other policy mandates), and the model must reflect this variation. In addition to workload driven by predictable factors, there will also be demand for services on an unpredictable as-needed basis, and that too must be accommodated in any attempt to estimate overall staffing requirements. Whatever their source, then, these demand indices should be translated into workload estimates represented in terms of staff person-hours or full-time equivalents (FTEs) needed to accomplish the work.2 This translation from drivers to required hours or FTEs should be empirically based, if possible. That is, it should be based either on a statistical relationship between person-hours or FTEs and demand factors, using recent data, or on direct observations of time required to perform tasks (job and task analysis). While expert judgment may substitute for the empirical estimates in the short run, a plan should be in place for empirical validation. 2 Intermediate steps might include estimating the numbers of specified activities required, given the factors driving demand, and the time required to complete each activity.
OCR for page 42
Staffing Standards for Aviation Safety Inspectors The Staffing Model Should Provide Staffing Demand Estimates That Are Based on Some Measures of Performance. The user must be able to say “We need enough employees with the right knowledge, skills, and abilities, in the right places, to perform x,” where x is a measure of work output, quality, and/or outcomes for each organizational entity modeled. One must be able to measure work, the amount of work accomplished, and the amount left undone. Preferably, the measure should incorporate a quality dimension. For the purposes of a staffing model, something as simple as a minimum time to complete a task may be appropriate. For example, if experience has shown that four hours are required to do a thorough job of Task x, then a record of having performed Task x in one hour would not be accepted as documentation of satisfactory performance, and the model should incorporate in its demand functions that Task x requires four staff hours each time it is performed. The Model Should Represent the Supply of Staff to Perform Work. The supply of staff FTEs must be translated into (or functionally related to) the capacity for carrying out the required work, using the same metrics (e.g., person-hours or FTEs) in which the demand-side workload is expressed. That is, the productive capability of the organization’s staff must be incorporated into the estimate of available capacity or, viewed a bit differently, the model should provide an estimate of the workload that can be accomplished with any given level of staffing (capacity). This involves calculating the policy-driven and practical limitations on the use of staff for the work to be modeled. In most staffing systems, these result in adjustments to full-time hours for training, leave, travel time, administrative tasks, and other nonmodeled activities. The Model Should Make a Performance-Based Supply-Demand Comparison. The demand side of the model should thus produce an estimate of the staffing necessary to satisfy workload demand at a satisfactory level of performance. The model then should be able to compare staffing necessary to meet demand with staffing available. Based on this comparison, the model, in its resource allocation role, should project the distribution of human resources that will best allow demand to be fulfilled, given organizational priorities and practical constraints. In addition, a sufficiency model should predict the workload that can be accomplished acceptably with available staffing. It should also provide
OCR for page 43
Staffing Standards for Aviation Safety Inspectors an estimate of the work requirements (if any) that will have to either remain undone or be performed with less than the required quality and thoroughness. The actual performance will depend on management decisions and staff follow-through, but at least there will be information available to guide those decisions. Some estimate of the consequences of these shortfalls in accomplishing the work should also be generated. Consequences or outcomes resulting from actual staffing compared to staffing necessary to meet demand can be described and measured on many levels; we consider three. First, the workload that cannot be accomplished because of inadequate staffing is quantified simply as backlog. This can be translated into a shortage of person-hours or FTEs. These are the additional hours or FTEs that would be necessary to meet workload demands consistent with policy. Second, the first level performance implications should be estimated. Examples include reduced frequencies of required activities or increased customer waiting time for processes to be completed. These immediate and measurable consequences commonly serve as performance measures for a system. A third, less immediate, level of outcome or performance is the effect that the deterioration of performance, as measured in the second level of description, will have on the final output of the system—in the FAA case, the timely provision of services that ensure aviation safety—and on safety itself. While this relationship is extremely important, and the raison d’etre for aviation inspection in the first place, such a relationship is extremely difficult to measure empirically. The first reason is that safety is generally very good, so that adverse events are rare. To establish such a relationship empirically requires natural variation in the data and outcomes that is unlikely to be present. Second, many factors not under the control of AFS may affect safety outcomes in U.S. aviation. Third, there has been little agreement until recently on a suitable measure of aviation safety.3 As a practical matter, therefore, it will be necessary to rely on outcomes more immediate than system safety per se in the development and validation of a staffing model. 3 The FAA has this year proposed a Composite Safety Indicator to quantify overall airline safety, for purposes of evaluating the effectiveness of FAA programs (Flight Plan 2006-2010, Federal Aviation Administration, 2006). This could be of some use in an effort to examine outcomes of ASI staffing, although it reflects the combined effects of many factors beyond the performance of ASIs. Also, because the Composite Safety Indicator is a rolling three-year average, it will mask short-term variations.
OCR for page 44
Staffing Standards for Aviation Safety Inspectors The Model Outputs Should Be Usable Both for Staffing and for Validation. For a model to meet commonly accepted scientific norms, the predictions it generates must be capable of empirical test. Elementary though it may seem, this requirement is often ignored in many so-called models through which appropriate staffing levels are estimated. Thus they are unable to predict the consequences of over- or understaffing through which their validity might be tested. Such models should be validated by generating predictions of how much of a specified workload various levels of staffing can accomplish, controlling for overtime and other factors used to stretch staffing in the short run, and comparing those predicted accomplishment levels with levels actually observed. Such validation might be possible using historical data during formative evaluation of the model, and certainly it should be pursued for continuing validation once the model is implemented. In addition to validation of the model’s predictions, standard errors of key model parameter estimates should be computed and reported, along with other measures of the goodness of fit between the model and the data. The Parameter Estimation Methods Should Be Appropriate to the System Modeled. Earlier in this chapter we describe methods for estimating the parameters of models in general. The parameters of a staffing model should be estimated using statistically sound, efficient estimation methods. The primary parameters to be estimated in a staffing model are the relationships among demand factors, workload, staffing inputs, and staffing productivity. Any of the estimation methods described earlier may be used, but for any given staffing model the decisions about parameter estimation should be made by expert modelers with input from people who understand the practical realities of the system to be modeled. It is especially important that the modelers understand the limitations of the data available for generating estimates, as well as the expected stability or change in operations and business practices between the period providing historical data and the period in which the model will be used. In principle, modelers should use the most rigorous, most empirical method that is practical. Any nonempirically based estimates should be empirically validated to the extent possible during pilot implementation of the model.
OCR for page 45
Staffing Standards for Aviation Safety Inspectors The Model Should Represent All Important System Dimensions at Appropriate Levels of Detail. The selection of factors to be included in any staffing model will be critical to the model’s success. Recall that the purpose of the model is to abstract from reality the essential elements, not to duplicate that reality’s complexity. If too much detailed data input is required to support the model, it is likely to be left on the shelf and not used, because implementation will be expensive, difficult, or impossible. If too little detail is included or if important drivers or moderating factors are omitted from the model, it will not provide valid predictions and will not be worth using. Again, the model design must be performed by experts with knowledge both of modeling techniques and of the system to be modeled. Summary The overarching goal for this chapter was to examine generic features, characteristics, and requirements of the two principal approaches to modeling (process and statistical approaches) in the context of the ASI staffing situation. Explicit consideration of these fundamentals is necessary for evaluating current ASI staffing models and any potential alternatives, for judging the merit and difficulty of making substantial improvements in those models, and for developing an entirely new approach, should that prove advisable. It is, in short, essential preparation for both the chapters to follow in which the specific components of our charge are addressed. In the committee’s view, the ASI situation does call for a formal modeling approach, one that supports staffing decisions in both the allocation and sufficiency functions through valid predictions of system-performance consequences. We wish to emphasize again that a staffing model is not, and should not be, the only tool used in the development of manpower or staffing plans (called staffing standards at the FAA). Other factors rightfully enter into the manpower planning process. But if a model is to be used, it should be the best model that the FAA can feasibly develop and implement. The question of how best to proceed toward achieving this goal can be answered only through the application of the principles presented in this chapter to current and previous ASI staffing models, alternatives derived from other large organizations, and consideration of an entirely new model. The following chapters address in depth these three applications together with characteristics of the aviation environment to which any ASI staffing model must be sensitive.
Representative terms from entire chapter: