5
Conclusions and Recommendations

In this report we have examined model features of most relevance to the staffing situation of aviation safety inspectors (ASIs), critically reviewed the modeling approaches that the Federal Aviation Administration (FAA) and other organizations have developed to guide staffing decisions, and identified unique human resource issues that should be addressed prior to initiating any manpower modeling efforts. We return now to the specific questions posed to us. In the following sections, we summarize our conclusions regarding the adequacy of current ASI staffing models, the potential afforded by models drawn from outside the FAA, and the merits of a completely new approach. Our specific recommendations follow each set of conclusions, and the final section provides an elaboration of our recommended approach.

Before proceeding, it is necessary to address one overarching issue that, while not directly related to our modeling charge, has an important bearing on all facets of the ASI staffing question. As explained earlier, staffing needs are heavily dependent on the capabilities of the individual ASIs, their fit with the prescribed work roles, and a host of organizational factors, such as management practices, culture, and other human resource considerations. In the course of our investigation, it became apparent that little can be gained through improved manpower modeling in the present case unless serious prior attention is accorded to these other critical human resource matters. Unless work is described with reasonable accuracy and the required skill sets are ensured through adequate recruitment, selection, training, and placement of personnel, estimates of the number



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 85
Staffing Standards for Aviation Safety Inspectors 5 Conclusions and Recommendations In this report we have examined model features of most relevance to the staffing situation of aviation safety inspectors (ASIs), critically reviewed the modeling approaches that the Federal Aviation Administration (FAA) and other organizations have developed to guide staffing decisions, and identified unique human resource issues that should be addressed prior to initiating any manpower modeling efforts. We return now to the specific questions posed to us. In the following sections, we summarize our conclusions regarding the adequacy of current ASI staffing models, the potential afforded by models drawn from outside the FAA, and the merits of a completely new approach. Our specific recommendations follow each set of conclusions, and the final section provides an elaboration of our recommended approach. Before proceeding, it is necessary to address one overarching issue that, while not directly related to our modeling charge, has an important bearing on all facets of the ASI staffing question. As explained earlier, staffing needs are heavily dependent on the capabilities of the individual ASIs, their fit with the prescribed work roles, and a host of organizational factors, such as management practices, culture, and other human resource considerations. In the course of our investigation, it became apparent that little can be gained through improved manpower modeling in the present case unless serious prior attention is accorded to these other critical human resource matters. Unless work is described with reasonable accuracy and the required skill sets are ensured through adequate recruitment, selection, training, and placement of personnel, estimates of the number

OCR for page 85
Staffing Standards for Aviation Safety Inspectors and distribution of ASIs required to sustain system performance will remain equivocal. Since our investigation was not explicitly directed toward these human resource issues, we cannot make explicit recommendations on the direction a preliminary effort of this sort should take. The evidence of weaknesses was sufficient, however, to justify our recommendation that such a prior effort is essential. Evaluation of the ASI Staffing Models Conclusions In reviewing the two comprehensive models of ASI staffing demand—the Automated Staffing Allocation Model (ASAM) and the aborted Holistic Staffing Model—against the features that we consider essential, we conclude that both fall short in certain areas. ASAM, as currently structured, appears deficient in at least the following respects: It does not predict the consequences of staffing shortfalls at any level and because of that it cannot be validated. It fails to account for some important factors affecting inspector workload. Many of its key parameters derive from expert judgment and have not been empirically validated. Hence it is not an empirically based model. The holistic approach, while potentially closer to the structure we consider appropriate, is lacking in the following respects: The documentation suggested statistically weak univariate methods for estimating parameters. The detail in its structure almost certainly exceeds a level that available data can support. There was no discernible plan for formal model validation,1 nor was it clear precisely what predictions the model would make regarding the consequences of alternative levels of staffing. Most importantly, the model was never developed or tested, so its potential utility and validity are unknown. 1 The contractor’s report describing the model (IBES, 2000) has sections devoted to model validation, but most of them address validation of the data, rather than the overall model predictions.

OCR for page 85
Staffing Standards for Aviation Safety Inspectors Recommendations In view of the above limitations, neither the ASAM nor the holistic model itself represents a promising framework in which to flesh out a more suitable ASI staffing model. Each has systemic deficiencies not easily remedied by piecemeal revisions to the current software. If the Flight Standards Service (AFS) were to continue to use the ASAM model without correcting the deficiencies, it would remain vulnerable to criticisms of the model’s validity and suitability for supporting staffing decisions. Staffing standards would continue to be developed without formal consideration of some important work drivers, and the model would remain unvalidated. In addition, we note that modification of an existing base of software code (particularly by someone other than the original developer) often proves more costly and less efficient than developing completely new code. We therefore recommend that any new modeling effort directed toward improving the FAA’s approach to addressing the ASI need structure for AFS not be constrained by over-reliance on the current modeling foundation. This is not to suggest, however, that nothing can be salvaged from these modeling efforts. Both ASAM, as it exists, and the holistic model, as it was described, provide a rich store of knowledge that would be useful for developing a new modeling approach. In particular, the existing documentation and the knowledge and experience of the FAA staff who were involved in the development of these models should be tapped in any new development effort. Furthermore, data-gathering activities, such as comprehensive job analyses, that were programmed into the holistic approach but never executed might profitably be revisited. Evaluation of Potential Alternatives Adapted from Other Organizations Conclusion Our analysis of approaches used by eight public-sector organizations that bear a resemblance to the FAA in certain key respects, as well as of the FAA’s staffing methodology for air traffic controllers, reveals that there is little potential for direct transfer—or adaptation—of any of these models to the FAA’s ASI staffing situation. In each case, unique features far outweigh the common ones, and the solutions that have evolved reflect that diversity.

OCR for page 85
Staffing Standards for Aviation Safety Inspectors Recommendation The ASI staffing challenge is sufficiently distinctive to rule out the option of importing, in whole or in part, an already developed model. Therefore, improvement over current practices can only be achieved through development of a new model, drawing on both the FAA’s previous experience with modeling efforts and careful consideration of the salient model properties described in this report. Model Development in Light of AFS/AIR Differences Conclusions The staffing situations for the AFS and the Aircraft Certification Service (AIR) are markedly different. With over 3,000 inspectors widely distributed across functional and geographical job domains and obvious deficiencies in the ASAM model, AFS is clearly in a position to benefit from (and justify the cost involved in) developing a new model. By contrast, it is difficult to justify a costly modeling effort for staffing the fewer than 200 AIR inspector positions, especially since the approach currently in use appears generally satisfactory. The main problem in this inspection domain seems to involve work recording systems rather than the staffing model. In particular, there is continuing uncertainty about future plans for the Manufacturing Inspection Management Information Subsystem and the Labor Distribution Reporting system. AIR management has told the committee that it is carefully weighing the effects of changes in FAA labor and services reporting, and it is taking action to ensure that the data needed to support its current approach to staffing are adequate. The challenge facing the AFS organization is considerably larger and more complex. To be demonstrably effective, a staffing model must incorporate accurate representations of workforce supply and demand, applying well-designed algorithms to produce accurate projections of staffing needs and the consequences of staffing shortfalls. It should also enable frequent updates and changes to ASI work processes. Most important of all, it must be integrally linked to appropriate measures of individual and system performance, without which its validity and utility cannot be established. Lacking these, any attempt to estimate the expected superiority of one staffing model over another, or the consequences and risks associated with understaffing or suboptimal distribution, is fruitless. In all the above respects, the current ASAM model is deficient. The holistic approach, while somewhat more promising, never materialized, and in view of its weaknesses does not appear to merit revisiting. What

OCR for page 85
Staffing Standards for Aviation Safety Inspectors performance measures currently exist are at best of dubious quality and utility. Recommendations AIR. The committee recommends continued effort aimed at improving the work recording systems for AIR inspectors rather than development of a new staffing model at this time. Should significant changes in the workload drivers appear in the future, the current AIR staffing approach might warrant another review. We recommend that AIR headquarters should be responsive to the concerns of ASIs and include them in any such improvement efforts. AFS. In the case of AFS, the situation clearly justifies a fresh analysis of the staffing need structure and development of a suitable model. Thus we recommend that the FAA initiate a systematic effort toward that end. Software tools and techniques for developing manpower and staffing models are readily available (e.g., Extend, SimScript, Flexsim, Arena, Micro Saint Sharp, and others; see Chapter 2). The use of such tools and software would reduce the cost of model development substantially while providing a powerful and usable modeling environment for predicting ASI staffing needs and the consequences of ASI staffing decisions. Since we recognize that this recommendation would involve a major effort encompassing a number of the considerations presented in this report, we elaborate our recommendations below. The approach should draw on the experience gained in developing ASAM and conceptualizing the holistic staffing approach, but it should not be constrained in an attempt to preserve the structure or core substance of either. While subtle, this distinction is important in the sense that an emphasis on preservation generally results in a more costly and less satisfactory end product than does starting afresh, especially when there are fundamental weaknesses in the initial structure—as clearly there are here. Therefore, the development process should include the following phases: Requirements definition, in which the questions that the ASI staffing model should address, data sources, and measures of effectiveness are specified; Model specification, in which a high-level software architecture is defined, including basic data flows, algorithms, and data structures; Model development, in which modern software engineering techniques and tools and an iterative development approach are utilized; and

OCR for page 85
Staffing Standards for Aviation Safety Inspectors Model verification and validation, through which it is demonstrated that the ASI staffing model software behaves as expected, and that the model produces sufficiently accurate predictions of the ASI staffing demand, supply, and supply-demand imbalance consequences. We recognize that the initial development and appropriate testing of a new model will require an up-front investment and take time. However, weighing that investment against the long-term benefits afforded by a model capable of estimating overall ASI staffing needs, optimal distribution, and understaffing consequences, we recommend making that investment. The modeling effort should be undertaken with the goal of supporting FAA decision making in both sufficiency and allocation decisions. To this end, it should embody as many of the desirable features identified in Chapter 2 as feasible. Most importantly, it should be empirically based, although certain relationships may necessarily be established through expert judgment—at least initially. Weighing the pros and cons of statistical versus process simulation models, we believe a hybrid approach may prove the most feasible and practical. We therefore recommend that the model designers explicitly consider which aspects of the model should be process-based and which based primarily on statistical relationships. As an example, routine tasks that have a long history of performance could most likely be modeled statistically, while new or modified tasks (like those associated with the Air Transportation Oversight System), for which there are few historical data on work processes and task performance times, may require more detailed process modeling. More precise specification than this would require a far more comprehensive analysis than was possible within the scope of the present study. Appropriate measures of system and individual performance are essential for both the development and validation of any improved staffing approach, irrespective of model properties and features. We therefore recommend that a concerted effort be invested at the outset in developing meaningful performance measures. We recognize that this is not a simple matter, and that heavy reliance on expert judgment at all levels will be necessary in order to devise measures that are both meaningful and widely accepted. Particular care should be taken not to sacrifice utility in the interest of convenience in this effort: the temptation to seize measures that are handy (and numeric) without proper regard for what they actually reflect is always present and must be guarded against.

OCR for page 85
Staffing Standards for Aviation Safety Inspectors One of the most significant weaknesses in the current ASI modeling practices is the burden of entering data to populate the model prior to making predictions. Hence we recommend that any future model should be designed so that it can be supported by institutionalized administrative databases, not by ad hoc surveys or other extraordinary data sources. There are existing databases, such as the Program Tracking and Reporting Subsystem and the National Vital Information Subsystem, that provide an excellent foundation on which to build. Such databases, maintained institutionally for purposes beyond staffing decisions, could be adapted to populate significant portions of the model without the need for manual data entry. This would help to ensure that the data are available, accurate, and timely, while reducing the burden of ad hoc data entry; it is a common practice in other staffing model applications, such as ISMAT, used by the U.S. Navy.2 In addition to model features per se, the FAA and those assisting with the staffing model development must consider a number of ancillary issues, mostly of a practical nature, that bear on model implementation. We therefore conclude with a set of recommendations addressing the most prominent of these considerations. Cost. It was difficult for the committee to estimate the cost of designing, developing, operating, and maintaining an ASI staffing model with the information available to us, and the estimate presented here is based on assumptions about the modeling environment that may not be accurate. Thus it should be taken as a rough estimate, with a high level of uncertainty. The greatest uncertainty is associated with the availability of the needed data in easily usable form. Other cost drivers for a modeling effort like this include the complexity of the organization, the number of variables to be modeled, the level of detail at which the model will operate, and the choice of method and model developers. Some of the FAA’s previous staffing models have apparently been of some use at the aggregate (AFS-wide) and regional levels, but they have not been useful at the facility level. It would certainly require more effort, and cost more, to design, test, and implement a model that is of proven value at all levels, including that of the individual facilities. Similarly, some stakeholders noted that the ASAM model fails to take 2 ISMAT (Integrated Simulation Manpower Analysis Tool) is a manpower analysis product developed for the United States Navy that can automatically import data from several established naval manpower databases that provide information on manpower supply and demand factors central to the manpower analysis issues addressed by the tool (Plott and Wenger, 2005).

OCR for page 85
Staffing Standards for Aviation Safety Inspectors account of some of the more subtle variables related to drivers of work demand (examples include number of designees overseen, age and condition of carrier fleets to be monitored), and they would like to see these factors considered in a new model. Again, the addition of variables and detail required in the design and test phases would result in increased cost, as would the greater precision of data required in the implementation phase. The FAA’s choice of a model design and development strategy will also affect the costs and the time (discussed below) needed to develop a model. We have noted that the development effort should be performed by a team that includes both subject matter experts (SMEs, people with an in-depth understanding of AFS operations and staffing issues) and professionals experienced in developing and implementing staffing models and systems. The FAA could assemble such a team relying exclusively on FAA employees, or it could engage a contractor, providing FAA employees as SMEs on a part-time basis with the contractor serving in the primary model development role. If the requisite staffing and modeling expertise is available in the current FAA workforce, the project could be performed in-house; otherwise, it would require a significant contractor investment. However, in deciding which strategy to pursue, careful consideration should be given to all associated costs, both direct and hidden. For example, on one hand, diverting FAA experts from their current responsibilities incurs hidden costs, as would any delay or compromise in end-product quality that a completely in-house effort might entail. On the other hand, direct investment in what is essentially a contracted “turnkey” effort would be substantial, but it would minimize the hidden costs associated with in-house development. What is essential is that whatever approach is selected should be chosen in full recognition of the level and term of investment required, and with a firm commitment to full completion. A repeat of the holistic experience must be avoided. The committee does not have the information on available talent and relative cost-value considerations to make a recommendation on which course should be pursued. We can, however, provide some rough estimates of the costs and time involved in a largely contracted effort. The committee’s best estimate, based on experience with organizations similar to AFS, is that it will take about $600-800K to design and build the modeling tool, $300-400K to initially populate the model with data and develop mechanisms to keep the data updated, and perhaps as much as $100K per year to keep the model and data current. Time. The previous experience of committee members who have performed or supported similar model development efforts suggests that

OCR for page 85
Staffing Standards for Aviation Safety Inspectors one to two years will be required to go through the development process that we recommend, if it is performed by an FAA-contractor team. Development of a model using only FAA in-house resources would be very likely to take longer (see above). By the end of this period, a working model should be in place. However, the model itself will continue to evolve over time as data are accumulated on its functioning, and inevitably adjustments will need to be made. We recommend that the development of a new model be undertaken in full recognition of this evolutionary requirement. Organizational constraints and culture. Perhaps the most critical determinants of a model’s long-term value are the organizational constraints and culture in which it is introduced and maintained. The committee would like to emphasize three points here. First, if the model is perceived as merely creating work for its care and feeding (e.g., gathering data to populate it) rather than as a valued aid to decision making, it will fail. Certainly, model acceptance depends in part on its ability to make predictions that are valid and consistent with ASI staff and management experience. However, even the most functional model will fail if misapplied or deemed by FAA decision makers and those affected as lacking in validity, utility, or significance. Not only must effort be directed toward developing and documenting a sound staffing model; once developed, its value must be actively promoted to those using it and affected by it. This may require an investment in a communications and training program to familiarize AFS employees with a new model and inform them of its advantages. Second, any effort of this sort will also be accepted most readily if those who are affected by its use are involved in the process of its development. ASIs and their managers should be consulted from the beginning, and they should have significant roles in the model design, development, and implementation. Because the Professional Airways Systems Specialists organization is an important stakeholder, its inclusion in the design and development process may facilitate the implementation of a new system. This will increase the likelihood that the resulting model will be easy to implement and that ASIs will become committed to the new system and be motivated to support it. Finally, as noted above, the FAA should address the human resource issues associated with changes to the ASI job and to AFS business processes before developing a new ASI staffing model. A new model of an old job, one that no longer is performed as it is modeled, will be of little value. Available data and improvements. As stated above, wherever possible,

OCR for page 85
Staffing Standards for Aviation Safety Inspectors the FAA should seek and take advantage of available data to populate new staffing models. Since a model’s predictions are only as good as the data that go into it, careful consideration should be given to the relative costs of modifying existing data-gathering systems versus creating new ones. Resources. The committee’s sense is that past ASI staffing models required a commitment of resources for development, maintenance, and use beyond what AVS or AFS management was able or willing to provide. This is not a problem unique to the FAA—modeling endeavors often begin with great ambitions and expectations, only to be undone by the weight of the work that is required to realize the ambitious goals (witness the holistic attempt). In a word, we recommend that the FAA undertake the development of an ASI staffing model with the features and supporting structure we have outlined above. However, it should undertake this effort only if it is institutionally committed to the development and maintenance of such a model. In addition, it will be critical to the success of any modeling effort that appropriate management resources be devoted to the effort. The managers should be skilled and experienced in managing software development projects and committed to the success of the effort. The skill with which the model development effort is managed will affect both the time to complete it and its eventual success. We strongly recommend that during the model design phase, the FAA should focus on what it is willing to invest versus what it is expecting to gain. Constant and explicit consideration of this trade-off is imperative during the early stages of ASI staffing model design and development. As noted above, investment includes not only initial model development cost, but also the costs of ongoing maintenance required to keep the database and model up to date and operating and the cost of continued use of the model. The gains should be viewed from the perspective of the breadth of ASI staffing questions that the model will be able to answer as well as the validity and utility of the answers it is able to yield. In a word, we recommend that the FAA conduct a serious cost-benefit analysis for any new modeling effort that is proposed. We hope that the processes and considerations presented in this report will prove useful in that endeavor.