Coda: The JPM Project and Accession Policy

Selecting personnel who will turn out to be successful on the job, particularly from a youthful and inexperienced applicant population, is a complicated business. Maintaining a standing army is also expensive. At the heart of these discussions of job proficiency, quality requirements, and trade-off models is the recognition that the public purse is not bottomless. The Department of Defense has an interest in attracting the best possible people into the military. At some point, however, the quality of the enlisted force becomes a function of what the nation can afford; high-quality personnel are more expensive to recruit and, having greater opportunities elsewhere, they are less likely than others to remain in the military for their career. In order to make reasonable budgetary decisions, Congress needs to be able to balance performance gains attributable to selecting those with better-than-average scores on the ASVAB against the costs of recruiting, training, and retaining high-quality personnel. And to improve their control over performance in the enlisted ranks, DoD and the Services need to be able to make more empirically grounded projections of their personnel quality requirements.

Although the problems are complex, and there is still room for improvement at every stage of the research and development, the results of the JPM Project to date indicate that the concept of linking selection standards to objective measures of job performance is basically sound. It appears that it will be feasible for human resource planners and policy makers to incorpo



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 207
Performance Assessment for the Workplace Coda: The JPM Project and Accession Policy Selecting personnel who will turn out to be successful on the job, particularly from a youthful and inexperienced applicant population, is a complicated business. Maintaining a standing army is also expensive. At the heart of these discussions of job proficiency, quality requirements, and trade-off models is the recognition that the public purse is not bottomless. The Department of Defense has an interest in attracting the best possible people into the military. At some point, however, the quality of the enlisted force becomes a function of what the nation can afford; high-quality personnel are more expensive to recruit and, having greater opportunities elsewhere, they are less likely than others to remain in the military for their career. In order to make reasonable budgetary decisions, Congress needs to be able to balance performance gains attributable to selecting those with better-than-average scores on the ASVAB against the costs of recruiting, training, and retaining high-quality personnel. And to improve their control over performance in the enlisted ranks, DoD and the Services need to be able to make more empirically grounded projections of their personnel quality requirements. Although the problems are complex, and there is still room for improvement at every stage of the research and development, the results of the JPM Project to date indicate that the concept of linking selection standards to objective measures of job performance is basically sound. It appears that it will be feasible for human resource planners and policy makers to incorpo

OCR for page 207
Performance Assessment for the Workplace rate empirical data derived from job performance into the decision process in a systematic way. Both for the Services' internal management purposes and for justification of Service quality requirements at the departmental level, this is desirable. Phase I of the JPM Project has demonstrated that it is possible to develop remarkably reliable hands-on measures of job performance. This was one of the biggest unknowns at the start of the project because there was very little experience of large-scale testing in the job sample mode. Success in this regard seems to have been due to the fact that the test items were designed as a sequence of discrete steps that involved specific, observable actions. This facilitated the scoring. Obviously, the nature of entry-level enlisted jobs lent itself to this kind of test item in a way that managerial or more cognitively loaded jobs would not. In addition, the reliability of the hands-on tests depended on careful training of the test administrators so that they would not give cues or otherwise distort the testing situation. (Such results should not be expected to generalize to situations in which supervisors score the performance of their subordinates or teachers their students.) Daily monitoring of scoring trends both within and among raters seems to have been an effective way to guard against unusually lenient or stringent scoring. The first phase of the JPM Project also demonstrated that reasonably valid measures of job proficiency can be developed. As anticipated, coverage of the total job domain is a problem with hands-on testing because the methodology is extremely time-consuming. Despite six to eight hours of testing time per individual, the analysis of measurement error due to the particular set of tasks selected for testing presented in Chapter 6 indicates that more time and more tasks would have produced better measures of job proficiency. In addition, there is still much to learn from the JPM data base about the relationships among the hands-on, walk-through performance tests, simulations, and written tests and about the substitutability of the less expensive and time-consuming measures for the hands-on tests. Nevertheless, the JPM Project has produced hands-on measures of job proficiency that are more credible than training grades and substantially more stable and accurate than the ubiquitous supervisor's rating. One of the most important implications of these findings is that the Armed Services Vocational Aptitude Battery has been shown to be moderately related to something fairly close to actual performance in 23 military jobs (comprising some 40 percent of the enlisted force) —and by inference to many more. Although the AFQT is not the optimal predictor that could be obtained from the ASVAB, the median corrected correlation of .38 with hands-on performance reported in Chapter 8 has considerable practical utility. When one remembers the uncertainties occasioned by the misnorming episode of 1976-1980, this is an important datum—enough to justify the use

OCR for page 207
Performance Assessment for the Workplace of the ASVAB in setting enlistment standards, if also modest enough to encourage the search for new predictors to add to the ASVAB. The development of cost/performance models for setting enlistment standards, now being explored by both DoD and the Services, has great potential relevance for accession policy. Until now, the standards-setting process has been largely based on an informal process of individual judgments and negotiations among the stakeholders. The manpower management models used by military planners for other purposes have simply assumed an appropriate enlistment standard or have used surrogates at quite some remove from job performance. If the JPM performance data can be successfully incorporated into trade-off models, the models will offer policy officials useful tools for estimating the probable effects on performance and/or costs of various scenarios—say a 10 percent reduction in recruiting budgets, a 20 percent reduction in force, or a downturn in the economy. The solutions provided by such models are not intended to and will not supplant the overarching judgment that policy officials must bring to bear, but they can challenge conventional assumptions and inject a solid core of empirical evidence into the decision process. One of the early runs of the DoD cost/ performance model, for example, has suggested the counterintuitive proposition that, in a situation of economic downturn, it is cost-effective to increase recruiting budgets: the marginal costs of increasing the proportion of high-quality personnel under these conditions will be outweighed by their performance contribution to the downsized force. There are a number of gaps in the JPM data that could limit the usefulness of the cost/performance methodology in operational settings. One of the most difficult challenges has been to find a satisfactory way to generalize the JPM results to performance in other military jobs. At least two lines of research might be fruitful. First, analysis of the personal attributes or traits required by a sample of military jobs (see Chapter 4) might provide clues for more homogeneous clusters of jobs around the JPM anchor jobs. In addition, a better understanding of the relative difficulty of military jobs has turned out to be of overriding importance in understanding the hands-on test scores and, by extension, in supporting the generalizability of the results to other, similar jobs. We have spoken at some length in this volume of the need to provide an absolute, or competency, interpretation of performance in a particular job in order to set enlistment standards—so that policy makers can address the question of how much performance is enough. If policy makers are only interested in optimizing costs, then relative comparisons among people as provided by the JPM data will suffice. From that data, analysts can construct various groups or ability mixes (based on different cutoff scores), compute their costs and overall performance outcomes, and see if there is a cutoff (or ability distribution) that minimizes cost per unit of productivity.

OCR for page 207
Performance Assessment for the Workplace But that analysis does not provide policy makers with a grounds for judging whether the optimal cost solution will produce an enlisted corps that can perform all first-term jobs competently. If policy makers want to be able to look at costs in the framework of the performance needed to accomplish the institutional mission, then the performance data will have to be interpretable with regard to an absolute standard of mastery or competency. By and large, the JPM measures were not developed as competency scales. However, some thought is being given to ways of providing external competency anchors for the data—say, by comparing the JPM cohort with the enlisted force in what military experts consider a very bad and a very good recruiting year and extrapolating the known job performance to the other two groups. This kind of thing seems well worth exploring. The full implications of the job performance measurement research for military policy makers—and for civilian-sector employers—remain to be worked out in coming years. It has produced a rich body of data and a wealth of methodological insights and advances. Most important of all, the JPM Project has defined the challenges for the next generation of research on performance assessment.