Modeling Cost and Performance for Military Enlistment: Report of a Workshop (1994)

The JPM Project was an ambitious effort to measure on-the-job performance of enlisted military personnel. The project offered researchers the opportunity to test hypotheses about differences between hands-on measures of job performance and paper-and-pencil surrogates of performance, about ethnic group differences in performance, and about gender difference in performance. Among the many concerns of the JPM Project and the committee, three were of primary interest. The first concern centered on the adequacy of test administration activities, such as scheduling, test security, and administration consistency from one individual to the next. A second concern was that the scaling of hands-on performance scores should go beyond rank ordering. That is, there was a need for a score scale that could be interpreted in terms of, at a minimum, acceptable and unacceptable performance, and preferably at finer gradations. A third concern centered on how job tasks should be selected. The committee recommended that stratified, random sampling of tasks be used rather than purposive sampling. They argued that purposive selection might capture only a certain type of task amenable to testing and might not be representative of the job, whereas stratified random sampling provided an unbiased selection of representative tasks and could more easily be defended.

Part II of this volume contains two papers dealing with various aspect of job performance measurement. In the first paper, Lauress Wise addresses the three concerns listed above in a thorough analysis of issues surrounding the validity of the JPM data and data from other sources such

as the Synthetic Validity (SYNVAL) project. He also examines the appropriateness of using these data for setting performance goals in the cost/performance trade-off model.

In the second paper in this section, Rodney McCloy pursues further the critical issue of generalizing performance results from jobs on which performance has been measured to jobs for which no data are available. This issue has been of particular importance because only a few jobs were selected for detailed study in the JPM Project and there was a need to generalize the findings to the several hundred jobs performed by first-term enlisted personnel. For the current model, the multilevel regression analysis method was recommended because of its contributions to performance prediction at the job level. The SYNVAL approach was considered but was deemed too time-consuming for the present project. McCloy discusses the application of the multilevel regression analysis in detail.

Lauress L. Wise

The times certainly are changing, particularly for the Department of Defense (DoD). The types of threats to which we must be ready to respond are changing; the size of the forces available to respond to these threats has decreased significantly and is likely to decrease further yet; and the resources available for recruiting, training, and equipping our forces have also declined dramatically. The debate continues as to how much we can afford to spend on defense in the post-cold war era and how much we can afford to cut. Efforts to keep missions, forces, and resources in some kind of balance are now focused on an emerging concept of readiness.

The DoD cost/performance trade-off model can play a central role in balancing readiness and resources. Those who sponsored the development of this model could not possibly have anticipated the importance of their efforts, but now that the model is nearing completion, the need for this type of linkage is all too obvious.

The model actually contains two separate linkages. Recruiting resources are linked to the levels of recruit quality, defined in terms of aptitude scores and educational attainment, obtained through application of the recruiting resources. In this first linkage, the model also suggests optimal mixtures of expenditures for recruiters, advertising, and incentives that will yield a given recruit quality mix with the smallest possible total cost. The second linkage is between recruit quality and performance in specific occupational specialties. The full cost/performance trade-off model takes specifications for required performance levels for each different job or family of jobs, deter-

mines a recruit quality mix that will yield the desired performance levels, and predicts the recruiting costs required to obtain this mix of recruit quality.

One more linkage is needed. The final step is to tie the emerging concept of readiness to levels of performance in different military specialties. This is, of course, a rather large step. Readiness is not yet a well-defined notion, but it is doubtlessly related to the number and effectiveness of different types of units, with unit effectiveness further related to individual performance levels. The focus of this paper is not, however, on how this last linkage might be achieved, but rather on how we might best set goals for performance levels today, while we are waiting for this final linkage to be created.

The DoD model has allowed us to replace the question of ''What level of recruit quality do we need?" with the question "What level of performance do we need?" The goal of this paper is to discuss issues and methods of trying to answer the latter question with information that is currently available to DoD personnel planners and policy makers.

The remainder of this paper is organized into four sections. The first section discusses issues related to the performance metric used in the DoD model. What is the meaning of the performance scale and what is a reasonable answer to the question "What level of performance do we need? The second section describes a normative approach to setting performance level goals. The general idea is to look at predicted performance levels for new recruits at different times and see how these levels varied across time and by job. At the very least, this normative approach will provide plausible ranges for performance level goals. The third section describes criterion-referenced approaches to setting performance level goals. In such an approach, judgments about the acceptability of different levels of performance are analyzed, and then additional judgments about minimum rates of acceptable performance are also collected. The final section lays out suggestions for additional research to further strengthen the support for specific performance level goals.

Performance in the DoD model is defined as percent-GO. The percent-GO scale is derived from the hands-on performance tests developed in the Joint-Service Job Performance Measurement/Enlistment Standards (JPM) Project. A general description of their development is provided by Wigdor and Green (1991:Chapter 4). More detailed descriptions of the development of these measures are provided by the researchers from each Service who worked on their development. Campbell et al. (1990) describe the

Army measures; the Air Force procedures are documented in Lipscomb and Hedge (1988). The description by Carey and Mayberry (1992) of the development and scoring of tests for Marine Corps mechanics specialties is a particularly good source, since this was one of the last efforts undertaken and it built on lessons learned in earlier efforts.

As described by Green and Wigdor (1991), the hands-on performance test scale is an attempt to create a domain-referenced metric in which scores reflect the percentage of relevant tasks that a job incumbent can perform successfully. They were developed as criteria for evaluating the success of selection and classification decisions. An estimate of the proportion of the job that the selected recruit could perform (after training and some specified amount of on-the-job experience and when sufficiently motivated) was judged the most valid measure of success on the job.

In general terms, hands-on test scores do provide at least relative information about success on the job that is reliable and valid. As such they are quite satisfactory for the purposes for which they are intended. There are several issues, however, that affect the level and linearity of the scores derived from them. Among others, these include the sampling of tasks, the scoring of tasks, and the way in which scores were combined across tasks.

In the JPM Project, a limited number of tasks was selected for measuring performance in each job. If these tasks were selected randomly from an exhaustive list of job tasks, generalization from scores on the hands-on tests to the entire domain of tasks would be simple and easy to defend. This was not, however, the case. Task sampling procedures varied somewhat across the Services. In nearly all cases, there was some attempt to cluster similar tasks and then sample separately from each cluster. In the Army, for example, a universe of up to 700 tasks (or task fragments) was consolidated into a list of 150 to 200 tasks; these tasks were then grouped into 6 to 9 task clusters. One, two, or possibly three tasks were then sampled from each of these clusters. This stratified sampling approach actually leads to a more carefully representative sample of tasks in comparison to simple random sampling. Technically, however, this approach also meant that tasks in different clusters were sampled with different probabilities. Statistical purists might require differential weighting of task results, inversely proportional to sampling probabilities, in order to create precise estimates of scores for the entire domain.

A second and more serious concern with the task sampling procedures is that many types of tasks were either excluded altogether or were selected with very low frequency. There was an attempt to collect judgments about the importance of each task as well as the frequency with which it was

pertormed. Few, if any, low importance or infrequent tasks were selected. In addition, some tasks were ruled out because it would be difficult or dangerous to collect work samples. Tasks for which poor performance by unsupervised recruits could result in damage to individuals or equipment were generally excluded. Tasks that were too easy (did not discriminate among incumbents) and, in a very few cases, too difficult were often excluded from consideration. A consequence of these exclusions is that performance on the sampled tasks generalized more precisely to performance on all job tasks that were judged moderately to highly important, were frequently performed, were not too dangerous, and were challenging enough to be at least a little difficult. This generalization is not necessarily bad, but the relevant domain should be kept in mind when it comes to setting performance standards. Higher performance levels would almost surely be expected for important and frequent tasks than for less important and less frequent tasks, but lower performance levels might be required for less dangerous tasks in comparison to more dangerous tasks; lower performance levels would also be expected for more difficult tasks in comparison to trivially easy tasks. In theory, these differences might offset each other, but to an unknown extent, so that performance on the sampled tasks might not be much different from performance across the entire job domain as called for by the Committee on the Performance of Military Personnel.

In their "idealized" description of a competency interpretation of the hands-on performance test scores, Green and Wigdor (1991:57) talk about the percentage of tasks in the job that an individual can do. Task performance is not, of course, dichotomous in most cases. For the most part, tasks were divided into a number (from 3 or 4 to as many as 20 or 30) of discrete steps, and criteria were established for successful performance of each of these steps. Naturally there were exceptions: the "type straight copy" task for Army clerks was scored in terms of words-per-minute adjusted for errors, one of the gunnery tasks for Marine Corps infantrymen was scored in terms of number of hits on target. For the most part, however, dichotomous scores were awarded for each of a discrete number of observable steps. In many or most cases, the criterion for successfully performing a step was clear and unambiguous. A mechanic changing a tire either did or did not tighten the lug nuts before replacing the cover, for example. In other cases, the criterion was somewhat arbitrary, as in "the grenade landed within some fixed (but mostly arbitrary) distance of the target'' or "the rifle was disassembled within an arbitrarily fixed amount of time." (These standards may have had some strong rationale, but they were not always obvious to the test developers.)

The discrete performance steps varied in terms of their criticality. If a job incumbent had to perform every step successfully in order to be considered successful on the task as a whole, than task success rates would be very low. The scoring generally focused on the process followed more than the overall output. In many cases, it was possible to achieve a satisfactory output even if some of the less critical steps were skipped or poorly performed. Weighting each of the individual steps according to their importance would have required an enormous amount of judgment by subject matter experts and would, in most cases, have led to less reliable overall scores. For purposes of differentiating high and low performers, the percentage of steps performed correctly, without regard to the importance of each step, proved quite satisfactory. When it comes to interpreting the resulting scores, however, it is in most cases impossible to say how many tasks an individual performed correctly because task standards were not generally established. Thus, the real interpretation of the hands-on test scores should be the percentage of task steps that an individual can perform correctly, not the percentage of tasks.

For the most part, the scores for each task were put onto a common metric—the percentage rather than number of steps performed successfully—and then averaged to create an overall score. Since the individual task scores were not dichotomous, there was some room for compensation with very high performance on one task (e.g., all steps completed successfully on a difficult task) compensating for somewhat lower performance on another task (e.g., several steps missed on a relatively easier task). As noted above, the tasks were not a simple random sample from a larger domain, and some form of task weighting—either by importance and frequency or by sampling probabilities—would have been possible. The fact that weights were not used should not create problems in interpretation so long as there were not highly significant interactions between task difficulty and importance or frequency. Some bias in the overall scale would also have resulted from the conversion from number to percentage of steps if there were a strong interaction between the number and difficulty of the steps within each task.

Several of the Services also examined different ways of grouping tasks or task steps into clusters in order to create meaningful subscores. The Army analyzed scores from six general task clusters: communications, vehicle operation and maintenance, basic soldiering, identifying targets or threats, technical or job-specific, and safety. Groupings of individual task steps into four knowledge and two skill categories were also analyzed. The Marine Corps created a matrix that mapped task steps onto different "behavioral elements." Although interesting, these subscores did not lead to

significant findings and have little bearing on the issue of setting overall performance standards.

In summary, the hands-on test scores derived from the JPM work do lend themselves to an interpretation of job competency. They are scaled in terms of the percentage of steps for important tasks that an individual will perform successfully. It is not unreasonable to interpret these scores in a general sense as the percentage of the central or important parts of the job that the individual can perform successfully. Since a great deal of aggregation is involved in setting performance requirements for the DoD model, a general sense interpretation is probably quite sufficient. Many of the issues raised above could be of significant concern if scores on individuals were being interpreted. Given the imprecision of the prediction of the performance scores from enlistment tests and high school credentials and the highly aggregated nature of the predictions, it seems reasonable to proceed with a general "percentage of job" interpretation.

One approach to setting performance level requirements is to ask what levels we have experienced in the past. This is essentially a normarive approach in which requirements for future years are tied to norms developed from prior years. If an important objective of the DoD model is to determine whether current quality levels are sufficient or perhaps excessive, then this normative approach is entirely circular, since performance level requirements will be tied back to current quality levels. Furthermore, we would be better off simply using aptitude scores to define quality requirements, since very little new information would be generated in linking performance requirements back to current or past quality levels.

At a more detailed level, however, several interesting questions can be addressed through analyses of normarive data. First is the question of the degree of variability in predicted performance levels across jobs. It may well be, for example, that observed differences in recruit quality are evened if high-quality recruits are more likely to be assigned to difficult jobs. A high-quality recruit assigned to a difficult job may end up being able to successfully perform the same percentage of job tasks as a lower-quality recruit in an easier job. If this were the case, then performance level requirements might generalize to new jobs more easily than quality requirements would.

Another question is how much predicted performance levels have varied over time, overall and by job. If performance levels have varied considerably, then using past performance levels to set future requirements would be questionable. If, however, performance levels (and performance level differences among jobs) are relatively stable across time, using past perfor-

mance levels as a benchmark would be more defensible, although, with dramatic changes in force levels, job requirements may not be as stable in the future as they have been in the past.

To examine these questions, the fiscal 1982 and 1989 accession cohorts were selected for analysis. The 1982 cohort was the earliest cohort for which the current form of the Armed Services Vocational Aptitude Battery (ASVAB, beginning with forms 8/9/10), was used exclusively in selection. This is important because the performance prediction equations in the linkage model are based on the subtests in the current ASVAB. Earlier ASVAB forms had different subtests and a number of assumptions would be required in generating AFQT and technical composite scores from these prior forms for use in the prediction equation. The 1989 cohort was the most recent for which data on job incumbents with at least two years of service are available. In addition, recruits from this cohort participated extensively in Operation Desert Storm and so some global assessment of their readiness is possible.

For each cohort, the active-duty roster as of 21 months after the end of the enlistment year was examined to identify incumbents in the 24 JPM specialties. The primary military occupational specialty (MOS) at time of enlistment was considered as the basis for sorting recruits into jobs, but it was discovered that many recruits are not enlisted directly into several of the JPM specialties. Consequently, it was decided to select for the JPM specialties on the basis of MOS codes at about 24 months of service. This decision meant that recruits who left service prior to 24 months were not included, and we were thus not modeling the exact enlistment policies. However, examining score distributions among job incumbents considered successful had many advantages and was deemed entirely appropriate.

The JPM samples included 24 different specialties. One of these specialties, Air Force avionics communications specialist, was deleted from the current study. The specialty code was changed prior to the 1989 accession year, and it was not possible to determine whether there was an appropriately comparable specialty.

ASVAB scores of record were obtained. A small number of cases in the 1982 cohort had enlisted using ASVAB forms 5, 6, or 7. These cases were deleted from the analyses since the ASVAB tests included in the AFQT and technical composites were not all available in these forms. Educational credential status was also obtained and coded as either high school

graduate or nongraduate. In these analyses, recruits with alternative high school diplomas were counted among the nongraduates.

Job performance prediction equations for the JPM specialties that were developed in the Linkage Project (McCloy et al., 1992) were used. These equations use AFQT and technical composite scores (expressed as sums of standardized subtest scores), educational level, time in service, and the interaction (product) of time in service and the technical composite in a prediction equation. The weights for each predictor are determined from job analysis information from the Dictionary of Occupational Titles. A constant value of 24 months was used for time in service so that predictions would reflect "average" first tour performance for a 3- to 4-year enlistment. Since predicted performance is a linear function of time in service, the average across the first tour will be equal to the value predicted for the midpoint. Ignoring the first six months as mostly training time, the midpoint would occur at month 21 for a 3-year tour and at month 27 for a 4-year tour.

The primary results were summarized in an analysis of variance with MOS (23 levels) and accession year (2 levels) treated as independent factors and predicted performance and the AFQT and technical composites each analyzed as dependent variables. One hypothesis tested with these analyses was that predicted performance might show smaller differences among jobs in comparison with the AFQT or technical composites. Separate intercepts (determined from job characteristics) were estimated for each job. It was plausible to believe that required performance score levels might be reasonably constant across jobs, even if input quality was not.

A second hypothesis tested was that predicted performance would show relatively smaller differences across recruiting years in comparison to the AFQT and technical composites, since predicted performance combines both AFQT and technical composite scores and the latter might be less affected by differences in recruiting conditions than the AFQT.

Across all jobs and both recruiting years, the mean test level was 68.4. Table 1 shows the sample sizes, and the mean and standard deviation of predicted performance scores for each job and entry cohort. Mean predicted performance scores by job and year are also plotted in the table. Table 2 shows the means for the AFQT and technical composites and predicted performance by year. These means are adjusted for differences in the MOS distributions for the two years. Table 3 shows F-ratios testing the

significance of differences across years, MOS, and the year-by-MOS interaction for these same three variables.

The first significant finding from these analyses was that there was virtually no change in mean predicted performance between the 1982 and the 1989 cohorts, overall or for any of the jobs analyzed. A statistically significant mean gain in AFQT was offset by a significant mean drop in technical scores between the 1982 and 1989 cohorts, resulting in no significant difference in predicted performance. Second, there was some consistent variation among jobs in predicted performance levels, with lows of around 60 for Army field artillery (13B) and truck driver (64C), Air Force 732 and two of the Marine Corps infantry jobs and highs above 75 for Air Force 328 and 426 jobs and Marine Corps 034. This variation is consistent with the assumption that higher competency levels might be required in more critical or complex jobs. The variation across jobs in predicted performance was much more significant (much greater F-ratio) than the varia-

tion in AFQT and technical scores. One reason for this is that the within job variance of predicted performance is small in comparison to the within job variance of the predictor composites. Predicted performance is critical for all jobs, and so is restricted in range. The predictor composites, particularly the technical composite, are not as critical for all jobs. Each composite is less restricted in range for those jobs for which it is less critical, leading to greater average within job variation. A conclusion that follows from this finding is that it is probably not sufficient to use a single average performance level for all jobs. Consequently, some judgmental procedure is needed to capture essential differences in performance level requirements for different jobs.

The Army investigated alternative approaches for setting job performance standards as part of the Synthetic Validity (SYNVAL) Project (Wise et al., 1991; Peterson et al., 1990). This project had two major objectives: (1) to investigate ways of generalizing performance prediction equations from a sample of jobs for which criterion data were available to the entire population of jobs and (2) to investigate ways of setting performance level standards for different jobs. To the extent that performance level standards could be linked to the test scales, the second objective relates directly to the topic of this chapter.

The SYNVAL project was conducted in three phases. The first phase involved pilot tests of job description and standard setting instruments for three jobs. The second phase involved a larger data collection with revised instruments and procedures on a larger sample of 7 jobs. In the final phase, further revisions to instruments and procedures were administered for a sample of 12 jobs to test additional issues, including generalization to one job for which no criterion data were available. The most directly relevant results with respect to standard setting come from Phases II and III.

Most work on standard setting has involved identification of a minimal passing score on a certification or criterion-referenced examination. This might seem to be exactly what is needed in setting performance standards for use with the DoD model. A common concern in education with minimum competency examinations is that they provide little motivation for students to achieve at levels well above the minimum. In setting enlistment standards, it is reasonable to ask whether it is acceptable to have all enlistees at the same minimum level or whether it would not be better to have a mix of skill levels within each occupational specialty. Particularly in situations involving teamwork, a mix of skill levels may be more optimal than inordinate homogeneity of skill levels. In the SYNVAL project, four different skill levels were defined for each job. These skill levels were tied to operational decisions that supervisors would make about job incumbents in an effort to derive cost implications for the different performance levels:

Unacceptable: the recruit cannot perform the job, is not likely to become an acceptable performer with additional training, and should be discharged;

Marginal: the recruit is not performing acceptably and should be given additional training to bring performance up to standard;

Acceptable: the recruit is performing at an acceptable level and making a positive contribution to force readiness; and

Outstanding: the recruit is performing well above minimal standards and should be given a promotion or other recognition for superior performance.

In setting performance level goals for the DoD model, standards for minimum performance should not be confused with performance level goals. Performance goals for the model should reflect a desired mix of abilities above the minimum. Ideally, economic analyses would be used to identify the mix of marginal, acceptable, and outstanding performers that is most cost-efficient for each job. Overall performance goals would then reflect an average of scores from these three performance levels, with each level weighted according to this optimal mix. As indicated below, the SYNVAL project results speak primarily to the first step of defining the different performance levels. Examples of the distributions across performance levels for incumbents in different jobs are provided, but economic analyses to identify more optimal mixes remain to be done.

Five different standard-setting instruments were administered to subject matter experts in the Phase II sample of jobs as described in Peterson et al. (1990). Standards obtained from the different protocols were compared with each other. The level of agreement among judges on the standards they provided was also examined for each approach. It was important that there be adequate agreement among judges on the standards before the standards can be used to determine selection criteria.

The overall objective of this project was to generalize to jobs for which no performance data were available. For this purpose, neither standards for individual tasks nor standards for the job as a whole were judged useful, although, in retrospect, the data on overall job standards were quite informative. The primary focus was on performance dimensions defined in terms of families of related job tasks. The job performance dimensions used for the standard-setting exercises came from a preliminary version of the hybrid taxonomy (used in the job analysis; see Peterson et al., 1990). A preliminary set of 24 dimensions were identified based on job components contained in the task categories and job activities taxonomy. Not all 24 dimensions were applicable for the Phase II jobs and thus the summary tables show only those dimensions relevant to Phase II jobs.

Three different proficiency categories based on the three minimum performance levels (cutoffs) that defined the performance levels described were examined. The three proficiency categories were unacceptable (less than marginal), unacceptable and marginal combined (less than acceptable), and outstanding (greater than acceptable). The last category was described as outstanding rather than less than outstanding to enhance interpretability.

Three different standard-setting protocols are referred to here:

1. Soldier-Based Protocol (Soldier Method). Under this protocol, judges

were asked to estimate the percentage of current job incumbents who are performing at each of the four levels of acceptability (e.g., what percentage is unacceptable) on a given performance dimension. This approach assumes that empirical data on soldier performance are available (in the form of hands-on tests scored GO/NO-GO) on a representative sample of the soldiers in question so that these "percentage-performing" estimates can be related to actual performance scores.

2. Critical Incident Protocol (Incident Method). Under this protocol, judges were presented with incidents that reflected varying levels of effectiveness on a particular performance dimension and asked to judge, for each incident, the acceptability level of soldiers whose typical performance was described by the incident.

3. Task-Based Protocols (Task-Hypothetical Soldier, Task-Detailed Percent-GO, and Task-Abbreviated Percent-GO Methods). Under these protocols, judges were presented with a list of specific tasks within each performance dimension (possibly from different MOS) and asked to make judgments about minimum percent-GO scores that a soldier should achieve to qualify as marginal, acceptable, and outstanding performers. Three types of judgments were collected. In the first condition, the hypothetic soldier (HS) approach, judges were presented detailed sets of hands-on test score sheets and corresponding summary percent-GO scores for 10 hypothetical soldiers and asked to rate the acceptability of each of these hypothetical soldiers (Task-HS Method). In the second condition, the detailed percent-GO (DPG) approach, judges were asked to rate the minimum percent-GO score for each level of acceptability on each specific task used to illustrate the dimension (Task-DPG Method). In the third condition, the abbreviated percent-GO approach, judges were given a list of tasks without detailed percent-GO scores or actual score sheet examples and asked to rate minimum percent-GO scores for tests on these types of tasks (Task-APG Method).

The five different standard-setting methods involved judgments that used very different metrics. The soldier-based method asked about the percentage of soldiers performing at each acceptability level; the critical incident method used a series of discrete behavioral items; and the task-based methods used judgments about acceptable levels of percent-GO scores.

A critical question in this research was the extent to which the different methods led to similar or distinct ability requirements. To answer this question, it was necessary to convert the standards derived from each approach to a common metric, making it possible to determine whether one of the methods led to significantly stricter or more lenient standards than the others and also to compare the level of agreement among judges using this same metric.

The soldier-based metric (percentage of soldiers performing at each

level) was used as the basis for comparison. If standards set with the other methods led to very different assessments of the percentage of soldiers performing at each level (in comparison to the judges' direct assessment), then the validity of these other methods would be questionable. Data from the Army's Project A on samples of incumbents in each MOS were used to estimate the percentage of soldiers performing above or below each of the standards set. The specific methods used to estimate the percentages of soldiers performing at or below specific critical incident or Percent-GO score levels are detailed in Whetzel and Wise (1990).

Table 4 shows the means and standard deviations of the judges' ratings of the percentage of soldiers performing at each acceptability level for each combination of performance dimension and MOS. There are some distinct differences in the judges' estimates of soldiers' ability across different MOS and dimensions. For example, for MANPADS crew members (MOS 16S), soldiers had high acceptability ratings for performance dimension 7 (detect targets), but relatively lower acceptability ratings on dimension 15 (operate vehicles). These differences reflect, in part, the appropriateness or importance of the dimension for the MOS (e.g., all crew members detect targets, but not all have to operate vehicles).

The standard deviations in the table are a measure of the degree of agreement among judges. These numbers also give an indication of the potential appropriateness of the dimension for the MOS. When there is more significant disagreement among judges, it may be because the dimension is poorly described or is not clearly appropriate for the MOS in question. To a certain extent, the standard deviations are related to the means—when there is more disagreement, the means tend to be closer to 50 percent of soldiers performing at a particular proficiency level. (Only very high or low scores are possible if nearly all of the judges consistently give high or low ratings.) In some cases, however, the standard deviations are greater than the means (e.g., the percentage of 16S soldiers rated unacceptable on the task operate vehicles or the percentage of motor transport drivers, MOS 88M, rated unacceptable on the task navigate). This can happen only when the distribution of ratings is highly skewed, with most judges giving low ratings (hence a low mean) and a few judges giving very high ratings (leading to a large standard deviation).

Similarly detailed analyses of results from each of the other methods are reported in Whetzel and Wise (1990). Table 5 shows comparisons of the overall results from each of the five methods. The methods varied considerably in terms of ''leniency": the soldier method suggested only 15 percentage of current job incumbents performed unacceptably and 25 per-

centage were less than fully acceptable, while the task-APG method implied that 43 percent were at unacceptable and 63 percent were at less than fully acceptable levels.

There were also notable differences in the reliabilities associated with the different methods. The task-based methods, particularly those based on percent-GO score ratings, had significantly higher single-rater reliabilities than the other methods. This appears to be a result of stereotypical beliefs that 60 or 70 percent correct should be the minimum "passing" score.

The task-DPG and task-APG methods are of particular interest because they use the same Percent-GO scale used in the DoD model. The only

difference between these two approaches is that, for the task-DPG method, a great deal of information is provided about the particular steps (items) that are considered in computing the percent-GO scores. It is reasonable to ask whether this additional information led to different standards or different levels of agreement among judges. In other words, did the extra information help judges to reach a common understanding or just confuse them?

Table 6 shows the means and standard deviations of the percent-GO scores that resulted from each method, rater group, and acceptability level. As can be seen from this table, the APG method usually led to slightly harsher ratings, but also very slightly smaller standard deviations than the DPG method. The differences were minimal at most.

In Phase III of the Army Synthetic Validity Project, standard-setting instruments were revised and used to collect data on 12 additional jobs. The task dimensions for which standards were set were also revised to make the dimensions parallel to the major categories in the revised job description instrument.

The task-based standard-setting instrument is most relevant to the issues in this paper. The Phase III version was simplified by eliminating detailed information about the task tests and eliminating the requirements for setting standards for individual tasks. For each performance dimension, three illustrative tasks were listed and then the number of soldiers at or below each percent-GO score level (in increments of 5 from 10 to 100) was provided. Raters were asked to draw lines between the score levels to indicate divisions between different performance levels (unacceptable versus marginal, marginal versus acceptable, and acceptable versus outstanding).

Table 7 shows the mean and standard deviation of the percent-GO cutoffs for each Phase III job, performance dimension, and performance level. As shown in this table, there was reasonable consistency across jobs, with

minimum scores of about 65 percent, 80 percent, and 95 percent for marginal, acceptable, and outstanding levels, respectively. It is difficult to tell the extent to which the small differences among jobs in the cutoff scores are reliable. Different task dimensions and different groups of judges were used with the different jobs, and the variation in results may well be associated with random and systematic factors associated with these differences. In any event, given all of the limitations on the accuracy of domain-referenced interpretations of the percent-GO scales, these differences in cutoffs would not appear to be of practical significance.

Table 8 shows the estimated percentage of current job incumbents at the lower and higher performance levels. The Phase III approach attempted to combine the task-based and soldier-based approaches by providing both criterion information (about the tasks) and normative information (about the proportion of soldiers at each level). As shown in the table, the results reflected this compromise with the proportion of soldiers judged unacceptable (28 percent) or less than fully acceptable (48 percent) falling midway between the Phase II results for the separate soldier and task-based methods (15 to 43 percent and 25 to 63 percent, respectively). The percentage of soldiers at the outstanding level (20 percent) also fell between the extremes of the Phase II methods (12 to 23 percent). Variation in the performance distributions across jobs was somewhat greater in comparison to the variation in score cutoffs, particularly at the high end of the scale. The percent-

age performing at an outstanding level varied from 15 percent for 54B to 33 percent for 29E.

The SYNVAL Project demonstrated both the promise and the difficulties of efforts to define comparable performance categories across different jobs. There were a number of arguments and concerns about differences in procedures and instruments and the reliability of individual judgments was not extremely high. One persistent finding was that standards set using the hands-on performance tests appeared harsh in comparison with direct estimates of performance level distributions. The consequence, in Phase III, that over a quarter of all recruits are performing unacceptably and should be terminated may be difficult to accept. That as many as half of the incumbents would benefit from additional training is much more credible and is consistent with current refresher training programs. At the upper end, the definition of outstanding performance is somewhat more subjective, and 20 percent outstanding is not unreasonable.

An important finding from the SYNVAL project was that cutoff scores for the percent-GO scale on the hands-on performance tests were reasonably similar across jobs. It would be reasonable to adopt 65, 80, and 95 percent cutoffs for all jobs, eliminating a requirement to collect new judgments for each new jobs. This is particularly important since this is the same metric used in the model. What remains is to identify factors associ-

ated with differences in optimal mixes of performance levels across different jobs. The key question seems to be "When are higher proportions of outstanding performers required?"

The performance metric used with the DoD model was constructed in such a way that a domain-referenced interpretation is at least plausible. There were some potentially offsetting restrictions on the domain of job performance covered by the hands-on performance tests. Although no absolute definition of successful performance was developed at the task level, there were clear criteria for success on individual task steps. For the global purposes required by the model, it is not unreasonable to interpret the scores as the percentage of the job that a recruit can perform successfully.

Normative data for two different entry cohorts showed significant variation across jobs in mean predicted performance but remarkable stability over time periods. Mean predicted performance scores ranged between 60 and 80 percent across different jobs with an overall average of 66 percent. Based on these data, it appears reasonable to use past predicted performance levels in setting performance targets for each job, but generalization across jobs will be somewhat limited.

In a more criterion-referenced approach, the Army Synthetic Validity Project analyzed procedures for defining different levels of job performance that are tied to possible economic consequences associated with good or poor performance. Task-based methods tied to the hands-on performance tests tended to yield stricter standards in comparison to direct judgments about the proportion of soldiers at each performance level. Standards set using the task-based methods were reasonably consistent across jobs. Performance below 65 percent was considered unacceptable, with the implication that the recruit should be discharged; from 65 to 80 percent was considered marginal, with the implication that additional training should be provided; from 80 to 95 percent was considered acceptable; and above 95 percent was considered outstanding with promotion or some other recognition deemed appropriate. Some variation among jobs in the proportion of incumbents at each performance level was observed for each of the standard-setting methods.

The normative and criterion-referenced approaches agreed that there was significant variation across jobs in performance levels. Normative data suggested that performance level targets of about 66 percent were consistent with current accession and readiness levels. The criterion-referenced approach implied that this was a minimally acceptable level and not necessarily a good target for average performance.

If we accept the results of the normative and criterion-referenced approaches summarized above, we can consider setting performance targets for the DoD model by multiplying the targeted number of recruits by an average performance level between 60 and 80 on the hands-on performance test percent-GO scale. Further research would be useful in defending more precise mean predicted performance level targets and, in particular, in supporting differences among occupational specialties in performance level targets. Several specific topics for additional research are discussed below.

The SYNVAL project attempted to link performance levels to operational decisions about individuals. This linkage was based entirely on expert judgments. A fruitful area for further research would be the development of better descriptions of what individuals at different performance levels can and cannot do. There has been a considerable effort in recent years to establish overall standards for educational achievement for use in interpreting results from the National Assessment of Educational Progress. Part of this process has involved analyses of items answered successfully by students at one level by not at the next lower level. A similar effort with the hands-on test performance levels would help in the development of explicit rationales for economic consequences of performance at specific levels.

While common performance level descriptions appeared feasible, there was considerable variation across jobs in the proportion of incumbents at each level. More systematic research is needed on the relationship of job characteristics (e.g., task complexity, the extent of teamwork, indicators of criticality of tasks, specific consequences of unsuccessful task performance) to different performance distribution targets. Most particularly, differences among jobs in the need for outstanding performers should be modeled.

As more concrete conceptions of factors relating to readiness emerge, it would be useful to relate these factors to level and heterogeneity of the performance of individuals in different units. In particular, analysis data from unit training exercises should prove useful in linking individual performance levels to indicators of unit effectiveness.

Campbell, C.H., Ford, P., Rumsey, M.G., Pulakos, E.D., Borman, W.C., Felker, D.B., de Vera, M.V., and Riegelhaupt, B.J. 1990 Development of multiple job performance measures in a representative sample of jobs. Personnel Psychology 43:277–300.

Carey, N.B., and Mayberry, P.W. 1992 Development and scoring of hands-on performance tests for mechanical maintenance specialties. CNA Research Memorandum 91–242 . Alexandria, Va.: Center for Naval Analyses.

Eitelberg, M.J. 1988 Manpower for Military Occupations. Alexandria, Va.: Office of the Assistant Secretary of Defense (Force Management and Personnel).

Green, B.F, Jr., and Wigdor, A.K. 1991 Measuring job competency. In A.K. Wigdor and B.F. Green, eds., Performance Assessment for the Workplace, Volume 2. Committee on the Performance of Military Personnel. Washington, D.C.: National Academy Press.

Laurence, J.H., and Ramsberger, P.F. 1991 Low-Aptitude Men in the Military: Who Profits, Who Pays? New York: Praeger Publishers.

Lipscomb, M.S., and Hedge, J.W. 1988 Job Performance Measurement: Topics in the Performance Measurement of Air Force Enlisted Personnel . Technical Report AFHRL-TP-87-58. Brooks Air Force Base, Tex.: Air Force Human Resources Laboratory.

McCloy, R.A., Harris, D.A., Barnes, J.D., Hogan, P.F., Smith, D.A., Clifton, D., and Sola, M. 1992 Accession Quality, Job Performance, and Cost: A Cost-Performance Tradeoff Model. HumRRo Report No. FR-PRD-92-11. Alexandria, Va.: Human Resources Research Organization.

Peterson, N.G., Owens-Kurtz, C., Hoffman, R.G., Arabian, J.M., and Whetzel, D.L. 1990 Army Synthetic Validity Project: Report of Phase II Results. Volume I. Army Research Institute Technical Report 892. Alexandria, Va.: United States Army Research Institute for the Behavioral and Social Sciences.

Whetzel, D.L., and Wise, L.L. 1990 Analysis of the standard setting data. In N.G. Peterson, C. Owens-Kurtz, R.G. Hoffman, J.M. Arabian, and D.L. Whetzel, eds., Army Synthetic Validity Project: Report of Phase II Results, Volume I. Army Research Institute Technical Report 892. Alexandria, Va.: United States Army Research Institute for the Behavioral and Social Sciences.

Wigdor, A.K., and Green, B.F., Jr., eds. 1991 Performance Assessment for the Workplace, Volume 1. Committee on the Performance of Military Personnel. Washington, D.C.: National Academy Press.

Wise, L.L., Peterson, N.G., Hoffman, R.G., Campbell, J.P., and Arabian, J.M. 1991 The Army Synthetic Validity Project: Report of Phase III Results, Volume 1. Army Research Institute Technical Report 922. Alexandria, Va.: United States Army Research Institute for the Behavioral and Social Sciences.

Rodney A. McCloy

Military manpower and personnel policy planners have pursued the goal of documenting the relationship between enlistment standards and job performance for over 10 years (Steadman, 1981; Waters et al., 1987). Prior to the Joint-Service Job Performance Measurement/Enlistment Standards (JPM) Project begun by the Department of Defense (DoD) in 1980, proponents of validity studies examining the Services' selection test, the Armed Services Vocational Aptitude Battery (ASVAB), had primarily relied on measures of success in training as criteria. The catalysts for the enormous JPM effort included the misnorming of ASVAB forms 6 and 7 that resulted in the accession of a disproportionate number of low-aptitude service men and women, a decrease in the number of 18–21-year-olds (i.e., the enlistmentage population), and the perpetual requirement of high-quality accessions (Laurence and Ramsberger, 1991). These events simultaneously focused attention on the need to relate the ASVAB to measures of job performance and the absence of such measures.

The outcome of this series of events was an all-Service effort to measure job performance and to determine the relationship between job performance and military enlistment standards. The steering committee for this effort established general guidelines for the work but encouraged diverse approaches to performance measurement in the interests of comparative research. To this end, each Service conducted its own JPM research program with its own specific goals and questions. As a result, the measures and samples across the Services are sometimes quite different. For ex

ample, the Army wished to limit the effect that job experience would have on the results from their JPM research project (Project A; Campbell, 1986). Hence, the range of months in service in the Army sample is small.¹ In contrast, the Marine Corps was keenly interested in the effect of job experience on performance and developed its performance measures to be applicable to soldiers in both their first and second tours. Accordingly, the range of experience in the Marine Corps sample is relatively large.

Although each Service developed several job performance measures (Knapp and Campbell, 1993), the Joint-Service steering committee selected the work sample (or ''hands-on") performance test as the measure to be "given [resource and scientific] primacy" and to serve as "the benchmark measure, the standard against which other, less faithful representations of job performance would be judged" (Wigdor and Green, 1991:60). Although some have questioned whether hands-on measures are the quintessential performance measures (Campbell et al., 1992), there is little debate over the notion that hands-on measures provide the best available assessment of an individual's job proficiency—the degree to which one can perform (as opposed to will perform) the requisite job tasks.

The advantages and disadvantages of hands-on measures are well known. The scientific primacy given them by the Committee on the Performance of Military Personnel is justifiable at least in part by their face validity and their being excellent measures of an individual's task proficiency. However, there can be limitations regarding the types of tasks they can assess (e.g., it would be difficult to assess a military policeman's proficiency at riot control using a hands-on measure), and their resource primacy is virtually required given their expense to develop (see Knapp and Campbell, 1993, and Wigdor and Green, 1991, for detailed descriptions of the development of the hands-on performance tests). To highlight this point, consider that hands-on tests were developed for only 33 jobs as part of the JPM Project (Knapp and Campbell, 1993), the most extensive performance measurement effort ever conducted.²

Much has been gained from the JPM research. The ASVAB has been shown to be a valid predictor of performance on the job as well as in training (Wigdor and Green, 1991). In addition, project research demonstrated that valid, reliable measures of individual job performance can be developed, including hands-on tests. JPM research supports the use of the ASVAB to select recruits into the military. But if a recruiter wished to predict an individual's performance score for a military job, he or she would

be limited to at best 45 jobs (those having some form of performance criterion) and 33 jobs that have hands-on measures. More desirable would be the capability to predict an individual's job performance for any military job, whether or not a hands-on measure (or other performance measures) had been developed for it.

Transporting validation results beyond a specific setting to other settings has been the concern of two methods in the industrial/organizational psychology literature: validity generalization and synthetic validation. In this paper, following a brief discussion of these two methods, a third method that can be used to provide performance predictions for jobs that are devoid of criterion data—multilevel regression—is introduced and discussed in detail. The application of multilevel regression models to the JPM data is presented, and results are also given from an investigation of the validity of the performance predictions derived from the multilevel equations.

For many years, psychologists specializing in personnel selection emphasized the need to demonstrate the validity of selection measures upon each new application—whether the goal was to predict performance in a different job, or for the same job in a different setting. The rationale for this approach was that the validity of a selection measure was specific to the situation. Indeed, one typically observed a rather large range in the magnitude of validity coefficients across various situations for the same test or for similar tests of the same construct (e.g., verbal ability). But conducting job-specific validity studies could be very expensive. Furthermore, for jobs containing a small number of incumbents, such studies would be likely to provide either unstable or nonsensical results.

Focusing on this latter shortcoming of the situational specificity hypothesis, Schmidt, Hunter, and their colleagues (e.g., Hunter and Hunter, 1984; Schmidt and Hunter, 1977; Schmidt et al., 1981; Schmidt et al., 1979) suggested that the majority of variation in observed validity coefficients across studies could be explained by statistical artifacts. This notion led to the conclusion that, contrary to the conventional wisdom, the validities of many selection measures (and cognitive ability measures in particular) were in fact generalizable across jobs and situations.

In validity generalization, a distribution of validities from numerous validation studies is created and then corrected for the statistical artifacts of sampling error, criterion unreliability, and predictor range restriction.³ The

result is a new distribution that more accurately reflects the degree of true variation in validity for a given set of predictors. If a large portion of the coefficients in the corrected distribution exceeds a value deemed to be meaningful, then one may conclude the validity generalizes across situations. In essence, validity generalization is a meta-analysis of coefficients from validation studies, and other recta-analytic approaches have been suggested (Hedges, 1988).

Concluding that validity coefficients generalize across situations, however, does not preclude variation in the coefficients across situations. The portion of the corrected distribution lying above the "meaningful" level may exhibit significant variation. If a substantial portion of the variation in the observed coefficients can be attributed to statistical artifacts (75 percent has served as the rule of thumb), however, then the mean of this distribution is considered the best estimate of the validity of the test(s) in question—situational specificity is rejected and the mean value is viewed as a population parameter (i.e., the correlation between the constructs in question). Hunter and Hunter (1984:80) reported that the validity generalization literature clearly indicates that most of the variance in the validity results for cognitive tests observed across studies is due to sampling error such that "for a given test-job combination, there is essentially no variation in validity across settings or time."

Although the findings of validity generalization research have considerably lightened the burden for personnel psychologists interested in demonstrating the validity of certain selection measures, the procedure is not without its critics, and many of its features are questioned (Schmitt et al., 1985; Sackett et al., 1985). Furthermore, the approach does not speak directly to the issue of obtaining performance predictions for jobs devoid of criteria. Although the corrected mean validity could be used to forecast performance scores, the approach is too indirect. Selection decisions are often based on prediction equations, which may in turn comprise a number of tests. The application of validity generalization results to this situation would require a number of validities for the test battery (i.e., composite) in question. Furthermore, even if such results were available, a more desirable approach would be to "focus the across-job analysis on the regression parameters of direct interest in the performance prediction, rather than on the correlations of the validity generalization analysis" (Bengt Muthén, personal communication, January 18, 1990).

A second alternative to the situational specificity hypothesis is synthetic validity (Lawshe, 1952:32), defined as "the inferring of validity in a specific situation." The basic approach is to derive the validity of some test

or test composite by reducing jobs into their components (i.e., behaviors, tasks, or individual attributes necessary for the job) via job analysis, determining the correlations between the components and performance in the job, and aggregating this validity information into a summary index of the expected validity. Wise et al. (1988:78–79) noted that "synthetic validation models assume that overall job performance can be expressed as the weighted or unweighted sum of individual performance components." Mossholder and Arvey (1984:323) pointed out that synthetic validity is not a specific type of validity (as opposed to content or construct validity), but rather "describes the logical process of inferring test-battery validity from predetermined validities of the tests for basic work components."

Several approaches to synthetic validation may be found in the literature, including the J-coefficient (Primoff, 1955) and attribute profiles from the Position Analysis Questionnaire (McCormick et al., 1972). Descriptions of these approaches are available in Crafts et al. (1988), Hollenbeck and Whitemer (1988), Mossholder and Arvey (1984), and Reynolds (1992). To illustrate the process of obtaining synthetic validity estimates, consider the Army's Synthetic Validation Project (SYNVAL) as an example.

The goal of the SYNVAL project (Wise et al., 1991) was to investigate the use of the results of the Army's JPM Project (Project A) to derive performance prediction equations for military occupational specialties (MOS) for which performance measures were not developed. SYNVAL researchers employed a validity estimation task to link 26 individual attributes, which roughly corresponded to the predictor data available on the Project A sample, to job components described in terms of three dimensions (i.e., tasks, activities, or individual attributes). These validity estimates were provided by experienced personnel psychologists.

The validity estimates, in concert with information regarding the importance, difficulty, and frequency of various job tasks or activities (component "criticality" weights) and empirical estimates of predictor construct intercorrelations, were used to generate synthetic equations for predicting job-specific and Army-wide job performance. The task and activity judgments were obtained from subject matter experts (noncommissioned officers and officers). Similar to previous results from a Project A validity estimation study (Wing et al., 1985), the estimates were found to have a high level of interrater agreement. Various strategies for weighting (1) the predictors in the component equations and (2) the component equations to form an overall equation were investigated.

A substantial advantage of the SYNVAL project is the capacity to compare the predicted scores generated by the synthetic equations to existing data. First, synthetic prediction equations were developed for the MOS having performance data. The synthetic equations were compared with ordinary least-squares prediction equations for the corresponding MOS, based

on data from Project A validation studies. The data had been corrected for range restriction and criterion unreliability. Validity coefficients for the synthetic equations were found to be slightly lower than the adjusted validity coefficients (adjusted for shrinkage) for the least-squares equations. The differential validity (i.e., the degree to which the validity coefficients for job-specific equations were higher when the equations were applied to their respective job than when applied to other jobs) of the synthetic equations was also somewhat lower than that evidenced by the least-squares equations.

Now that personnel psychologists have provided the estimated linkages between the attributes and Army job components, all that remains is to obtain criticality estimates of each component from subject matter experts for any Army MOS without criteria or for any new MOS. This information can then be combined with the estimated attribute-component relationships to form a synthetic validity equation for the job.

An important characteristic of the synthetic validation approach used in SYNVAL is that synthetic equations could have been developed without any criterion data whatsoever. The presence of actual job performance data, however, allowed validation of the synthetic validity procedure. Although the SYNVAL project demonstrated the synthetic validation technique to approximate optimal (i.e., least-squares) equations closely, some researchers might be wary of deriving equations that rely so heavily on the judgments of psychologists or job incumbents.

A procedure is available that operates directly on the job-analytic information to provide estimated performance equations for jobs without criterion data (although this procedure does require the presence of criterion data for at least a subset of jobs). The procedure is presented by describing its use in the Linkage project (Harris et al., 1991; McCloy et al., 1992), the project that marked the beginning of the enlistment standards portion of the JPM Project. The goal of the Linkage project was to use JPM Project data to investigate the relationship (i.e., linkage) between job performance and enlistment standards. To explore this relationship fully, it was imperative that the method of analysis provide equations that were generalizable to the entire set of military jobs. The synthetic validity approach used in SYNVAL was not selected for the Linkage project because the cost of implementing the approach would have been prohibitive. Specifically, the SYNVAL approach would have necessitated an extensive data collection, because job component information and validity estimates of the predictor/job component linkages were not available for non-Army jobs. In the next section, multilevel regression analysis is proffered as an alternative method for deriving job performance predictions for jobs without criterion data.

Suppose that some new selection measures have been developed for predicting performance, and it is of interest to investigate their predictive validity for several jobs. In this example, we have a criterion (e.g., a score from a hands-on test of job performance) P_ij for person i in job j. We assume that P_ij depends on an individual's aptitude test score (call it A_ij; this could be a set of test scores) and some set of other individual characteristics such as education and time in service (call this O_ij). We further assume that the effects of these independent variables could differ across jobs and that the jobs are a random sample of the total set of jobs. Thus, the model is

where α_j is a job-specific intercept, β_j and γ_j are job-specific slopes, and ε_ij is an error term. This is the general form for the equation linking individual job performance to enlistment standards—the linkage equation.

This model says that α_j, β_j, and γ_j can, in principle, vary across jobs. Multilevel regression allows one to quantify the variation in these parameters and to determine if the variation is statistically significant. The variation is addressed by assuming that the parameters themselves have a stochastic structure. Namely,

This formulation says that the intercept for job j (α_j) has two components: α, the mean of all the α_j's (note the lack of the j subscript), and a_j, a component that can be viewed as the amount by which job j's intercept differs from the average job j's intercept (i.e., differs from α). Note that the model assumes the distributions of a_j, b_j, and c_j are to be normal; their joint distribution is assumed to be multivariate normal. Although a_j, b_j, and c_j are completely determined for any specific job, the multilevel model conceives of these components as random, because the sample of jobs is assumed to be chosen at random. If the jobs are picked at random, these components are likewise random. Thus, coefficients modeled to vary across groups (here, jobs) may be labeled random effects (indeed, multilevel models are sometimes called random effects models), whereas coefficients mod-

eled to remain constant across groups may be labeled fixed effects. The variance components represent the variance of the random effects across jobs. For example, σ²_a is the variance across jobs of the a_j's and therefore of the α_j's, because α is the same for all jobs.

A multilevel regression model was chosen for the linkage equation because the JPM data are multilevel or "nested." Specifically, in the JPM database, individuals are nested within jobs.⁴ Ordinary least-squares (OLS) regression models are inappropriate for multilevel data. To see why this is so, consider a simpler version of equation (2) in which only the intercept (α) is allowed to vary across jobs (i.e., estimate α_j). Thus, the model is

and α_j is modeled by equation (2). Substituting equation (2) into equation (5) results in a residual term of

implying that the residuals from two individuals in the same job are correlated (i.e., individuals within a job share the same error component, a_j). The same situation obtains for the other parameters as well. Therefore, applying the ordinary regression model to these data would result in biased standard errors for the regression parameters (generally biased downward, increasing the chance of a Type I error; see Green, 1990:478 for more details).

A key feature of the random effects (a_j, b_j, c_j) is that a portion of the variation they represent may be systematic. Consider the general form of the linkage equation given by equation (1). The parameters in this equation possess a j subscript, signifying they may vary across jobs. One might surmise that some of the across-job variation in the parameters may be due to job characteristics (e.g., cognitive demands, demand for psychomotor ability). If so, variables assessing those job characteristics (call them M_j) thought to contribute to this variation could be included in the multilevel regression model. Thus equation (2) would become

where π_α is a weight applied to the job characteristic variables M_j, and η_αj is the residual random effect. To the extent that the job characteristic variables were predictive of the parameter variation across jobs, the amount of error in the prediction system would be reduced.

Thus, the linkage equation contains not only individual characteristic information, but also variables that assess various characteristics of the military jobs. Before a discussion of the specific form taken by equation (1) in the Linkage project, the variables constituting it deserve comment.

As noted above, equation (1) gives the general form of the linkage equation. Before discussing the specific form taken by equation (1) in the Linkage project, the variables constituting it deserve comment.

Measures of individual characteristics were obtained from the Services' JPM data files. The measures were: (1) hands-on job performance test scores (percentage correct),⁵ (2) educational attainment (high school diploma graduate or non-high school graduate), (3) experience (total months of service), (4) 10 ASVAB subtest standard scores,⁶ and (5) MOS code. The data were available for 8,464 individuals from 24 jobs studied in the JPM Project. A total of 24 MOS were included in the Linkage project:

The small number of jobs for which performance data were available made reducing the number of predictors in the performance model advisable. Using each of the ASVAB subtest scores as a predictor along with the other measures of individual characteristics and job characteristics would have involved estimating too many parameters, whereas using only the Armed Forces Qualification Test (AFQT) or some other general ability factor might have missed important job differences. The solution to this problem was to use nonoverlapping ASVAB composite scores, thus reducing the number of predictors while retaining as much ability information as possible. The ASVAB composite scores were calculated from a database used to develop the ASVAB-to-Reading Grade Level (RGL) conversion table (Waters et al., 1988) and have been discussed elsewhere (e.g., Campbell, 1986). There are four ASVAB factors:

Quantitative = MK + AR⁸

Speed = NO + CS

Technical = AS + MC + EI

Verbal = PC + WK + GS.

The quantitative and verbal factors form the AFQT score:

To supplement the AFQT, the technical composite was also selected. The speed composite was not used in subsequent modeling efforts because it was not a significant predictor of hands-on performance.

Descriptive statistics were calculated on the individual characteristics for the JPM Project sample (N = 8,464). Means and standard deviations for the hands-on performance (HOPT) test score, the AFQT composite score, the technical composite score (TECH), months of service experience (TIS), and educational attainment¹⁰ (EDUC) are presented in Table 1. Means and standard deviations for AFQT, TECH, and EDUC calculated from the Waters et al. data are in parentheses. Note that the test score distributions for the JPM job incumbents do not differ greatly from the distributions obtained from military recruits. Table 2 contains job-specific means and standard deviations for the individual characteristics for each of the jobs in the JPM Project sample.

Development of the job-level variables for the multilevel model was based on an analysis of civilian jobs. Because job-characteristic data for entry-level military jobs were not available, alternative sources of this type

of information were sought. A readily available source of job-characteristic information is the Dictionary of Occupational Titles (DOT) (U.S. Department of Labor, 1977). The Department of Labor database used to compile the most recent edition of the DOT was obtained from the National Technical Information Service. The database contains the DOT codes for ratings of worker functions and worker traits for approximately 12,000 civilian occupations. Military-civilian job matches were obtained from the Military-Civilian Crosscode Project database (Wright, 1984). Each military job so matched adopted the ratings of its civilian counterpart.

Because the Linkage project was concerned with first-term job performance, only cross-code matches for entry-level jobs were obtained from DoD. The database contained military-civilian equivalents for 965 entry-level military occupations. Once the military-civilian equivalents were obtained, 44 DOT job-characteristic variables describing the civilian jobs were used to characterize the military occupations. These variables represent a variety of job functions and worker traits, including job complexity, training time, aptitude and temperament requirements, physical demands, and environmental conditions.

Job-level composite scores were developed to decrease the number of predictors in the performance equation. To calculate job-level composite scores from the full set of 44 DOT variables, principal components analyses were performed using the 965 military occupations from the cross-code database. An orthogonal component structure was maintained to produce nonoverlapping composites of job characteristics.

A four-factor solution with orthogonal varimax rotation was selected as most appropriate; the four factors accounted for 48 percent of the total variance in the original variables. The first component accounted for 18.5 percent of the variance and consisted of 16 variables that deal mainly with working with things, suggesting that this component reflects the extent to which manual labor is a part of the job. The second component accounted for 15 percent of the total variance and consisted of 10 variables that reflect the cognitive complexity of work. The third component accounted for 8.6 percent of the variance and included 9 variables dealing with unpleasant working conditions . The fourth component accounted for 5.9 percent of the variance and contained 12 items dealing with fine motor control and coordination needed in some jobs.

Using the results of the four-component solution, component scores were obtained for the 925 military jobs having complete data (i.e., observations on all 44 occupational variables). These job-specific component scores were then used as the job-characteristic variables in the multilevel regression model of military job performance.¹¹

For the Linkage project, the general specification given in equation (1) is expanded so that job performance is a function of the characteristics of individuals (AFQT standard score, ASVAB technical composite score, time in service, and education). Job performance, in turn, is operationalized as scores on the job-specific hands-on performance tests developed for each service during the JPM Project. Thus, the prediction equation takes the form:

where P_ij is the hands-on performance test (HOPT) score for person i in job j; A_ij is the AFQT composite score; T_ij is the ASVAB technical composite (TECH) score; E_ij is educational attainment (EDUC); X_ij is time in service (TIS); and α_j, β_j, _j, γ, δ_j and ρ are model parameters. Note that γ has no j subscript because the effect of education was found not to vary across jobs:

The structure of the model parameters for the linkage equation is the following:

where α, β, δ, and are the mean values of the parameters across all jobs (note the lack of the j subscript), the π's are vectors of coefficients constrained to be the same across jobs (i.e., they are ''fixed" coefficients), M_j is a vector of four standardized component scores that describe job characteristics (working with things, cognitive complexity, unpleasant working conditions, and fine motor control) described earlier, and the η's are random variation.¹² (To generalize the model to the universe of first-term military

jobs, the job-level coefficients—the π's—cannot be job-specific.) Because the linkage equation contains both individual- and job-level predictors, it qualifies as a multilevel model, with individuals being level one and jobs being level two.¹³

This structure for the model parameters assumes that some of their variation (except for γ) is due to characteristics of the jobs. The M_j variables represent characteristics of jobs believed to influence an individual's performance. The inclusion of such job-characteristic information is our attempt to generalize from our small sample of jobs (the 24 JPM jobs having hands-on criterion data) to the population of military jobs. To the extent that some portion of the parameter variation is due to job characteristics and the proper job-characteristic variables (M_j's) are included in the multilevel model, the amount of variance in the parameters that is unaccounted for can be reduced, thereby increasing the accuracy of prediction or, equivalently, decreasing the standard error of estimate.

The M_j variables reduce the uncertainty in the job-specific parameters by absorbing some of the variation across jobs that would be part of the random effect if the M_j variables were not in the model. For example, for the job-specific intercept α_j, the term π_αM_j models part of the variation in intercept parameters across jobs that otherwise would be part of the random effect α_j Including the second-level variables should reduce the uncertainty in the estimation of the α_j's. This same logic holds for all other model parameters.

The multilevel model may be approximated by a fixed-effects (i.e., conventional OLS) regression model. Substituting equations (8)–(11) into (7) gives the following:

Multiplying through and collecting terms yields:

where

Thus, a model containing the job-characteristic variables and interactions between the job-characteristic variables and the variables whose effects are

to vary across occupations may be used to estimate the structural parameters (regression coefficients) in the multilevel analysis. The standard errors of the parameter estimates for this model will be biased, however, due to the failure of the fixed-effects regression to adequately model the correlations among errors in the multilevel error structure. The standard errors will typically be smaller than they should be, thereby increasing the probability of a Type I error.

To determine if the simpler fixed-effects approximation adequately characterized the Linkage data, equation (14) was estimated and compared with the multilevel model. The two sets of parameter estimates were sufficiently different to suggest retaining the multilevel model (see Harris et al., 1991, for a detailed description of the formulation of the performance equation).

The linkage equation's parameters were estimated using the VARCL software package for the analysis of multilevel data (Longford, 1988). This program uses maximum likelihood estimation to obtain parameter values for the model. The specific, unstandardized parameter estimates for the linkage equation are given in Table 3, along with their associated standard errors. (Table 1 contains the means and standard deviations of the individual-characteristic variables.) Also included in this table are values giving the square root of the variance (designated σ) of each parameter across jobs (i.e., the random effects remaining after taking into account job-characteristic information) and their respective standard errors (SE_σ).

The standard errors of the fixed-effect parameter estimates (e.g., &#946, ) indicate that all of these values are significant. Thus, the variables in the linkage equation demonstrate statistically reliable predictive relationships with job performance for the total sample of jobs. The SE_σ values indicate that the σ values for the random-effect parameters are also significant, suggesting reliable variation remains in these parameters across the sample of jobs, even after the job-characteristic variables are used to model this variation.

Examination of the standard errors for the job-level parameter estimates (e.g., π_α1, π_δ3) shows how well the four job-characteristic variables model the across-job variation in the parameter values for each variable. Specifically, the factor scores do account for a statistically significant portion of the variation in the TECH parameter but do not do as well for the remaining parameters, the values of the intercept term being of particular note (i.e., π_α1 through π_α4). Combined with the results of the variation in the random effects (the σ values), we conclude that there might be other moderators (i.e., M_j variables) that would better model the variation of the parameters of the primary linkage equation.

Take note that the application of multilevel analysis in the Linkage project is rather atypical. That is, most applications of multilevel regression analysis involve many groups (e.g., 100–200 schools) with relatively few members in each group (e.g., 20–30 students). In the present analyses, there are relatively few groups (i.e., 24 jobs) that contain many members (sample sizes ranging from 83 to 940). Although this complicates estimation of the variance components for the job-level parameters (i.e., the σ's in Table 3), these components are estimated with enough precision to be statistically significant.

The VARCL output also includes a covariance matrix of the random effects. This matrix provides information regarding the degree to which the parameter values across jobs for one variable covary with the parameter values across jobs for another variable. The covariances for the linkage parameters are presented in Table 4, along with the corresponding correlations. As an example of the information provided in the table, the substantial negative correlation between the intercept and TECH indicates that the TECH parameter tends to be smaller in jobs having a higher overall mean performance level.

Although VARCL provides standard errors of the model parameters, no standard error of estimate is printed. One may be calculated, however, by taking the square root of the equation:

where V(•) and Cov(•,•) are the variance and covariance of the random effects, respectively. The terms A, T, E, and X are the mean AFQT, TECH, EDUC, and TIS values for the job under consideration (see Harris et al., 1991). Note that not terms including job-characteristic variables appear in equation (16). This is because the parameters of the job-characteristic variables (the π's) are fixed (the values of the job-characteristic variable are constants for a given job).

The information for this equation is available form Tables 3 and 4. There are two sources of variability in the job-specific parameters. The largest is the variance of the coefficients across jobs (σ²_α, σ²_β, , σ²_δ), the square roots of which are given in Table 3. This source arises because the coefficient α_j is imperfectly estimated by (α + π_αM_j), β_j by (β + π_βM_j), and so on. Residual error variance ( i.e., the η's), conditional on the job-characteristic variables, remains in the parameter estimates. The variance of these errors across the population of jobs is the variance component, σ².

The second (and usually smaller) source of variation is the variance of the estimate of the mean effects (e.g. , ,). These are the standard errors of the parameters given in Table 3. Thus, for the TECH parameter _j,

where is the variance component for the slope _j and SE () is the standard error of (the estimate of the mean of the _j 's). The intercept contributes only its random effect component to the variability of _j, because a shift in the mean of the intercepts does not contribute to variability in the _ij values. Slopes that are constrained to be the same across all jobs (i.e., are a fixed, but not a random, effect) contribute only their [SE(•)]² component. Thus, for the EDUC parameter,

The covariance terms inn equation (16) are the covariances between the conditional (or residual) random effects (the random effects not accounted for by the second-level variables—the job characteristic). These values are given in Table 4. Note that no covariances are given between the random parameters that vary across jobs and the parameters constrained to be the same across jobs (i.e., the fixed parameters γ and ρ). Because the fixed parameters do not vary across jobs, their covariance terms equal zero.

Table 5 contains the means of the predicted performance scores and associated standard errors of estimate for the 24 JPM jobs. The values of the standard error range from 6.80 for Army MOS 64C (motor transport operator) to 9.62 for the Navy Rating ET (electronics technician). In no instance does a 95-percent confidence interval around the predicted scores yield an implausible value (i.e., less than zero or grater than 100).

Much of the variation in prediction is due to scaling difference across jobs (specifically, intercept variation introduced by the differences in mean HOPT scores across jobs). The HOPT scores were not standardized within jobs but rather remain in their original metric. By retaining the original HOPT metric, the mean differences in HOPT scores across jobs remain. Part of the mean differences in performance scores is attributable to differences in job difficulty. Another contributor to the mean difference in HOPT across jobs is differences in test difficulty, with the difficulty of a given hands-on test being determined by comparing that test to other tests that could be constructed for a given job. For example, recall that the Marine Corps JPM sample h as a wider range of experience than the Army. The Marine Corps HOPT for rifleman (the Marine Corps's infantry MOS) contains items that assess performance of second-tour tasks. Clearly, this test would be more difficult for first-term soldiers than a test assessing

performance on tasks that are performed during the first tour. In contrast, the Army's infantry HOPT assesses performance on first-term tasks only. This difference in performance test design is reflected in the mean HOPT scores for Marine Corps rifleman (mean = 52.62, SD = 8.96) and Army infantryman (mean = 69.92, SD = 7.42) given in Table 2.

Job difficulty variance is desirable to retain, but test difficulty variance is not. Although standardizing within job would remove the unwanted test difficulty differences, the desired job differences would also be lost. For the approach of leaving the scores in their original metric to be tenable, one must assume that the variance due to test difficulty across jobs is uncorrelated with the individual- and job-characteristic variables. Considering the job-characteristic variables, there is no reason to believe that the characteristics of the job and the difficulty of the test are related. It is possible that more difficult jobs have more difficult tests. If this is due to the content of the job, however, then this reflects job difficulty, not test difficulty, and is not a

concern. Thus, for any given job, tests may be relatively easy or relatively difficult. For example, there is no reason to suspect that tests for jobs requiring performance in difficult working conditions would be easier or more difficult than tests for other jobs. Test comparisons as difficult or easy must be made within a job. Comparisons across jobs are confounded by job difficulty.

The reasoning for a negligible correlation between test difficulty and individual-characteristic variables runs parallel to that for job characteristics. Although intelligent people are placed in difficult jobs, this relationship is based on job content, not test difficulty. There is no reason to believe that people with certain individual characteristics are disproportionately placed into jobs that have very difficult or very easy tests–again, difficult or easy represents a relative comparison within a job rather than across jobs.

As mentioned above, the linkage to be made in the Linkage project was the relationship between measures of recruit quality (i.e., enlistment standards) and job performance. The multilevel performance equation given in equation (7) characterizes this relationship for the 24 JPM occupations. In addition, this prediction model is the primary linkage equation—it is the progenitor of all job-specific linkage equations. That is, this model's parameter estimates (given in Table 3) are used to calculate the parameters for equations that allow job-specific performance predictions.

The principal advantage of the primary linkage equation is that it allows performance predictions for jobs having no criterion data. Using ordinary regression, performance scores can be estimated for individuals without criterion data by weighting their predictor information by the appropriate regression coefficients. Performance data, however, are needed for some individuals in that job before the job-specific equation may be estimated. By including job-characteristic variables in our multilevel model, job-specific parameters can be derived for any job having job-characteristic data. These parameters are functions of the job-characteristic variables and, together with the fixed effects of EDUC and the interaction between TECH and TIS, constitute job-specific linkage equations.

Returning to Table 3, the value associated with a Greek letter represents the mean effect of the variable across jobs (e.g., β = .074). The parameters subscripted from 1 to 4 (e.g., π_β1) signify the values in the π vector that are applied to the four component scores, respectively. For AFQT, these values are -.030, .001, -.020, and -.036. Substituting these values into equations (8) through (11) allows the estimation of job-specific parameters. Equations (8) through (11) also demonstrate that these esti-

mated job-specific parameter estimates are deviations from the mean parameter estimate—the degree of deviation being a function of the job's factor scores. (Note that γ and ρ, fixed across jobs, do not have any corresponding subscripted values.) Consider the Army MOS 11B (infantryman). The four factor scores for this MOS are -0.68, -2.41, 2.33, and 0.18. Substituting these M_j values and the multilevel parameter estimates just given into equation (9) yields the AFQT parameter (β_j) for predicting job performance for infantrymen:

Substituting into equations (8), (10), and (11), the same procedure yields the remaining parameters for MOS 11B:

The coefficients for the job-specific linkage equations for the 24 JPM jobs used in the Linkage project are given in Table 6.

The same procedure affords job-specific parameters for jobs without criterion data. For example, continuing with the AFQT parameter, the following job-specific value is obtained for the Army MOS combat engineer (12B) using its factor scores of -.51, -3.09, 1.90, and -.97:

and for the other parameters,

Note that the value for β and the four π_β values remain constant in the β_j equations for both MOS; the equations differ only in the M_j values.

Performance equations may also be generated for groups of jobs. For example, jobs were grouped into 9 of the 10 DoD occupation codes (see

Table 7) in the Linkage project.¹⁴ Scores on the four factors for the nine job groups were obtained by calculating the weighted (by sample size) mean of the factor scores across the jobs in each of the nine occupational codes. As above, the weighted mean factor scores would be inserted into the primary linkage equation to generate performance equations for each of the occupational codes.

The model also may be amended to include additional or different individual and job characteristics. All that is required is to reestimate the primary linkage equation with the new variables in the model so that new parameter values may be obtained. The procedure just described still applies.

The capacity to generate prediction equations for jobs without criterion data (given that job-characteristic data are available) is a very attractive feature of the multilevel primary linkage equation. Nevertheless, at least two issues should be addressed with regard to the ability of the primary equation to generate job-specific linkage equations that yield quality predictions for jobs without criterion data.

First, the primary equation was estimated on a sample of only 24 jobs. Although these jobs have various desirable qualities—for example, they are high-density jobs, and they are representative in the sense that they span important job groups in the Services (e.g., mechanical, administrative, and combat occupations)—there are still several limitations inherent in them for the Linkage project. For example, they do not span the job-characteristic space defined by the four component scores (e.g., most of the jobs are low m cognitive complexity), and 24 jobs is not a large number of cases for estimating across-job variability. As a result, there is some question about the degree to which the parameters from the primary linkage equation and the corresponding job-specific equations would change if any one of the 24 jobs were deleted from the sample.

Second, quite apart from the capability simply to generate job-specific linkage equations for jobs devoid of criterion data and the independence of those equations from the 24 jobs included in the estimation sample is the issue of how well those generated equations actually predict performance in the out-of-sample jobs. Such information is crucial for evaluating the validity of the performance equation. Hence, there are two primary issues to be addressed: (1) the sensitivity of the primary linkage equation to the jobs m the estimation sample and (2) the validity of the job-specific linkage equations that are generated by the primary linkage equation.

Parameter sensitivity results were reported by McCloy et al. (1992). They estimated the linkage equation 24 times, once as each job in the sample was removed from consideration. Distributions of the parameters from the resulting 23-job equations were obtained, and ratios of predicted performance scores were calculated both across AFQT categories within jobs and within AFQT category across jobs. They found that (1) the parameters were not unduly influenced by the presence of any particular JPM job in the sample of 24 and (2) the predicted performance scores evidenced reasonable stability, suggesting that the performance equations yield consistent results and are mostly unaffected by the presence or absence of specific jobs in the estimation sample.

The second question refers to the validity of the job-specific linkage equations generated by the primary linkage equation. The results of the validity analyses are given here. The capacity to generate predicted performance scores (via job-specific linkage equations) for individuals in jobs for which no criterion data are available begs the question of how well the job-specific linkage equations predict performance for various jobs—in particular, any out-of-sample jobs without criterion data.

Answering this question requires jobs that have criterion data but were not part of the estimation sample for the primary linkage equation. There are essentially two ways such a situation could arise: (1) manufacture such a situation out of the extant sample by using a holdout procedure or (2) obtain relevant data on one or more new jobs after estimating the original primary linkage equation. Both conditions obtained in the present analyses.

The primary linkage equation can be used to generate a job-specific equation for any job having job-characteristic data, whether it appears in the estimation sample or not. The quality of the predictions from any of these equations is of interest, but perhaps the most stringent test of the linkage methodology lies in the prediction of performance scores for out-of-sample jobs. To investigate the validity of the job-specific linkage equations, two types of analyses were performed.

First, each of the 24 JPM jobs was held out of the sample and a ''reduced" primary equation was estimated on the remaining 23 jobs. This process resulted in 24 reduced equations. These reduced equations were used to generate a job-specific equation for their corresponding holdout (i.e,, out-of-sample) job. The existence of criterion data for each holdout job permitted the observed performance scores for each job to be correlated with the performance scores predicted by the corresponding job-specific linkage equation.

Second, job-specific linkage equations were generated from the 24-job

primary linkage equation for two Navy ratings[ electrician's mate (EM) and gasoline turbine mechanic (GSM)] and five Marine Corps jobs [organizational automotive mechanic (3521), helicopter mechanic CH-46 (6112), helicopter mechanic CH-53 (6113), helicopter mechanic U/AH-1 (6114), and helicopter mechanic CH-53E (6115)] that were not part of the original estimation sample. These equations yielded predicted performance scores for the individuals in the additional Navy jobs. As in the holdout analyses, the correlation between the observed and predicted performance scores wa obtained.

When conducting cross-validation, one typically splits the total sample into two random subsamples, developing a prediction equation on the first sample and applying that equation to the second sample. But this method underestimates the R² expected if the regression equation were developed using the entire sample and then applied to the population, because the best set of regression weights (i.e., the full-sample weights) is not used (Campbell, 1990). The present analyses do not match typical cross-validation procedures exactly: job-specific equations were (1) generated from a primary linkage equation that in turn was estimated from a sample of 23 (holdout analyses) or 24 (new Navy and Marine Corps specialties) jobs and (2) applied to their respective jobs that were not part of the estimation sample. That is, the validation sample was a job that was not part of the original sample of 23 (holdout analyses or 24 (new Navy and Marine Corps specialties) jobs, rather tan a random subsample of the original test of the ability of the job-specific linkage equations to provide accurate predictions of actual performance scores for out-of-sample jobs. Such information is vital because these situations reproduce the scenario in which the primary linkage equation would be implemented by manpower planners.

The results of the analyses are presented in Table 8, which contains (1) the sample size for each job (N), (2) the squared multiple correlation for the least-squares job-specific regression equations (R²_OLS), (3) the squared multiple correlation for the job-specific linkage equation generated from the reduced 23-job "primary" equation (R²_cv) (i.e., the correlation between the predicted performance scores taken from the job-specific linkage equation and the actual observed performance scores), (4) the difference between the values of R²_OLS and R²_cv, and (5) R²_OLS values adjusted using various shrinkage formulae (R²_adj).

Two feature of the first two columns of R² values are of note: (1) the values are quite variable, ranging from .065 to .508 for R²_OLS and .031 to .461 for R²cv, and (2) R²_OLS >R²_cv. The latter finding is expected, given that