Development of the Rating Scales

The next step in the field test of the content model was to develop rating scales that individuals could use to assess the extent to which each descriptor was required for each occupational unit.


For each of the 52 Abilities descriptors included in the taxonomy, the scale used to rate the importance for a given occupation ranged from (1) not important to (5) extremely important. The scale used to rate the level of each Ability is defined by “behavioral anchors”—brief descriptions of specific work behaviors—provided to assist individuals in making ratings. To generate the anchors, the development team asked panels of subject-matter experts to suggest multiple examples to illustrate different levels of each Ability. Another, independent set of panels of subject-matter experts was asked, for each Ability descriptor, to place the level represented by each anchor on a quantitative scale. For each Ability descriptor, the development team chose behavioral anchors that covered selected points on the scale, were scaled with high agreement by the subject-matter experts, and were also judged to be relevant for rating occupational (e.g., not educational) requirements. The selected anchors were then included in the final prototype rating scale. Although behavioral anchors for many Abilities descriptors had previously been developed over the course of the Fleishman job analysis research program (Fleishman, 1992), the anchors were apparently rescaled for the O*NET application (Peterson et al., 1999, p. 185).

The current rating scales used by trained occupational analysts to assess the importance and level of Abilities descriptors are largely unchanged from those developed in the prototype development project. These scales present problems for users of the resulting data. For example, Figure 4-1 depicts the current scale used to rate the level and importance of the descriptor Ability “static strength.” One issue that becomes apparent in this example emerges from the question posed to the raters. The question, “What level of static strength is needed to perform your current job?” implies a dichotomy of the form “can perform the job” versus “cannot perform the job.” The meaning of perform is unclear. All research on job performance assessment yields a continuous distribution of performance differences across job holders. Does perform mean perform at a minimally acceptable level, an average level, or a very high level? The ability requirements may be different for performing at the minimum level versus performing at a high level, and the performance referent should be made clear to raters.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement