These correlations suggest that the ranking of items does not change for practical purposes regardless of which scale, importance or level, is employed. Although correlations between the importance and the level scale might be smaller if individual-level data were used, such data are not made available to the public and, therefore, O*NET users rely solely on the aggregate ratings (i.e., average ratings across approximately 25 respondents) extracted from the O*NET 14.0 database for these analyses.

An analysis of variance components using the same database supported the finding that the type of scale, namely level or importance, has practically negligible effects (3 percent or less variance) on the ratings:

Descriptor

Percentage of Variance Accounted for by Scale (i.e., importance versus level)

Generalized Work Activities

.50

Abilities

3.00

Skills

1.54

Interests

1.32

Knowledge

1.31

Between the two scales, the questionable and often disconcerting behavioral anchors (see Chapter 4) placed at the various points of the level scale strengthen the case for its elimination. The elimination of the level scale will cut more than 150 items from the surveys, thereby cutting survey costs and possibly increasing response rates. Unlike the level scale, other scales (e.g., frequency or duration) may provide independent, valuable, and incremental occupational information above and beyond the information provided by the importance scale, and their potential inclusion warrants further cost-benefit analysis.

We believe there are compelling reasons for at least the temporary suspension of the procedure currently employed to measure the ability and the skill domains, which are rated by trained analysts on the basis of a methodically assembled yet paper-based description of the job. First, these analysts do not have a chance to interview or observe actual occupational incumbents to help them formulate their ratings. The evidence indicating adequate interrater reliability among analysts suggests that they consistently rate abilities and skills, but interrater agreement does not imply validity. Second, a factor analysis of the ability ratings in the 14.0 database confirms the presence of substantial data redundancy among the ratings of the 52 abilities included in this particular domain. That is, a single factor accounts for 43 percent of the variance in ability ratings. There is also quite a bit of empirical redundancy between the two domains currently populated by analyst ratings, namely the ability and skill domains, on one hand, and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement