Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 195
Appendix A Dissent Juan I. Sanchez and David H. Autor ISSuES OF CONTENTION We believe the report could have provided more expedited solutions on three issues: (1) deletion of scales, (2) weaknesses of descriptors and scales, and (3) whether information on some content domains could be eliminated. In our opinion, none of the recommendations in the report acknowledges that these concerns can be at least partly addressed through immediate modification of the O*NET measurement model without diminishing the utility of the overall database; we believe that such modifications would be supported by the extant O*NET data and do not need to await further research. ARGuMENTS IN SuPPORT OF THE MINORITY vIEW We believe the latest O*NET 14.0 database provides empirical evidence of redundancy of descriptors between scales and also within and across do- mains. First, the data strongly suggest that the importance and level scales are highly redundant. Indeed, their bivariate correlations computed across the various items in each domain and across the 832 occupations included in the 14.0 database are as follows: Domain Importance × Level Pearson Correlation Generalized Work Activities .92** Abilities .97** Skills .95** Interests .97** Knowledge .97** ** = p < .01 

OCR for page 195
 A DATABASE FOR A CHANGING ECONOMY These correlations suggest that the ranking of items does not change for practical purposes regardless of which scale, importance or level, is em- ployed. Although correlations between the importance and the level scale might be smaller if individual-level data were used, such data are not made available to the public and, therefore, O*NET users rely solely on the ag- gregate ratings (i.e., average ratings across approximately 25 respondents) extracted from the O*NET 14.0 database for these analyses. An analysis of variance components using the same database supported the finding that the type of scale, namely level or importance, has practically negligible effects (3 percent or less variance) on the ratings: Percentage of Variance Accounted for by Descriptor Scale (i.e., importance versus level) Generalized Work Activities .50 Abilities 3.00 Skills 1.54 Interests 1.32 Knowledge 1.31 Between the two scales, the questionable and often disconcerting be- havioral anchors (see Chapter 4) placed at the various points of the level scale strengthen the case for its elimination. The elimination of the level scale will cut more than 150 items from the surveys, thereby cutting survey costs and possibly increasing response rates. Unlike the level scale, other scales (e.g., frequency or duration) may provide independent, valuable, and incremental occupational information above and beyond the information provided by the importance scale, and their potential inclusion warrants further cost-benefit analysis. We believe there are compelling reasons for at least the temporary suspension of the procedure currently employed to measure the ability and the skill domains, which are rated by trained analysts on the basis of a methodically assembled yet paper-based description of the job. First, these analysts do not have a chance to interview or observe actual occupational incumbents to help them formulate their ratings. The evidence indicating adequate interrater reliability among analysts suggests that they consistently rate abilities and skills, but interrater agreement does not imply validity. Second, a factor analysis of the ability ratings in the 14.0 database confirms the presence of substantial data redundancy among the ratings of the 52 abilities included in this particular domain. That is, a single factor accounts for 43 percent of the variance in ability ratings. There is also quite a bit of empirical redundancy between the two domains currently populated by analyst ratings, namely the ability and skill domains, on one hand, and

OCR for page 195
 APPENDIX A the domains of generalized work activities and work context populated by incumbent ratings, on the other. That is, a statistical regression of any ability or any skill rating on the set of generalized work activities and work context ratings reveals statistically reliable multiple R correlations ranging from .65 to .98, p < .01, even after correcting for shrinkage. Therefore, analyst-based ability and skill ratings can be reliably predicted using simple linear combinations of incumbent-based ratings in other domains. Eliminating analyst-based ratings of abilities and skills would cut al- most 90 additional items from the surveys, hence lowering data collection costs and possibly increasing response rates. However, current uses of skill and ability ratings would not need to be disrupted until a better measure- ment procedure to estimate these domains is developed. Indeed, current users could resort to the mechanical estimates based on incumbent rat- ings from other domains, which provide practically equivalent values (as mentioned, Pearson R correlations between analyst ratings and mechanical estimates range from .65 to .98). In future waves of data collection, these estimates could be automatically computed and added to the O*NET da- tabase in lieu of the analyst-based ratings.

OCR for page 195