Appendix A
Prototype Evaluation

An initial set of studies was conducted with the O*NET™ instruments. Data were collected from incumbents in about 30 occupations out of an initial targeted sample of 70 occupations. In addition, occupational analysts rated 1,122 occupations on a subset of O*NET™ descriptors using information from the Dictionary of Occupational Titles to guide their ratings.

Return rates were disappointing for the questionnaires mailed to incumbents. Though 60 percent of all incumbents who were given questionnaires completed them, many establishments that initially agreed to participate failed to do so (only 27 percent actually participated). A number of factors may have contributed to the disappointing participation rates, suggesting that future data collections would benefit from shorter questionnaires, less burden on each establishment, monetary or other incentives for points of contact within each establishment and for incumbents, and greater use of organizations willing to volunteer their services or showing interest in applying O*NET™ results. However, results for the reliability of the questionnaires, as measured by interrater reliability coefficients, were acceptable. In addition, the correlations between the mean ratings of incumbents and the mean ratings of analysts were sufficiently high to warrant the use



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement