The Content Model
The content model is the basic building block of the O*NET system and refers to the prescribed set of variables used to describe occupations (see Figure 1-1). Each of these variables—a particular aptitude, skill, or education requirement—is referred to as a descriptor. All uses of O*NET rely on descriptions of occupations using the descriptors specified by the content model. If important features of occupations are not included in the content model, or irrelevant features are included, or the descriptors that are included are flawed, then the applications of O*NET are suboptimal. Simply put, the importance of the validity, completeness, and usability of the content model cannot be overemphasized.
This chapter discusses the developmental history of the O*NET content model and the research literature on which the model is based. Following a brief introduction, the first section focuses on the content model of the Dictionary of Occupational Titles (DOT), the predecessor to O*NET. The second section discusses the content model proposed by the Advisory Panel for the Dictionary of Occupational Titles (APDOT) for a new and different DOT; this ultimately became O*NET. The third section discusses the O*NET prototype content model, focusing especially on the development of descriptors, many of which are included in the current O*NET content model. The fourth section discusses the field test of the prototype and development of the current content model. The final section presents the panel’s conclusions and recommendations related to the content model.
THE CONTENT MODEL AS A TAXONOMY
The content model is intended to be a taxonomy of occupational descriptors. An important first step in developing a taxonomy is to define the unit of analysis. In O*NET, this is the occupation, rather than the job or position. An occupation is broader than a specific job or specific position, and it is not idiosyncratic to a particular organization, industry, or setting. A particular occupation could include several jobs if the general responsibilities, activities, and requirements for the various jobs are substantially similar. For example, the occupation “commercial aircraft pilot” could include many different jobs as a function of type of organization, type of aircraft, and business function (see Chapter 3 for further discussion).
With occupations as the unit of analysis and the characteristics of these occupations, such as the physical and cognitive abilities they require of workers, to be included in a taxonomy of descriptors, a number of other important questions become relevant:
How general or specific will the descriptors of occupational requirements be? For example, when considering how to develop a taxonomy of the abilities required by occupations, previous research provides taxonomies that define abilities in terms of two groups of factors (clusters of more specific abilities), four groups of factors, eight groups of factors, or some greater number (see Carroll, 1993).
Given a particular level of generality/specificity, should the set of descriptors of a particular occupational requirement (e.g., knowledge, ability) be a representative sample of all possible descriptors of that requirement, or should it represent the entire universe of descriptors? Previous research-based taxonomies of human abilities (Carroll, 1993) are meant to describe the full range of human abilities, rather than a representative sample. Is this exhaustive approach to be applied when developing descriptors for all of the occupational requirements?
Should each descriptor of occupational requirements be applicable to every occupation (i.e., unit of analysis)? To return to the example of abilities, previous taxonomies have been designed to include descriptors that can be measured across all individuals. They do not include any descriptors of abilities that are applicable only to men or only to women.
Is the taxonomy to include genuine taxons, such as those in biology? In biology, a taxon is defined as a group of organisms with common characteristics, in varying degrees of distinction, such as a phylum, order, family, genus, or species. Common physical
characteristics among the organisms allow clear separation between groups and subgroups (Meehl and Golden, 1982). In the social sciences, where there are rarely such distinguishing variables, a taxonomy must live with classification variables that are continuous and errors of classification that cannot be reduced to zero. The boundaries between different groups and subgroups of similar occupational characteristics are always blurred, and the clarity of naming and defining different occupational characteristics must always be a function of the rules for minimizing the errors of classification given the alternative ways of describing skills.
Can the taxonomy be designed to serve a wide range of purposes among diverse users?
The developers of the O*NET content model addressed all these issues, if not directly, then by default. There have been four major milestones in the development of today’s content model. The first was the final edition of DOT, which incorporated a content model (U.S. Department of Labor, 1991). Second, in 1993, APDOT recommended a set of specifications for a content model to be included in a new and different DOT, which they referred to as the “new DOT” (Advisory Panel for the Dictionary of Occupational Titles, 1993). Third, following the advisory panel’s recommendations, the U.S. Department of Labor (DOL) funded the O*NET development project, which produced a prototype O*NET content model in 1997 (Peterson et al., 1997). Fourth, the prototype was revised in 1998-1999, leading to the current content model. Since 2000, the National Center for O*NET Development (the O*NET Center) has collected data related to the descriptors in this content model.
THE DICTIONARY OF OCCUPATIONAL TITLES CONTENT MODEL
The 1991 revision of the DOT contains descriptions of over 12,000 occupational titles, referred to as DOT codes. All information was obtained by trained occupational analysts who visited each work site, interviewed job incumbents, and observed them at work. The analysts were told to follow standardized procedures when observing and interviewing job incumbents and creating descriptions using a common format or content model. The 1991 volume included, for each DOT code:
A written description. The analysts followed a detailed standard protocol to describe the specific tasks, duties, responsibilities, etc., carried out by the incumbent. These occupation-specific task descriptions led to the criticism that the DOT lacked a common
language that could be used to describe a variety of different jobs (National Research Council, 1980). However, the current O*NET content model includes occupation-specific tasks as well as Detailed Work Activities (DWA) that are derived from multiple analyses of task information and are somewhat similar to the task descriptions similar to those included in the DOT.
Aptitude ratings. Following a prescribed procedure, the analysts rated each occupational title on the level of each of nine aptitudes that were derived from the nine constructs measured by the General Aptitude Test Battery (GATB) (Dvorak, 1947). The GATB included four cognitive ability subtests, two perceptual speed subtests, and three psychomotor subtests.
Temperament requirements. A temperament requirement was defined as an occupational situation to which a job incumbent must adjust. The analysts were provided with 10 situations and asked to identify the 2 most critical for each occupational title. They were not asked to equate these situational requirements with established individual personality traits.
Interests. The domain of work interests was represented by 10 interest factors configured as 5 bipolar preferences. The factors were derived from factor analyses of data from the Kuder Preference Record and Strong Vocational Interest Blank by Cottle (1950). These factors predate the Holland typology of work interests that is currently widely used in career guidance (i.e., Holland and Gottfredson, 1976). The analysts designated the two factors that were the most critical requirements of a particular occupational title and were not asked to rate the importance or level of the factors.
Physical demands. The analysts also rated each occupational title in terms of six physical demands. They rated the level of strength required by the job using a five-point scale ranging from sedentary to light, medium, heavy, or very heavy. The other five physical demands: (1) climbing/balancing, (2) stooping/crawling, (3) reaching/handling, (4) talking/hearing, and (5) seeing were designated as present or absent.
Job environment. The analysts indicated the presence or absence of extreme cold, extreme heat, dampness and/or humidity, noise and/or vibration, and whether the occupational title was performed primarily indoors, outdoors, or both.
Specific vocational preparation. The job analysts rated the level of Specific Vocational Preparation required by the occupational title, independent of general education, using a 9-point scale
ranging from 1 (receiving a short demonstration only) to 9 (over 10 years of specific preparation).
General educational development. Based on extensive specifications and rater training, the job analysts rated the level of reasoning, mathematics, and language skill required by the occupational title.
Complexity. The analysts rated the occupational title in terms of the involvement of the job tasks with data and information, interactions with people, and usage of things (such as equipment, tools, and vehicles).
In sum, DOT was based on an extensive content model that included a broad spectrum of descriptors that were used to describe occupational titles for various purposes. With the exception of the task information in the written description, all of the descriptors were cross-functional and could be used to describe all occupational titles. These descriptors constituted a common language, and a number of them have their analogs in the current O*NET content model.
THE APDOT CONTENT MODEL
The secretary of labor convened APDOT in 1990, charging it to recommend strategies for collecting, analyzing, and disseminating occupational information.
The advisory panel proposed a new model that specified the most important types of information about occupations and workers that should be included in what it called the “new DOT,” which evolved to become O*NET (Advisory Panel for the Dictionary of Occupational Titles, 1993). At the most general level, this content model organized information into four broad domains that are similar to the most general domains in O*NET: (1) worker attributes, (2) work context, (3) labor market context, and (4) work content and outcomes (see Figure 1-1). It also included more specific domains of information, such as Occupation-Specific Knowledge and Generalized Work Activities (GWAs). The advisory panel envisioned that information related to these domains would be incorporated into a flexible electronic database, rather than being published as a printed book.
The domains and occupational descriptors included in this content model were based on extensive research, reviews of current practice, and expert testimony. However, the advisory panel did not discuss in any detail why each domain was critical, nor did it outline the specific uses of each domain (Advisory Panel for the Dictionary of Occupational Titles, 1993). Consequently, the APDOT content model is open to the criticism that it simply includes everything that occupational analysis researchers, practi-
tioners, and users of occupational data were discussing at that time, albeit in meaningful and organized ways. Overall, in comparison to the previous DOT content model, the APDOT content model was more comprehensive, systematically structured, and reflective of what had been learned about occupational analysis between 1930 and 1990. Of course, the advisory panel did not engage in the formidable data collection and measurement issues presented by implementation of the model. That was the task of the O*NET prototype development project.
Although the advisory panel criticized the DOT codes as being too job-specific, its content model included Occupation-Specific Knowledge, Occupation-Specific Skills, and Duties/Tasks Performed (Advisory Panel for the Dictionary of Occupational Titles, 1993). It appears that the advisory panel wanted to include both cross-occupation information and occupation-specific information.
The advisory panel proposed (p. 15) that the single most important capability of the new occupational information system would be to use multiple approaches to identify transferable skills. Because the advisory panel viewed several different groups of descriptors as skills (e.g., tasks, work activities, aptitudes), it created the potential for future confusion concerning the specifications for the skills taxonomy in the O*NET content model. The advisory panel made three other recommendations that informed the panel’s review of the current O*NET content model and data collection methods (Advisory Panel for the Dictionary of Occupational Titles, 1993):
The content model should describe occupational characteristics in such a way that any person can evaluate his or her capability to perform in an occupation. Consequently, difficult to understand technical terms should be avoided.
The Department of Labor should commit to “an ongoing research and development agenda” and should “recruit, train, and maintain a core staff of methodologically sophisticated professionals to manage the DOT program” (1993, p. 7).
The content model itself should be subjected to continuous evaluation and enhancements, based on future developments in research and practice. In no sense should it be cast in concrete or viewed as unchangeable.
THE O*NET PROTOTYPE CONTENT MODEL
To carry out the advisory panel’s recommendation to develop a “new DOT,” the Employment and Training Administration contracted with a consortium of consulting firms with expertise in occupational analysis as
well as survey design and administration to conduct the necessary research and development. The project to develop the prototype for the new occupational information system was completed in 1996, and the research and development process has been described in several publications (Peterson et al., 1997, 1999, 2001). During the course of the project, the term “new DOT” was replaced by the Occupational Information Network (O*NET) in order to sever any direct linkage to the DOT and reflect the move to an electronic database.
As envisioned by the advisory panel, the goal of the O*NET project was to develop a prototype of a new national occupational information content model and offer evidence substantiating “proof of concept” for such a model. The advisory panel expected that the prototype would then undergo additional development and revision before becoming the basis for the principal data collections.
The prototype project team attempted to develop a content model that followed the advisory panel’s recommendations, corrected the DOT’s deficiencies, reflected new research and practice in occupational analysis, and incorporated the capability to adapt to a changing labor market and workforce. To advance these multiple goals, the specifications for the content taxonomies would be crucial.
The resulting prototype content model adopted the overarching framework of the APDOT content model, which divided all domains of occupational descriptors into three groups describing (1) the work itself, (2) the “worker” (i.e., job applicant or job incumbent), or (3) the context in which work takes place. Curiously, neither the advisory panel nor the prototype project team explains why these three facets of occupations should be described, except to say that the database should be a complete description of occupations and that multiple windows (i.e., domains of information) are necessary to accommodate different user needs (Peterson et al., 1999, p. 13). Compared with the APDOT content model, the O*NET prototype gave less attention to occupation-specific descriptors and concentrated to a greater extent on so-called cross-occupation descriptors, or variables that could be measured meaningfully in all occupations and along which occupations would be expected to vary. As discussed later in this chapter, it is not easy to distinguish between cross-occupation and occupation-specific descriptors.
Another objective for the O*NET prototype was to develop taxonomies of descriptor variables in each domain that are hierarchical in nature. The goal from the outset was to describe occupations at varying levels of generalizability or specificity in each domain. Consequently, the descriptors were to be nested in hierarchical levels. For example, there are 17 specific descriptors in the Work Styles taxonomy, which are in turn nested in 7 more general styles (see Appendix B). This approach implies that the
database would include “score profiles” for each occupation at each level of the hierarchy.
Because many of the taxonomies of descriptors created in the O*NET prototype project are included in the current content model, it is important to examine their development and research base. The prototype project proceeded by constituting a separate development team for each of the content domains, reflecting the goal of providing multiple windows, or domains of information, to meet the needs of different users. The teams’ results are described briefly below.
The current content model incorporates the taxonomy of Abilities descriptors developed in the prototype. The prototype team’s (and current) definition of ability is consistent with the very large research literature investigating the nature of human abilities (e.g., Carroll, 1993). That is, an Ability is a relatively enduring attribute that reflects an individual’s capability for performing a particular category of tasks (e.g., verbal, mathematical, physical, motor, psychomotor, sensory). This underlying capability for performing a designated category of tasks remains relatively stable over much of the individual’s life span, with certain physical or motor abilities perhaps being exceptions.
The literature documents literally hundreds, if not thousands, of well-researched standardized tests of various human abilities (e.g., Geisinger et al., 2007), and investigations of the empirically estimated covariances among sets of such tests have produced considerable evidence for hierarchical taxonomic structures of human abilities (Carroll, 1993). In light of O*NET’s objective of producing the most comprehensive occupationally relevant taxonomy, the development team incorporated the Fleishman and Reilly (1992) taxonomy of abilities to create the O*NET Abilities domain. There are 52 Abilities at the most specific level, 15 more general Abilities at the next level, and 4 Abilities at the most general level (Fleishman, Costanza, and Marshall-Mies, 1999). The abilities at one level are fully nested in the next higher level (see Appendix B).
The process of developing and administering surveys to gather information related to these abilities as occupational requirements has proven complex (see Chapter 4). The 52 Ability descriptors were used to describe occupations by asking raters (job incumbents or occupational analysts) to rate both the required level of the ability and the importance of the ability for being able “to perform the job.” Neither the metrics for the scales nor the frame of reference is without criticism and could benefit from additional research and development. The same issues exist for the other descriptor domains as well. An additional difficulty is that the descriptors themselves
are defined only very briefly, and considerable expertise in the psychology of individual differences is required to understand some of them.
The current O*NET content model incorporates the taxonomy of Work Styles developed by the prototype team (see Appendix B). Work Styles refer to the personality or dispositional requirements of occupations. Similar to Abilities, Work Styles are seen as relatively stable attributes of individuals that affect their ability to perform different types of occupations. Work Styles are included in both the prototype and current O*NET content models because decades of previous research have shown them to be important determinants of individual differences in occupational performance, particularly components of performance having to do with teamwork, peer leadership, supervision, and management (Borman, Kubisiak, and Schneider, 1999). The term “work styles” was used, rather than “personality,” because of the intent to emphasize personal characteristics that are occupationally related.
The Work Styles taxonomy was developed by a careful review of two different literatures. The first consists of research focused on developing taxonomies of personality dimensions and facets, such as the Big Five (Costa and McCrae, 1992). The second literature examined research that reflects the validity of personality characteristics for predicting different aspects of job performance (e.g., Barrick and Mount, 1991; Hough, 1992). The research team focusing on Work Styles (Borman, Kubisiak, and Schneider, 1999) synthesized the results of these two bodies of research and suggested a taxonomy of Work Styles incorporating 17 factors fully nested in 7 higher order factors. The taxonomy is shown in Appendix B. The Work Styles domain in O*NET is analogous to the Personal Qualities domain in the APDOT content model, which was to include information on an individual’s characteristic, habitual, or typical manner of thinking, feeling, behaving, or responding as it relates to work (e.g., sociability, integrity).
The Work Styles taxonomy seems well grounded in previous research. However, identifying the most appropriate raters for what are essentially the personality requirements of occupations is perhaps more difficult to accomplish (see Chapter 4). In addition, the question remains about whether the judgments about the level of each work style that is required for an occupation should be with reference to minimal, average, or high performance. There is also recent evidence that for some occupations the relationships of certain personality factors to performance may be curvilinear (e.g., Benson and Campbell, 2007). The O*NET prototype specifications seem to imply only linear relationships, or at least monotonically increasing
predicted performance as a function of increasing scores on the personality factor.
The current O*NET content model uses the taxonomy of Occupational Interests adopted in the prototype content model (Sager, 1997; Sager et al., 1999). The prototype development team adopted Holland’s typology of six types of occupational interests: realistic, investigative, artistic, social, enterprising, and conventional (Holland and Gottfredson, 1976), or RIASEC. The typology is based on a long record of vocational interest research going back over three decades, and its construct validity is really not in doubt. However, unlike the other taxonomies included in the prototype model, the descriptors are not hierarchical.
The taxonomy of descriptors of Work Values in the current O*NET content model is nearly identical to that of the prototype content model. While interests are defined as the pattern of individual likes and dislikes for a large set of different activities, school subjects, and occupations, Work Values refer to an individual’s evaluation of the importance of certain characteristics of the work environment for determining their job satisfaction (Sager, 1999). After a consideration of the research literature on the assessment of work values, the prototype team adopted a taxonomy developed by the Minnesota Studies in Work Adjustment (Dawis and Lofquist, 1984). In that model, the individual’s self-rated importance for each value is assessed with the Minnesota Importance Questionnaire and the degree to which the job or occupation provides the opportunity to satisfy such values is rated by supervisors/managers using the Minnesota Job Description Questionnaire.
The prototype Work Values taxonomy incorporated two levels, with 21 descriptors of Work Values grouped into 6 higher order factors. The current O*NET content model changes the names of a few of the higher order factors, but the 21 Work Values are unchanged (see Appendix B).
The taxonomy of Knowledge descriptors in the current O*NET content model is unchanged from the prototype content model. The prototype development team defined occupation-relevant Knowledge as a collection of discrete but interrelated facts and information about a particular domain having to do with performance in an occupation. The team adhered to the findings from cognitive psychology (e.g., Chi, Glaser, and Rees, 1983;
Lesgold, 1984) that knowledge is not very useful as a determinant of performance until it is embedded in an organized and meaningful cognitive structure.
The project team focusing on knowledge had to confront the difficult questions of how general or specific the Knowledge descriptors should be and how the Knowledge domains most critical for employment could be identified (Costanza, Fleishman, and Marshall-Mies, 1999). The team adopted the initial taxonomy of Knowledge descriptor from the Fleishman Job Analysis Survey (F-JAS), including 49 job-relevant Knowledge descriptors intended to cover (at a very general level) the knowledge requirements for all occupations in the labor force (Fleishman, 1992). The prototype development team compared the 49 F-JAS descriptors with the results of recent job analysis work at the Office of Personnel Management (Corts and Gowing, 1992). They also compared the descriptors to the Classification of Instructional Programs (Morgan, Hunt, and Carpenter, 1990) and to the preliminary findings of the National Occupational Information Coordinating Committee, which was attempting to develop a hierarchical taxonomy of educational preparation content. After many rounds of expert panel judgments concerning the comprehensiveness and suitability of alternative taxonomies, the prototype development team settled on a 2-level taxonomy consisting of 33 basic descriptors grouped into 10 more general categories. Each of the 33 descriptors was given a short (one or two sentence) specification.
The Knowledge descriptors are at a very high level of generality, which may suffice for some purposes, but not for others. For example, the subdisciplines in psychology are very different, and a knowledge requirement labeled “psychology” may not be informative. Also, the knowledge requirements for many technical services (e.g., plumber) and equipment operators (e.g., geothermal installation equipment operators, airplane pilots) do not appear to be included.
The prototype content model included a taxonomy of the education, training, and other preparation required for occupations (Anderson, 1999). At the most general level, the descriptors were grouped into 7 categories: general education level, instructional program required (42 descriptors), subject-area education level (15 descriptors), licenses required (2 descriptors), requirement to obtain a license (6 descriptors), who requires the license (3 descriptors), and related work experience (4 descriptors). In the field test of the prototype content model (described in Chapter 4), job incumbents answered seven factual questions related to each category of information. Although these descriptors were relatively occupation specific,
multiple surveys of users of DOT had shown that they valued information on the amount and type of education needed to enter an occupation (Anderson, 1997). The Occupational Preparation taxonomy of the prototype content model was analogous to Specific Vocational Preparation in DOT, but it provided much more information. The current O*NET content model incorporates a less extensive taxonomy of occupational preparation, with five broad categories of information: general education level, related work experience required for hiring, onsite training required to perform the job, on-the-job training required, and number of years of apprenticeship required.
The current O*NET content model includes a taxonomy of Skills that is nearly identical to the Skills taxonomy in the prototype content model. This was the most difficult domain of occupational performance requirements to conceptualize, in terms of a taxonomy of descriptors. At the same time, both the advisory panel and many users of O*NET view information on skills as crucial. For example, presenters at our panel workshops described using O*NET to identify transferable skills to assist in matching individuals with jobs (see Chapters 6 and 7). However, simply defining the term “skill” is difficult, and the O*NET prototype development reports (Peterson et al., 1997, 1999) did not do so with any degree of concreteness or clarity. The lack of clarity is shown in the following statements, which reflect varying definitions of skill (Peterson et al., 1999, pp. 50-57):
“In the past, skills have been defined in terms of specific task performance, educational requirements (e.g., the 3 Rs), or as a set of new capacities such as critical and creative thinking.”
“Skilled performance is a function of knowledge expertise as well as acquisition of a set of strategies, procedures, and processes for acquiring and working with information.”
“The procedures for acquiring and working with knowledge appear to represent the key components of skills.”
“Consequently, skills are not necessarily enduring characteristics of individuals. They depend on experience and practice. Second, they can be defined at different levels of generality; and thirdly, skills cannot be defined apart from some performance domains involving the acquisition of certain types of knowledge.”
Given these attempts to define skill, the prototype development team then made the unsubstantiated statement that “sociotechnical systems theory” is the most widely accepted model for workplace behavior (Peterson
et al., 1999). Although we know of no evidence to support this statement, the project team used the sociotechnical systems framework to suggest that there are five categories of cross-functional skills (problem-solving skills, technical skills, social skills, systems skills, and resource management skills). The project team went on to identify more specific skills in each of the five categories, but it did not always provide a clear rationale for what was included or excluded. For example, there is a long-standing model of the problem-solving process that suggests there are several stages in the process, such as problem identification, information gathering, idea generation, etc. The prototype development team turned each of these stages into a separate cross-functional skill in the category of complex problem-solving skills. The team identified these stages as separate skills despite considerable evidence that suggests experts do not go through such stages when solving ill-structured problems. The current O*NET content model does not include these separate skills.
For the technical skills category, the team developed the taxonomy of descriptors by examining a sample of 48 job analyses (not identified) and inferring the existence of 12 distinct technical skills that enabled technical performance in them. Again, several of these technical skills had not been named in previous research. To develop the taxonomy in the social skills category, the team primarily drew on the social psychology literature (e.g., Cantor and Kihlstrom, 1987). However, the substantial—and occupationally relevant—literature on the training of interpersonal skills (e.g., Decker and Nathan, 1985; Goldstein and Sorcher, 1974; Latham and Saari, 1979) was not mentioned. Consideration of this literature would most likely have produced a somewhat different social skills taxonomy. The current social skills taxonomy in the O*NET content model is identical to the prototype.
The systems skills category included six individual descriptors of cross-functional skills, but they were given very little concrete specification (see Chapter 9 for a critique of one such descriptor, systems evaluation). The O*NET prototype development project final report (Peterson et al., 1997) appears to be the first place these six skills have been named. One possible exception is “visioning skill,” which is at least semantically related in the management literature to the notion of leaders as visionaries. However, visioning does not have a research base and the current O*NET content model does not include it.
At the highest level of the prototype and current Skills taxonomies are the two broad categories, cross-functional and basic skills. Basic skills are further separated into content and process skills. Content skills include reading, writing, and arithmetic as well as listening skill and skill at using the scientific method. Process skills are “cognitive information processing skills that facilitate learning.” However, Lohman (1994a, 1994b) has
warned against trying to convert hypothesized general cognitive processes into measures of individual differences.
The O*NET prototype development team did not define or specify the skills domain very clearly, admittedly a difficult task. As a consequence, there is considerable potential for confusing descriptors from the Skills taxonomy with descriptors with similar names from other domains (e.g., Abilities or GWAs). In addition, many of the specific skill descriptors were first named and defined as part of the O*NET prototype development project. The rationale for creating them is not always clear. The Skills taxonomy would have benefitted from considerable additional development and review.
Generalized Work Activities
The taxonomy of GWAs in the current O*NET content model is nearly identical to the taxonomy in the prototype content model. The prototype development team carefully defined these descriptors of the basic characteristics of the work itself. According to Jeanneret et al. (1999), GWAs are not tasks, they are not responsibilities, and they are not dimensions of performance. They are the underlying behavioral components of tasks, such that performance on a specific task, no matter how broad or narrow, could be a function of more than one GWA, and a particular GWA could be a component of performance on more than one job task or responsibility.
For example, in the world of sports, “throwing a ball” might be a GWA. It underlies any number of sports tasks, and any sports task that involves throwing a ball also is a function of additional GWAs (e.g., processing information about the batter, if the task is for a baseball pitcher to throw strikes). Describing work in terms of GWAs is seen as more general (nomothetic), rather than idiosyncratic, and as better than describing work in terms of tasks for the purposes of making comparisons between occupations and building useful data archives relating many different personal attributes to performance capabilities. These are difficult distinctions to make, and the GWAs are characterized somewhat differently in the O*NET book (Peterson et al., 1999) and the O*NET monograph published in Personnel Psychology (Peterson et al., 2001). They are referred to as worker oriented in the former and more work oriented in the latter.
The taxonomy of GWAs was based on an exhaustive review of the job analysis research literature that used general activity requirement assessment rather than specific task requirement assessment, such as the Position Analysis Questionnaire (McCormick, Jeanneret, and Meecham, 1969, 1972) or the Occupational Analysis Inventory (Cunningham, 1988). The developers of the GWA taxonomy reviewed job analytic data obtained
from all corners of the labor force. Their scholarship was detailed and comprehensive.
The resulting taxonomy adopted an information-processing model of work behavior incorporating at the most general level four categories of descriptors involving (1) information input, (2) mental processes using that input, (3) work output, and (4) interactions with others. The GWA taxonomy had 42 descriptors at the most specific level, 9 more general categories at the next level into which the more specific 42 descriptors were grouped, and finally the 4 general categories listed above. An outline of the full prototype taxonomy is shown in Appendix B. The current content model uses the same terms and incorporates all but 1 of these 42 descriptors.
Some of the ambiguities in the prototype and current GWA descriptors now become more readily apparent. For example, why is the current and prototype GWA “Repairing and maintaining mechanical equipment” not a task? Why is it not a skill? What makes it an underlying Generalized Work Activity? Many of the GWAs are very abstract and perhaps difficult for incumbents to relate to their own work. The information-processing model may exacerbate this problem by suggesting inclusion of such GWAs as “processing information” and “making decisions and solving problems,” which are very difficult to specify in any concrete way (see Lohman, 1994a, 1994b).
The prototype Work Context taxonomy was revised more extensively than the other taxonomies before its inclusion in the current O*NET content model. The prototype development team viewed information on the Work Context as critical for the O*NET system because it is important to identify health and safety hazards; investigate contextual effects on performance, satisfaction, group cohesion, and organizational effectiveness; help design selection systems; provide more realistic job previews; and improve compensation systems (Peterson et al., 1999). However, the team did not confront the question of what was and was not “context” or the question of the extent to which a particular context descriptor was specific to settings (e.g., a particular establishment) rather than occupations, which is the unit of analysis for O*NET. The unanswered question: Is the variability in contexts greater across settings for a particular occupation than the variability across occupations? In general, context variables were not defined except to say that they are “moderator” variables, without saying what relationships were being moderated (Strong et al., 1999).
The Work Context taxonomy was developed via an extensive search of the literature on context effects in many different disciplines, including a review of instruments used to measure these effects, as well as searches
of the literatures on environmental health and occupational health (Strong et al., 1999). After considerable review by subject-matter experts, the team created a three-level taxonomy of Work Context. In the field test of the prototype content model, over 50 questionnaire items were used to assess the first two levels of the taxonomy, with additional items to assess the most detailed level. Following the field test, the first two levels of the taxonomy were retained (see Appendix B), but the number of specific descriptors of Work Context was reduced from 97 to 57.
The Organizational Context taxonomy is based on the assumption that the nature of an occupation will vary as a function of the nature of the organization in which it is embedded. This assumption raises questions. For example, is the occupation of economics professor described differently at a large public research university and a small liberal arts college? If so, should the differences be regarded as sampling error or as substantive occupational differences, and is the design of the O*NET database equipped to handle the latter?
The Organizational Context taxonomy was generated by examining the literatures on organization theory, organizational development, organizational design, organizational performance assessment, organizational sociology, and organizational psychology (Arad, Hanson, and Schneider, 1999). The resulting large number of organizational context variables that had shown important effects in previous research was organized into a four-level hierarchical taxonomic structure that seemed to best reflect the research results. At the top of the hierarchy, the context variables were designated as either structural characteristics or social processes. There are 6 second-order factors (type of industry, organizational structure, human resource systems and practices, culture, goals, and roles), 16 third-order factors, and 35 specific descriptors. The descriptors include such things as leadership characteristics, organization size, skill variety, autonomy, recruitment planning, and operations. In the field test of the prototype content model, information related to many descriptors was obtained from a single manager representing the organization, and information related to other descriptors (e.g., autonomy, skill variety, leader behavior) was obtained by aggregating the perceptions of individual job incumbents.
In general, the Organizational Context descriptors cover a wealth of information, but it is not clear how such data should be incorporated in the O*NET database, since they are not tied to occupations. Although this taxonomy of descriptors is included in the current O*NET content model (see Figure 1-1), the O*NET Center does not collect data on Organizational
Context and does not include information on Organizational Context in the current O*NET database.
As discussed above, the Advisory Panel for the Dictionary of Occupational Titles (1993) proposed a content model including occupation-specific data in the Knowledge and Skills domains. However, what the O*NET prototype project team began developing were occupation-specific Tasks and Work Activities. At the time the O*NET prototype development project was concluded in 1996, the team had completed only one pilot study of occupation-specific descriptors. This pilot was the precursor to the development of the DWAs descriptor taxonomy in subsequent years.
DEVELOPMENT OF THE CURRENT O*NET CONTENT MODEL
Once the prototype content model was fully developed, the project team conducted a field test of the prototype questionnaires. The team anticipated that the prototype content model would be revised as more research data became available and as user needs and the characteristics of the labor force itself changed (Peterson et al., 1999). Such changes could take the form of additions, deletions, or revisions of the descriptors in a domain; changing item formats and response scales; or even deleting or adding entire domains. However, despite the expectation that the prototype content model was not set in concrete, few changes were made as a result of the field test data and subsequent questionnaire evaluations.
Field Tests of the Prototype
The first step in testing the prototype content model was to define the unit of analysis by creating a new occupational classification system to replace the system of over 12,000 titles included in DOT.
The process used to create a new occupational classification system was summarized in O*NET Data Dictionary-Release 1.0, Appendix D (National Center for O*NET Development, 1998). That report indicates that the process entailed use of the occupational classification system adopted by the Bureau of Labor Statistics to administer the Occupational Employment Survey, the development of crosswalks to DOT title codes, cluster analyses of DOT data, analysis and aggregation of DOT task statements, and multiple reviews by subject-matter experts. The process resulted in identification of 1,122 occupations, referred to as “occupational units.” As described in the 1998 report, even though DOT titles and task data contributed to the original formation of the occupational units, these were not the outcome of
a simple clustering of DOT titles, nor were they meant to represent simple aggregations of DOT titles. Thus, it is not unreasonable to conclude that, at the end of this development process, each occupational unit had its own identity—consisting of a title, definition, and task description—independent of, though partially informed by, DOT titles and task content.
The next step was to collect two sets of data related to these occupational units. The first data set contained ratings made by occupational analysts using the written descriptions of occupational units. It became the first O*NET database that could be analyzed to establish proof of concept. The second data set contained ratings by job incumbents, obtained through a survey of incumbents in a targeted set of 80 occupational units; the response rate to the survey was low, however.
Despite the limited response to the job incumbent survey, the two data sets were adequate to support a preliminary evaluation of the prototype content model. At least 5 analyst ratings were available for all 1,122 occupational units, and 30-35 of the 80 occupational units targeted in the survey were rated by at least 4 (mean = 10) incumbents (Peterson et al., 1997). The psychometric properties of both the analyst and incumbent ratings were generally encouraging. There were analyst versus incumbent differences in the expected direction (e.g., incumbents rated their jobs as more complex and demanding than did the analysts), but the differences were not startling. The various descriptor domains could discriminate among occupations, and there was reasonable variation across the domain descriptors for a specific occupation. However, exploratory principal components analyses tended to yield a smaller number of factors within domains than the investigators had hoped. For example the GWA descriptor covariances yielded only three factors corresponding roughly to data, people, and things—the same factors that had been used to rate the complexity of occupational titles in DOT. The researchers identified a similar three-factor structure for the Skills taxonomy, although they were labeled cognitive skills, organizational skills, and technical skills (Mumford, Peterson, and Childs, 1999).
The limitations of these data sets were thoroughly discussed by Peterson et al. (1999). However, despite these limitations, the results pertaining to the reliability and discriminant validity of the domain descriptors were encouraging and positive enough in the investigators’ judgment to establish proof of concept for the prototype content model.
To address the low response to the job incumbent surveys, DOL commissioned a working group to test and revise the questionnaires. The revisions made on the basis of this review led to the questionnaires currently used to collect O*NET data (see Chapter 4).
Subsequent to the field test and prototype revisions, the O*NET Center made several other changes to the content model and to the occupational classification system. These changes are discussed below, and changes to the occupational classification system are discussed in Chapter 3.
Organizational Context, Work Values, and Occupational Interests
Although the Organizational Context domain is a part of the current content model (see Figure 1-1), the O*NET Center collects no data related to this domain. It does not routinely collect data related to the Work Values and Occupational Interests domains as part of its primary data collection program, but it has used other methods to gather data related to these two domains. To generate updated information on Work Values, the O*NET Center engaged experts in vocational psychology to rate occupations (Rounds et al., 2008a). The raters were provided with information on the occupation, including the O*NET written description, Skills, GWAs, Work Context, Work Styles, Education, and wages. They applied a method developed earlier (McCloy et al., 1999) to assign an occupational values profile to each occupation. To develop updated information on Occupational Interests, the O*NET Center uses a similar approach, engaging trained occupational analysts. The analysts are provided with updated information on the occupation, including O*NET information, and apply a specified method to create an occupational interest profile for each occupation in the database (Rounds et al., 2008b).
Development of Detailed Work Activities
Consistent with the recommendations of the Advisory Panel for the Dictionary of Occupational Titles (1993), the O*NET Center facilitated the development of a taxonomy of more occupationally specific descriptors that came to be called DWAs. This taxonomy is included in the current O*NET content model and information on DWAs for each occupation is available as part of the work activities information in the database viewed through the website O*NET OnLine. However, the O*NET Center does not include DWAs in the core content model data files it makes available for downloading, providing these descriptors along with other information as supplemental data (http://www.onetcenter.org/supplemental.html).
In developing this new taxonomy, the O*NET Center specified a DWA as a descriptor of work activities that was intermediate in generality between specific occupational tasks and the GWAs described above. The goal was to create descriptors general enough to permit cross-occupational
matching, but specific enough to permit specific occupational differentiation (National Center for O*NET Development, 2003). To achieve this goal, the O*NET Center indicated that the DWAs should not overlap, or be redundant with, other content model descriptors, such as Abilities, Skills, and Knowledge, and should be found in more than one occupation. At the same time, the O*NET Center viewed any DWA found in more than 100 occupations as too broad to be useful and suggested it should be eliminated (p. 7). The O*NET Center sought to develop approximately 15-20 DWAs for each occupation.
The O*NET Center’s development of DWAs built on the results of an earlier effort by the state of Oregon to develop a set of similar descriptors. That study reviewed the descriptors used in the Bureau of Labor Statistics’ Occupational Employment Statistics survey, the Classification of Instructional Programs, and the task information from the DOT to develop “skill statements.” The O*NET Center created additional statements and the resulting pool was edited for redundancy, clarity, cross-occupational relevance, and comprehensiveness of coverage to create an initial pool of 2,345 DWAs. In several rounds of review, subject-matter experts eliminated some DWAs from the pool and added others in order to meet the specifications listed above. In addition, the developers edited the DWAs for clarity, understandability, and appropriateness. Through this process, the O*NET Center created 2,165 DWAs and assigned them to occupations. Some examples are as follows: (1) adhere to government aviation regulations, (2) analyze dental data, (3) use airbrush techniques, and (4) apply appropriate physical restraints.
In keeping with this design, each DWA was assigned to a single GWA, and the best fitting assignment of DWAs to GWAs has significant convergent and divergent validity. Consequently, DWAs are fully nested within GWAs, and users of O*NET OnLine who access either the details report or the custom report for an occupation see a list of “work activities” which displays DWAs nested within GWAs.
Development of Tools and Technology
Consistent with another advisory panel recommendation, the O*NET Center began in 2006 to develop another domain of more occupationally specific descriptors, known as Tools and Technologies or T2. This domain is included in the current O*NET content model (see Figure 1-1).
The purpose of this effort is to incorporate additional contextual detail and occupation-specific descriptors into O*NET. The O*NET Center does not routinely collect data related to this domain through its main data collection program. Instead, the principal sources of T2 information are Internet-based searches and information gathered from various O*NET
customers, supplemented by review of printed publications and review by subject-matter experts (Dierdorff, Drewes, and Norton, 2006). To date, the O*NET Center has gathered this type of information for 427 occupations, approximately half of those in the current database. Like the DWA data, the T2 data can be viewed through the O*NET OnLine website, but they are not included in the core O*NET database made available for download. T2 data are available for download as supplemental information.
Lay Job Titles and Code Connector
The O*NET Center collects a “lay titles file,” which is made available to users through a web-based application, the O*NET Code Connector. This application allows the user to match a particular job title to O*NET occupations. The Code Connector is designed especially to help workforce professionals determine the correct O*NET occupational codes for a specific job that the professional is helping an individual with. Although designed for the workforce development community, many human resource management professionals using O*NET find this feature valuable.
O*NET’s Evolutionary History Recapitulated
There is a direct evolutionary progression from DOT through APDOT to the O*NET prototype to the current database, O*NET 14.0, and there are many similarities between the DOT and O*NET content models. The DOT content model was built in stages from the basic task descriptions, to aptitude requirements, to education requirements, to physical requirements, to vocational interest requirements, to work complexity (in terms of data, people, and things), and to temperament requirements. Virtually every taxonomy of descriptors in the current O*NET content model has an analog in the DOT, and most of these taxonomies (in both the DOT and O*NET) are cross-functional in nature. Although most elements of the O*NET content model provide cross-functional information, the model also includes occupation-specific tasks, as well as two elements that are to a certain degree occupation specific—DWA and T2 descriptors. The major difference between DOT and O*NET is the specification for the unit of analysis. The 1991 DOT provided information on over 12,000 occupational titles, whereas O*NET provides information on 1,102 occupations.
Another difference is that, by the 1980s, the cross-functional domains of occupational descriptors included in DOT had become rather dated. Research on the characteristics of jobs had progressed considerably beyond the DOT’s aptitude, interests, temperament, and education requirements and work activity taxonomies. As a consequence, the advisory panel proposed very comprehensive recommendations for what the “new DOT,”
which became O*NET, should look like (Advisory Panel for the Dictionary of Occupational Titles, 1993). These recommendations do not appear to be based on a detailed analysis of specific user needs for occupational data. Instead, the APDOT content model may have represented an attempt to anticipate an unspecified user’s future needs, whatever they might be, by incorporating everything about jobs that had been studied, in the name of explaining occupational choices, occupational performance, and work/occupational satisfaction. This included findings from the research literature in job analysis, performance measurement, cognitive psychology, and other fields.
Despite the breadth of the APDOT content model, the O*NET prototype development project came very close to total success in making it operational. The postprototype development of the DWAs and T2 information was in direct response to the proposed APDOT content model. One advisory panel recommendation that has not been implemented is the development of performance standards and critical performance outcomes for each occupation.
CONCLUSIONS AND RECOMMENDATIONS
The construct validity of the taxonomies of descriptors is very uneven across the different domains included in the content model. For example, in the Abilities domain, the descriptors reflect a long history of psychological research on the nature and measurement of human abilities, but many of the descriptors in the Skills domain lack such an extensive research base. In addition, the different domains lack detailed and concrete specifications and often include descriptors with the same name, making it difficult to distinguish among them. “Problem solving” appears in the Abilities, Skills, Work Styles, and GWA domains. Although there may legitimately be both a problem-solving Ability and a problem-solving Skill, the content model does not clearly distinguish the different meanings of problem solving in these two different domains.
This history of the research and development of the content model, as well as user experiences discussed in this report, raise serious questions about the completeness of some of the individual domains and the extent to which each uses the most appropriate level of generality or specificity for its descriptors. For example, in the Knowledge domain, the taxonomy includes a general descriptor for building and construction, but no more specialized Knowledge descriptors, such as plumbing or carpentry. In addition, by design, the GWAs have a very high level of generality. The lack of more specific descriptors in the Knowledge, Skills, and Abilities domains (specifically, physical abilities) limits the usefulness of O*NET data for human resource management and the determination of disabilities, and the
incomplete development of the DWAs domain limits its potential use in retraining displaced workers. These limitations, and the recommendations to address them, appear in Part II.
Research leading to revisions of elements of the content model, designed to address these weaknesses, could prove disruptive to current users of O*NET data. By maintaining the continuity of the core elements of the content model over the past decade, the O*NET Center has encouraged developers to incorporate O*NET data in a variety of useful tools and applications (see Chapter 6) and supported longitudinal research on changes in the labor market. Nevertheless, the panel members agreed that research to revise the content model could ultimately lead to long-term benefits to O*NET users that would outweigh short-term disruptions. Both the Advisory Panel for the Dictionary of Occupational Titles (1993) and the O*NET prototype investigators (Peterson et al., 1999) viewed the O*NET content model as needing continuous improvement, yet there have been few changes since the mid-1990s.
Recommendation: The Department of Labor should commission research to improve the content model, beginning with the skills and knowledge taxonomies. The goals of the research should be to reduce the redundancy of descriptors within and across taxonomies, distinguish more clearly among the taxonomies, enhance completeness, and specify descriptor variables in concrete and meaningful terms.
This research to refine the taxonomies would be aided by an analysis to identify which descriptors in each domain yield high standard errors across raters. Patterns of descriptors with high standard errors in a particular domain could provide insight into rater understanding of the rating scales and inform research and revisions of that domain.
Collecting data aligned with a simpler, more rational content model would require shorter surveys, freeing resources to improve data collection along other dimensions, such as increasing the number of detailed occupations in the classification system or reducing the time interval between successive waves of data collection.
To date, there has been little effort to use the hierarchical structure of the various domain taxonomies in the content model. Occupations are not “profiled” on the higher level variables represented by grouping descriptors into meaningful higher level categories. In addition, there has been little recent empirical research on the factor structure of the domains. Research is needed to determine whether improvements in the descriptor taxonomies and in the measurement scales would yield clearer and better differentiated factor structures. However, if the descriptors in a domain lack clear, dis-
tinct, and meaningful specifications, raters will have trouble distinguishing among them, and fewer distinguishable factors would result.
Recommendation: The Department of Labor should commission a series of studies to develop occupational profiles at each hierarchical level of each taxonomy of descriptors. For example, in addition to profiling an occupation on all 52 Ability requirements, each occupation could be profiled on the smaller number of higher level Abilities formed by grouping specific abilities into higher order factors. Developing these higher level profiles would not involve collecting new data. It would involve considerable analysis of the existing O*NET database and much effort devoted to writing the specifications for each of the higher order factors.
The panel thinks that if significant and steady progress were made on these recommendations, the usability and construct validity of the O*NET content model information would be significantly enhanced. The O*NET database would also gain in flexibility for meeting a greater number of user needs.
Technical expertise is needed to assist DOL in carrying out these two research recommendations and the larger research agenda outlined in Chapter 10, as well as to determine what other research will strengthen the quality of O*NET and enhance its use. Although the research agenda represents the panel’s best judgment about important research needs at this time, important new questions about O*NET will emerge in the coming years. There is a pressing need for a sustained program of research and evaluation of O*NET to guide ongoing improvements in the content model, occupational classification system, data collection methods, and the usability of the database.
Recommendation: The Department of Labor should establish and support an external technical advisory board, comprised of senior scientists, to develop a research agenda for O*NET that will prioritize research suggestions from its members, the department, the O*NET Center, the user advisory board recommended below, and other sources. At a minimum, it should meet twice yearly, once to establish research priorities for the coming year and develop requests for proposals reflecting these priorities and once to review and rank proposals submitted by academic researchers or contractors.
The panel anticipates that the technical advisory board will prioritize research suggestions according to criteria that include the extent to which the research suggestion complies with the corpus of scientific literature and
current best practices, its cost, and the feasibility of implementation. The board will also weigh the research suggestion’s potential to improve the reliability and validity of the data in the content model, to reduce or extend the length of the questionnaires, to enhance collection of complete, accurate data, or to negatively affect longitudinal research based on O*NET or user-designed platforms.
In Chapter 6, the committee proposes a user advisory board to communicate users’ needs. DOL should not wait to initiate the research recommended in this report until the technical and user advisory boards have been constituted and are fully functioning, but should proceed with continuous improvement initiatives using its traditional advisers until these boards can be established. In addition, DOL should establish mechanisms for ongoing communication between the user advisory board and the technical advisory board.
Advisory Panel for the Dictionary of Occupational Titles. (1993). The new DOT: A database of occupational titles for the twenty-first century (final report). Washington, DC: U.S. Department of Labor, Employment and Training Administration.
Anderson, L.E. (1997). Education, training, experience, and licensure/certification: Evidence for the reliability and validity of measures. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, E.A. Fleishman, and K.Y. Levin (Eds.), O*NET final technical report. Volume I, II, and III. Available: http://www.eric.ed.gov/ [accessed July 2009].
Anderson, L.E. (1999). Occupational preparation: Education training, experience, and licensure/certification. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 91-104). Washington, DC: American Psychological Association.
Arad, S., Hanson, M.A., and Schneider, R.J. (1999). Organizational context. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 147-174). Washington, DC: American Psychological Association.
Barrick, M.R., and Mount, M.K. (1991). The big five personality dimensions and job performance. Personnel Psychology, 44, 1-26.
Benson, M.J., and Campbell, J.P. (2007). To be, or not to be linear: An expanded representation of personality and its relationship to leadership performance. International Journal of Selection and Assessment, 15, 232-249.
Borman, W.C., Kubisiak, U.C., and Schneider, R.J. (1999). Work styles. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 213-226). Washington, DC: American Psychological Association.
Cantor, R., and Kihlstrom, J.F. (1987). Personality and social intelligence. Englewood Cliffs, NJ: Prentice-Hall.
Carroll, J.B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press.
Chi, M.T., Glaser, R., and Rees, E. (1983). Expertise in problem solving. In R. Sternberg (Ed.), Advances in the psychology of human intelligence. Hillsdale, NJ: Erlbaum.
Corts, D.B., and Gowing, M.K. (1992). Dimensions of effective behavior: Executives, managers, and supervisors. Washington, DC: U.S. Office of Personnel Management, Personnel Research and Development.
Costa, P.T., Jr., and McCrae, R.R. (1992). Revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI) professional manual. Odessa, FL: Psychological Assessment Resources.
Costanza, D.P., Fleishman, E.A., and Marshall-Mies, J. (1999). Knowledges. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 71-90). Washington, DC: American Psychological Association.
Cottle, W.C. (1950). A factorial study of the multiphasic, Strong, Kuder, and Bell inventories using a population of adult males. Psychometrika, 15, 25-47.
Cunningham, J.W. (1988). Occupation analysis inventory. In S. Gael (Ed.), The job analysis handbook for business, industry, and government (pp. 975-990). New York: Wiley.
Dawis, R.V., and Lofquist, L.H. (1984). A psychological theory of work adjustment: An individual-differences model and its implications. Minneapolis: University of Minnesota.
Decker, P.J., and Nathan, B.R. (1985). Behavior modeling training: Principles and applications. New York: Praeger.
Dierdorff, E.C., Drewes, D.W., and Norton, J.J. (2006). O*NET tools and technology: A synopsis of data development procedures. Raleigh, NC: National Center for O*NET Development. Available: http://www.onetcenter.org/reports/T2Development.html [accessed July 2009].
Dvorak, B.J. (1947). The new USES General Aptitude Test Battery. Journal of Applied Psychology, 31, 372-376.
Fleishman, E.A. (1992). Fleishman Job Analysis Survey (F-JAS). Bethesda, MD: Management Research Institute.
Fleishman, E.A., and Reilly, M.E. (1992). Handbook of human abilities: Definitions, measurements, and job task requirements. Potomac, MD: Management Research Institute.
Fleishman, E.A., Costanza, D.P., and Marshall-Mies, J. (1999). Abilities. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 175-195). Washington, DC: American Psychological Association.
Geisinger, K.F., Spies, R.A., Carlson, J.F., and Plake, B.S. (Eds.). (2007). The seventeenth mental measurements yearbook. Lincoln, NE: Buros Institute of Mental Measurement.
Goldstein, A.P., and Sorcher, M. (1974). Changing supervisory behavior. New York: Pergamon.
Holland, J.L., and Gottfredson, G.D. (1976). Using a typology of persons and environments to explain careers: Some extensions and clarifications. Counseling Psychologist, 6, 20-29.
Hough, L.M. (1992). The “big five” personality variables—construct confusion: Description versus prediction. Human Performance, 5, 139-155.
Jeanneret, P.R., Borman, W.C., Kubisiak, U.C., and Hanson, M.A. (1999). Generalized work activities. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 105-125). Washington, DC: American Psychological Association.
Latham, G.P., and Saari, L.M., (1979). Application of social learning theory to training supervisors through behavioral modeling. Journal of Applied Psychology, 64, 239-246.
Lesgold, A.M. (1984). Acquiring expertise. In J.R. Anderson and S.M. Kosslyn (Eds.), Tutorials in learning and memory: Essays in honor of Gordon Bower. New York: Freeman.
Lohman, D.F. (1994a). Component scores as residual variation (or why the intercept correlates best). Intelligence, 19, 1-11.
Lohman, D.F. (1994b). Implication of cognitive psychology for ability testing: Three critical assumptions. In M.G. Rumsey, C.B. Walker, and J.H. Harris (Eds.), Personnel selection and classification (pp. 145-172). Hillsdale, NJ: Erlbaum.
McCloy, R.A., Waugh, G., Medsker, G., Wall, J., Rivkin, D., and Lewis, P. (1999). Determining the occupational reinforcer patterns for O*NET occupational units (Volumes I and II). Raleigh, NC: National Center for O*NET Development. Available: http://www.onetcenter.org/reports/ORP.html [accessed July 2009].
McCormick, E.J., Jeanneret, P.R., and Meecham, R.C. (1969). The development and background of the Position Analysis Questionnaire (PAQ). (Tech. Rep. #5). Lafayette, IN: Purdue University, Occupational Research Center.
McCormick, E.J., Jeanneret, P.R., and Mecham, R.C. (1972). A study of job characteristics and job dimensions as based on the Position Analysis Questionnaire (PAQ). Journal of Applied Psychology, 56, 347-368.
Meehl, P.E., and Golden, R. (1982). Taxonomic methods. In P. Kendall and J. Butcher (Eds.), Handbook of research methods in clinical psychology (pp. 127-181). New York: Wiley.
Morgan, R.L., Hunt, E.S., and Carpenter, J.M. (1990). Classification of instructional programs. Washington, DC: National Center for Education Statistics.
Mumford, M.D., Peterson, N.G., and Childs, R.A. (1999). Basic and cross-functional skills. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 49-69). Washington, DC: American Psychological Association.
National Center for O*NET Development. (1998). Appendix D: The development of the Occupational Information Network (O*NET) analyst database. Raleigh, NC: Author. Available: http://www.onetcenter.org/reports/appendix_d.html [accessed November 2009].
National Center for O*NET Development. (2003). Summary report: Updating the detailed work activities. Raleigh, NC: Author. Available: http://www.onetcenter.org/dl_files/DWA_summary.pdf [accessed July 2009].
National Research Council. (1980). Work, jobs, and occupations: A critical review of the Dictionary of Occupational Titles. A.R. Miller, D.J. Treiman, P.S. Cain, and P.A. Roos (Eds.).Washington, DC: National Academy Press. Available: http://www.nap.edu/catalog.php?record_id=92 [accessed May 2009].
Peterson, N.G., Mumford, M.D., Borman, W.C., Jeanneret, P.R., Fleishman, E.A., and Levin, K.Y. (Eds.). (1997). O*NET Final Technical Report (Vols. 1-3). Salt Lake City: Utah Department of Employment Security, on behalf of the U.S. Department of Labor Employment and Training Administration. Available: http://www.eric.ed.gov/ [accessed March 2009].
Peterson, N.G., Mumford, M.D., Borman, W.C., Jeanneret, P.R., and Fleishman, E.A. (Eds.). (1999). An occupational information system for the 21st century: The development of O*NET. Washington, DC: American Psychological Association.
Peterson, N.G., Mumford, M.D., Borman, W.C., Jeanneret, P.R., Fleishman, E.A., Levin, K.Y., Campion, M.A., Mayfield, M.S., Morgeson, F.P., Pearlman, K., Gowing, M.K., Lancaster, A.R., Silver, M.B., and Dye, D.M. (2001). Understanding work using the Occupational Information Network (O*NET): Implications for practice and research. Personnel Psychology, 54, 451-492.
Rounds, J., Armstrong, P.I., Liao, H., Rivkin, D., and Lewis, P. (2008a). Second generation occupational value profiles for the O*NET system: Summary. Raleigh, NC: National Center for O*NET Development. Available: http://www.onetcenter.org/reports/SecondOVP_Summary.html [accessed July 2009].
Rounds, J., Armstrong, P.I., Liao, H., Lewis, P. and Rivkin, D. (2008b). Second generation occupational interest profiles for the O*NET system: Summary. Available: http://www.onetcenter.org/reports/SecondOIP_Summary.html [accessed July 2009].
Sager, C.E. (1997). Occupational interests and values: Evidence for the reliability and validity of the occupational interest codes and the values measures. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, E.A. Fleishman, and K.Y. Levin (Eds.), O*NET final technical report, volumes I, II and III. Salt Lake City: Utah State Department of Workforce Services on behalf of the U.S. Department of Labor, Employment and Training Administration. Available: http://www.eric.ed.gov/ [accessed July 2009].
Sager, C.E. (1999). Occupational interests and values. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 197-211). Washington, DC: American Psychological Association.
Sager, C.E., Mumford, M.D., Baughman, W.A., and Childs, R.A. (1999). Occupation-specific descriptors: Approaches, procedures, and findings. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 227-235). Washington, DC: American Psychological Association.
Strong, M.H., Jeanneret, P.R., McPhail, S.M., Blakley, B.R., and D’Egidio, E.L. (1999). Work context: Taxonomy and measurement of the work environment. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 127-145). Washington, DC: American Psychological Association.
U.S. Department of Labor. (1991). Dictionary of occupational titles (4th ed.). Washington, DC: Author.