2
The Content Model

The content model is the basic building block of the O*NET system and refers to the prescribed set of variables used to describe occupations (see Figure 1-1). Each of these variables—a particular aptitude, skill, or education requirement—is referred to as a descriptor. All uses of O*NET rely on descriptions of occupations using the descriptors specified by the content model. If important features of occupations are not included in the content model, or irrelevant features are included, or the descriptors that are included are flawed, then the applications of O*NET are suboptimal. Simply put, the importance of the validity, completeness, and usability of the content model cannot be overemphasized.

This chapter discusses the developmental history of the O*NET content model and the research literature on which the model is based. Following a brief introduction, the first section focuses on the content model of the Dictionary of Occupational Titles (DOT), the predecessor to O*NET. The second section discusses the content model proposed by the Advisory Panel for the Dictionary of Occupational Titles (APDOT) for a new and different DOT; this ultimately became O*NET. The third section discusses the O*NET prototype content model, focusing especially on the development of descriptors, many of which are included in the current O*NET content model. The fourth section discusses the field test of the prototype and development of the current content model. The final section presents the panel’s conclusions and recommendations related to the content model.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 21
2 The Content Model T he content model is the basic building block of the O*NET system and refers to the prescribed set of variables used to describe occupa- tions (see Figure 1-1). Each of these variables—a particular aptitude, skill, or education requirement—is referred to as a descriptor. All uses of O*NET rely on descriptions of occupations using the descriptors specified by the content model. If important features of occupations are not included in the content model, or irrelevant features are included, or the descriptors that are included are flawed, then the applications of O*NET are subopti- mal. Simply put, the importance of the validity, completeness, and usability of the content model cannot be overemphasized. This chapter discusses the developmental history of the O*NET content model and the research literature on which the model is based. Following a brief introduction, the first section focuses on the content model of the Dictionary of Occupational Titles (DOT), the predecessor to O*NET. The second section discusses the content model proposed by the Advisory Panel for the Dictionary of Occupational Titles (APDOT) for a new and differ- ent DOT; this ultimately became O*NET. The third section discusses the O*NET prototype content model, focusing especially on the development of descriptors, many of which are included in the current O*NET content model. The fourth section discusses the field test of the prototype and devel- opment of the current content model. The final section presents the panel’s conclusions and recommendations related to the content model. 

OCR for page 21
22 A DATABASE FOR A CHANGING ECONOMY THE CONTENT MODEL AS A TAXONOMY The content model is intended to be a taxonomy of occupational de- scriptors. An important first step in developing a taxonomy is to define the unit of analysis. In O*NET, this is the occupation, rather than the job or position. An occupation is broader than a specific job or specific position, and it is not idiosyncratic to a particular organization, industry, or setting. A particular occupation could include several jobs if the general responsi- bilities, activities, and requirements for the various jobs are substantially similar. For example, the occupation “commercial aircraft pilot” could include many different jobs as a function of type of organization, type of aircraft, and business function (see Chapter 3 for further discussion). With occupations as the unit of analysis and the characteristics of these occupations, such as the physical and cognitive abilities they require of workers, to be included in a taxonomy of descriptors, a number of other important questions become relevant: 1. How general or specific will the descriptors of occupational re- quirements be? For example, when considering how to develop a taxonomy of the abilities required by occupations, previous re- search provides taxonomies that define abilities in terms of two groups of factors (clusters of more specific abilities), four groups of factors, eight groups of factors, or some greater number (see Carroll, 1993). 2. Given a particular level of generality/specificity, should the set of descriptors of a particular occupational requirement (e.g., knowl- edge, ability) be a representative sample of all possible descriptors of that requirement, or should it represent the entire universe of descriptors? Previous research-based taxonomies of human abili- ties (Carroll, 1993) are meant to describe the full range of human abilities, rather than a representative sample. Is this exhaustive approach to be applied when developing descriptors for all of the occupational requirements? 3. Should each descriptor of occupational requirements be applicable to every occupation (i.e., unit of analysis)? To return to the exam- ple of abilities, previous taxonomies have been designed to include descriptors that can be measured across all individuals. They do not include any descriptors of abilities that are applicable only to men or only to women. 4. Is the taxonomy to include genuine taxons, such as those in biol- ogy? In biology, a taxon is defined as a group of organisms with common characteristics, in varying degrees of distinction, such as a phylum, order, family, genus, or species. Common physical char-

OCR for page 21
23 THE CONTENT MODEL acteristics among the organisms allow clear separation between groups and subgroups (Meehl and Golden, 1982). In the social sciences, where there are rarely such distinguishing variables, a taxonomy must live with classification variables that are continu- ous and errors of classification that cannot be reduced to zero. The boundaries between different groups and subgroups of similar occupational characteristics are always blurred, and the clarity of naming and defining different occupational characteristics must always be a function of the rules for minimizing the errors of clas- sification given the alternative ways of describing skills. 5. Can the taxonomy be designed to serve a wide range of purposes among diverse users? The developers of the O*NET content model addressed all these issues, if not directly, then by default. There have been four major milestones in the development of today’s content model. The first was the final edition of DOT, which incorporated a content model (U.S. Department of Labor, 1991). Second, in 1993, APDOT recommended a set of specifications for a content model to be included in a new and different DOT, which they referred to as the “new DOT” (Advisory Panel for the Dictionary of Occupational Titles, 1993). Third, following the advisory panel’s recom- mendations, the U.S. Department of Labor (DOL) funded the O*NET development project, which produced a prototype O*NET content model in 1997 (Peterson et al., 1997). Fourth, the prototype was revised in 1998- 1999, leading to the current content model. Since 2000, the National Center for O*NET Development (the O*NET Center) has collected data related to the descriptors in this content model. THE Dictionary of occupational titles CONTENT MODEL The 1991 revision of the DOT contains descriptions of over 12,000 oc- cupational titles, referred to as DOT codes. All information was obtained by trained occupational analysts who visited each work site, interviewed job incumbents, and observed them at work. The analysts were told to fol- low standardized procedures when observing and interviewing job incum- bents and creating descriptions using a common format or content model. The 1991 volume included, for each DOT code: 1. A written description. The analysts followed a detailed standard protocol to describe the specific tasks, duties, responsibilities, etc., carried out by the incumbent. These occupation-specific task de- scriptions led to the criticism that the DOT lacked a common

OCR for page 21
 A DATABASE FOR A CHANGING ECONOMY language that could be used to describe a variety of different jobs (National Research Council, 1980). However, the current O*NET content model includes occupation-specific tasks as well as Detailed Work Activities (DWA) that are derived from multiple analyses of task information and are somewhat similar to the task descriptions similar to those included in the DOT. 2. Aptitude ratings. Following a prescribed procedure, the analysts rated each occupational title on the level of each of nine aptitudes that were derived from the nine constructs measured by the General Aptitude Test Battery (GATB) (Dvorak, 1947). The GATB included four cognitive ability subtests, two perceptual speed subtests, and three psychomotor subtests. 3. Temperament requirements. A temperament requirement was de- fined as an occupational situation to which a job incumbent must adjust. The analysts were provided with 10 situations and asked to identify the 2 most critical for each occupational title. They were not asked to equate these situational requirements with established individual personality traits. 4. Interests. The domain of work interests was represented by 10 interest factors configured as 5 bipolar preferences. The factors were derived from factor analyses of data from the Kuder Pref- erence Record and Strong Vocational Interest Blank by Cottle (1950). These factors predate the Holland typology of work inter- ests that is currently widely used in career guidance (i.e., Holland and Gottfredson, 1976). The analysts designated the two factors that were the most critical requirements of a particular occupa- tional title and were not asked to rate the importance or level of the factors. 5. Physical demands. The analysts also rated each occupational title in terms of six physical demands. They rated the level of strength required by the job using a five-point scale ranging from sedentary to light, medium, heavy, or very heavy. The other five physical de- mands: (1) climbing/balancing, (2) stooping/crawling, (3) reaching/ handling, (4) talking/hearing, and (5) seeing were designated as present or absent. 6. Job environment. The analysts indicated the presence or absence of extreme cold, extreme heat, dampness and/or humidity, noise and/or vibration, and whether the occupational title was performed primarily indoors, outdoors, or both. 7. Specific vocational preparation. The job analysts rated the level of Specific Vocational Preparation required by the occupational title, independent of general education, using a 9-point scale

OCR for page 21
 THE CONTENT MODEL ranging from 1 (receiving a short demonstration only) to 9 (over 10 years of specific preparation). 8. General educational development. Based on extensive specifica- tions and rater training, the job analysts rated the level of reason- ing, mathematics, and language skill required by the occupational title. 9. Complexity. The analysts rated the occupational title in terms of the involvement of the job tasks with data and information, inter- actions with people, and usage of things (such as equipment, tools, and vehicles). In sum, DOT was based on an extensive content model that included a broad spectrum of descriptors that were used to describe occupational titles for various purposes. With the exception of the task information in the written description, all of the descriptors were cross-functional and could be used to describe all occupational titles. These descriptors constituted a common language, and a number of them have their analogs in the current O*NET content model. THE APDOT CONTENT MODEL The secretary of labor convened APDOT in 1990, charging it to recom- mend strategies for collecting, analyzing, and disseminating occupational information. The advisory panel proposed a new model that specified the most im- portant types of information about occupations and workers that should be included in what it called the “new DOT,” which evolved to become O*NET (Advisory Panel for the Dictionary of Occupational Titles, 1993). At the most general level, this content model organized information into four broad domains that are similar to the most general domains in O*NET: (1) worker attributes, (2) work context, (3) labor market context, and (4) work content and outcomes (see Figure 1-1). It also included more spe- cific domains of information, such as Occupation-Specific Knowledge and Generalized Work Activities (GWAs). The advisory panel envisioned that information related to these domains would be incorporated into a flexible electronic database, rather than being published as a printed book. The domains and occupational descriptors included in this content model were based on extensive research, reviews of current practice, and expert testimony. However, the advisory panel did not discuss in any detail why each domain was critical, nor did it outline the specific uses of each domain (Advisory Panel for the Dictionary of Occupational Titles, 1993). Consequently, the APDOT content model is open to the criticism that it simply includes everything that occupational analysis researchers, practi-

OCR for page 21
 A DATABASE FOR A CHANGING ECONOMY tioners, and users of occupational data were discussing at that time, albeit in meaningful and organized ways. Overall, in comparison to the previous DOT content model, the APDOT content model was more comprehensive, systematically structured, and reflective of what had been learned about occupational analysis between 1930 and 1990. Of course, the advisory panel did not engage in the formidable data collection and measurement issues presented by implementation of the model. That was the task of the O*NET prototype development project. Although the advisory panel criticized the DOT codes as being too job-specific, its content model included Occupation-Specific Knowledge, Occupation-Specific Skills, and Duties/Tasks Performed (Advisory Panel for the Dictionary of Occupational Titles, 1993). It appears that the advisory panel wanted to include both cross-occupation information and occupa- tion-specific information. The advisory panel proposed (p. 15) that the single most important capability of the new occupational information system would be to use multiple approaches to identify transferable skills. Because the advisory panel viewed several different groups of descriptors as skills (e.g., tasks, work activities, aptitudes), it created the potential for future confusion concerning the specifications for the skills taxonomy in the O*NET con- tent model. The advisory panel made three other recommendations that informed the panel’s review of the current O*NET content model and data collection methods (Advisory Panel for the Dictionary of Occupational Titles, 1993): 1. The content model should describe occupational characteristics in such a way that any person can evaluate his or her capability to perform in an occupation. Consequently, difficult to understand technical terms should be avoided. 2. The Department of Labor should commit to “an ongoing research and development agenda” and should “recruit, train, and maintain a core staff of methodologically sophisticated professionals to man- age the DOT program” (1993, p. 7). 3. The content model itself should be subjected to continuous evalua- tion and enhancements, based on future developments in research and practice. In no sense should it be cast in concrete or viewed as unchangeable. THE O*NET PROTOTYPE CONTENT MODEL To carry out the advisory panel’s recommendation to develop a “new DOT,” the Employment and Training Administration contracted with a consortium of consulting firms with expertise in occupational analysis as

OCR for page 21
 THE CONTENT MODEL well as survey design and administration to conduct the necessary research and development. The project to develop the prototype for the new occu- pational information system was completed in 1996, and the research and development process has been described in several publications (Peterson et al., 1997, 1999, 2001). During the course of the project, the term “new DOT” was replaced by the Occupational Information Network (O*NET) in order to sever any direct linkage to the DOT and reflect the move to an electronic database. As envisioned by the advisory panel, the goal of the O*NET project was to develop a prototype of a new national occupational information content model and offer evidence substantiating “proof of concept” for such a model. The advisory panel expected that the prototype would then undergo additional development and revision before becoming the basis for the principal data collections. The prototype project team attempted to develop a content model that followed the advisory panel’s recommendations, corrected the DOT’s deficiencies, reflected new research and practice in occupational analysis, and incorporated the capability to adapt to a changing labor market and workforce. To advance these multiple goals, the specifications for the con- tent taxonomies would be crucial. The resulting prototype content model adopted the overarching frame- work of the APDOT content model, which divided all domains of occupa- tional descriptors into three groups describing (1) the work itself, (2) the “worker” (i.e., job applicant or job incumbent), or (3) the context in which work takes place. Curiously, neither the advisory panel nor the prototype project team explains why these three facets of occupations should be de- scribed, except to say that the database should be a complete description of occupations and that multiple windows (i.e., domains of information) are necessary to accommodate different user needs (Peterson et al., 1999, p. 13). Compared with the APDOT content model, the O*NET prototype gave less attention to occupation-specific descriptors and concentrated to a greater extent on so-called cross-occupation descriptors, or variables that could be measured meaningfully in all occupations and along which occu- pations would be expected to vary. As discussed later in this chapter, it is not easy to distinguish between cross-occupation and occupation-specific descriptors. Another objective for the O*NET prototype was to develop taxono- mies of descriptor variables in each domain that are hierarchical in nature. The goal from the outset was to describe occupations at varying levels of generalizability or specificity in each domain. Consequently, the descrip- tors were to be nested in hierarchical levels. For example, there are 17 specific descriptors in the Work Styles taxonomy, which are in turn nested in 7 more general styles (see Appendix B). This approach implies that the

OCR for page 21
 A DATABASE FOR A CHANGING ECONOMY database would include “score profiles” for each occupation at each level of the hierarchy. Because many of the taxonomies of descriptors created in the O*NET prototype project are included in the current content model, it is important to examine their development and research base. The prototype project pro- ceeded by constituting a separate development team for each of the content domains, reflecting the goal of providing multiple windows, or domains of information, to meet the needs of different users. The teams’ results are described briefly below. Abilities The current content model incorporates the taxonomy of Abilities descriptors developed in the prototype. The prototype team’s (and current) definition of ability is consistent with the very large research literature in- vestigating the nature of human abilities (e.g., Carroll, 1993). That is, an Ability is a relatively enduring attribute that reflects an individual’s capabil- ity for performing a particular category of tasks (e.g., verbal, mathemati- cal, physical, motor, psychomotor, sensory). This underlying capability for performing a designated category of tasks remains relatively stable over much of the individual’s life span, with certain physical or motor abilities perhaps being exceptions. The literature documents literally hundreds, if not thousands, of well- researched standardized tests of various human abilities (e.g., Geisinger et al., 2007), and investigations of the empirically estimated covariances among sets of such tests have produced considerable evidence for hierar- chical taxonomic structures of human abilities (Carroll, 1993). In light of O*NET’s objective of producing the most comprehensive occupationally relevant taxonomy, the development team incorporated the Fleishman and Reilly (1992) taxonomy of abilities to create the O*NET Abilities domain. There are 52 Abilities at the most specific level, 15 more general Abili- ties at the next level, and 4 Abilities at the most general level (Fleishman, Costanza, and Marshall-Mies, 1999). The abilities at one level are fully nested in the next higher level (see Appendix B). The process of developing and administering surveys to gather infor- mation related to these abilities as occupational requirements has proven complex (see Chapter 4). The 52 Ability descriptors were used to describe occupations by asking raters (job incumbents or occupational analysts) to rate both the required level of the ability and the importance of the ability for being able “to perform the job.” Neither the metrics for the scales nor the frame of reference is without criticism and could benefit from additional research and development. The same issues exist for the other descriptor domains as well. An additional difficulty is that the descriptors themselves

OCR for page 21
 THE CONTENT MODEL are defined only very briefly, and considerable expertise in the psychology of individual differences is required to understand some of them. Work Styles The current O*NET content model incorporates the taxonomy of Work Styles developed by the prototype team (see Appendix B). Work Styles refer to the personality or dispositional requirements of occupations. Similar to Abilities, Work Styles are seen as relatively stable attributes of individuals that affect their ability to perform different types of occupa- tions. Work Styles are included in both the prototype and current O*NET content models because decades of previous research have shown them to be important determinants of individual differences in occupational perfor- mance, particularly components of performance having to do with team- work, peer leadership, supervision, and management (Borman, Kubisiak, and Schneider, 1999). The term “work styles” was used, rather than “per- sonality,” because of the intent to emphasize personal characteristics that are occupationally related. The Work Styles taxonomy was developed by a careful review of two different literatures. The first consists of research focused on devel- oping taxonomies of personality dimensions and facets, such as the Big Five (Costa and McCrae, 1992). The second literature examined research that reflects the validity of personality characteristics for predicting differ- ent aspects of job performance (e.g., Barrick and Mount, 1991; Hough, 1992). The research team focusing on Work Styles (Borman, Kubisiak, and Schneider, 1999) synthesized the results of these two bodies of research and suggested a taxonomy of Work Styles incorporating 17 factors fully nested in 7 higher order factors. The taxonomy is shown in Appendix B. The Work Styles domain in O*NET is analogous to the Personal Qualities domain in the APDOT content model, which was to include information on an individual’s characteristic, habitual, or typical manner of thinking, feeling, behaving, or responding as it relates to work (e.g., sociability, integrity). The Work Styles taxonomy seems well grounded in previous research. However, identifying the most appropriate raters for what are essentially the personality requirements of occupations is perhaps more difficult to ac- complish (see Chapter 4). In addition, the question remains about whether the judgments about the level of each work style that is required for an occupation should be with reference to minimal, average, or high perfor- mance. There is also recent evidence that for some occupations the rela- tionships of certain personality factors to performance may be curvilinear (e.g., Benson and Campbell, 2007). The O*NET prototype specifications seem to imply only linear relationships, or at least monotonically increasing

OCR for page 21
0 A DATABASE FOR A CHANGING ECONOMY predicted performance as a function of increasing scores on the personality factor. Occupational Interests The current O*NET content model uses the taxonomy of Occupational Interests adopted in the prototype content model (Sager, 1997; Sager et al., 1999). The prototype development team adopted Holland’s typology of six types of occupational interests: realistic, investigative, artistic, social, enter- prising, and conventional (Holland and Gottfredson, 1976), or RIASEC. The typology is based on a long record of vocational interest research going back over three decades, and its construct validity is really not in doubt. However, unlike the other taxonomies included in the prototype model, the descriptors are not hierarchical. Work values The taxonomy of descriptors of Work Values in the current O*NET content model is nearly identical to that of the prototype content model. While interests are defined as the pattern of individual likes and dislikes for a large set of different activities, school subjects, and occupations, Work Values refer to an individual’s evaluation of the importance of certain char- acteristics of the work environment for determining their job satisfaction (Sager, 1999). After a consideration of the research literature on the assess- ment of work values, the prototype team adopted a taxonomy developed by the Minnesota Studies in Work Adjustment (Dawis and Lofquist, 1984). In that model, the individual’s self-rated importance for each value is assessed with the Minnesota Importance Questionnaire and the degree to which the job or occupation provides the opportunity to satisfy such values is rated by supervisors/managers using the Minnesota Job Description Questionnaire. The prototype Work Values taxonomy incorporated two levels, with 21 descriptors of Work Values grouped into 6 higher order factors. The current O*NET content model changes the names of a few of the higher order fac- tors, but the 21 Work Values are unchanged (see Appendix B). job-Relevant knowledge The taxonomy of Knowledge descriptors in the current O*NET con- tent model is unchanged from the prototype content model. The prototype development team defined occupation-relevant Knowledge as a collection of discrete but interrelated facts and information about a particular domain having to do with performance in an occupation. The team adhered to the findings from cognitive psychology (e.g., Chi, Glaser, and Rees, 1983;

OCR for page 21
 THE CONTENT MODEL Lesgold, 1984) that knowledge is not very useful as a determinant of per- formance until it is embedded in an organized and meaningful cognitive structure. The project team focusing on knowledge had to confront the difficult questions of how general or specific the Knowledge descriptors should be and how the Knowledge domains most critical for employment could be identified (Costanza, Fleishman, and Marshall-Mies, 1999). The team adopted the initial taxonomy of Knowledge descriptor from the Fleishman Job Analysis Survey (F-JAS), including 49 job-relevant Knowledge descrip- tors intended to cover (at a very general level) the knowledge requirements for all occupations in the labor force (Fleishman, 1992). The prototype development team compared the 49 F-JAS descriptors with the results of recent job analysis work at the Office of Personnel Management (Corts and Gowing, 1992). They also compared the descriptors to the Classifi- cation of Instructional Programs (Morgan, Hunt, and Carpenter, 1990) and to the preliminary findings of the National Occupational Information Coordinating Committee, which was attempting to develop a hierarchi- cal taxonomy of educational preparation content. After many rounds of expert panel judgments concerning the comprehensiveness and suitability of alternative taxonomies, the prototype development team settled on a 2-level taxonomy consisting of 33 basic descriptors grouped into 10 more general categories. Each of the 33 descriptors was given a short (one or two sentence) specification. The Knowledge descriptors are at a very high level of generality, which may suffice for some purposes, but not for others. For example, the sub- disciplines in psychology are very different, and a knowledge requirement labeled “psychology” may not be informative. Also, the knowledge require- ments for many technical services (e.g., plumber) and equipment operators (e.g., geothermal installation equipment operators, airplane pilots) do not appear to be included. Occupational Preparation The prototype content model included a taxonomy of the education, training, and other preparation required for occupations (Anderson, 1999). At the most general level, the descriptors were grouped into 7 categories: general education level, instructional program required (42 descriptors), subject-area education level (15 descriptors), licenses required (2 descrip- tors), requirement to obtain a license (6 descriptors), who requires the license (3 descriptors), and related work experience (4 descriptors). In the field test of the prototype content model (described in Chapter 4), job incumbents answered seven factual questions related to each category of information. Although these descriptors were relatively occupation specific,

OCR for page 21
 A DATABASE FOR A CHANGING ECONOMY a simple clustering of DOT titles, nor were they meant to represent simple aggregations of DOT titles. Thus, it is not unreasonable to conclude that, at the end of this development process, each occupational unit had its own identity—consisting of a title, definition, and task description—independent of, though partially informed by, DOT titles and task content. The next step was to collect two sets of data related to these occu- pational units. The first data set contained ratings made by occupational analysts using the written descriptions of occupational units. It became the first O*NET database that could be analyzed to establish proof of concept. The second data set contained ratings by job incumbents, obtained through a survey of incumbents in a targeted set of 80 occupational units; the re- sponse rate to the survey was low, however. Despite the limited response to the job incumbent survey, the two data sets were adequate to support a preliminary evaluation of the prototype content model. At least 5 analyst ratings were available for all 1,122 occu- pational units, and 30-35 of the 80 occupational units targeted in the survey were rated by at least 4 (mean = 10) incumbents (Peterson et al., 1997). The psychometric properties of both the analyst and incumbent ratings were generally encouraging. There were analyst versus incumbent differences in the expected direction (e.g., incumbents rated their jobs as more complex and demanding than did the analysts), but the differences were not star- tling. The various descriptor domains could discriminate among occupa- tions, and there was reasonable variation across the domain descriptors for a specific occupation. However, exploratory principal components analyses tended to yield a smaller number of factors within domains than the inves- tigators had hoped. For example the GWA descriptor covariances yielded only three factors corresponding roughly to data, people, and things—the same factors that had been used to rate the complexity of occupational titles in DOT. The researchers identified a similar three-factor structure for the Skills taxonomy, although they were labeled cognitive skills, organizational skills, and technical skills (Mumford, Peterson, and Childs, 1999). The limitations of these data sets were thoroughly discussed by Peterson et al. (1999). However, despite these limitations, the results pertaining to the reliability and discriminant validity of the domain descriptors were en- couraging and positive enough in the investigators’ judgment to establish proof of concept for the prototype content model. To address the low response to the job incumbent surveys, DOL com- missioned a working group to test and revise the questionnaires. The revi- sions made on the basis of this review led to the questionnaires currently used to collect O*NET data (see Chapter 4).

OCR for page 21
 THE CONTENT MODEL Subsequent Changes Subsequent to the field test and prototype revisions, the O*NET Center made several other changes to the content model and to the occupational classification system. These changes are discussed below, and changes to the occupational classification system are discussed in Chapter 3. Organizational Context, Work values, and Occupational Interests Although the Organizational Context domain is a part of the current content model (see Figure 1-1), the O*NET Center collects no data re- lated to this domain. It does not routinely collect data related to the Work Values and Occupational Interests domains as part of its primary data collection program, but it has used other methods to gather data related to these two domains. To generate updated information on Work Values, the O*NET Center engaged experts in vocational psychology to rate oc- cupations (Rounds et al., 2008a). The raters were provided with informa- tion on the occupation, including the O*NET written description, Skills, GWAs, Work Context, Work Styles, Education, and wages. They applied a method developed earlier (McCloy et al., 1999) to assign an occupational values profile to each occupation. To develop updated information on Oc- cupational Interests, the O*NET Center uses a similar approach, engaging trained occupational analysts. The analysts are provided with updated information on the occupation, including O*NET information, and apply a specified method to create an occupational interest profile for each oc- cupation in the database (Rounds et al., 2008b). Development of Detailed Work Activities Consistent with the recommendations of the Advisory Panel for the Dictionary of Occupational Titles (1993), the O*NET Center facilitated the development of a taxonomy of more occupationally specific descriptors that came to be called DWAs. This taxonomy is included in the current O*NET content model and information on DWAs for each occupation is available as part of the work activities information in the database viewed through the website O*NET OnLine. However, the O*NET Center does not include DWAs in the core content model data files it makes available for downloading, providing these descriptors along with other information as supplemental data (http://www.onetcenter.org/supplemental.html). In developing this new taxonomy, the O*NET Center specified a DWA as a descriptor of work activities that was intermediate in generality be- tween specific occupational tasks and the GWAs described above. The goal was to create descriptors general enough to permit cross-occupational

OCR for page 21
40 A DATABASE FOR A CHANGING ECONOMY matching, but specific enough to permit specific occupational differentiation (National Center for O*NET Development, 2003). To achieve this goal, the O*NET Center indicated that the DWAs should not overlap, or be redundant with, other content model descriptors, such as Abilities, Skills, and Knowledge, and should be found in more than one occupation. At the same time, the O*NET Center viewed any DWA found in more than 100 occupations as too broad to be useful and suggested it should be eliminated (p. 7). The O*NET Center sought to develop approximately 15-20 DWAs for each occupation. The O*NET Center’s development of DWAs built on the results of an earlier effort by the state of Oregon to develop a set of similar descrip- tors. That study reviewed the descriptors used in the Bureau of Labor Statistics’ Occupational Employment Statistics survey, the Classification of Instructional Programs, and the task information from the DOT to develop “skill statements.” The O*NET Center created additional statements and the resulting pool was edited for redundancy, clarity, cross-occupational relevance, and comprehensiveness of coverage to create an initial pool of 2,345 DWAs. In several rounds of review, subject-matter experts eliminated some DWAs from the pool and added others in order to meet the specifica- tions listed above. In addition, the developers edited the DWAs for clarity, understandability, and appropriateness. Through this process, the O*NET Center created 2,165 DWAs and assigned them to occupations. Some ex- amples are as follows: (1) adhere to government aviation regulations, (2) analyze dental data, (3) use airbrush techniques, and (4) apply appropriate physical restraints. In keeping with this design, each DWA was assigned to a single GWA, and the best fitting assignment of DWAs to GWAs has significant conver- gent and divergent validity. Consequently, DWAs are fully nested within GWAs, and users of O*NET OnLine who access either the details report or the custom report for an occupation see a list of “work activities” which displays DWAs nested within GWAs. Development of Tools and Technology Consistent with another advisory panel recommendation, the O*NET Center began in 2006 to develop another domain of more occupationally specific descriptors, known as Tools and Technologies or T2. This domain is included in the current O*NET content model (see Figure 1-1). The purpose of this effort is to incorporate additional contextual de- tail and occupation-specific descriptors into O*NET. The O*NET Center does not routinely collect data related to this domain through its main data collection program. Instead, the principal sources of T2 information are Internet-based searches and information gathered from various O*NET

OCR for page 21
41 THE CONTENT MODEL customers, supplemented by review of printed publications and review by subject-matter experts (Dierdorff, Drewes, and Norton, 2006). To date, the O*NET Center has gathered this type of information for 427 occupations, approximately half of those in the current database. Like the DWA data, the T2 data can be viewed through the O*NET OnLine website, but they are not included in the core O*NET database made available for download. T2 data are available for download as supplemental information. Lay Job Titles and Code Connector The O*NET Center collects a “lay titles file,” which is made available to users through a web-based application, the O*NET Code Connector. This application allows the user to match a particular job title to O*NET occupations. The Code Connector is designed especially to help work- force professionals determine the correct O*NET occupational codes for a specific job that the professional is helping an individual with. Although designed for the workforce development community, many human resource management professionals using O*NET find this feature valuable. O*NET’s Evolutionary History Recapitulated There is a direct evolutionary progression from DOT through APDOT to the O*NET prototype to the current database, O*NET 14.0, and there are many similarities between the DOT and O*NET content models. The DOT content model was built in stages from the basic task descriptions, to aptitude requirements, to education requirements, to physical requirements, to vocational interest requirements, to work complexity (in terms of data, people, and things), and to temperament requirements. Virtually every tax- onomy of descriptors in the current O*NET content model has an analog in the DOT, and most of these taxonomies (in both the DOT and O*NET) are cross-functional in nature. Although most elements of the O*NET con- tent model provide cross-functional information, the model also includes occupation-specific tasks, as well as two elements that are to a certain de- gree occupation specific—DWA and T2 descriptors. The major difference between DOT and O*NET is the specification for the unit of analysis. The 1991 DOT provided information on over 12,000 occupational titles, whereas O*NET provides information on 1,102 occupations. Another difference is that, by the 1980s, the cross-functional domains of occupational descriptors included in DOT had become rather dated. Research on the characteristics of jobs had progressed considerably beyond the DOT’s aptitude, interests, temperament, and education requirements and work activity taxonomies. As a consequence, the advisory panel pro- posed very comprehensive recommendations for what the “new DOT,”

OCR for page 21
42 A DATABASE FOR A CHANGING ECONOMY which became O*NET, should look like (Advisory Panel for the Dictionary of Occupational Titles, 1993). These recommendations do not appear to be based on a detailed analysis of specific user needs for occupational data. Instead, the APDOT content model may have represented an attempt to anticipate an unspecified user’s future needs, whatever they might be, by incorporating everything about jobs that had been studied, in the name of explaining occupational choices, occupational performance, and work/oc- cupational satisfaction. This included findings from the research literature in job analysis, performance measurement, cognitive psychology, and other fields. Despite the breadth of the APDOT content model, the O*NET pro- totype development project came very close to total success in making it operational. The postprototype development of the DWAs and T2 infor- mation was in direct response to the proposed APDOT content model. One advisory panel recommendation that has not been implemented is the development of performance standards and critical performance outcomes for each occupation. CONCLUSIONS AND RECOMMENDATIONS The construct validity of the taxonomies of descriptors is very uneven across the different domains included in the content model. For example, in the Abilities domain, the descriptors reflect a long history of psychological research on the nature and measurement of human abilities, but many of the descriptors in the Skills domain lack such an extensive research base. In addition, the different domains lack detailed and concrete specifications and often include descriptors with the same name, making it difficult to distinguish among them. “Problem solving” appears in the Abilities, Skills, Work Styles, and GWA domains. Although there may legitimately be both a problem-solving Ability and a problem-solving Skill, the content model does not clearly distinguish the different meanings of problem solving in these two different domains. This history of the research and development of the content model, as well as user experiences discussed in this report, raise serious questions about the completeness of some of the individual domains and the extent to which each uses the most appropriate level of generality or specificity for its descriptors. For example, in the Knowledge domain, the taxonomy includes a general descriptor for building and construction, but no more specialized Knowledge descriptors, such as plumbing or carpentry. In addi- tion, by design, the GWAs have a very high level of generality. The lack of more specific descriptors in the Knowledge, Skills, and Abilities domains (specifically, physical abilities) limits the usefulness of O*NET data for hu- man resource management and the determination of disabilities, and the

OCR for page 21
43 THE CONTENT MODEL incomplete development of the DWAs domain limits its potential use in retraining displaced workers. These limitations, and the recommendations to address them, appear in Part II. Research leading to revisions of elements of the content model, de- signed to address these weaknesses, could prove disruptive to current users of O*NET data. By maintaining the continuity of the core elements of the content model over the past decade, the O*NET Center has encouraged developers to incorporate O*NET data in a variety of useful tools and ap- plications (see Chapter 6) and supported longitudinal research on changes in the labor market. Nevertheless, the panel members agreed that research to revise the content model could ultimately lead to long-term benefits to O*NET users that would outweigh short-term disruptions. Both the Advi- sory Panel for the Dictionary of Occupational Titles (1993) and the O*NET prototype investigators (Peterson et al., 1999) viewed the O*NET content model as needing continuous improvement, yet there have been few changes since the mid-1990s. Recommendation: The Department of Labor should commission re- search to improve the content model, beginning with the skills and knowledge taxonomies. The goals of the research should be to reduce the redundancy of descriptors within and across taxonomies, distin- guish more clearly among the taxonomies, enhance completeness, and specify descriptor variables in concrete and meaningful terms. This research to refine the taxonomies would be aided by an analysis to identify which descriptors in each domain yield high standard errors across raters. Patterns of descriptors with high standard errors in a particular domain could provide insight into rater understanding of the rating scales and inform research and revisions of that domain. Collecting data aligned with a simpler, more rational content model would require shorter surveys, freeing resources to improve data collection along other dimensions, such as increasing the number of detailed occu- pations in the classification system or reducing the time interval between successive waves of data collection. To date, there has been little effort to use the hierarchical structure of the various domain taxonomies in the content model. Occupations are not “profiled” on the higher level variables represented by grouping descriptors into meaningful higher level categories. In addition, there has been little recent empirical research on the factor structure of the domains. Research is needed to determine whether improvements in the descriptor taxonomies and in the measurement scales would yield clearer and better differentiated factor structures. However, if the descriptors in a domain lack clear, dis-

OCR for page 21
44 A DATABASE FOR A CHANGING ECONOMY tinct, and meaningful specifications, raters will have trouble distinguishing among them, and fewer distinguishable factors would result. Recommendation: The Department of Labor should commission a se- ries of studies to develop occupational profiles at each hierarchical level of each taxonomy of descriptors. For example, in addition to profiling an occupation on all 52 Ability requirements, each occupation could be profiled on the smaller number of higher level Abilities formed by grouping specific abilities into higher order factors. Developing these higher level profiles would not involve collecting new data. It would involve considerable analysis of the existing O*NET database and much effort devoted to writing the specifications for each of the higher order factors. The panel thinks that if significant and steady progress were made on these recommendations, the usability and construct validity of the O*NET content model information would be significantly enhanced. The O*NET database would also gain in flexibility for meeting a greater number of user needs. Technical expertise is needed to assist DOL in carrying out these two research recommendations and the larger research agenda outlined in Chap- ter 10, as well as to determine what other research will strengthen the qual- ity of O*NET and enhance its use. Although the research agenda represents the panel’s best judgment about important research needs at this time, important new questions about O*NET will emerge in the coming years. There is a pressing need for a sustained program of research and evalua- tion of O*NET to guide ongoing improvements in the content model, oc- cupational classification system, data collection methods, and the usability of the database. Recommendation: The Department of Labor should establish and sup- port an external technical advisory board, comprised of senior scientists, to develop a research agenda for O*NET that will prioritize research suggestions from its members, the department, the O*NET Center, the user advisory board recommended below, and other sources. At a minimum, it should meet twice yearly, once to establish research priori- ties for the coming year and develop requests for proposals reflecting these priorities and once to review and rank proposals submitted by academic researchers or contractors. The panel anticipates that the technical advisory board will prioritize research suggestions according to criteria that include the extent to which the research suggestion complies with the corpus of scientific literature and

OCR for page 21
45 THE CONTENT MODEL current best practices, its cost, and the feasibility of implementation. The board will also weigh the research suggestion’s potential to improve the reliability and validity of the data in the content model, to reduce or extend the length of the questionnaires, to enhance collection of complete, accurate data, or to negatively affect longitudinal research based on O*NET or user- designed platforms. In Chapter 6, the committee proposes a user advisory board to com- municate users’ needs. DOL should not wait to initiate the research rec- ommended in this report until the technical and user advisory boards have been constituted and are fully functioning, but should proceed with continuous improvement initiatives using its traditional advisers until these boards can be established. In addition, DOL should establish mechanisms for ongoing communication between the user advisory board and the tech- nical advisory board. REFERENCES Advisory Panel for the Dictionary of Occupational Titles. (1993). The new DOT: A database of occupational titles for the twenty-first century (final report). Washington, DC: U.S. Department of Labor, Employment and Training Administration. Anderson, L.E. (1997). Education, training, experience, and licensure/certification: Evidence for the reliability and validity of measures. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, E.A. Fleishman, and K.Y. Levin (Eds.), O*NET final technical report. Volume I, II, and III. Available: http://www.eric.ed.gov/ [accessed July 2009]. Anderson, L.E. (1999). Occupational preparation: Education training, experience, and licensure/certification. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 91-104). Washington, DC: American Psychological Association. Arad, S., Hanson, M.A., and Schneider, R.J. (1999). Organizational context. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupa- tional information system for the 21st century: The development of O*NET (pp. 147- 174). Washington, DC: American Psychological Association. Barrick, M.R., and Mount, M.K. (1991). The big five personality dimensions and job perfor- mance. Personnel Psychology, 44, 1-26. Benson, M.J., and Campbell, J.P. (2007). To be, or not to be linear: An expanded representa- tion of personality and its relationship to leadership performance. International Journal of Selection and Assessment, 15, 232-249. Borman, W.C., Kubisiak, U.C., and Schneider, R.J. (1999). Work styles. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupa- tional information system for the 21st century: The development of O*NET (pp. 213- 226). Washington, DC: American Psychological Association. Cantor, R., and Kihlstrom, J.F. (1987). Personality and social intelligence. Englewood Cliffs, NJ: Prentice-Hall. Carroll, J.B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press. Chi, M.T., Glaser, R., and Rees, E. (1983). Expertise in problem solving. In R. Sternberg (Ed.), Advances in the psychology of human intelligence. Hillsdale, NJ: Erlbaum.

OCR for page 21
46 A DATABASE FOR A CHANGING ECONOMY Corts, D.B., and Gowing, M.K. (1992). Dimensions of effective behavior: Executives, manag- ers, and supervisors. Washington, DC: U.S. Office of Personnel Management, Personnel Research and Development. Costa, P.T., Jr., and McCrae, R.R. (1992). Revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI) professional manual. Odessa, FL: Psychologi- cal Assessment Resources. Costanza, D.P., Fleishman, E.A., and Marshall-Mies, J. (1999). Knowledges. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupa- tional information system for the 21st century: The development of O*NET (pp. 71-90). Washington, DC: American Psychological Association. Cottle, W.C. (1950). A factorial study of the multiphasic, Strong, Kuder, and Bell inventories using a population of adult males. Psychometrika, 15, 25-47. Cunningham, J.W. (1988). Occupation analysis inventory. In S. Gael (Ed.), The job analysis handbook for business, industry, and government (pp. 975-990). New York: Wiley. Dawis, R.V., and Lofquist, L.H. (1984). A psychological theory of work adjustment: An indi- vidual-differences model and its implications. Minneapolis: University of Minnesota. Decker, P.J., and Nathan, B.R. (1985). Behavior modeling training: Principles and applica- tions. New York: Praeger. Dierdorff, E.C., Drewes, D.W., and Norton, J.J. (2006). O*NET tools and technology: A synopsis of data development procedures. Raleigh, NC: National Center for O*NET Development. Available: http://www.onetcenter.org/reports/T2Development.html [ac- cessed July 2009]. Dvorak, B.J. (1947). The new USES General Aptitude Test Battery. Journal of Applied Psy- chology, 31, 372-376. Fleishman, E.A. (1992). Fleishman Job Analysis Survey (F-JAS). Bethesda, MD: Management Research Institute. Fleishman, E.A., and Reilly, M.E. (1992). Handbook of human abilities: Definitions, measure- ments, and job task requirements. Potomac, MD: Management Research Institute. Fleishman, E.A., Costanza, D.P., and Marshall-Mies, J. (1999). Abilities. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupa- tional information system for the 21st century: The development of O*NET (pp. 175- 195). Washington, DC: American Psychological Association. Geisinger, K.F., Spies, R.A., Carlson, J.F., and Plake, B.S. (Eds.). (2007). The seventeenth men- tal measurements yearbook. Lincoln, NE: Buros Institute of Mental Measurement. Goldstein, A.P., and Sorcher, M. (1974). Changing supervisory behavior. New York: Pergamon. Holland, J.L., and Gottfredson, G.D. (1976). Using a typology of persons and environments to explain careers: Some extensions and clarifications. Counseling Psychologist, 6, 20-29. Hough, L.M. (1992). The “big five” personality variables—construct confusion: Description versus prediction. Human Performance, 5, 139-155. Jeanneret, P.R., Borman, W.C., Kubisiak, U.C., and Hanson, M.A. (1999). Generalized work activities. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The develop- ment of O*NET (pp. 105-125). Washington, DC: American Psychological Association. Latham, G.P., and Saari, L.M., (1979). Application of social learning theory to training super- visors through behavioral modeling. Journal of Applied Psychology, 64, 239-246. Lesgold, A.M. (1984). Acquiring expertise. In J.R. Anderson and S.M. Kosslyn (Eds.), Tutori- als in learning and memory: Essays in honor of Gordon Bower. New York: Freeman. Lohman, D.F. (1994a). Component scores as residual variation (or why the intercept correlates best). Intelligence, 19, 1-11.

OCR for page 21
47 THE CONTENT MODEL Lohman, D.F. (1994b). Implication of cognitive psychology for ability testing: Three critical assumptions. In M.G. Rumsey, C.B. Walker, and J.H. Harris (Eds.), Personnel selection and classification (pp. 145-172). Hillsdale, NJ: Erlbaum. McCloy, R.A., Waugh, G., Medsker, G., Wall, J., Rivkin, D., and Lewis, P. (1999). Determin- ing the occupational reinforcer patterns for O*NET occupational units (Volumes I and II). Raleigh, NC: National Center for O*NET Development. Available: http://www. onetcenter.org/reports/ORP.html [accessed July 2009]. McCormick, E.J., Jeanneret, P.R., and Meecham, R.C. (1969). The development and back- ground of the Position Analysis Questionnaire (PAQ). (Tech. Rep. #5). Lafayette, IN: Purdue University, Occupational Research Center. McCormick, E.J., Jeanneret, P.R., and Mecham, R.C. (1972). A study of job characteristics and job dimensions as based on the Position Analysis Questionnaire (PAQ). Journal of Applied Psychology, 56, 347-368. Meehl, P.E., and Golden, R. (1982). Taxonomic methods. In P. Kendall and J. Butcher (Eds.), Handbook of research methods in clinical psychology (pp. 127-181). New York: Wiley. Morgan, R.L., Hunt, E.S., and Carpenter, J.M. (1990). Classification of instructional pro- grams. Washington, DC: National Center for Education Statistics. Mumford, M.D., Peterson, N.G., and Childs, R.A. (1999). Basic and cross-functional skills. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 49-69). Washington, DC: American Psychological Association. National Center for O*NET Development. (1998). Appendix D: The development of the Oc- cupational Information Network (O*NET) analyst database. Raleigh, NC: Author. Avail- able: http://www.onetcenter.org/reports/appendix_d.html [accessed November 2009]. National Center for O*NET Development. (2003). Summary report: Updating the detailed work activities. Raleigh, NC: Author. Available: http://www.onetcenter.org/dl_files/DWA_ summary.pdf [accessed July 2009]. National Research Council. (1980). Work, jobs, and occupations: A critical review of the Dictionary of Occupational Titles. A.R. Miller, D.J. Treiman, P.S. Cain, and P.A. Roos (Eds.).Washington, DC: National Academy Press. Available: http://www.nap.edu/catalog. php?record_id=92 [accessed May 2009]. Peterson, N.G., Mumford, M.D., Borman, W.C., Jeanneret, P.R., Fleishman, E.A., and Levin, K.Y. (Eds.). (1997). O*NET Final Technical Report (Vols. 1-3). Salt Lake City: Utah Department of Employment Security, on behalf of the U.S. Department of Labor Employ- ment and Training Administration. Available: http://www.eric.ed.gov/ [accessed March 2009]. Peterson, N.G., Mumford, M.D., Borman, W.C., Jeanneret, P.R., and Fleishman, E.A. (Eds.). (1999). An occupational information system for the 21st century: The development of O*NET. Washington, DC: American Psychological Association. Peterson, N.G., Mumford, M.D., Borman, W.C., Jeanneret, P.R., Fleishman, E.A., Levin, K.Y., Campion, M.A., Mayfield, M.S., Morgeson, F.P., Pearlman, K., Gowing, M.K., Lancaster, A.R., Silver, M.B., and Dye, D.M. (2001). Understanding work using the Occupational Information Network (O*NET): Implications for practice and research. Personnel Psy- chology, 54, 451-492. Rounds, J., Armstrong, P.I., Liao, H., Rivkin, D., and Lewis, P. (2008a). Second genera- tion occupational value profiles for the O*NET system: Summary. Raleigh, NC: National Center for O*NET Development. Available: http://www.onetcenter.org/reports/ SecondOVP_Summary.html [accessed July 2009]. Rounds, J., Armstrong, P.I., Liao, H., Lewis, P. and Rivkin, D. (2008b). Second generation occupational interest profiles for the O*NET system: Summary. Available: http://www. onetcenter.org/reports/SecondOIP_Summary.html [accessed July 2009].

OCR for page 21
48 A DATABASE FOR A CHANGING ECONOMY Sager, C.E. (1997). Occupational interests and values: Evidence for the reliability and valid- ity of the occupational interest codes and the values measures. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, E.A. Fleishman, and K.Y. Levin (Eds.), O*NET final technical report, volumes I, II and III. Salt Lake City: Utah State Department of Workforce Services on behalf of the U.S. Department of Labor, Employment and Training Administration. Available: http://www.eric.ed.gov/ [accessed July 2009]. Sager, C.E. (1999). Occupational interests and values. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 197-211). Washington, DC: American Psychological Association. Sager, C.E., Mumford, M.D., Baughman, W.A., and Childs, R.A. (1999). Occupation-specific descriptors: Approaches, procedures, and findings. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 227-235). Washington, DC: American Psychological Association. Strong, M.H., Jeanneret, P.R., McPhail, S.M., Blakley, B.R., and D’Egidio, E.L. (1999). Work context: Taxonomy and measurement of the work environment. In N.G. Peterson, M.D. Mumford, W.C. Borman, P.R. Jeanneret, and E.A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 127-145). Washington, DC: American Psychological Association. U.S. Department of Labor. (1991). Dictionary of occupational titles (4th ed.). Washington, DC: Author.