6— Twenty-First Century Measures for Twenty-First Century Work

Kenneth Pearlman

Introduction

This paper explores the possible roles of assessment in promoting and facilitating the development of skills required in the emerging world of work. The paper draws heavily on earlier work (Pearlman, 1993, 1994a,b) and develops the following basic arguments:

  1. ''Skill" is not a singular or unitary concept. There are a number of skill types with differing implications for how we conceptualize skill gaps and skill transferability.
  2. The meaning and value of work in the twenty-first century, especially in emerging high-performance organizations, will be increasingly dependent on emerging theories of job performance and on the meaning of the term "contextual performance." Contextual performance (as distinct from job-specific, technical or task performance) involves activities that, whether or not they are formally prescribed (part of a job description), are not specific to a particular job or area of work specialization but rather support the organizational, social, and psychological environment in which job-specific or technical or task performance occurs (see also Resnick, Chapter 11, this volume). This definition encompasses such activities as facilitation of peer, team, or unit performance; commitment to, promotion of, and generation of enthusiasm for organization or unit goals, practices, and policies; organizational "citizenship" or image—enhancing behavior; and various forms of prosocial, service—oriented, or organizational commitment behavior, which simultaneously implies the avoidance of behavior that would harm the organization or work unit. In other words, contextual performance is the "surround" of what we have traditionally thought of as "real" job performance.


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 136
--> 6— Twenty-First Century Measures for Twenty-First Century Work Kenneth Pearlman Introduction This paper explores the possible roles of assessment in promoting and facilitating the development of skills required in the emerging world of work. The paper draws heavily on earlier work (Pearlman, 1993, 1994a,b) and develops the following basic arguments: ''Skill" is not a singular or unitary concept. There are a number of skill types with differing implications for how we conceptualize skill gaps and skill transferability. The meaning and value of work in the twenty-first century, especially in emerging high-performance organizations, will be increasingly dependent on emerging theories of job performance and on the meaning of the term "contextual performance." Contextual performance (as distinct from job-specific, technical or task performance) involves activities that, whether or not they are formally prescribed (part of a job description), are not specific to a particular job or area of work specialization but rather support the organizational, social, and psychological environment in which job-specific or technical or task performance occurs (see also Resnick, Chapter 11, this volume). This definition encompasses such activities as facilitation of peer, team, or unit performance; commitment to, promotion of, and generation of enthusiasm for organization or unit goals, practices, and policies; organizational "citizenship" or image—enhancing behavior; and various forms of prosocial, service—oriented, or organizational commitment behavior, which simultaneously implies the avoidance of behavior that would harm the organization or work unit. In other words, contextual performance is the "surround" of what we have traditionally thought of as "real" job performance.

OCR for page 136
--> As the emerging workplace increasingly blurs distinctions among jobs, and even threatens the very concept of a "job" (Bridges, 1994; Pearlman, 1995), contextual performance becomes increasingly important to organizational survival—it is the organizational analog of the medium becoming the message. Cross-functional skills such as teamwork, communication, leadership, coaching/mentoring, conflict management, negotiating, customer service, decision making, managing resources, and information gathering and analysis are among the most important for effective contextual performance and employment stability and security for workers. Unfortunately, such skills are also the most problematic to define, assess, and develop, largely due to the absence of rigorous, comprehensive, work-analytic or construct-oriented research on such skills. There is as yet no systematic mapping of such skills to either the content or the context of the emerging workplace. The utility of programs and initiatives designed to shape and motivate the education, training, or development of the skills and knowledge needed in the emerging workplace depends on research and information that is incomplete in several key respects, such as the relative importance and the relative trainability of different types of skills. The above points present numerous challenges for assessment, the most urgent of which is the need for technically sound and widely deployable measures of cross-functional skills. On a system level, there is a need for better integration of the three conventional roles of assessment: diagnosis (enabling inferences regarding what has and has not been learned); prediction function (enabling inferences regarding future performance or behavior); and evaluation (enabling inferences regarding level, status, or progress of either individuals or institutions, which can influence the degree and direction of individual and institutional investment in skill, knowledge, and ability development). The remainder of this paper builds toward a vision of twenty-first century assessment that links four key themes: the changing demographic and organizational context of work, changing concepts of skill and competence, the need to map changing skills definitions to changing definitions of work, and the resultant (and formidable) challenges to assessment posed by these changes. To set the stage for this analysis, I begin with a brief review of the legislative, policy, and research contexts. Background ITEM: A report issued recently by the National Association of Secondary School Principals (1996) proposed that an employer dissatisfied with a recent high school graduate's job performance should be able to send the employee back to high school for additional training. This was one of several recommendations for sweeping structural changes in our education system included in their study, Breaking Ranks: Changing an American Institution. The report states that:

OCR for page 136
--> buyers can get [faulty products] fixed or replaced if they do not work properly. We understand the difference between inanimate objects and human beings, but that does not excuse high schools from the obligation to guarantee the quality of the young people they educate. The report said that personnel managers often complain about the deficiencies of 18 and 19 year olds who cannot do simple arithmetic and lack basic writing skills. Forcing the schools to guarantee their "product"—symbolized by the diploma—would encourage them to raise their standards for graduation. A dissatisfied employer could file a complaint with the high school. If the young person's education was deficient by entry-level standards, the high school would have to provide additional training or arrange for the graduate to attend classes at a local community college or in a special remedial program. ITEM: In a speech at an October 1993 Business for Social Responsibility Conference, the renowned Massachusetts Institute of Technology economist Lester Thurow expressed his ambivalence about Motorola University, widely regarded as one of the best-private sector training enterprises in the country: "On one level," he said, "I think it is the most fantastic thing any big company has ever done in America. On another level, it tells the local school system, 'You don't have to perform because even if you turn out a lousy product, we'll re-educate them later,' so you deliver a very bad signal to the system." Thurow cited the work of John Bishop at Cornell University, who has argued against the use of the Scholastic Aptitude Test (SAT) for making college admissions decisions in favor of wide-range achievement tests and Advanced Placement tests (Bishop, 1988). Bishop holds that use of the SAT by colleges and universities gives American high schools an excuse for failure by, in effect, sending the message that no matter how poor a job has been done by the K-12 education system, a student with a high IQ will be accepted anyway. Thurow argues, as does Bishop, that universities should insist that high IQ is not a substitute for performance and should let high schools know that they will be judged on performance, not whether they have high-IQ students. Thurow went on to make the point that it is only through well-conceived and well-designed systems, and systematic national efforts to change our philosophy and approach to education and training in this country that we will be able to begin to offer the type of "product guarantee" now increasingly being sought by employers and educators alike. ITEM: A recent report of a survey of New Jersey employers conducted for the New Jersey Business-Higher Education Forum (Van Horn, 1995:22-23) concluded that: there is wide agreement that more must be done to strengthen the bond between higher education and employers. Many employers say they are having difficulty hiring college graduates who have the skills they need. Employers are placing greater emphasis on teamwork, communications skills, problem-solving, and creative thinking. Business people often say that faculty members do not know enough about the world of work and are thus ill-prepared to teach necessary work

OCR for page 136
--> skills. … Many postsecondary institutions are responding to employers' concerns, but significant resistance persists. Leaders from higher education complain that employers do not adequately communicate what they need from college graduates. … Some academics resent what they regard as crass attempts to transform colleges and universities into "vocational schools" and to subvert the nobler purpose of education for its own sake. Suspicion and resentment are fueled because the pressure for greater higher education/workplace connections is coming primarily from business leaders and politicians rather than from the faculty and academic administrators. … Significant progress has been made on the crucial step of stipulating the knowledge, skills, and abilities vital to employers. A consensus is emerging on the basic elements. Efforts by industrial sectors to specify clusters of necessary skills are promising and could contribute lessons for other areas of the economy. Less progress has been made on assessing the performance of college students in acquiring the skills and abilities desired by employers. Designing such measures is difficult and also problematic because many of the desired skills and abilities are not taught in colleges and university classrooms and laboratories. … Questions still must be resolved about what should be taught in schools, what should be taught on the job, and what cannot be taught at all. These represent but a few of the most recent examples of a series of alarms that, beginning with the widely cited 1983 report A Nation at Risk: The Imperative for Educational Reform (National Commission on Excellence in Education, 1983), have been sounded over the past 15 years by various federal and state governments, special commissions, and task forces (Johnston and Packer, 1987; National Center on Education and the Economy, 1990; National Governors' Association, 1992; Secretary's Commission on Achieving Necessary Skills [SCANS], 1991). These reports reflect a widely perceived national crisis—the growing gap between the demands of a new American workplace driven by the emerging global economy and the supply of workers who possess the skills and knowledge to function effectively in such a workplace. It is a crisis that has called into question the efficacy of our education system—both K-12 and postsecondary education—as well as the way we assess the capabilities of individuals and the effectiveness of educational and training institutions. At the national level, this crisis has been the impetus for a number of federal initiatives over the past several years, each directed in one way or another toward remediation of America's perceived "skills gap" and the attainment of "a high-performance economy—one characterized by high skills, high wages, and full employment—in which every human being's resources are put to their best use" (SCANS, 1992). Secretary's Commission on Achieving Necessary Skills The Secretary's Commission on Achieving Necessary Skills (SCANS) was

OCR for page 136
--> chartered as one aspect of the "America 2000" strategy to achieve a set of national goals in education agreed to by the president and the nation's governors (SCANS, 1991, 1992). SCANS was asked to examine the demands of the workplace and the capacities of young people to meet those demands. Specifically, it was asked to define the skills needed for employment; to propose acceptable levels of proficiency in those skills, to suggest effective ways to assess proficiency; and to develop a dissemination strategy for the nation's schools, businesses, and homes. SCANS found that more than half of the nation's students leave school without the knowledge or foundation required to obtain and hold a good job. This is directly attributed to the workplace changes that have resulted from globalization and new technology growth, creating conditions that have fundamentally changed the terms for entry into the workplace. The centerpiece of the SCANS work is the delineation of a set of five "competencies" (productive use of resources, interpersonal skills, information, systems, and technology) and three categories of "foundation skills" (basic skills, thinking skills, and personal qualities) believed to lie at the heart of job performance and to reflect essential preparatory requirements for all students, both those going directly to work and those planning further education. The vision was for this work to constitute the blueprint that would drive needed changes in education (curriculum development, school organization, teacher training, and instructional materials and technology); the workplace (work-based learning, public-/private-sector training coalitions), and associated assessment systems (a national assessment system that helps students understand what they need to learn and that certifies achieved levels of the SCANS skills and competencies). An ongoing National Job Analysis Study is being conducted by American College Testing to pursue further validation and elaboration of the SCANS framework (American College Testing, 1994). Advisory Panel for the Dictionary of Occupational Titles The Advisory Panel for the Dictionary of Occupational Titles (APDOT) was chartered in 1990 to recommend to the secretary of labor strategies for collecting, analyzing, and disseminating occupational information to revise or replace the aging Dictionary. This initiative, like SCANS, grew out of the recognition that investment in people's skills and restructuring of workplaces into high-performance organizations were critical to our country's ability to remain competitive in today's global economy. Moreover, it was recognized that the development of a national occupational information system—one that provided a common language for describing both people (i.e., skills, abilities, and knowledge) and work (both content and context)—was in turn an essential requirement for meeting such needs. A major output of APDOT was a "content model" (see Appendix) that provides, in some detail, a framework or taxonomy, for the collection, assessment, and dissemination of occupational information that is both worker oriented (skills, knowledge, abilities) and work oriented (work tasks and outputs, as well

OCR for page 136
--> as the surrounding work, organizational, and labor market context) (APDOT, 1993). APDOT's vision was that this content model, when fully operationalized and deployed in the form of an automated database available in multiple media and formats, would form the basis for a national occupational information infrastructure, based on a common language of person- and work-related information, that would support a myriad of applications and work-force investment strategies involving skill and knowledge development, education, assessment, and vocational and career counseling. A prototype of this system, covering about 20 percent of the occupations in the U.S. economy, is nearly complete (American Institutes for Research, 1995). Goals 2000 and Skills Standards In 1994 the Clinton administration launched a collaborative effort between the U.S. Departments of Labor and Education to promote the development of a nationwide system of voluntary, industry-based skill standards. The Goals 2000: Educate America Act established a National Skills Standards Board to encourage, promote, and assist partnerships representing business, labor, educators, and others in the development of industry-related skill standards. This initiative was an outgrowth of the recognition that there is currently little systematic connection between the skills needed in the workplace and those imparted through education and training. The problem is exacerbated by the limited range of nationally recognized credentials. These problems result in increased hiring and training costs, restricted employment opportunities, lack of quality assurance, and a direct threat to the country's ability to compete effectively in the emerging and rapidly changing global economy. The stated purpose of a national system of industry-based skill standards is to identify the knowledge, skill, and ability levels needed for successful workplace performance. Such a system would also ensure a common, standardized way to classify and describe the skills needed for particular occupations and would utilize a variety of evaluation techniques to assess the skills possessed by individual workers. In so doing, it would aid communication among employers, educators, trainers, and workers regarding specific skill levels and needs and would ensure that workers have the portable skills required in today's dynamic economy. The development of broadly defined skill standards is viewed, in the words of then-Labor Secretary Robert Reich (1991), as "the cornerstone of [a] work force development system," which, when linked to educational standards, "will help create a seamless system of lifelong learning opportunities with certificates of mastery and competency that are accepted and recognized by employers" and which will "enhance America's ability to match skills and jobs." There are many anticipated advantages of a system of voluntary, industry-driven skill standards and certification. For industries it would be a

OCR for page 136
--> vehicle to inform training providers and prospective employees of the skills required for employment. For employers it would reduce the costs and legal risks associated with the assessment of job candidates and would make employment decisions more objective. For unions it would increase members' job security through access to competency-based training and certification. For workers, it would protect against dislocation, enable them to pursue career advancement, and enhance their ability to reenter the work force by having a work portfolio that is based on training to industry standards. It would help trainers and educators determine appropriate training services to offer. It would help government to protect the integrity of public expenditures by requiring that employment-related training meet industry standards where they exist. The fundamental challenge of Goals 2000 is how to most effectively ensure that today's and tomorrow's students and workers can acquire and maintain the skills necessary to be productive in a rapidly changing workplace and thereby contribute to the country's continued competitiveness in the emerging global economy. To accomplish this, Goals 2000 has adopted the strategy of building a national system of skill standards and skill certification intended to focus, motivate, and reward the skills that are to be learned on the job, in training, and in school. It has further adopted the strategies of addressing this issue using (1) job families or occupational clusters, as opposed to individual jobs or occupations, as the primary units of analysis within which skill standards and certification criteria would be developed and (2) skills that are sufficiently broad so as to be transferable across relatively wide ranges of work. School-to-Work Opportunities Act The 1994 School-to-Work Opportunities Act provides seed money to states and local partnerships of business, labor, government, education, and community organizations to develop school-to-work systems. Rather than creating a government program, the intent of this law is to establish an infrastructure for a national system based on existing models of school-to-work transition, such as career academies, youth apprenticeships, technical preparation, and cooperative education. The law allows states and their partners to work together in efforts at education reform, worker preparation, and economic development to create a system that will prepare young people for the high-wage, high-skill jobs of the emerging twenty-first century workplace. The legislation prescribes no single model of how to accomplish this; rather, it encourages states and their partnerships to design school-to-work systems that make the most sense for them. However, all such systems would share the common goal of providing every American student with (1) relevant education (allowing students to explore different careers and see what skills are required in different work environments), (2) skills (obtained from structured training and work-based learning experiences), and (3) valued credentials (establishing industry-standard benchmarks

OCR for page 136
--> and developing education and training standards that ensure that proper education is received for each career). In addition, the legislation sets forth three elements viewed as fundamental to efforts to create a national school-to-work system: school-based learning, work-based learning, and connecting activities. School-based learning involves curriculum restructuring so that students can see the relationship between academics and work. It includes such elements as project work in teams, teacher-employer interaction, and workplace-relevant interdisciplinary teaching. Work-based learning involves (1) employer-provided learning experiences to develop broad transferable skills and (2) the study of complex subject matter and important workplace skills (e.g., teamwork, problem solving, meeting commitments) in a real-life "hands-on" environment that provides feedback and mentorships. Connecting activities include program coordination and administration, school and business staff exchanges, and career counseling. The above initiatives address the challenge of how best to ensure the continued competitiveness of the American economy in the global marketplace through investment in a highly competent, knowledgeable, skilled, and flexible work force. Central to all of these initiatives are issues of skills, skill transferability, skill standards, changes needed in our systems of education and training, the nature of work and work performance, and the role of assessment. Significantly, each of these initiatives, implicitly or explicitly, recognizes the need for systems that integrate and leverage these elements in an optimal way. This implies the need for theories, conceptual models, or frameworks by which the nature of these elements and their interrelationships can be understood. Skills Skill is not a unitary concept. In fact, there is currently no single, generally accepted definition of "skill" in the professional or scientific literature. The term has been used to refer to a wide range of personal characteristics, traits, work preferences, broad aptitudes, basic abilities, generic competencies, specialized skills, and specialized knowledge, creating a contemporary tower of Babel in that the same terms are often used to denote different classes of skills and different terms are often used to denote the same classes of skills. This lack of an accepted vocabulary or a "common skills language" has been a major obstacle to developing appropriate strategies for addressing many critical skills issues, such as transferability, gaps, and the setting of standards. As one example of this problem, consider the lack of clarity that permeates many current discussions of the growing "skills gap" in this country, a term used to refer to the difference between the demand for and the supply of "work-ready" people. At times the skills gap involves or implies problems in fundamental aptitudes or abilities (e.g., seen as current work-force literacy and numeracy deficiencies). At times it involves problems in relatively generic or cross-functional

OCR for page 136
--> skills (e.g., seen as the need for greater degrees of interpersonal, teamwork, and decision-making skills among production workers who have been reorganized into semiautonomous teams). And at times the gap refers to problems in very specialized skills or knowledge (e.g., seen as the need for workers to become knowledgeable about and proficient in the use of new technology). More recently, it has become apparent that the gap of greatest concern to many employers is not about skills at all but about attitudes and personal qualities, such as integrity, reliability, and dependability (Cappelli, 1995; National Center for the Educational Quality of the Workforce, 1994; Zemsky, Chapter 3, this volume). The lack of clarity and specificity about the origins and meaning of the concept of a skills gap makes it difficult to propose appropriate corrective policies. Multiple Categories of Skills This problem was recognized in the content model developed by APDOT and shown in the Appendix. The model defines the specifications for a comprehensive occupational information infrastructure intended to have utility for multiple work-force investment initiatives. Rather than attempting to define skills in a unitary fashion, the content model proposes a set of five categories of skills-related information as a provisional framework for defining and understanding the full range of attributes commonly referred to as skills. These five categories are (1) aptitudes and abilities, (2) workplace basic skills, (3) cross-functional skills, (4) occupation-specific skills, and (5) occupation-specific knowledge. An additional category, personal qualities (defined as "an individual's characteristic, habitual, or typical manner of thinking, feeling, behaving, or responding with respect to oneself, others, situations, or events) was not considered by APDOT as part of the set of skills-related information descriptor categories because it refers to personality traits, values, and attitudes rather than skills per se. It is, however, potentially relevant to the present discussion. These skills-related information categories can be conceived of as a continuum that varies in the level of description and application. At one end of the continuum are the very general and relatively few aptitudes, abilities, and basic workplace skills, consisting of perhaps 15 to 30 elements, that are expected to be applicable to very wide ranges of jobs. At the other end of the continuum are the fairly specific and relatively many occupation-specific skills and occupation-specific knowledge, consisting of thousands of elements, that are expected to be applicable to relatively narrower ranges of jobs. The cross-functional skills represent a moderate level of generality and a moderate number of elements (perhaps 30 to 50), and encompass skills that are expected to be applicable to relatively wide ranges of jobs but that fall far short of the presumably near-universal applicability of such basic workplace skills as reading and writing. The human attributes denoted by such terms in fact differ in a number of important ways, as discussed below.

OCR for page 136
--> How Broadly or Narrowly Attributes Are Defined and Described Skills-related attributes vary in the degree of generality or specificity with which they are defined. For example, verbal ability represents a broader level of description than reading comprehension skill, which in turn represents a broader level of description than the ability to read and understand corporate real estate contracts. Similarly, skill at carpentry represents a broader level of definition than skill at inside finishing, which in turn represents a broader level of definition than skill at hanging interior doors. Applicability and Relevance (or Transferability) Across Different Jobs Different classes of attributes also differ in terms of their applicability and hence transferability (or portability) across jobs and job families. For example, a skill such as organizing and prioritizing work tasks is obviously applicable to a much wider range of jobs than such a skill as repairing watches. Basic skills such as reading and arithmetic computation are nearly universally applicable, whereas highly specialized knowledge, such as knowledge of the physical properties of solenoid magnets, is relevant to very few jobs. Modifiability or Trainability of the Attribute A skill or attribute's trainability refers to how well it lends itself to being learned quickly or to higher degrees, successfully transferred from the learning setting to the application setting, and retained over time. There is evidence that personal qualities (personality traits, values, and attitudes) and general aptitudes and abilities are trainable or modifiable to very limited degrees and only with substantial investments of time and effort (Ackerman and Humphreys, 1990:260; Gottfredson, 1986b:386-389; Humphreys, 1989). Basic workplace skills, cross-functional skills, and occupation-specific skills and knowledge, on the other hand, are, by definition, acquired and hence trainable but within limits defined by an individual's degree of general aptitudes and abilities and possession of the particular personal qualities that underlie or are related to the attribute to be trained (Hunter, 1986). In addition, the trainability of such skills varies as a function of their complexity, with simpler, less abstract knowledge and more routinized skills more readily trainable than more complex, more abstract knowledge and more dynamic or adaptive skills, such as those that require frequent or constant adaptation to changing situations or conditions (Ackerman, 1987; Kanfer and Ackerman, 1989). Applicability or Relevance of an Attribute to Different Settings or Purposes Not all types of attributes are equally applicable or relevant to different

OCR for page 136
--> purposes. For example, in the employment domain it is appropriate to use measures of aptitudes and abilities and/or basic workplace skills to select among inexperienced applicants (such as recent high school graduates) for jobs in which they will receive subsequent company training (e.g., entry-level computer programming). This is because the goal in such a situation is to identify individuals with the highest capacity or potential for mastering the training. It would make no sense to test programming skill or knowledge in such a situation. On the other hand, a test of programming skill or knowledge (i.e., a measure of developed skills or knowledge) would be appropriate for selecting among applicants with prior computer programming experience for higher-level programming positions. As another example, Bishop (1988) has argued that measures of developed skill and knowledge are more effective incentives for learning than are aptitude measures. In addition, various categories of skills-related information also differ in the degree to which they lend themselves to the setting of skill standards and the establishment of skill certification criteria. There is evidence to support the view that both meaningful definition of a particular attribute and meaningful determination of an appropriate required level of that attribute (e.g., to certify attainment of some established performance or knowledge level) are more feasible for fairly specialized skills or knowledge than for more general attributes, such as aptitudes and abilities, personal qualities, and cross-functional skills. The Manner in Which Attributes Can Be Measured There is a wide variety of methods and techniques by which human attributes can be measured or assessed. These include paper-and-pencil tests, physical ability tests, performance tests, assessment centers and job simulations, work samples, interviews, structured training and experience evaluations, trainability tests, personality tests, direct job performance observation and assessment, direct training performance assessment, education or training course grades, level of education, amount of experience or seniority, and work product assessment. Such techniques, however, are not equally applicable to different categories of worker attributes. For example, paper-and-pencil tests, which can provide reasonably good measures of certain kinds of specialized factual knowledge, do not lend themselves to measurement of various types of interpersonal skills (such as teamwork, leadership, and persuasiveness) or oral communication skills (such as speaking and listening skills). The Reliability and Validity with Which Attributes Can Be Measured Different categories of worker attributes also vary in the degree of precision or stability (reliability) and the degree of accuracy (validity) with which they can be measured. In general, aptitudes and abilities, basic workplace skills, and occupation-specific skills and knowledge can be measured with reasonably high degrees of reliability and validity when assessed using appropriate methods. In

OCR for page 136
--> When such standards are used for certification purposes, an additional (and very high-stakes) step is required, namely, to establish which level of performance will be required for certification (or qualification, mastery, or whatever delineations are to be used)—in other words, to decide "where to make the cut." Failure to carefully and appropriately set such skill certification levels creates the risk of either of two types of errors, which decision theorists call "false positives" and "false negatives," each having different—but inevitably dysfunctional—consequences for both individuals and organizations. False positives refer to situations in which standards are set too low, resulting in incompetent or insufficiently skilled people being considered qualified, or certified, and consequently assigned to jobs, training, or other activities for which they are not equipped. This is obviously harmful to the receiving organization but can also result in possible psychological and even physical damage to individuals so misclassified. More insidiously, the proliferation of such effects over time will inevitably lower both the perceived and the actual market value of the certification or qualification standard, producing a de facto system of credentialism rather than a truly competence-based system. False negatives refer to situations in which standards are set too high, resulting in competent or highly skilled people being disqualified or not certified. This results in potentially significant costs in terms of lost opportunities for organizations, especially if the skills in question are rare or in high demand. Such errors are additionally costly in terms of the cynicism and sense of unfairness they are likely to engender among the individuals who are so misclassified. To the above considerations can be added the even more sobering fact that there are no "objective" procedures for setting skill or performance standards—all such procedures ultimately rely (directly or indirectly) on human judgment. However, one can enhance the developmental rigor of the standards-setting process, and hence the ultimate credibility and utility of the resulting system, by paying careful attention to the numerous methodological and operational issues that affect this judgment process. These issues include such factors as the types and qualifications of judges (job incumbents, supervisors, outside subject matter experts, etc.); the referent population represented by the judges (i.e., the scope or domain of their expertise and the frame of reference implied by this domain—task specific, job specific, job-family-wide—which in turn affects both the breadth or narrowness of the standard and its content); the number of judges to be used; the training provided to judges; and the amount and type of stimulus material and instructional information provided to judges. The above generic process for deriving and setting skill and certification standards is relatively straightforward, if not necessarily easy. However, when applied to cross-functional skills, it becomes considerably less straightforward because the available evidence suggests that such level setting is much more tractable for highly specific skills or knowledge than for more general skills or attributes. This is because specific skills and knowledge typically have fairly

OCR for page 136
--> singular and objectively specifiable behavioral (i.e., performance) referents. For example, it is not hard to envision or find examples of definable and differentiable levels for such specific skills as word processing (through such indices as words-per-minute or error-rate levels) or for such specific areas of knowledge as emergency medical procedures (such as through degree of mastery, as measured by a job knowledge test, of the required component area knowledge, such as cardiopulmonary resuscitation or first aid). Nor is it hard to envision the establishment of performance standards for given jobs or tasks that can be meaningfully linked to such levels. In such cases the establishment of meaningful performance standards and associated skill or knowledge levels is possible because of their more or less objectively specifiable performance referents. However, this sort of endeavor is much more problematic when applied to cross-functional skills, which have no such unitary or easily specifiable behavioral or performance referents. That is, they refer to the ability to perform classes, groupings, or aggregates of more specific tasks, behaviors, or skills. This can be readily seen in Table 6-1, discussed earlier, in which information gathering was operationally defined differently for each of three jobs, depending on the underlying task or occupation-specific skill structure. There is thus no unitary performance referent for this skill that applies across all the jobs. As I have discussed in more detail elsewhere (Pearlman, 1994a), the prospects for our being able to establish meaningful proficiency levels (for people) that are linked to required performance levels (for work or jobs) are at best extremely challenging when cross-functional skills are dealt with. Without the type of rigorous underpinnings of work and job analysis discussed earlier in this paper, assessment standards for cross-functional skills are likely to be either lacking in critical measurement properties (reliability and validity) or not very useful for an integrated national system. What about the appropriateness and usefulness of a skill certification framework for cross-functional skills? First, it is important to recognize that the certification model has conventionally been used in a criterion-referenced measurement context. In this context, certification represents attainment or mastery of some specifiable—and specific—body of knowledge or some unambiguously definable—and defined—type, amount, or level of skill. Once certified, an individual is expected to have sufficient competence to carry out the basic tasks or functions implied by this certification in a manner that assures consumers or recipients of the associated product or service of a certain basic level of quality and conformance to generally accepted norms or standards of performance. As noted above, such a model readily lends itself to application with more narrowly defined, occupation-specific skills and knowledge, whereas its application to cross-functional skills, which can connote a multiplicity of performance or behavioral referents, is more problematic. For example, what would it mean to be certified in "decision making" or "problem solving" (two of the SCANS foundation skills) or "works with cultural diversity" or ''exercises leadership" (two of the SCANS "workplace competencies")? It would be easy to argue that one can have attained

OCR for page 136
--> a certain "level" of, or have been "certified" on, such cross-functional skills but still not know how to do anything (i.e., in any concrete, job-specific sense). It is always somewhat artificial and hence problematic to specify a given level of any skill or human attribute that will result in effective versus ineffective performance or similar delineations, such as work ready/not work ready, certified/noncertified, or novice/apprentice/journeyman/master/expert. This is because work or job performance, however it is measured or operationalized, virtually always represents an underlying continuum (i.e., from low to high or poor to outstanding), and most human attributes tend to be linearly related to job performance (meaning the more of the attribute possessed, the higher the performance). However, the problem is mitigated in the realm of more specialized skills and knowledge because of their more objective and specifiable behavioral or performance referents. This point also raises the more general concern that, by falsely implying a discontinuity of the attribute-performance relationship, skill certifications, if not appropriately positioned, or if based on insufficiently rigorous standards, could end up inadvertently promoting a minimum competency rather than a high-performance mindset, by signaling to students and worker trainees that development or acquisition of a certain amount of a skill or attribute is "enough." It could thus have the unintended effect of constraining rather than promoting both upward and lateral occupational mobility and discouraging rather than motivating the lifelong growth and development of valued skills. Thus, the challenges of operationalizing the measurement of cross-functional skills in terms of both instruments and systems that optimize their diagnostic, predictive, and evaluative utility are indeed formidable. The near-term success of such endeavors is constrained by a number of key limitations in our current research and knowledge base, particularly with respect to the mapping of skills and knowledge to both the content and the context of the emerging twenty-first century world of work. There are, nevertheless, some reasons for optimism, at least from a technical standpoint, that interim solutions can be developed on the basis of currently available methods, instruments, and technology, while the research needed to support more rigorous, longer-term solutions (some of which is already under way) is being completed. The key elements of such possibilities are outlined below. The bigger obstacles are likely to arise at the social and policy levels, where, as Thurow (1993) suggests, Americans resist the imposition of national standards in general, and local school boards in particular resist the imposition of national educational standards and assessments. A Vision Of Twenty-First Century Assessment There is little doubt that widely recognized, highly valued, and commonly understood standards of performance excellence are probably the single most

OCR for page 136
--> important external drivers of individuals' skill and ability development in both our educational institutions and our workplaces. There is also little doubt that assessment can play a crucial role in promoting such standards, provided it is directed to the right kinds of skills and deployed in a way that motivates, rather than disengages, the target populations. What might such a system look like? The hypotheses and conclusions drawn throughout this paper suggest that an optimal approach to assessment for the twenty-first century workplace might consist of the following principles, characteristics, or elements: A sound basis in empirical, systematic, and comprehensive work and job analysis that will have established the following: a clear mapping or linkage between worker-oriented attributes (skills-related information categories) and both work content and context across the occupational spectrum. well-founded taxonomies of both broader (cross-functional) and narrower (occupation-specific skill and knowledge) types of skills-related information. a system of empirically based job families or occupational clusters that indicate occupational interrelationships and career lattices in terms of work content, work context, and skills-related information and that show which skills are transferable and to what degree among occupational areas of subareas. Adoption of a tiered approach to skill and knowledge development, in which: the development of broader skill domains (basic workplace skills and cross-functional skills) are targeted for K-12 and vocational-technical institutions, and more advanced or specialized skill and knowledge domains for postsecondary education and specialized training institutions. contemporary principles of education reform (e.g., SCANS, 1992) are adopted in K-12 education to incorporate the integration of cross-functional skill development with traditional academics. performance-based assessments, in conjunction with locally developed and teacher-made assessments, are used within the K-12 domain as tools for diagnosis, student feedback, and development planning but not for skill certification or program evaluation. Establishment of a network of standardized, national assessment centers to measure important cross-functional skills; widely deployable and practically feasible through the use of video-based technology (e.g., video teleconferencing, videotaping for later evaluation by others), as well as emerging multimedia-PC-based Internet/Web-based technology; with such centers characterized by the following: an exclusively individual/developmental focus during grades 9 through 11, with opportunities for multiple "trials," detailed feedback and coaching, and multiple skill remediation mechanisms and support materials (e.g., lists of key experiences and learning activities targeted to particular cross-functional skills).

OCR for page 136
--> an evaluative focus at grade 12, with skill-level results used for high-stakes purposes (high school diploma, employment recommendations, postsecondary referral and placement decisions, institutional evaluation, etc.). The use of standardized measures of relatively occupation-specific skills and knowledge and academic achievement for evaluative purposes (standards setting and certification) in secondary and postsecondary education contexts, with results used for high-stakes purposes (high school diploma, college or university degree, employment recommendations, postsecondary referral and placement decisions, institutional evaluation, etc.). The above approach suggests some means by which the diagnostic, predictive, and evaluative/incentive functions of assessment might be usefully integrated in a manner that is consistent with our current research and knowledge base. It combines traditional measurement practices (e.g., paper-and-pencil tests) with newer practices and technology in ways that are both technically appropriate and suitable for the various purposes to which the different types of measures would be applied. Although much research is still needed to expand and refine our knowledge base with respect to the effective measurement of cross-functional skills, the approach suggested above could yield significant educational, social, and economic benefits while the needed research continues. Appendix: Content Model To help revitalize the American economy, the Advisory Panel for the Dictionary of Occupational Titles (APDOT) is recommending a national electronic database system that collects, produces, and maintains accurate, reliable, and valid information on occupations. The new Database of Occupational Titles (DOT) would serve as a national benchmark and provide a common language for all users of occupational information. The APDOT proposes the following content model as a framework for the new DOT. The model is intended to provide a coherent and integrated system that identifies the most important types of information about jobs and workers that APDOT believes should be considered for inclusion in the new DOT. APDOT views this content model as an initial point of departure and as being subject to further research and analysis as well as administrative decisions that will be made during implementation. APDOT expects that specifics of the descriptors will be designed and developed based on future intensive research and that descriptors will be included when supporting data meet professional standards for reliability, validity, and generalizability. This content model has been drawn from a thorough analysis of user survey results, public comments, and a wide-ranging review of research in such areas as job and skill analysis, human individual differences, and organization analysis. It embodies a view of occupational analysis that reflects the characteristics of

OCR for page 136
--> both occupations (through the use of "job-oriented" descriptors) and people (through the use of "worker-oriented" descriptors) as well as the broader labor market. This content model is not intended to imply that information or data regarding all of its components can or should be collected as part of a single job analysis instrument or even as part of the job analysis process. Some information may more appropriately lend itself to determination through other forms of research or data collection. For example, worker aptitude/ability patterns may be developed FIGURE 6-A The new DOT content model.

OCR for page 136
--> through aptitude test validation studies. In addition, some descriptors may be obtained through linkages with other databases and information sources. For example, the development of such descriptors as occupational outlook information, labor market trends and occupational demographics may be completed by linking with appropriate databases developed by sources outside the DOT. The content model is organized into four sections intended to represent the major elements of a systems model of work: Worker Attributes (Section I), reflecting input variables; Work Context (Section II), reflecting the organizational, social, and physical environment or system in which a job is performed; Work Content and Outcomes (Section III), reflecting output variables; and Labor Market Context (Section IV), reflecting the broader economic system of which all jobs are a part. The Content Model is shown schematically in Figure 6-A, the new DOT content model. Each section defines, provides examples of, and in some cases lists more specific elements of a set of descriptor categories. References Ackerman, P.L. 1987 Individual differences in skill learning: An integration of psychometric and information processing perspectives. Psychological Bulletin 102:3-27. Ackerman, P.L., and L.G. Humphreys 1990 Individual differences theory in industrial and organizational psychology. Pp. 223-282 in Handbook of Industrial and Organizational Psychology, 2d ed., vol. 1, M.D. Dunnette and L.M. Hough, eds. Palo Alto, CA: Consulting Psychologists Press. Advisory Panel for the Dictionary of Occupational Titles 1993 The New DOT: A Database of Occupational Titles for the Twenty-first Century. Washington, DC: U.S. Department of Labor. American College Testing 1991 Work Keys: National System for Teaching and Assessing Employability Skills. Iowa City: American College Testing. 1994 Performing a National Job Analysis Study: Overview of Methodology and Procedure. Iowa City: American College Testing. American Institutes for Research 1995 Development of Prototype Occupational Information Network (O*NET) Content Model. Salt Lake City: Utah Department of Employment Security. Bailey, T. 1990 Jobs of the future and skills they will require: New thinking on an old debate. American Educator 14(1):10-15,40-44. Baker, E.L., H.F. O'Neil, Jr., and R.L. Linn 1993 Policy and validity prospects for performance-based assessments. American Psychologist 48:1210-1218. Bishop, J.H. 1988 Employment testing and incentives to learn. Journal of Vocational Behavior 33:404-423. Borman, W.C., M.A. Hanson, S.H. Oppler, E.D. Pulakos, and L.A. White 1993 Role of early supervisory experience in supervisor performance. Journal of Applied Psychology 78:443-449.

OCR for page 136
--> Borman, W.C., and S.J. Motowidlo 1993 Contextual aspects of job performance. Pp. 71-98 in Personnel Selection in Organizations, N.Schmitt, W.C. Borman, and associates, eds. San Francisco: Jossey-Bass. Bridges, W. 1994 Job Shift: How to Prosper in a World Without Jobs. Reading, MA: Addison-Wesley. Burke, M.J., and K. Pearlman 1988 Recruiting, selecting, and matching people to jobs. Pp. 97-142 in Productivity in Organizations, J.P. Campbell, R.J. Campbell, and associates, eds. San Francisco: Jossey-Bass. Campbell, J.P., J.J. McHenry, and L.L. Wise 1990 Modeling job performance in a population of jobs. Personnel Psychology 43:313-333. Campbell, J. P., R.A. McCloy, S.H. Oppler, and C.E. Sager 1993 A theory of job performance. Pp. 35-70 in Personnel Selection in Organizations, N. Schmitt, W.C. Borman, and associates, eds. San Francisco: Jossey-Bass. Cappelli, P. 1992 College and the Workplace: How Should We Assess Student Performance? Philadelphia: National Center for the Educational Quality of the Workforce. 1995 Is the "Skills Gap" Really About Attitudes? California Management Review 37(4):108-124. Carnevale, A.P., L.J. Gainer, and A.S. Meltzer 1988 Workplace Basics: The Skills Employers Want. Washington, DC: American Society for Training and Development. Coates, J.F., J. Jarratt, and J.B. Mahaffie 1990 Future Work: Seven Critical Forces Reshaping Work and the Work Force in North America. San Francisco: Jossey-Bass. Gaugler, B.B., and G.C. Thornton III 1989 Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology 74:611-618. Gitomer, D.H. 1992 Cognitive Science Implications for Revising the DOT. In the unpublished report, Implications of Cognitive Psychology and Cognitive Task Analysis for the Revision of the Dictionary of Occupational Titles, American Psychological Association, Washington, DC. Gottfredson, L.S. 1981 A Skills Map: The General and Specific Competencies Required by Different Occupations. Progress report. Baltimore: Center for Social Organization of Schools, Johns Hopkins University. 1983 The Validity of an Occupational Classification Based on Job Competencies for Assessing Employability. EDRS No. ED 246 239. Baltimore: Center for Social Organization of Schools, Johns Hopkins University. 1986a Occupational aptitude patterns map: Development and implications for a theory of job aptitude requirements. Journal of Vocational Behavior 29:254-291. 1986b Societal consequences of the g factor in employment. Journal of Vocational Behavior 29:340-362. Grubb, N.W., et al. 1992 Betwixt and Between: Education, Skills, and Employment in Sub-baccalaureate Labor Markets. National Center for Research in Vocational Education Report. Berkeley: University of California. Hough, L.M. 1984 Development and evaluation of the "accomplishment record" method of selecting and promoting professionals. Journal of Applied Psychology 69:135-146.

OCR for page 136
--> Humphreys, L.G. 1989 Intelligence: Three kinds of instability and their consequences for policy. Pp. 193-216 in Intelligence: Measurement, Theory, and Public Policy, R.L. Linn, ed. Urbana: University of Illinois Press. Hunter, J.E. 1983 A causal analysis of cognitive ability, job knowledge, job performance, and supervisory ratings. Pp. 257-266 in Performance Measurement and Theory, F. Landy, S. Zedeck, and J. Cleveland, eds. Hillsdale, NJ: Lawrence Erlbaum Associates. 1986 Cognitive ability, cognitive aptitudes, job knowledge, and job performance. Journal of Vocational Behavior 29:340-362. Janz, T. 1989 The patterned behavior description interview: The best prophet of the future is the past. Pp. 158-168 in The Employment Interview: Theory, Research and Practice, R.W. Eder and G.R. Ferris, eds. Newbury Park, CA: Sage. Johnston, W.B., and A.E. Packer 1987 Workforce 2000. Indianapolis, IN: Hudson Institute. Kanfer, R., and P.L. Ackerman 1989 Motivation and cognitive abilities: An integrative-aptitude-treatment interaction approach to skill acquisition. Journal of Applied Psychology 74:657-690. Latham, G.P., L.M. Saari, E.D. Pursell, and M.A. Campion 1980 The situational interview. Journal of Applied Psychology 65:422-427. Meridian Corporation 1991 The Changing World of Work: Implications for the DOT Review Initiative. Report prepared for the Advisory Panel for the Dictionary of Occupational Titles. Washington, DC: U.S. Department of Labor. Michigan Department of Education 1989 The Michigan Employability Survey. Ann Arbor: Michigan Department of Education. Motowidlo, S.J., M.D. Dunnette, and G.W. Carter 1990 An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology 75:640-647. Motowidlo, S.J., G.W. Carter, M.D. Dunnette, N. Tippins, S. Werner, J.R. Burnett, and M.J. Vaughn 1992 Studies of the structured behavioral interview. Journal of Applied Psychology 77:571-587. National Association of Secondary School Principals 1996 Breaking Ranks: Changing an American Institution. Reston, VA: National Association of Secondary School Principals. National Center for the Educational Quality of the Workforce 1994 First Findings: The EQW National Employer Survey. Philadelphia: University of Pennsylvania. National Center on Education and the Economy 1990 America's Choice: High Skills or Low Wages. Rochester, NY: National Center on Education. National Commission on Excellence in Education 1983 A Nation at Risk: The Imperative for Educational Reform. Washington, DC: U.S. Government Printing Office. National Governors' Association 1992 Enhancing Skills for a Competitive World. Washington, DC: National Governors' Association. Naughton, T.J., and D. Outcalt 1988 Development and test of an occupational taxonomy based on job characteristics theory . Journal of Vocational Behavior 32:16-36.

OCR for page 136
--> Neidig, R.D., and P.J. Neidig 1984 Multiple assessment center exercises and job relatedness. Journal of Applied Psychology 69:182-186. Offerman, L.R., and M.K. Gowing 1993 Personnel selection in the future: The impact of changing demographics and the nature of work. Pp. 385-417 in Personnel Selection in Organizations , N. Schmitt, W.C. Borman, and associates, eds. San Francisco: Jossey-Bass. Olsten Forum on Human Resource Issues and Trends 1994 Skills for Success. Westbury, NY: Olsten Corp. O'Neill, H.F., Jr., K. Allred, and E.L. Baker 1992 Measurement of Workforce Readiness: Design of Prototype Measures . CSE Technical Report 344. Los Angeles: Center for the Study of Evaluation and National Center for Research on Evaluation, Standards, and Student Testing, University of California. Pearlman, K. 1980 Job families: A review and discussion of their implications for personnel selection. Psychological Bulletin 87:1-28. 1993 The Skills Standards Project and the Redesign of the Nation's Occupational Classification System. Unpublished paper prepared for the U.S. Department of Labor, contract no. 93-509, by Western Test Development Center, Utah Department of Employment Security , Salt Lake City. 1994a The Development of Occupational/Skill Clusters for Goals 2000: Suggested Approaches and Key Issues. Unpublished paper prepared for the U.S. Department of Labor under contract to DTI, 2361 Jefferson Davis Highway, Suite 500, Arlington, VA 22202. 1994b Job Families in the United States Employment Service: Review, Analysis, and Recommendations. Unpublished technical report prepared for the U.S. Department of Labor, contract no. 92-442, by Western Test Development Center, Utah Department of Employment Security, Salt Lake City. 1995 Is "Job" Dead? Implications of Changing Concepts of Work for I/O Science and Practice. Panel discussion conducted at the 10th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL, May. Reich, R.B. 1991 The Work of Nations: Preparing Ourselves for 21st Century Capitalism . New York: Alfred A. Knopf. Russell, C.J. 1987 Person characteristic versus role congruency explanations for assessment center ratings. Academy of Management Journal 30:817-826. Sackett, P.R., and G.F. Dreher 1982 Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology 67:401-410. 1984 Situation specificity of behavior and assessment center validation strategies: A rejoinder to Neidig & Neidig. Journal of Applied Psychology 69:187-190. Schmidt, F.L., and J.E. Hunter 1992 Development of a causal model: Processes determining job performance. Current Directions in Psychological Science 1:89-92. Schmidt, F.L., J.R. Caplan, S.E. Bemis, R. Decuir, L. Dunn, and L. Antone 1979 The Behavioral Consistency Method of Unassembled Examining. Washington, DC: Office of Personnel Management. Schmidt, F.L., J.E. Hunter, and A.N. Outerbridge 1986 Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology 71:432-439.

OCR for page 136
--> Secretary's Commission on Achieving Necessary Skills 1991 What Work Requires of Schools: A SCANS Report for America 2000 . Washington, DC: U.S. Department of Labor. 1992 Learning a Living: A Blueprint for High Performance: A SCANS Report for America 2000. Washington, DC: U.S. Department of Labor. Thornton, G.C., III, and W.C. Byham 1982 Assessment Centers and Managerial Performance. New York: Academic Press. Thurow, L. 1993 Unpublished speech presented at the Business for Social Responsibility Conference, Washington, DC, October 21. Tucker, M.S. 1993 Prepared Remarks on Title IV, Goals 2000: Educate America Act before the House Subcommittee on Elementary, Secondary, and Vocational Education. Rochester, NY: National Center on Education and the Economy. U.S. Department of Labor 1977 Dictionary of Occupational Titles, 4th ed. Washington, DC: U.S. Department of Labor. 1979a Guide for Occupational Exploration. Washington, DC: U.S. Department of Labor. 1979b Manual for the USES Geneal Aptitude Test Battery. Section II: Occupational Apitude Pattern Structure. Washington, DC: U.S. Department of Labor. 1979c Manual for the USES General Aptitude Test Battery. Section II-A: Development of the Occupational Aptitude Pattern Structure. Washington, DC: U.S. Department of Labor. 1981 Selected Characteristics of Occupations Defined in the Dictionary of Occupational Titles. Washington, DC: U.S. Department of Labor. 1991 Dictionary of Occupational Titles, 4th ed., rev. Washington, DC: U.S. Department of Labor. Van Horn, C. 1995 Enhancing the Connection Between Higher Education and the Workplace: A Survey of Employers. Denver: State Higher Education Executive Officers and the Education Commission of the States. Wetrogan, L.I., D.M. Olson, and H.M. Sperling 1983 A Systemic Model of Work Performance. Unpublished paper presented at the 91st annual meeting of the American Psychological Association, Anaheim, CA, August.