Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
Performance Assessment for the Workplace Performance Assessment for the Workplace VOLUME I Alexandra K. Wigdor and Bert F. Green, Jr., Editors Committee on the Performance of Military Personnel Commission on Behavioral and Social Sciences and Education National Research Council NATIONAL ACADEMY PRESS Washington, D.C. 1991
OCR for page R2
Performance Assessment for the Workplace NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Frank Press is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Stuart Bondurant is acting president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Frank Press and Dr. Robert M. White are chairman and vice chairman, respectively, of the National Research Council. This project was supported by the Office of the Assistant Secretary of Defense (Force Management and Personnel). Library of Congress Catalog Card No. 91-67187 International Standard Book Number 0-309-04538-X Additional copies of this report are available from: National Academy Press 2101 Constitution Avenue N.W. Washington, D.C. 20418 S375 Printed in the United States of America
OCR for page R3
Performance Assessment for the Workplace COMMITTEE ON THE PERFORMANCE OF MILITARY PERSONNEL BERT F. GREEN, JR. (Chair), Department of Psychology, Johns Hopkins University JERALD G. BACHMAN, Institute for Social Research, University of Michigan V. JON BENTZ, Elmhurst, Ill. LLOYD BOND, School of Education, University of North Carolina, Greensboro RICHARD V.L. COOPER, Ernst & Young, Chicago, Ill. RICHARD DANZIG, Latham & Watkins, Washington, D.C. FRANK J. LANDY, Department of Psychology, Pennsylvania State University ROBERT L. LINN, School of Education, University of Colorado, Boulder JOHN W. ROBERTS (USAF, ret.), San Antonio, Tex. DONALD B. RUBIN, Department of Statistics, Harvard University MADY W. SEGAL, Department of Sociology, University of Maryland RICHARD J. SHAVELSON, Graduate School of Education, University of California, Santa Barbara H.P. VAN COTT, Committee on Human Factors, National Research Council HAROLD WOOL,* Bethesda, Maryland ALEXANDRA K. WIGDOR, Study Director CAROLYN J. SAX, Administrative Assistant *Member, 1983-1985
OCR for page R4
Performance Assessment for the Workplace This page in the original is blank.
OCR for page R5
Performance Assessment for the Workplace Contents Preface vii Overview 1 1 Psychological Testing and the Challenge of the Criterion 15 2 Policy Goals and Testing 31 3 Improving Job Performance Criteria for Selection Tests 56 4 The Development of Job Performance Measures 73 5 The Testing of Personnel 103 6 Evaluating the Quality of Performance Measures: Reliability 116 7 Evaluating the Quality of Performance Measures: Content Representativeness 128 8 Evaluating the Quality of Performance Measures: Criterion-Related Validity Evidence 141 9 The Management of Human Resources 184 Coda: The JPM Project and Accession Policy 207 References 211 Appendix A Service Bibliographies 225 Appendix B Biographical Sketches 247 Index 251
OCR for page R6
Performance Assessment for the Workplace CONTENTS OF VOLUME II Implications of Cognitive Psychology for Measuring Job Performance Robert Glaser, Alan Lesgold, and Sherrie Gott Work Samples as Measures of Performance Frederick D. Smith Measuring Job Competency Bert F. Green, Jr., and Alexandra K. Wigdor The Evaluation of Alternative Measures of Job Performance Linda S. Gottfredson Range Restriction Adjustments in the Prediction of Military Job Performance Stephen B. Dunbar and Robert L. Linn Alternatives to the Validity Coefficient for Reporting the Test-Criterion Relationship Linda J. Allred Generalizability of Military Performance Measurements: I. Individual Performance Richard J. Shavelson Procedures for Eliciting and Using Judgments of the Value of Observed Behaviors on Military Job Performance Tests Richard M. Jaeger and Sallie Keller-McNulty Exploring Strategies for Clustering Military Occupations Paul R. Sackett
OCR for page R7
Performance Assessment for the Workplace Preface This is the sixth and final report of the Committee on the Performance of Military Personnel, which was established at the request of the Department of Defense (DoD) in 1983 to provide scientific oversight of the Joint-Service Job Performance Measurement/Enlistment Standards (JPM) Project. That pioneering research effort, now 10 years under way, proposed to develop robust measures of performance in entry-level military jobs so that, for the first time, military enlistment standards could be linked to performance on the job. The JPM Project was launched during a time when the transition to a system of voluntary military service was still a matter of public controversy and the issue of whether the Services could attract enough qualified people into the enlisted ranks a central policy concern. Its fruition is coming after years of great recruiting success, at a time when budgetary stringencies have recast the policy debate to address the cost-effectiveness of high-quality enlisted personnel. Because of the complexity of the JPM Project research goals, as well as the pressing need to establish the credibility of military selection and classification procedures after technical errors in computing enlistment test scores were discovered in 1980, DoD turned to the National Academy of Sciences to provide independent technical review of the research as it progressed. Created by act of Congress in 1863 as a private, nonprofit corporation, the National Academy of Sciences enjoys a unique position as an official, yet independent, adviser to the federal government. The Academy accomplishes its work through committees of experts established within the National Research Council. It has demonstrated consistently that it can call on
OCR for page R8
Performance Assessment for the Workplace the very best scientific and technical expertise nationwide. The independence and objectivity of NRC committees are carefully protected: the Academy does not reply to competitive requests for proposals; members of study committees serve without monetary compensation and are carefully screened for possible conflicts of interest; and the reports produced by committees are subject to thoroughgoing review before being transmitted to sponsors. The Committee on the Performance of Military Personnel was selected in accordance with this tradition of excellence and independence. Housed within the Commission on Behavioral and Social Sciences and Education, its membership was drawn from the scientific and policy fields of greatest relevance to the JPM Project: psychometrics, performance measurement, industrial and organizational psychology, research design and statistics, military sociology, economics, human resource management, and military manpower policy. The committee included both scholars and practitioners—those whose scientific sophistication represents the state of the art, as well as those who understand firsthand the exigencies of real-world applications. General John Roberts (USAF, Ret.), former head of Air Training Command, brought to committee deliberations experience at the highest policy levels with military selection, classification, and training issues. John Bentz offered the insights gleaned from 35 years as director of personnel research for a major nationwide retailing company. Biographies of all committee members and staff appear in Appendix B. The first phase of the JPM Project entailed the development of measures of performance for a sample of military jobs—jobs like jet engine mechanic, avionics communication specialist, infantryman, and medical technician. The goal was to determine how well the Armed Services Vocational Aptitude Battery (ASVAB), the test that is used throughout the military to assess enlistment eligibility, predicts performance on these high-fidelity measures of job performance. The committee 's advisory activities in the early years centered on evaluation of the research design and the development of measurement instruments (e.g., hands-on tests, written job knowledge tests, supervisor ratings) and issues in hands-on test administration. Later the focus turned to evaluation of the quality of the measures and analysis of the performance data to understand the relationships among the various new performance measures and to ASVAB scores and other input characteristics. (The second phase of the project, now well under way, is exploring the use of cost/performance trade-off models for setting enlistment standards. A new Committee on Military Enlistment Standards has been established by the National Academy of Sciences to advise on that work.) In order to enhance its understanding of the job performance measurement research and to place the project in proper context, members of the Committee on the Performance of Military Personnel also learned about military entrance processing, entry-level jobs in the military, technical training,
OCR for page R9
Performance Assessment for the Workplace and the general outlines of how entrance standards are currently set. Committee and staff members made a series of site visits to Army, Navy, Marine Corps, and Air Force bases to see enlisted personnel at work, to talk to their supervisors about the content of entry-level jobs, and to observe the administration of hands-on tests. They experienced at firsthand the operation of military entrance processing stations, visited training facilities, and also made a number of visits to military personnel research laboratories to gather information and discuss the progress of the research. To facilitate the exchange of ideas, over the years the committee invited JPM Project scientists as well as other experts to explore solutions to specific technical problems in a series of workshops. And, to supplement its own activities, the committee has called on outside experts to prepare background papers on various aspects of the issues involved. A selection of these papers is presented in the companion volume to this report. This book is a distillation of the experiences and insights and many things learned by the committee over the years since its founding in 1983 and, we hope, conveys at least a modicum of what the Service research scientists learned as well. It represents many years of faithful attention by committee members to the task at hand, frequent delight in the collegial nature of the enterprise, and a lot of sheer hard work. In the course of preparing the report, each of the members took an active role in drafting sections or chapters, providing data, leading discussions, and reading and commenting on successive drafts. Robert Linn bore primary responsibility for drafting the chapter on criterion-related validity. Richard Shavelson made major contributions to several chapters. Frank Landy and Lloyd Bond organized team efforts on the development of performance measures and standardization issues, respectively. John Roberts drafted sections on manpower policy issues. Our work has benefited over the years from the contributions of many inside and outside the military. We were received with unfailing courtesy and openness at numerous military installations and provided with as much information as we could think to request. We developed a productive working relationship with the Job Performance Measurement Working Group, which coordinates the research effort, under the successive leadership of Col. Charles R. Curran, Lt. Col. Burt Itoga, Lt. Col. Dickie Harris, and Lt. Col. Thomas Ulrich. David J. Armor, former principal deputy to the Assistant Secretary of Defense (Force Management and Personnel) and currently a member of the follow-on Committee on Military Enlistment Standards, stimulated many a good discussion of technical challenges. Our workshops were enlivened by the presentations of many of the military researchers and their contractors, among others Milton H. Maier, Lawrence Hanser, Gerald Laabs, Col. Rodger D. Ballentine, Paul W. Mayberry, John Campbell, Leonard P. Kroeker, and Jerry W. Hedge.
OCR for page R10
Performance Assessment for the Workplace In addition to the authors of papers that appear in the accompanying volume, Brian K. Waters, Janice H. Laurence, and Wayne J. Camara contributed a separately published paper on personnel enlistment and classification procedures in the U.S. military. Stephen B. Dunbar 's analysis of JPM data forms the basis of the discussion of group-to-group differences in prediction systems presented in Chapter 8. Gail P. Baxter and James R. Valadez of the University of California at Santa Barbara assisted Richard J. Shavelson with background materials. Carolyn Sax, the committee's administrative assistant, was indispensable in keeping track of everything from errant committee members to errant commas. We could not have managed half so well without her quiet competence. Thanks are due as well to her extremely able predecessor, Diane Goldman, to research associates Hilda Wing and Anne S. Mavor, and to Christine McShane, who has edited all of the committee's reports and in the process become something of an expert on job performance measurement in the military. A special note of thanks is due to W.S. Sellman, director for accession policy in the Office of the Assistant Secretary of Defense (Force Management and Personnel), who had the tenacity needed to see the JPM Project through to fruition and a great enough respect for science to subject himself to the ministrations of an independent advisory committee. Bert F. Green, Jr., Chair Alexandra K. Wigdor, Study Director Committee on the Performance of Military Personnel
OCR for page R11
Performance Assessment for the Workplace Performance Assessment for the Workplace
OCR for page R12
Performance Assessment for the Workplace This page in the original is blank.