Developing New Norms

Developing new norms is a necessary (but not sufficient) condition for the continued use of GATB for referral or selection, and it would improve the utility of GATB even if limited to occupational counseling. Note, however, that development of new norms may not resolve the more fundamental problem of modest validities and high adverse impact.

Computerizing the GATB

The component of the improvement plan that focuses on computerizing the GATB could be strengthened by focusing more attention on findings from experiments in computerization of other large-scale testing programs. One question is whether there is reasonable likelihood that a computerized GATB will do any better at ameliorating the problems of modest validity and high adverse impact of the paper-and-pencil GATB. Research on test delivery mode and its effects on group differences is therefore worthwhile (although it is not clear that this is the intended focus of the research as described in the improvement plan). In addition, a potentially important line of research would be to explore ways in which computer technology could bring about significant changes in test format or in the measurement of performance. On this issue, closer coordination with research under way in other settings--on computerized versions of the Armed Services Vocational Aptitude Battery (ASVAB), Graduate Records Examinations, and professional licensure examinations under development or in use--would be the first step. But simply computerizing the existing GATB does not appear to be a reasonable investment, given the more basic problems that warrant in-depth attention.

GENERAL CONCLUSIONS AND RECOMMENDATIONS

The 1991 Civil Rights Act and the Americans with Disabilities Act afford employers and governmental agencies a challenge to engage in joint research, share research findings, and generally mount a concerted effort to study alternative selection procedures with higher predictive validity and less adverse impact than the GATB. Our basic recommendation is that the department explore ways to respond to this challenge.

Incremental changes to the GATB are not likely to achieve the dual objectives of improved predictive validity and reduced adverse impact. But alternative methods of selection, which have been the target of research in industrial psychology for some time, are difficult and costly to develop. Therefore, we recommend that the Department of Labor selectively continue its GATB improvement plan, concentrating attention and resources on those activities that are most likely to yield meaningful improvements to test validity, thereby significantly reducing adverse impact. The department should explore ways to expand the research program to allow for greater attention to alternative methods of assessment and to inclusion of additional sources of data, such as biographical information, in the selection



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 7
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Developing New Norms Developing new norms is a necessary (but not sufficient) condition for the continued use of GATB for referral or selection, and it would improve the utility of GATB even if limited to occupational counseling. Note, however, that development of new norms may not resolve the more fundamental problem of modest validities and high adverse impact. Computerizing the GATB The component of the improvement plan that focuses on computerizing the GATB could be strengthened by focusing more attention on findings from experiments in computerization of other large-scale testing programs. One question is whether there is reasonable likelihood that a computerized GATB will do any better at ameliorating the problems of modest validity and high adverse impact of the paper-and-pencil GATB. Research on test delivery mode and its effects on group differences is therefore worthwhile (although it is not clear that this is the intended focus of the research as described in the improvement plan). In addition, a potentially important line of research would be to explore ways in which computer technology could bring about significant changes in test format or in the measurement of performance. On this issue, closer coordination with research under way in other settings--on computerized versions of the Armed Services Vocational Aptitude Battery (ASVAB), Graduate Records Examinations, and professional licensure examinations under development or in use--would be the first step. But simply computerizing the existing GATB does not appear to be a reasonable investment, given the more basic problems that warrant in-depth attention. GENERAL CONCLUSIONS AND RECOMMENDATIONS The 1991 Civil Rights Act and the Americans with Disabilities Act afford employers and governmental agencies a challenge to engage in joint research, share research findings, and generally mount a concerted effort to study alternative selection procedures with higher predictive validity and less adverse impact than the GATB. Our basic recommendation is that the department explore ways to respond to this challenge. Incremental changes to the GATB are not likely to achieve the dual objectives of improved predictive validity and reduced adverse impact. But alternative methods of selection, which have been the target of research in industrial psychology for some time, are difficult and costly to develop. Therefore, we recommend that the Department of Labor selectively continue its GATB improvement plan, concentrating attention and resources on those activities that are most likely to yield meaningful improvements to test validity, thereby significantly reducing adverse impact. The department should explore ways to expand the research program to allow for greater attention to alternative methods of assessment and to inclusion of additional sources of data, such as biographical information, in the selection

OCR for page 7
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project process. Reliance on cognitive ability, as measured by the GATB (and similar tests), ignores other abilities (including those in the affective domain) that are important determinants of productive work performance. The board further recommends collaboration and coordination with other federal agencies involved in large-scale testing, to economize on costs and maximize the benefits of research. It would be difficult to overemphasize the potential value of joint efforts with the Department of Defense, the Department of Education, and the Office of Personnel Management (OPM), leading to exchanges of research findings and eventual improved coordination of policies. A first question, of course, is what success other agencies have experienced with alternative testing programs, such as the ASVAB and OPM employment tests. In order to build stronger bridges within and across federal agencies, the Department of Labor should consider taking the lead in working with private-sector employers, in particular, to assist them in understanding the limitations of the GATB and the potential utility of alternative selection methods available, even in the near term. Finally, we recommend that the department not view the GATB improvement project as a single, one-time effort; rather, it should conceptualize the program as part of a long-term and sustained effort involving development, experimentation, and validation of a variety of approaches to employment testing and selection. The board urges the department to broaden the GATB program into a more comprehensive effort, structured to enable close coordination with other projects (within the department as well as in other federal agencies) that bear on the definition and assessment of workplace literacy and occupational competency. The results of such an expanded scientific program would be of interest to the Department of Justice and the Equal Employment Opportunity Commission, because of their use of the Uniform Guidelines on Employee Selection Procedures, as well as to the federal agencies directly involved in testing research and development.