APPENDIX

U.S. EMPLOYMENT SERVICE'S WORKPLAN FOR THE GENERAL APTITUDE TEST BATTERY (GATB) IMPROVEMENT PROJECT



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project APPENDIX U.S. EMPLOYMENT SERVICE'S WORKPLAN FOR THE GENERAL APTITUDE TEST BATTERY (GATB) IMPROVEMENT PROJECT

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project U.S. Department of Labor Assistant Secretary for Employment and Training Washington, D.C. 20210 NOV 8 1993 Dr. Michael J. Feuer Director Board on Testing and Assessment National Research Council, HA-178 Commission on Behavioral and Social Sciences and Education 2101 Constitution Avenue, N.W. Washington, DC 20418 Dear Dr. Feuer: This letter transmits the United States Employment Service's Workplan for the General Aptitude Test Battery (GATB) Improvement Project to the National Academy of Science's (NAS) Board on Testing and Assessment (Board). As we discussed at the Board Meeting on September 23, 1993, the Department of Labor (the Department) wants to assure that the GATB Improvement Project addresses the major NAS concerns expressed in the 1989 report Fairness in Employment Testing. To accomplish this end, the Department is seeking the Board's review of the technical adequacy of the Workplan and the work products that are under development in the United States Employment Service. The Department staff were pleased with the initial meeting of the Board and look forward to working closely with you and the panel of experts. To facilitate the Board's review process, the Department staff have identified specific questions that they would like to see addressed as the Board reviews the Workplan. We believe that structuring the review process in this manner will help us complete the GATB Improvement Project in an appropriate and timely manner. Questions for the Board Does the workplan adequately address the major technical issues raised in the NAS report? Are there any significant gaps/flaws in the basic approach reflected in the workplan?

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project In particular, does the workplan reflect an adequate research response to recommendations in the 1989 NAS report in the key areas of reducing score differences and adverse impact, and testing persons with disabilities? Thank you for your time and attention, we look forward to the Board 's response, if possible by the end of January 1994. Sincerely, Raymond J. Uhalde Acting Deputy Assistant Secretary Employment and Training Administration Enclosure

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project This page in the original is blank.

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project ASSESSMENT RESEARCH AND DEVELOPMENT PROGRAM GATB IMPROVEMENT PROJECT Midterm Status Summary Background In 1987, the National Academy of Sciences (NAS) was commissioned by the U.S. Department of Labor (DOL) to conduct a thorough evaluation of the General Aptitude Test Battery (GATB) and the implications of its use as a tool for employment counselling and referral by the U.S. Employment Service. Following the publication of the NAS findings in 1989 (Hartigan & Wigdor, 1989), DOL announced a proposed policy on GATB use in July of 1990. This proposed policy, published in the Federal Register (July 24, 1990), called for suspending GATB use for employment referral purposes during a research period, although it could continue to be used for counselling. The Federal Register notice also outlined a GATB research plan, which called for a comprehensive program to improve validation methods and accelerate the validation process, develop additional forms of the GATB, reduce susceptibility to coaching and reduce GATB speededness, develop standards for physical test security, develop new norms, computerize the GATB, and provide guidance on proper use of tests. A final decision regarding the DOL 's policy on the use of the GATB was held in abeyance during Congressional debate on the Civil Rights Act of 1991. During this period, however, steps were taken to implement the proposed research plan. In December of 1991, the DOL's policy decision regarding the use of the GATB was announced in the Federal Register (December 18, 1991). The GATB could continue to be used for counselling as well as employment referral purposes, with the caveat that GATB users were responsible for compliance with all applicable Federal and State statutes and regulations. However, DOL policy also prohibited use of the GATB as the sole factor in employment referral, and, consistent with the Civil Rights Act of 1991, the use of within-group scoring was precluded. Many questions remained, however, about the GATB. The GATB research plan outlined in the Federal Register, the NAS report, and calls by employers for the continued use of the GATB for employment referral all underscored a need for DOL to: commission reviews and research by independent experts concerning job clustering and score weighting, reducing the number of items in some sections of the GATB, and the effect of correcting for guessing on the test; develop new forms of the GATB (Forms E and F) that will better provide for retesting of applicants and will help to ensure the integrity of scores; October 22, 1993

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project validate the GATB for occupations that are new to the U.S. economy or that have changed significantly during the last ten years; construct a computerized adaptive version of the GATB; enhance guidelines for maintaining the security of the GATB and specifying the individuals and organizations authorized to use the test; and research assessment methods that may reduce score differences among racial/ethnic groups. Since implementation of the GATB Improvement plan, many research projects have been started and are currently on going. This workplan reflects the results of a mid-term review of progress, preliminary findings, and next steps generated from work efforts completed to date. The workplan is divided into six areas: within each area, projects currently underway that address the research area are summarized. GATB Validity Research Reduce GATB Score Differences among Racial/Ethnic Groups Maintain the GATB Develop New Norms Computerized GATB Provide Guidance on Proper Use of the GATB

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project GATB IMPROVEMENT PLAN - Midterm Status Summary Area 1: GATB Validity Research. The NAS report noted an unexpected decline in GATB validities in recent studies and somewhat lower validities for blacks. In the area of validity research, the NAS recommended updating documentation of GATB validation research, continuing validation research to assure adequate coverage, particularly for new and changed occupations, updating procedures for future validation research and, further examination of optimal methods for clustering jobs. The objective of Area 1 GATB Validity Research is to improve the validity of the GATB, address the NAS concerns regarding minorities, persons with disabilities and older workers and develop optimal clustering and weighting. Series of Issue Papers. To examine issues and options for addressing GATB validity, expert papers have been commissioned which will contribute to the overall success of the improvement plan and the results will be incorporated into related research projects. Issues Paper on Job Clustering and Score Weighting. There are over 12,000 jobs in the U.S. economy, and it is practically infeasible and scientifically unnecessary to develop different assessment tools for predicting success in each. Instead, instruments can be developed to predict success in groups of jobs. The question is how job(s) should be grouped. The immediate focus of this paper is to respond to questions posed by the NAS regarding the basis for the current aggregation of jobs into five job families and the associated score weightings of the GATB. This paper will also address potential methods for clustering jobs for new assessments under consideration. Dr. Kenneth Pearlman, a member of the Advisory Panel for the Dictionary of Occupational Titles (APDOT) and expert in the area of job clustering, has completed a review of all previously-conducted research, respond specifically to related NAS concerns, and made recommendations as to optimal method(s) for grouping jobs for ETA. End Product: (1) issues aper (2) Analysis of paper and recommendations for integration into workplan

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Advisory Panel for the Dictionary of Occupational Titles: Analysis of Assessment Implications of the APDOT Report on Use of GATB. The Advisory Panel for the Dictionary of Occupational Titles submitted a final report to the Secretary of Labor June 1993. That report has implications for the way in which the GATB and other assessment instruments are developed or used. The APDOT report is being analyzed to identify information which may improve or enhance assessment research efforts and to determine if the changes it recommended in the DOT (e.g., in job analysis methodology, jobs clustering, additional or different worker characteristics such as skills) necessitate further changes in assessment development and use. A plan to take appropriate action will be developed, if warranted by the analysis. End Product: Analysis of APDOT report and recommendations for integration into workplan Issues Paper on Assessing Job Performance. Dr. Frank Landy, a leading expert in job performance measurement, has completed a paper that summarizes the cumulative research in using supervisory ratings of job performance on the job in validation studies. Dr. Landy's findings support continued use of the supervisory rating as criterion measures in validation research. His recommendations have been implemented in the Validation Methods Research project discussed below. End Product: Issue paper Validation Studies. Additional validation studies for various occupations are underway that contribute to the overall validation program of the GATB. These studies together will form an overall report of GATB Improvement Program validation work. Validity Research of Most and Least Complex Jobs. No one validation study can definitively determine the usefulness of the GATB; instead, trends across numerous validation studies must be examined. The NAS report stated that the GATB is valid for most jobs, but was unwilling to assert universal validity. Most of the validation studies to date have been conducted on jobs of moderate complexity. These two studies concentrate on the most and least complex jobs. The expected outcome of these two studies is to improve the coverage of the GATB validity data base by providing more data on which to determine the validity of the GATB for low

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project and high complexity jobs. In addition, this research will address the extent to which the results of research conducted on one set of jobs (i.e., moderate complexity jobs) can be generalized to all jobs, and will thus explore the parameters of validity occupationally. End Product: Technical report Validity Research on the Occupation of Paralegal. This project examined the validity of the GATB for predicting success in one of the fastest growing occupations in the U.S. workforce. This study, begun prior to the NAS report, provides direct evidence regarding GATB validity for this important occupation and demonstrates improved methodological procedures that are reflected in other GATB validity research projects in the plan. End Product: Technical report Validity Research on the Occupation of Word Processor. As with the Paralegal project, the purpose of this project is to examine the validity of the GATB for predicting success in one of the occupations in the U.S. workforce that has changed and grown most significantly during the last decade. This study, begun prior to the NAS report, provides direct evidence regarding GATB validity for this important occupation and demonstrates improved methodological procedures implemented in other GATB validity research projects in the plan. End Product: Technical report New, Emerging and Changed Occupations Validation Studies. The validity of all tests needs to be examined continuously as jobs in the U.S. economy change. This project examines the validity of the GATB for occupations that are either new or that have changed significantly during the last ten years according to job analysis data collected by the Occupational Analysis system. This study will provide direct evidence for the validity of the GATB for important occupations. End Product: Technical report Validation Methods Research. This study has the following objectives: to examine the decline in GATB validities; implement improved methodological

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project techniques, especially criterion measurement; obtain a better estimations of the GATB validity; expand the GATB database and develop staff expertise for developing criterion measures. This study (as well as others in the GATB improvement project) will analyze minority subsamples and other special groups. End Product: Technical report Other Validity Research. Validation Guide. An internal guide is being developed for use in the nationwide Assessment Research and Development Program in staff training and as a reference tool to ensure that future validation studies are conducted following scientifically accepted standardized methodological procedures. It will set the standards and procedures for developing research designs, identifying research samples, collecting data, conducting statistical analyses, and developing technical reports and journal articles to disseminate the results of our research. End Product: Guide on validation research procedures GATB Manual Section III. Development. The NAS report recommended better documentation of GATB research. As GATB validation research is conducted, documentation is being updated to reflect the results of that research. Relevant research by others will be documented. This documentation will be rigorous and also user-friendly. GATB Manual Section III. Development will summarize this type of information regarding the GATB. The new documentation will assist test users in understanding GATB while supporting its use under professional standards. End Product: Revised documentation containing virtually all relevant research on GATB suitable for management and technical review In addition to the above validation projects, other projects described elsewhere in this workplan are contributing to the validity research: the speed/power study, the Test Development Guide; the paper on administration of the GATB to wheelchair users, a paper by Dr. Ragosta on Testing Persons with Disabilities; data collection test training; and improving test aesthetics.

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Area 2: Reduce GATB Score Differences among Racial/Ethnic Groups. The objective of this area is to examine procedures to reduce GATB score differences among racial/ethnic groups. While some improvements can be made to the GATB, it may not be possible to modify the GATB structure, design or scoring to significantly reduce GATB score differences. A more promising approach for narrowing gaps in score differences appears to lie in applying additional job-related assessment methods such as biographical information and measurements of work values and attitudes to the assessment process. It may also be more promising to use the GATB as part of a comprehensive assessment program in which the GATB is only one of a series of instruments used for assessment purposes. Literature Review and Options Paper for Reducing Score Differences. This project will review and identify potential concepts for reducing adverse impact in assessment systems. The paper will address modifications and provide options which hold promise for reducing score differences in the GATB, including both potential changes to the GATB as well as alternative scoring procedures. End Product: Options paper Literature Review and Options Paper for Alternative Methods of Assessment. This project will review and identify methods and approaches for alternative assessment that will assist in reducing adverse impact. In addition to using previous reviews, it will examine the potential of new concepts and methods, including portfolios and self assessment. End Product: Options paper Operational Biodata Instrument for Clerical Occupations. As recommended by the NAS report, this project investigates the usefulness of scored biographical information (biodata) as an alternative or supplement to other assessment procedures. The initial project, using clerical occupations, is following a construct based, empirical approach in which items written to measure constructs important for successful job performance (e.g., achievement motivation, interpersonal skills, and conscientiousness) are evaluated using large samples with respect to psychometric characteristics and criterion related validity. A large scale pilot study (n=1,000) in 1993 will be followed by a nationwide study (n=10,000) in 1994. End Product: Technical report, implementation plan, biodata instrument and manuals

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Options Paper for Developing Self Assessment Instruments and their use in Career Counseling, Job Training and Job Referral. This paper is an expansion of the literature search and options paper for alternative methods of assessment described in area 2B. With an increasing use of the self service approach in employment and training offices, there is a need to develop assessment instruments that can be used in such settings. This project will examine alternative approaches, their advantages and disadvantages and develop recommendations. End Product: Options paper Guide for Assessing Persons With Disabilities. Preliminary study indicates that it may not be feasible to develop forms of the GATB for assessing persons with various disabilities. Instead it may be more appropriate to develop a guide which focuses on the best methods and approaches to assessing persons with disabilities. Dr. Ragosta of the Educational Testing Service, a leading expert in assessment of persons with disabilities has developed a paper that provides information and recommendations regarding research and operational use of tests for persons with disabilities. This paper will be useful in preparation of the Guide. In addition this guide will also be developed with informed consultation with advocacy groups. End Product: Technical assistance guide Other projects described elsewhere in the workplan contributing to reducing score differences are: development of new forms, item analysis, item sensitivity review, projects to analyze speededness, reducing coaching, and development of the CAT-GATB. Area 3: Maintain the GATB. The NAS report made recommendations regarding GATB maintenance. These recommendations fall into two major categories: (a) improving the technical quality of the GATB, and (b) improving the security of test materials. In response to these recommendations, the following research areas were identified:

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Develop additional forms of the GATB. Operational forms of the GATB need to be replaced periodically to guard against over-exposure and compromise and to allow for more frequent retesting on a different form of the test. Improvements in the format and appearance of the test booklets and other materials will also be made. Test questions will continue to be screened and selected for fairness to various racial/ethnic, age, and gender groups using both editorial analysis and statistical methods such as Item Response Theory. Reduce susceptibility to coaching. A modified scoring procedure to include a penalty for wrong answers is needed to reduce or eliminate the advantage of guessing or answering at random. Instructions to examinees regarding guessing will also be included in administration instructions. Care will be taken not to change the meaning (validity) of the test. Reduce speededness of the GATB. “Speededness” is of concern because of the possibility of achieving a spuriously high score by answering, at random, the items the person is unable to consider in the time allowed. In addition, some have hypothesized that speeded tests may disadvantage some groups (e.g., older workers) more than others. It also seems advisable to investigate the possibility of reducing the number of test questions or extending the time limits. Care will be taken to avoid operational problems or changing the meaning of the test scores. Develop standards for the physical security of testing material. Currently there are no instructions to GATB users regarding the confidentiality and security of testing material. Physical security of tests is important because widespread compromise could destroy the value of a very expensive test form. Listed below are the projects underway to address these areas of concern: Development of Experimental GATB Forms E & F. To address issues of speededness, susceptibility to coaching, and test security, a project to develop alternate forms of the GATB is being undertaken. The development plan for the new forms include reducing the number of test items, examining time limits, including instructions to examinees about guessing, and including a scoring correction for guessing. These components of the developmental plan will help address the NAS concerns about speededness and coachability. Additionally, the development of alternate forms will address the test security issue discussed in the NAS report. End Product: Experimental Forms E & F

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Speededness of GATB Power Tests Study. A literature review and an empirical study is being conducted to examine the extent to which GATB power tests given under current (i.e. speeded) and non-speeded conditions measure similar or identical constructs, and the extent to which speededness influences the magnitude of differences between scores of minority and majority group members. This study will provide recommendations and information which will help to decide upon the appropriate levels of speededness of GATB Forms E & F power tests and of future forms of GATB power tests. End Product: (1) Literature review and options paper (2) Technical report with recommendations for development of future forms of the GATB Operational GATB Forms E & F. Following an examination of the technical merits of Experimental Forms E & F, these forms will be made available for administration to certified users of the GATB. These new forms will address the technical issues (e.g., coachability, speededness) raised by the NAS report. Additionally, the introduction of the new forms will help address the test security issue raised by the report. When the new forms are introduced, as suggested by the NAS report, they will be accompanied by new test administration and interpretation guidelines. These guidance materials are discussed in more detail under Area 6 of this work plan. End Product: Operational Forms of E & F System to Develop New Forms of the GATB. This project will develop a systematic method for the ongoing development and equating of other alternate forms of the GATB. As the NAS report pointed out, regular development of alternate forms of the test to replace old forms is essential to test security. Much of the methodology for the system will be based on the results of E & F development. The following projects will help develop this system. Test Development Guide. A key component of any ongoing assessment program is to institute a systematic approach to test development. A goal of the ARD Program is to set up a system of continuous GATB form development, as suggested by the NAS report. The purpose of the Test Development Guide project is to specify a systematic test development approach that GATB test developers can follow. The Guide will provide guidance on developing alternate forms of the GATB. Additionally, other developers of cognitive ability tests will be able to use the Guide as a reference. The Guide will include information on design and development plan, item writing and review, data collection, data analysis and research

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project standards. Much of the information presented will reflect procedures used to develop GATB Forms E & F. Additionally the Guide will serve as a blueprint for the ARD Program to develop new forms of the GATB. End Product: Guide to developing new test forms Test Aesthetics. The appearance and format of the GATB and related materials have remained largely unchanged for decades, and the NAS report suggested improvements in these areas. This project will update the format of GATB and GATB test administration materials to make them more user friendly. For example, a modification suggested by the NAS report was to reduce the size of bubbles examinees are required to blacken when answering test questions. Currently a number of such aesthetic changes have been implement as part of the new GATB forms E & F. End Product: Improved aesthetics of GATB; manual on methods to improve aesthetics Development of a Test Security Manual. The NAS report recommended the development of guidelines and instructions for GATB test users regarding the confidentiality and security of testing materials. The purpose of this project is develop a security manual that will provide instructions for the physical security of GATB and other USES assessment materials. End Product: Manual on instructions for maintaining security Development of Computerized Test Materials Tracking System. A major concern of the NAS report was test security. The purpose of this project is to develop a computerized library system that will allow USES to track test materials to improve the control and monitoring of test distribution throughout the country by USES. The present goal is to have the tracking system in place with the implementation of GATB Forms E & F. Additionally, modifications to the system will enable the tracking of other USES assessment materials so that USES program staff and management know the location of test materials. This will greatly enhance security of test instruments and facilitate the inventory process. End Product: Computerized tracking system of GATB materials

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Technical Report on Score Increases Due to Retesting. When an individual completes a test, he or she may become familiar with the questions and format and learn “shortcuts” for answering the questions efficiently and correctly. Therefore, when the individual takes the test again he or she may score better. This paper examines the GATB data base to determine to what extent examinees score better the second time they take the test. The results of this paper will guide retesting policy and will help to contribute to the scientific literature in an area that is extremely limited in research. End Product: Technical report New Experimental Forms of GATB Psychomotor (dexterity) Tests. The NAS identified a number of critical issues regarding the adequacy of the psychomotor tests developed to date. In particular it recommended “more reliable measurement of the psychomotor aptitudes deserves a place on the GATB research agenda.” This project will examine the issues associated with psychomotor tests, the NAS concerns and will develop possible options for assessing psychomotor skills including the GATB tests. A paper will be completed that reports on the examination of this area, present options and recommendations for developing psychomotor tests including options and recommendations for the GATB. End Product: Options paper with recommendations for integration into the workplan Development of Forms C & D Technical Documentation. This report will document the steps followed and the results obtained when Forms C and D of the GATB were developed. This report is being used to help guide the current development of Forms E and F and the continued development of forms in the future. End Product: Technical report documenting development of Forms C & D

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project Area 4: Develop New Norms. New GATB norms based on current data are needed to provide more accurate information to counselees and other persons taking the GATB regarding their standing relative to other people. Options Paper for Developing New Norms. A paper will be prepared which will present norming study design options, describe the type of information that would be available based on each option, describe the approximate resource requirements associated with each option and evaluate the feasibility of each option. End Product: Paper Develop New Norms. An appropriate option will be selected from the options paper to develop new norms. A research design will be developed and implemented. End Product: Improved norms for GATB Area 5: Computerized GATB. Although not a specific recommendation of the NAS report, a computer adaptive version of some parts of the GATB could address several of the concerns noted in the report and improve operational efficiency. Sometimes termed “tailored testing,” this advance in testing technology efficiently determines the ability level of each individual by presenting only items of appropriate difficulty (i.e., an individual does not have to take items that are clearly too easy or too difficult). Potentially, a computer adaptive GATB (CAT-GATB) could: (1) reduce the cost of testing by reducing test administration and scoring demands, (2) reduce or eliminate the test security problem, (3) reduce examinee frustration by presenting examinees with items appropriate to their ability levels, (4) improve testing accuracy, and (5) open the possibility of different test formats including essay and free response. Computerized GATB. Efforts to computerize portions of the GATB are underway. Three and possibly four of the 12 GATB subtests are amenable to computer adaptive testing, two or three others can be computer administered, and the five psychomotor subtests are not easily configured for either computer adaptive or administration modes. Software is currently being developed. The

OCR for page 9
Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project software will be easily modifiable so that it can be used to administer and score virtually any fixed response format assessment instrument. End Product: Computer adaptive software program for the administration of GATB. Area 6: Provide Guidance on Proper Use of the GATB. The NAS report made a number of recommendations regarding test use; other general principles are embodied in various professional standards and legal guidelines (e.g., Principles for Validation and Use of Personnel Selection Procedures, Standards for Educational and Psychological Testing, Uniform Guidelines on Employee Selection Procedures). The successful approach to meeting the objectives in this area may be by building upon the work accomplished in Area 2 For example, the proper use of the GATB may be as a tool in a comprehensive approach to assessment. Listed below are projects that will result in test guidance materials for GATB users. The Employment Counseling System: A Guide and Automated Resource Database. This database is designed to assist ETA programs with automated guidance and resources for employment counseling. It will provide information on the counseling process, client groups, counseling techniques, community resources and a number of assessment instruments. It will also discuss the integration of assessment into the counseling process and provide specific guidance on conducting an assessment program. End Product: Guides and automated resource database

OCR for page 9