Evaluation of the U.S. Employment Service Workplan for the GATB Improvement Project (1995)

NOV 8 1993

Dear Dr. Feuer:

This letter transmits the United States Employment Service's Workplan for the General Aptitude Test Battery (GATB) Improvement Project to the National Academy of Science's (NAS) Board on Testing and Assessment (Board).

As we discussed at the Board Meeting on September 23, 1993, the Department of Labor (the Department) wants to assure that the GATB Improvement Project addresses the major NAS concerns expressed in the 1989 report Fairness in Employment Testing. To accomplish this end, the Department is seeking the Board's review of the technical adequacy of the Workplan and the work products that are under development in the United States Employment Service.

The Department staff were pleased with the initial meeting of the Board and look forward to working closely with you and the panel of experts. To facilitate the Board's review process, the Department staff have identified specific questions that they would like to see addressed as the Board reviews the Workplan. We believe that structuring the review process in this manner will help us complete the GATB Improvement Project in an appropriate and timely manner.

Questions for the Board

1. Does the workplan adequately address the major technical issues raised in the NAS report?

2. Are there any significant gaps/flaws in the basic approach reflected in the workplan?

3. In particular, does the workplan reflect an adequate research response to recommendations in the 1989 NAS report in the key areas of reducing score differences and adverse impact, and testing persons with disabilities?

Thank you for your time and attention, we look forward to the Board 's response, if possible by the end of January 1994.

In 1987, the National Academy of Sciences (NAS) was commissioned by the U.S. Department of Labor (DOL) to conduct a thorough evaluation of the General Aptitude Test Battery (GATB) and the implications of its use as a tool for employment counselling and referral by the U.S. Employment Service. Following the publication of the NAS findings in 1989 (Hartigan & Wigdor, 1989), DOL announced a proposed policy on GATB use in July of 1990. This proposed policy, published in the Federal Register (July 24, 1990), called for suspending GATB use for employment referral purposes during a research period, although it could continue to be used for counselling.

The Federal Register notice also outlined a GATB research plan, which called for a comprehensive program to improve validation methods and accelerate the validation process, develop additional forms of the GATB, reduce susceptibility to coaching and reduce GATB speededness, develop standards for physical test security, develop new norms, computerize the GATB, and provide guidance on proper use of tests. A final decision regarding the DOL 's policy on the use of the GATB was held in abeyance during Congressional debate on the Civil Rights Act of 1991. During this period, however, steps were taken to implement the proposed research plan. In December of 1991, the DOL's policy decision regarding the use of the GATB was announced in the Federal Register (December 18, 1991). The GATB could continue to be used for counselling as well as employment referral purposes, with the caveat that GATB users were responsible for compliance with all applicable Federal and State statutes and regulations. However, DOL policy also prohibited use of the GATB as the sole factor in employment referral, and, consistent with the Civil Rights Act of 1991, the use of within-group scoring was precluded.

Many questions remained, however, about the GATB. The GATB research plan outlined in the Federal Register, the NAS report, and calls by employers for the continued use of the GATB for employment referral all underscored a need for DOL to:

• commission reviews and research by independent experts concerning job clustering and score weighting, reducing the number of items in some sections of the GATB, and the effect of correcting for guessing on the test;

• develop new forms of the GATB (Forms E and F) that will better provide for retesting of applicants and will help to ensure the integrity of scores;

October 22, 1993

• validate the GATB for occupations that are new to the U.S. economy or that have changed significantly during the last ten years;

• construct a computerized adaptive version of the GATB;

• enhance guidelines for maintaining the security of the GATB and specifying the individuals and organizations authorized to use the test; and

• research assessment methods that may reduce score differences among racial/ethnic groups.

Since implementation of the GATB Improvement plan, many research projects have been started and are currently on going. This workplan reflects the results of a mid-term review of progress, preliminary findings, and next steps generated from work efforts completed to date.

The workplan is divided into six areas: within each area, projects currently underway that address the research area are summarized.

1. GATB Validity Research

2. Reduce GATB Score Differences among Racial/Ethnic Groups

3. Maintain the GATB

4. Develop New Norms

5. Computerized GATB

6. Provide Guidance on Proper Use of the GATB

The NAS report noted an unexpected decline in GATB validities in recent studies and somewhat lower validities for blacks. In the area of validity research, the NAS recommended updating documentation of GATB validation research, continuing validation research to assure adequate coverage, particularly for new and changed occupations, updating procedures for future validation research and, further examination of optimal methods for clustering jobs.

The objective of Area 1 GATB Validity Research is to improve the validity of the GATB, address the NAS concerns regarding minorities, persons with disabilities and older workers and develop optimal clustering and weighting.

1. Series of Issue Papers. To examine issues and options for addressing GATB validity, expert papers have been commissioned which will contribute to the overall success of the improvement plan and the results will be incorporated into related research projects.

   1. Issues Paper on Job Clustering and Score Weighting. There are over 12,000 jobs in the U.S. economy, and it is practically infeasible and scientifically unnecessary to develop different assessment tools for predicting success in each. Instead, instruments can be developed to predict success in groups of jobs. The question is how job(s) should be grouped. The immediate focus of this paper is to respond to questions posed by the NAS regarding the basis for the current aggregation of jobs into five job families and the associated score weightings of the GATB. This paper will also address potential methods for clustering jobs for new assessments under consideration. Dr. Kenneth Pearlman, a member of the Advisory Panel for the Dictionary of Occupational Titles (APDOT) and expert in the area of job clustering, has completed a review of all previously-conducted research, respond specifically to related NAS concerns, and made recommendations as to optimal method(s) for grouping jobs for ETA.

      End Product: (1) issues aper

      (2) Analysis of paper and recommendations for integration into workplan