Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
11 In Whose Interest: Potential Effects of the VG-GATB Referral System The previous chapter described the U.S. Employment Service's (USES) conceptual model of local office operations under the VG-GATB Referral System. Although few offices have as yet switched entirely to the new system, most using some combination of old and new procedures, it is reasonable to ask what the effects might be if the full-scale version of the VG-GATB Referral System envisioned in the mode! were to be widely adopted in the Job Service. Indeed, the Department of Labor did ask the committee to consider the likely impact of widespread adoption on employers, minority job seekers, people with handicapping conditions, and veterans. Our treatment of the question, covered in this chapter, is necessarily partly conjectural. There is very little systematic evidence available from the pilot studies that speaks to the question of impact on the various Job Service clients. In addition, the national reporting of data each year on aggregate Job Service operations was discontinued in 1985, so there is no longitudinal data base from which to glean any general before-and-after comparisons. Nevertheless, we have ourselves gathered, and we have received from our liaison group members and others in the interested communities, enough information to be able to suggest certain likely effects and to recommend alterations in USES policy in the interest of particular client populations. 209
2~0 ASSESSMENT OF THE VG-GATB PROM EFFECTS ON EMPLOYERS The VG-CiATB Referral System is designed, in the first instance, to serve the interests of employers. Overall, GATB validities support the notion that referral on the basis of rank-ordered GATB scores will, in general, bring modest increases in the estimated job performance of the referred group of job candidates. In other words, using the VG-GATB Referral System as an initial screen will on average provide the employer with a somewhat more productive applicant pool than a nontested population would provide. This effect is, of course, not independent of the wages, benefits, and working conditions offered by the employer; the more competitive workers will actively seek referral to the top employers in a community. Moreover, the potential benefits to any given employer would be diluted as the number of competing employers using the test-based referral program increases. Empirical support for the finding that top-down referral of tested applicants would in general provide the employer with a set of applicants with better-than-average estimated performance on the job is provided by the Madigan et al. (1987) study of sewing machine operators, described in the previous chapter. Workers with GATB Job Family V scores in the top quarter needed 13.6 weeks on average to reach a given production standard, whereas those in the lowest quarter of the distribution took 16.4 weeks to meet the standard. Correlations between GATB scores and monthly production averages ranged from .15 to .24. In addition, there was somewhat less attrition among the higher scorers. These results indicate that the clothing manufacturer could have had a slightly more productive work force by hiring on the basis of test scores, assuming there were enough job seekers in the area to allow for selectivity. If the job order for some 900 workers absorbed virtually all available labor in the area around the five plants studied, then the test data would be irrelevant. A number of employers, liaison group members as well as others, contacted the committee about their experiences with the VG-GATB. As might be expected, those who took the effort to write or phone had found the test-based referral system valuable. One large midwestern corpora- tion, which has used the Job Service as its sole supplier of semiskilled labor for many years, was able to compare the traditional and the VG-GATB procedures. With the advent of the VG-GATB system, the company began to look at candidates on the basis of test scores. The director of personnel reported a number of positive effects. First, the tested cohorts were more likely than their predecessors to show up for the entire application process, which involves repeat visits and multiple screens. Supervisors found that the workers selected on the basis of test scores worked out better on the floor, both in terms of performance and
POTENTIAL EFFECTS OF THE VG-GATB ~FE~L SYSTEM 21 ~ reduced absenteeism. One of the most interesting points made was that top-down referral of tested candidates produced a group with better literacy skills, which was considered a great boon in the auto industry because of the need to retrain large segments of the work force every three to five years. Certain large employers have been attracted to the Job Service because of the introduction of the testing program. The director of the Virginia State Employment Commission and a member of the liaison group, Ralph Cantrell, reported to the committee that the two largest employers in the state list jobs with the Virginia Job Service solely because of its use of the test. Several employers new to the state have required test-based screen- ing, including a major Japanese firm, which indicated that it had chosen to locate there at least in part because of the VG-GATB testing program. Cantrell also notes that a number of Virginia employers who have operations in other states report that the plants that select according to test score outproduce their counterparts. An Informal Survey of Employers Thanks to the good offices of the Employers' National Job Service Committee (ENJSC), these impressions can be fleshed out somewhat with information derived from an informal survey of employers who have used the VG-GATB system. A questionnaire, based on a set of questions provided by the committee, was put together by the ENJSC steering committee and sent to the state-level Job Service Employers' Committees for distribution to employers known to be Job Service clients. Some responses were returned to the ENJSC members for forwarding; others came directly to the committee. In all, some 500 employers answered the questionnaire. Our discussion of the information provided by these employers must be prefaced by a caution about overreliance on the responses. The question- naire was constructed informally and was not pretested. That some questions were occasionally misunderstood is apparent from the results. Moreover, the respondents cannot be considered representative of Job Service clients since the sample was not scientifically drawn. Within these limitations, however, we can learn a good deal about the practices and attitudes of 500 employers who have experience with their state Employ- ment Service and the VG-GATB procedures. Most of the employers who took part were medium-sized or small, with 64 percent reporting under 250 employees. The types of jobs they reported filling through the Job Service are largely blue-collar and clerical ones, although nearly a third of the respondents also use it to fill some technical positions.
2~2 ASSESSMENT OF THE VG-GATB PROM How Respondents Use the VG-GATB Referral System Virtually all the respondents reported that they use the VG-GATB system as a preliminary screen only. In all, 99 percent interview the applicants referred by the Employment Service, 94 percent require a written application, 80 percent reported using reference checks, and more than half administer a physical examination. A comparatively small number, about 16 percent, reported administering additional job-specific tests. (This response could be misleading; one suspects that many clerical job orders would require a typing test in addition to the GATB.) Over half of the 500 employers in the survey said that they include specific selection criteria in their job order. The two most common requirements listed were job experience and the imposition of a minimum cutoff score on the GATE (each criterion used by about 51 percent of respondents), although over a third also reported imposing an education requirement. Some 72 of the 500 respondents said that they have the Job Service interview candidates for them. One issue of interest to the committee was the number of candidates the employers request per job opening. Among the 500 employers who participated in this informal survey, there was a good deal of variety. Many left the number of referrals up to the Job Service local office. A small number reported that they request just one or two applicants per opening, and an equally small number ask for 11 or more. Most of those who stipulate a referral ratio ask for between 3 and 10 applicants per opening; the median is 5. This information suggests that USES could consider referral models that required two to five people to be referred for each job opening without going beyond the common practice of many employers. Attitudes Toward the VG-GATB Referral System A large majority of the respondents were favorably disposed toward the VG-GATB system. It is important to remember that this group cannot be considered representative of Job Service clients in general or of employ- ers who have experience with the VG-GATB. We do not know how many employers received the questionnaire and did not respond, nor have we polled employers who choose to use the Job Service counselors in the traditional way rather than requesting tested applicants. This group of 500 respondents should probably be looked on as predisposed to a positive evaluation of the effectiveness of test-based referral. This does not mean that their judgments should be disregarded, just that they cannot be accepted as definitive.
POTENTIAL EFFECTS OF THE VG-GATB REFEREE SYSTEM 2~3 Close to 90 percent of the respondents felt that the VG-GATB Referral System improved the company's use of human resources. Among the benefits most frequently checked off were decreased training time and reduced interviewing time. Over a third claimed increased productivity and a like number claimed reduced turnover. Only about 7 percent of respondents said that the system did not improve the use of human resources. At the same time, about 37 percent wrote in disadvantages associated with the VG-GATB system. The most frequent complaint was the increased time lag between the employer's request and the filling of the job order. Equal Employment Opportunity ant! Affirmative Action More than three-quarters of the respondents reported that they have an affirmative action program in place. (Employers with fewer than 15 employees are not subject to Title VII of the Civil Rights Act, so the question would not necessarily be applicable to all who returned the questionnaire. In addition, a small proportion of the questionnaires omitted the question.) This would suggest that the general thrust of the USES policy of adjusting the scores of minority-group candidates is in step with the equal employment opportunity goals of many private-sector employers. However, the responses to questions about preferred scoring methods suggest that many employers may not be very clear about that aspect of the VG-GATB system. On the questionnaire, the within-group score adjustment used in the VG-GATB system to avoid screening out minority job seekers was explained. The questionnaire then asked which of four scoring methods respondents find most advantageous: within-group per- centile scores, total-group percentile scores, a combination of the two, or pass/fail. It provided a very brief explanation of the first two. Some 20 percent of respondents indicated a preference for a pass/fail scoring method, which means they either do not understand the new system or are not involved with it. Only about 10 percent chose the within-group option, and a few of the respondents wrote in that they did not realize that the VG-GATB system included such score adjustments. About equal numbers of respondents chose the other two options, with 37 percent of respondents supporting the reporting of total-group scores and 35 percent supporting a system combining total-group and within-group scores. During the course of its study, the committee heard a good deal of support for within-group scoring, particularly from larger employers. Because large employers of necessity have a trained personnel manage- ment staff, often including members with some expertise in tests and
2 ~ 4 ASSESSMENT OF THE VG-GATB PROGRAM measurement, they may well be more aware of the implications of the various scoring procedures than this group of medium-sized and small employers seems to be. These responses appear to show a need for the Employment Service to provide employers with better information about the test scores reported and what they mean. We have specific recom- mendations on the subject in Chapter 13 on referral and score reporting. Possible Negative Effects To this point, the potential effects of further use of the VG-GATB appear to be largely salutary as far as employers are concerned. In general, selecting the group of candidates to refer on the basis of rank-ordered test scores should present the employer with a modestly enriched pool of candidates to choose from. There is theoretical as well as some empirical evidence to support this claim. The main drawback for the individual employer so far mentioned is that in some areas it takes longer for candidates to show up at the employer's door. If computerization of local office operations proceeds apace and if regional job information networks develop, this problem may be eased. A different sort of problem could well emerge if the Department of Labor decides to support the spread of the VG-GATB system. This problem has to do with jobs that require significant prior preparation. The claim is being made that the GATE predicts performance for all 12,000 jobs in the U.S. economy, and that it predicts best for the cognitively more complex jobs. Although these claims may be true, they could also badly mislead employers. Until now, the preponderance of test-based referrals has been in less-skilled occupations, occupations that do not require extensive prior education and/or training. If for legal, economic, or other reasons employers eliminated more job-specific testing or assess- ment and depended on this test of general cognitive abilities to fill jobs with more stringent requirements, the more pertinent information about an applicant's qualifications could be lost. This is not merely an academic concern. At least one state is using the VG-GATB to construct the list of eligibles for state merit system jobs, including jobs such as social worker. EFFECTS ON JOB SEEKERS Logistics The most obvious effect of the VG-GATB Referral System on job seekers is that they have to take a three-hour examination in order to be considered for referral. As we have noted, most local offices have implemented the new procedures as a supplement to, not a replacement
POTENTIAL EFFECTS OF THE VG-GATB REFEREE SYSTEM 2~5 of, the old counselor-based system. But the conceptual model of a full-scale VG-GATB system would close off the alternate route for most applicants, making the test a necessary hurdle. Since many local offices appear to administer the GATE just once a week, the applicant is likely to be required to make two trips to the Job Service office, once to register and again to take the test. In contrast to the old system, this is not necessarily inefficient from the applicant's point of view. USES field staff who have helped implement the VG-GATB system argue that it is much less irksome for job seekers. They point out that in large, busy Job Service offices, applicants often have to wait around for hours until a job counselor is available. With the new procedures, applicants are given an appointment for a specific day and time, and thereafter could receive referrals by telephone. There is some evidence of user attitudes that supports this assessment. K.D. Scott and colleagues (1987), as part of a larger pilot study commis- sioned by the State of Virginia, surveyed all job applicants who registered with the Roanoke Job Service Office and took the GATE between January and the end of April 1986. Almost 70 percent of the applicants, when presented with six possible reactions from "strongly disagree" to "strongly agree," agreed or strongly agreed that they "did not waste a lot of time at the IS." Over 80 percent found the explanation of the testing and referral process easy to understand, and about 88 percent found the people at the Job Service helpful and courteous. This degree of positive response suggests that, in terms of logistics, the VG-GATB system can work well for job seekers. Perceptions of Fairness User perceptions of the more substantive benefits of VG-GATB referral are somewhat less positive. A primary justification for standardized testing, and therefore for VG-GATB referral, is that all applicants are judged by an objective standard in this case, an objective measure of abilities that are related to job performance. Proponents of the VG-GATB point out that eliminating the subjective judgment of counselors, who may be more or less able to match up suitable workers and jobs, can be particularly important for individuals who incite deep-seated social or cultural prejudices. We agree that GATB-based referral could have a beneficial impact on job seekers at least on competitive job seekers by reducing the possi- bility of personal prejudice from referral decisions and by giving the Job Service staff a better picture of the relative abilities of all the registrants currently in the files. Particularly in heavy-volume Job Service offices, the identification of prospective candidates could be less hit or miss.
2~6 ASSESSMENT OF THE VG-GATB PROM However, there are other aspects to the question. Tests of general ability such as the GATE are particularly valuable in assessing a youth population, people without a lot of life and job experience. The Armed Forces, for example, use a similar test to enlist and classify young men and women, most of whom are between the ages of 17 and 23. Although most have a high school education, few have specialized training or extensive job experience. In that circumstance, it is difficult to propose an alternative method of gauging an applicant's probability of success in the military. The Job Service deals with a very different clientele. It includes people of all ages, many of whom have years of relevant job experience. For such applicants, GATE scores are not the most important informa- tion. The applicant's past record would give an employer a better sense of likely future performance. Exclusive use of VG-GATB referral could well preclude the candidate from being able to present or the employer from being able to consider-that record. For example, the committee has heard the complaint of a worker who apparently worked success- fully for more than 15 years at a semiskilled job in the auto industry. Now, after a period of illness, the worker is trying to get back into the same line of work. But the firm has entered an agreement with the Job Service so that access to the job is solely by test score. The worker is being told that his test scores do not qualify him for the work that he performed for many years. Without getting into the merits of the particular case, the general point is well made that exclusive reliance on the VG-GATB system would deprive some experienced workers of the chance to compete for jobs they have already shown that they can in fact perform. There appears to be a significant degree of skepticism among job seekers about making testing a necessary prerequisite to referral. Veter- ans' organizations have protested the requirement on the grounds that many unemployed veterans neither desire nor need to take such a general test of abilities since, with an average age of 50, they have proven work experience and training on which to be judged. Among some population groups, the very existence of the VG-GATB will become a powerful screen; many people simply will not use the Job Service. For example, in one New Jersey office the experiment with the VG-GATB was cut short when the applicant population, a largely minority population, simply stopped using the Job Service. The Roanoke survey (Scott et al., 1987) provides additional evidence of unease with the test-based approach to job referral. (Although there is less generalized resistance to tests in Roanoke than in the New Jersey community mentioned above, note that the kind of people who aborted the New Jersey experiment would not appear in the Roanoke survey
POTENTIAL EFFECTS OF THE VG-GATB ~FE~ SYSTEM 2~7 sample, since only those who registered and took the GATB during a specified period were given the questionnaire.) The Roanoke applicants were asked to evaluate the following statements on a six-point scale from "strongly disagree" to "strongly agree": · The GATB was a fair test of my job abilities. · A test is a fair way to decide which people should be referred to a job. Of the 1,064 usable questionnaires, some 60 percent of respondents registered at least slight agreement with each of the propositions, and 45 percent agreed or strongly agreed. But put the other way, about 40 percent of applicants did not perceive of the procedure as fair. (This perception of unfairness increased somewhat in a follow-up survey 12 weeks after registration and testing; 50 percent of those who responded had not yet been placed in a job.) There were no significant differences between minority and majority applicants with regard to the fairness questions, but applicants with advanced degrees (beyond high school) were less favorably disposed toward the GATB than others, which has implications for the USES hope that the VG-GATB Referral System will make the Employment Service a more mainline employment agency and help it shed its past reputation as a provider of last resort. The evidence is far too limited to draw any broad conclusions about the public acceptability of test-based referral. Still, it seems to suggest that the elimination of other routes to referral could narrow the population of job seekers willing to work through the Job Service, and, in some communities, drastically so. Low-Scoring Applicants For job seekers with very low scores on the GATB, VG-GATB referral will have predictable and completely negative effects. Such applicants will be referred only to the least desirable jobs, if they are referred at all. As we suggest elsewhere, there is a very real possibility that if the VG-GATB system came to dominate the Employment Service system, the effect would be to create a class of perpetually unemployed people, identifiable from early school years as those who are the poor performers on tests of general cognitive ability. Industrial society seems to require less than full employment. It does not require, however, that one category of citizen, low scorers on cognitive tests, be fated to permanent unem- ployment. Our analysis of GATB validities demonstrates that there is too much error in GATB scores for any argument from economic necessity to be compelling. Even low-scoring applicants have a reasonable chance of performing better than average on the job.
2 ~ ~ ASSESSMENT OF THE VG-GATB PROGRAM If the conceptual model of the full-service VG-GATB office incorpo- rated special procedures for working with low-scoring applicants, our concern about the rigidities and limitations of a referral system based on cognitive test scores would not be as great. But in the VG-GATB system as it is conceived, with the exception of statutorily mandated special programs to assist handicapped veterans and perhaps handicapped appli- cants more generally, those most in need of special assistance in finding work would be funneled out of the Employment Service system. We realize that this design is a reflection of the economic stringency that has descended on the Employment Service both nationally and in the states. It is, nevertheless, one of the important considerations that leads us to recommend that VG-GATB referral in no instance be the only set of procedures used in any local office. Older Applicants The evidence presented in Chapter 8 indicates that GATE validities vary with the mean age of workers in the validation samples. More specifically, the validity of the cognitive composite (GVN) tended to be somewhat lower in studies in which the average age of the workers was higher. The evidence also suggests the possibility that GATE scores decline with age. The worker samples with higher mean age have notably lower composite scores, particularly for the spatial (SPQ) and psychomo- tor (KFM) composites. This evidence raises the possibility that older workers workers whose skills and experience testify to their ability to do the jo~will tend to be excluded by the VG-GATB system. EFFECTS ON MINORITY JOB SEEKERS The single most important question with regard to the effects of the VG-GATB Referral System on minority job seekers is whether the government-in the near term the Justice Department, ultimately the courts find score adjustments a legally and constitutionally acceptable means of furthering equal employment opportunity goals. Chapter 13, on referral methods and score reporting, presents some scientific reasons in support of such a policy. But of course we cannot predict the ultimate outcome. Recent experience in the Employment Service system indicates that minorities have been referred and employed at the same rate as the majority group, although on the average to somewhat lower-paying jobs. So long as the VG-GATB Referral System includes within-group score adjustments, that should continue to be the case. A recent study of the
POTENTIAL EFFECTS OF THE VG-GATB ~FE~L SYSTEM 2~9 application of the VG-GATB system in local Job Service offices in Salt Lake City, Provo, and Logan, Utah (Robins, 1988), sustains these expectations: minority applicants (who make up just 7.2 percent of the work force) were placed at a slightly higher rate and in jobs of slightly longer duration than the majority-group applicants. Furthermore, the patterns of service delivery to majority and minority applicants were similar for VG-GATB and regular local offices. This would not be the case if VG-GATB referral were not accompanied by score adjustments of the magnitude of the within-group percentile scores. Without such adjustments, referral on the basis of rank-ordered test scores would have very severe adverse impact on black and Hispanic applicants. Empirical evidence indicates that the difference in average test scores between majority- and minority-group members ranges from 0.5 to 1 standard deviation. When the average score difference is 1 standard deviation, then referral in order of test score will mean that if 20 percent of an applicant pool is referred to a job, about 20 percent of majority applicants would be referred, compared with 3 percent of minority applicants. Even if a minimum competency system were used, with all referred at random above a given cutoff score, a cutoff score at the 4th percentile of the majority group would be required for minority applicants to be referred at four-fifths the rate of majority candidates. We conclude that the VG-GATB Referral System will be viable only if it includes some kind of adjustment of minority scores, so long as the government is committed to a policy of equal employment opportunity that looks to the effects of employment practices on racial and ethnic minority groups. Assessing Applicants Who Have Marginal English Skills Foreign-born applicants, whose command of the English (or perhaps any written) language is marginal, cannot be reasonably assessed with the GATB. In some states, particularly in the Southwest and the West, such people constitute a large proportion of the likely Job Service client population. The problem was well expressed by a professional vocational evaluator working in the state of California (letter from Julia Edgcomb, dated November 23, 1988~. Between 40 and 60 percent of this evaluator s clients are non-English-speaking; most of them have a maximum of six years of formal education, with three years being common. Many have been working since the age of 10 and are now trying to find careers outside farmwork. The GATE will portray these job seekers as of very low cognitive abilities because of language difficulties, lack of formal educa
220 ASSESSMENT OF THE VG-GATB PROGRAM talon, and lack of experience with paper-and-pencil tests. Yet many of them, in the writer s experience, are very bright and can demonstrate job-relevant skills in hands-on work simulations. By exclusive use of an instrument that does not permit such people to show their strengths, the Employment Service system would serve neither employers nor work- ers needs as well as it might. EFFECTS ON PEOPLE WITH HANDICAPS The populations most at risk of misassessment in a test-based referral system include people with handicapping conditions. Like certain racial and ethnic subgroups, people with many kinds of handicaps have lower scores on average than the majority group. Yet for other population groups, although there are some inconsistencies, the test scores have a fundamental comparability of meaning. No such claim can be made for the scores of people with handicaps. As a consequence, referral based solely on test scores not only would tend to screen out such applicants, but also would do so arbitrarily. The difficulties of using standardized ability tests to assess people with handicaps were described in an earlier report emanating from the Na- tional Research Council (Sherman and Robinson, 19821. The goal of many such people is to participate fully in American life, a goal explicitly recognized as a right in such federal legislation as the Rehabilitation Act of 1973. In order to enter the economic mainstream, people with handi- caps need to be able to demonstrate their qualifications. This is more or less difficult depending on the particular handicapping condition. Ability testing poses a quandary. If, on one hand, a test reflects a person s disability rather than the skills actually needed on the job, it becomes a barrier, a vehicle that frustrates that person s attempt to demonstrate job capabilities. If, on the other hand, there is no way for people with disabilities to compete with others for jobs or educational opportunities, they are all too likely to be sidelined. The most obvious problem is that standardized tests often cannot be administered satisfactorily to people with physical handicaps. Various modifications have been attempted, for example, to provide Braille or oral versions for blind test takers or to modify the response mode for those with motor handicaps. Indeed, the history of nonverbal performance tests began with tests for hearing-impaired people (Anastasi, 1988:290-2971. However, such modifications of format and/or content have unknown effects on the construct validity of the instrument and on the meaning of test scores. The complexities of developing appropriate modifications have emerged gradually over the years. Some are fundamentally logistical,
POTENTIAL EFFECTS OF THE VG-GATB REFEREE SYSTEM 22) whereas others would have to be called epistemological. In the first category is the following kind of problem: the Employment Service of a northern state recently developed a videotape of a sign language admin- istration of the GATE for use with deaf applicants. Because of its history of aptitude test research for the deaf, the USES Southern Test Develop- ment Field Center in North Carolina was asked to review the videotape. One of the things noted by their sign language specialist was the difference between the signing dialects in the two states. The sign that to its Nebraska audience means "exercise" is used in North Carolina to mean "ice cream" or "cold." Although use of a nonlocal signer could obviously result in some very confused Job Service clients, this kind of problem in redesigning stan- dardized tests for people with handicaps is easy to comprehend, and a good deal of progress has been made over the years in coming up with solutions. The most thorough and detailed documentation of modifica- tions to an employment test was that provided for the Professional and Administrative Career Examination (PACE), a test developed and admin- istered for years to entry-level federal employees by the Office of Personnel Management (OPM) (Nester, 1984~. In its modifications for visually and hearing-impaired applicants, for example, OPM developed careful procedures to review item content for appropriateness, deleting items involving color or spatial patterns in tests modified for the blind, and so on. (Unfortunately, much of OPM's work in this area came to an end with the elimination of the PACE in 1981 because of EEO problems.) The far more difficult issue concerns what is being measured. Are the cognitive processes used by blind test takers the same as those of the sighted? What is the the relation between verbal or linguistic abilities and cognitive abilities? If language is merely a vehicle for such cognitive abilities, is it possible for people who are profoundly hearing impaired to demonstrate cognitive abilities through other vehicles? Does test perfor- mance bear the same relationship to job performance for people with visual or hearing impairment as for others? Although research at the Educational Testing Service (ETS) and OPM has brought some interest- ing insights (Nester, 1984; Willingham et al., 1988), we are a long way from knowing the answers to these questions. Recent Research Findings Many of the research issues posed by the earlier National Research Council's Panel on Testing of Handicapped People (Sherman and Robin- son, 1982) have been addressed in a four-year research project carried out by ETS and described by Willingham et al. (19881. The study involved two academic entrance examinations, the Scholastic Aptitude Test Bat
222 ASSESSMENT OF THE VG-GATB PROGRAM tery (SAT) and the Graduate Record Examination (GRE). The subjects were a pool of handicapped applicants and a comparison nonhandicapped group attending the same colleges (for the SAT) and graduate schools (for the ORE). Admittedly, that research based on a college population and using tests designed to predict academic performance is not directly relevant to GATE applications, but its consideration may offer some guidance. Some general features should be noted here. The handicapping condi- tions of the students in the study were four: hearing impairment, learning disability, motor handicaps, and visual impairment. These disabilities are listed in the order of the students' disparity from the nonhandicapped group. Clearly the language impairment of the deaf and (to a somewhat lesser extent) the learning disabled represents an extreme, special diffi- culty with the standard tests, especially the verbal part (U.S. Department of Labor, 1982~. For these two groups the SAT-V(erbal) and the SAT- M(ath) are more independent than in the other handicapped groups or the nonhandicapped group. Neither the test scores nor high school grades predicted college performance as well as they did for the nonhandicapped group. Both scores and predictions are better for people with motor handicaps and visual impairment. The National Research Council panel was especially anxious that research address the issue of comparability, that is, whether the standard and modified tests measure the same abilities and, if so, whether the scores from modified tests could somehow be transformed so as to yield the same scale of measurement as the standardized tests. The experience of the ETS research effort seems to indicate that such a transformation process is not within reach in the present state of the art. Among other things, any such transformations would depend on the availability of some measure of the degree of handicap. The biggest stumbling block to score comparability is the sheer difficulty of gathering sufficient data to establish comparable scales empirically. Although the SAT is one of the three largest testing programs in the country and is supported by an extensive data base, the ETS research team had great difficulty assembling an "even minimally suffi- cient" data set to study the validity of scores on the modified versions of the SAT for predicting first-year grade-point average. It was virtually impossible to take account of degree of handicap, and even within type of handicap, the numbers of students for whom all data were available were often small. Instead, the ETS authors look to more individualized testing proce- dures to improve the testing of handicapped applicants. In particular they emphasize the importance, in future research, of establishing an appro- priate test time for each handicapping condition. Their report suggests
POTENTIAL EFFECTS OF THE VG-GATB ~FE~E SYSTEM 223 that the greatest variable across modifications for the different groups was time some handicapped groups taking as much as 12 hours for what was a 3-hour test for the nonhandicapped group. Experience with the GATB There has not been comparably extensive research on the use of modified GATB tests or test procedures for the handicapped. A series of technical reports from the Employment Security Commission of North Carolina, from the early 1970s to 1980s, does provide, however, consid- erable information on modified instruction for the deaf. Several studies by the Employment Security Commission of North Carolina (1971, 1972, 1973) compared scores of high school juniors and seniors from the North Carolina School for the Deaf on the GATB with those on the Nonreading Aptitude Test Battery (NATB) and also com- pared their scores with nondeaf high school students, regular and educa- tionally deprived. The principal variable in these studies concerned modifications in the instructions whether through sign language by an expert signer or by newly written instructions that held vocabulary and sentence structures down to lower-grade usage in the elementary schools. Such modifications had a favorable effect on the scores for the deaf, but the scores for abilities G. V, and N still remained significantly lower than those of the nondeaf group. The authors point out that, although the NATE may give a clearer picture of abilities for the educationally deprived group, it does not for the deaf, whose scores on the GATB are either the same or slightly better than on the NATB. Although subsequent Department of Labor memoranda recommended the use of sign language interpreters for deaf applicants, particularly in a regional office where there were many deaf applicants, there was no clear advantage for the signed over the modified written instructions. The clear disparity between the deaf and the nondeaf on verbal and cognitive factors (G. V, and N) would be especially penalizing in a referral system based on the VG-GATB, since the abilities in which the deaf show more normal scores are less heavily weighted in computing total scores. We are not aware of validation studies with any of the handicapped groups that relate their GATB score to job performance. Such work, following the pattern of the ETS research cited above, would be useful. We have seen a Technical Assistance Guide from DOL that offers guidance to the local offices in testing the handicapped. The suggestions are quite general and mainly involve careful, patient accommodation to the needs of the applicants addressed globally without particular sug- gestions for different handicapped groups. Most of the suggestions, including the use of sign language, having the test administrator record
224 ASSESSMENT OF THE VG-GATB PROGRAM responses for applicants with motor handicaps, and so on, take time. In Chapter 5 we discussed the highly speeded character of the GATB. The time pressure that all applicants encounter would be particularly discon- certing for those with handicaps. Should Handicapped Applicants Take the GATB? Given the thin research base on modified forms of the GATB and, more generally, the great uncertainties about the meaning of the scores achieved by many applicants with handicapping conditions, policy mak- ers might wonder if such applicants should be tested at all. The wisest course is to leave the decision to the applicant. If appropriately modified tests are available, and if they are knowledgeably administered, they might provide the applicant with a means of demonstrating skills and capabilities relevant to job performance. The committee has received communications from a number of job counselors and vocational rehabilitation specialists who have long expe- rience with the GATB and who have found it a useful assessment tool. They also emphasized that it is not always the best method for assessing the capabilities of disabled job seekers and that the imposition of any one instrument could be very damaging to the prospects of such applicants. The inflexible implementation of VG-GATB procedures would cer- tainly not be in the interest of job seekers with handicaps. Referral by rank-ordered scores would tend to deprive them of the chance even to be in the running for jobs, and, as we said at the beginning, often arbitrarily. The committee was informed of one such instance. A candidate for a state job as an accounting clerk, who has disabilities including a hearing impairment and cerebral palsy, met the requirements, was placed on the list of eligibles, and was sent out on several job interviews. However, before the applicant found a position the VG-GATB system was imple- mented and all on the list of eligibles had to take the test. On the new list of eligibles, this applicant no longer made the cutoff. Above all, we recommend that the Employment Service continue to provide the services of job counselors to work with handicapped appli- cants to find the best means of assessing their skills and of bringing their performance potential to the attention of likely employers. Since state rehabilitation agencies have a wealth of expertise in vocational counseling for people with handicaps, it could well make sense to coordinate their work more closely with that of the Employment Service. One possible option would be to detail state rehabilitation agency counselors to Job Service offices that serve a sizable handicapped clientele.
POTENTIAL EFFECTS OF THE VG-GATB REFEREE SYSTEM 225 In any case, handicapped applicants can be served fairly and appropri- ately only if the Employment Service system remains flexible and responsive to the particular circumstances of each of them. EFFECTS ON VETERANS Because the Public Employment Service is a source of benefits for veterans of military service, the introduction of new referral procedures necessarily involves the interests of this constituency. Representatives of military veterans have expressed concern that adoption of the VG-GATB Referral System could conflict with the preference or priority in federal employment and training services granted to most veterans as a matter of law. Three major veterans' groups have communicated with the committee: the American Legion, the Veterans of Foreign Wars, and the Disabled American Veterans. All have called for the Department of Labor to enunciate a nationally consistent policy for implementing veterans' pref- erence in the Employment Service system. They have also expressed opposition to exclusive use of the VG-GATB system for referral by the Job Service on the grounds that many veterans (average age 50) have no desire to take a written test, preferring instead to be referred on the basis of work experience and training. The American Legion has adopted no official position on the question of using within-group score adjustments to prevent the test-based referral system from adversely affecting the employment opportunities of protected minorities (Rhoades, 1988~. The Veterans of Foreign Wars has formally adopted the view that such adjustments represent an abridgement of the entitlements of nonminority veterans since no equivalent adjustments are made to veterans' scores (Veterans of Foreign Wars, 1987~. The History of Veterans' Preference The first federal law granting employment preference to military veterans was passed at the end of the Civil War. By joint resolution, Congress mandated in March 1865 that persons who had been honorably discharged due to injury or illness incurred in the line of duty should be preferred for appointment to civil jobs, provided that they "possess the business capacity necessary for the proper discharge of the duties of such offices." The hiring preference accorded to disabled veterans was reaf- firmed in the act creating the modern civil service in 1883, although the eligibility requirements were sufficiently restrictive in these early years that only a few hundred cases came before the Civil Service Commission annually.
226 ASSESSMENT OF THE VG-GATB PROGRAM Throughout the years, the operational definition of veterans prefer- ence has varied widely; as policy makers were persuaded at one point in time by those who pursued efficiency in government through merit hiring, and at another by those who considered public employment a rightful reward for military service. For example, in 1881 the Attorney General construed veterans preference narrowly to mean that a disabled veteran should be granted preference only when his qualifica- tions were equal to those of other candidates what might be called a tie-breaker rule. This interpretation was overturned in 1910 when the Attorney General ruled that the law required absolute preference for disabled veterans on the list of eligibles (U.S. Civil Service Commis- sion, 1955; Manela, 1976~. In the immediate aftermath of World War I, the scope of veterans preference was expanded dramatically to include not just disabled veter- ans, but all honorably discharged soldiers, sailors, and marines, the widows of such servicemen, and the wives of disabled veterans who were not themselves qualified to hold civil service positions. Until 1944, authority to promulgate the rules implementing the Preference Act of 1919 lay with the president. Executive Order 3152 (Aug. 18, 1919) gave absolute preference to all veterans passing competitive examinations with a minimum score of 65. As a consequence, claims to the Civil Service Commission for veterans preference went from between 600 and 900 per year to 60,000 or 70,000 per year. Questions were rapidly raised about the sacrifice in efficiency in the civil service caused by placing all veterans at the top of the registers of eligibles. With the support of the American Legion, the Civil Service Commis- sion in 1923 recommended to the president that a system of adding points be instituted 5 points for veterans and 10 points for disabled veterans- in place of absolute priority above a minimum cutoff score. It would, the commission argued, provide a substantial benefit without being seri- ously detrimental to efficiency. Absolute priority was restored to veterans with military service-connected disabilities in 1929 because of complaints that the simple addition of 10 points was not resulting in a sufficient number of appointments for such veterans. After years of negotiation, the three major veterans organizations (the American Legion, the Veterans of Foreign Wars, and the Disabled American Veterans), the Civil Service Commission, and the House and Senate Committees on Civil Service reached a compromise arrange- ment one that attempted to strike a balance between the principles of merit and reward for military service. This compromise agreement was codified in the Veterans Preference Act of 1944. In broad outline, the act provided preference for veterans in federal employment as follows:
POTENTIAL EFFECTS OF THE VG-GATB REFER^L SYSTEM 227 Civil Service examinations. Disabled veterans are given an additional 10 points on competitive examinations; other veterans receive 5 extra points; and, in certain circumstances, spouses, widows, and mothers receive 10 points above their earned score. (Since 1953, the points have been added only to the scores of veterans who reach a passing score.) Eligibility lists. The names of 5-point veterans are placed on the list of eligibles in order of their augmented score; in cases of ties with nonvet- erans, the veteran precedes. Until 1953, disabled veterans continued to be placed at the top of the list of eligibles; since then, the additional preference of absolute priority on the list of eligibles has been restricted to veterans who receive disability compensation or who have a military service-connected disability and is not available for scientific and profes- sional positions of GS-9 or higher. Civil Service appointments. The 1944 act codified the so-called rule of three, according to which the top three names on the register of eligibles for a particular position are forwarded to the appointing officer for consider- ation. The selecting officer can pass over a veteran in favor of competing nonveterans only with the approval of the Civil Service Commission. The Public Employment Service and Veterans' Preference Since its own creation by the Wagner-Peyser Act of 1933, the Public Employment Service has had major responsibility for promoting veter- ans' employment. The act provided for veterans' preference and in addition instructed that a federal employee with the title State Veterans' Employment Representative be detailed to every state. The Service- men's Readjustment Act of 1944 created the position of Local Veterans' Employment Representative so that every full-service local office of the Employment Service has an official in place who oversees local activities to ensure that veterans are accorded all priorities, privileges, and services to which they are entitled by federal law. The statutory basis for these entitlements is found in Title 38, Chapter 41, of the United States Code, which authorizes a national program of job counseling, training, and placement services for eligible veterans and for certain spouses of veterans. Eligibility is defined very broadly to include any person who has served on active duty for at least 180 days and has been discharged other than dishonorably, or who was discharged due to a military service~onnected disability. Since 1980, the statute has placed responsibility for veterans' employment and training services with the Assistant Secretary of Labor for Veterans' Employment. The State Directors for Veterans' Employment and, nowadays, the Assistant State Directors (one for every 250,000 eligible veterans and spouses), are administratively responsible to the Assistant Secretary.
228 ASSESSMENT OF THE VG-GATB PROM From the moment a veteran enters a local Job Service office, special services are made available. The Local Veterans' Employment Repre- sentative is to supervise the registration of veterans and provide a special orientation on available training, counseling, and placement services, including federally funded employment and training programs under the Emergency Veterans' lob Training Act of 1983 or the Disabled Veterans' Outreach Program; to promote job development activities such as job fairs; to maintain a data base with information about job opportunities; to encourage employers and labor unions to employ those eligible for veterans' preference and to conduct apprenticeship and on-thejob train- ing programs for them; to supervise job listings of federal contractors to ensure compliance with veterans' set-asides; to identify and assist veter- ans with readjustment problems; and so on (Fraas, 19831. Veterans' Priority in Referral The VG-GATB Referral System, because it would dramatically alter referral procedures in the Employment Service system, has been a topic of concern to the veterans' organizations and to the Assistant Secretary of Labor for Veterans' Employment. Although definitive policies have not yet been enunciated by the agency for the guidance of states that are using VG-GATB referral procedures, some general outlines of Department of Labor policy are visible. The basic principle of referral priority is that qualified veterans shall be referred before qualified nonveterans. Definition of the word qualified is the crux of the matter. In the narrowest interpretation, it would mean that between two applicants with exactly the same test scores, the veteran would be placed ahead of the nonveteran on the referral roster. In the most generous decision rule, all veterans who met the employer-imposed cutoff score (if such there be) would be referred before all nonveterans. The Department of Labor seems to have decided that a number of alternatives for defining qualified are acceptable; neither the Assistant Secretary for Veterans' Employment nor USES has promoted a particu- lar procedure for implementing veterans' priority in the VG-GATB Referral System. And there are dramatic differences in the methods chosen by the states to keep the VG-GATB program in compliance with the statutory requirements. Of the 35 states reporting their procedures in a 1987 survey by the Department of Labor, 24 schedule veterans ahead of others for testing and/or referral. Those that grant veterans referral priority typically put a 24-hour hold on new job orders, during which time only veterans are contacted from the VG-GATB call-in list. In ranking candidates, 16 states accord veterans absolute priority regardless of test score, ranking all veterans ahead of all nonveterans on
POTENTIAL EFFECTS OF THE VG-GATB ~FE~L SYSTEM 229 the VG-GATB call-in list. One state applies this rule only for those in the 50th to 99th percentiles. Eleven states apply a docile method in which veterans within each decile are referred first from the top down; one state uses a decile variation in which veterans within 10 percentile points are referred first, from the top down. Three states add from 5 to 10 points to veterans' percentile scores and apply veterans' priority in the case of ties. Another three states apply only the tie-breaker rule. It is noteworthy that the absolute-priority rule adopted by 16 states is far more generous than the preference accorded veterans in traditional federal civil service hiring (procedures for entry-level hiring are currently undergoing change). In that system, test scores are augmented by 5 to 10 points (depending on disability status). Referrals are made from a single list of eligibles three at a time, from the top down. The hiring official has discretion in selecting among the three, except that if a veteran in the group of three is not the one selected, the decision must be approved by the Office of Personnel Management. Although there has not been uniformity of procedures for extending priority in referral in the past, the veterans' organizations have ex- pressed sharp dissatisfaction with this laissez-faire approach. They are calling for the Department of Labor to issue explicit guidelines or rules for a uniform system of veterans' priority under VG-GATB referral procedures, because the welfare of veterans is a national responsibility and they should therefore be accorded the same treatment in every state. Although they would prefer absolute priority, the veterans' organizations are not pushing for a specific rule. In a formal statement to the committee, Dennis Rhoades, spokesman for the American Legion and member of the committee's liaison group, described two possible approaches to providing veterans' preference in a local office operating under the VG-GATB system: the first would be to continue to rely on the method that times referrals, sending only qualified veterans (in rank order) in response to a job order for the first 24 or 48 hours. The second means would be to apply ordering preference within a range of percentile scores. Although not coming out in favor of a particular approach, Rhoades, like his colleagues in the other veterans' organiza- tions, did reject a simple tie-breaker rule as incompatible with the spirit of the law (Rhoades, 19881. Veterans and Within-Group Score Adjustments Questions have been raised about a possible conflict between the VG-GATB within-group scoring procedure and veterans' statutory entitlement to preferential treatment in employment referral. The American Legion maintains a neutral position on score adjustments
230 ASSESSMENT OF THE VG-GATB PROGRAM intended to promote equal employment opportunity for minority-group members "if there is validity established for their use." The Veterans of Foreign Wars, however, has taken the position that the conversion of minority scores is illegal because it deprives nonminority veterans of their legal right to priority in referral. The point was illustrated in a letter (dated May 3, 1988) to the Assistant Secretary of Labor for Veterans' Employment and Training from Robert L. Jones as follows: if a white veteran receives a Job Family IV percentile score of 41, any black applicant with an identical raw score will receive a percentile score of 81; the score for a Hispanic applicant would be 63, and that for a Native American, 59. There is no doubt that within-group score adjustments would have the elect of drawing minority applicants (veteran and nonveteran alike) into the referral pool while excluding majority-group veterans with the same raw scores. It is also true that government grants of preferential treatment can never be absolute; one person's preference qualifies that of the other. Whether the preference granted by statute to veterans takes precedence over federal attempts to remedy the effects of discrimination against blacks and other protected groups in this case seems to be an open question. Some have suggested that the way to resolve the issue is to compute adjustments in the scores of veterans as a group. Were there average group differences between veterans and nonveterans, it would be possible to consider such a solution. However, based on admittedly limited data comparing scores in all five job families for two groups defined by veteran status, it appears that veterans and nonveterans have roughly the same mean percentile scores. For example, the 1987-1988 figures for one local office in Michigan show: Average Percentile Score Job Family Veterans (N= 1,100) Nonveterans (N= 3,900) IV 54 52 V 53 55 The statewide applicant pool in Utah in early 1988 shows: Average Percentile Score Job Family Veterans (N= 9,958) Total (N= 93,504) IV 50.36 52.14 V 47.00 53.82
POTENTIAL EFFECTS OF THE VG-GATB REFEREE SYSTEM 23) Unless there are larger average group differences, development of con- version tables for veterans would not significantly change their chances of referral. But a test-based system that does not have some adjustment like the current within-group conversions would seriously reduce the chances of minority veterans. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Employers Conclusions 1. Although empirical evidence is sparse, the use of VG-GATB scores top-down to refer applicants appears to offer an employer benefits such as modest improvements in worker performance and reduced training time. Certainly a large majority of the employers who communicated with the committee perceived the VG-GATB system to improve their use of human resources. 2. However, these benefits will tend to attenuate as more employers who compete in the same labor market adopt VG-GATB procedures. Job Seekers Conclusions 1. Compared with subjective procedures, test-based referral can po- tentially reduce the risk that racial or ethnic prejudice will influence referral decisions. 2. At the same time, tests are fallible and they give a narrow reading on human capabilities. To limit all job seekers to a single test-based modality would artifically restrict the opportunities of many applicants. 3. The VG-GATB system has the potential disadvantage for older, experienced workers of basing referral on less relevant information (test score) when more relevant information (past job performance) is avail- able. 4. The VG-GATB Referral System, were it to be the only mode of referral offered through the Employment Service system, would consign the lowest-scoring applicants to receiving little or no assistance in finding work when, in fact, many such applicants could perform satisfactorily on the job. 5. Without the kind of adjustments currently made to the scores of black, Hispanic, and in some cases Native American applicants, the VG-GATB Referral System would have severely adverse impact on
232 ASSESSMENT OF THE VG-GATB PROGRAM members of these demographic groups. The severity of the adverse impact leads us to conclude that the VG-GATB Referral System is probably not viable without adjustment of minority scores so long as the government is committed to a policy of equal employment opportunity that looks to the effects of employment practices on racial and ethnic minority groups. 6. It is not reasonable to use the GATB to estimate the abilities of foreign- born applicants who have a marginal command of the English language. Recommendations We recommend that no job seeker be obliged to take the GATB; every local office that uses VG-GATB referral should maintain an alternate referral path for those who choose not to take the test. 2. Because tests provide only partial information about future job performance, we recommend that local Job Service offices that adopt the VG-GATB Referral System continue to use multiple criteria in choosing which applicants to refer. People with Handicapping Conditions Findings 1. The central scientific questions concerning the use of standardized tests to estimate the academic or job performance of people with handicaps have revolved around the question of comparability. When such tests are modified to accommodate visual, hearing, motor, or other handicaps, do the modified and regular instruments measure the same abilities? Even if they do, are the resulting scores comparable? 2. Recent research carried out by the Educational Testing Service to investigate these questions in terms of its major cognitive test batteries, particularly the Scholastic Aptitude Test Battery, demonstrates the extreme difficulty of gathering sufficient data to answer these questions empirically. Although the SAT is one of the three largest testing programs in the country and is supported by an extensive data base, the ETS research team had great difficulty assembling an "even minimally suffi- cient" data set to study the validity of scores on modified versions of the SAT for predicting first-year college grade-point average. It was virtually impossible to take account of degree of handicap, and even within type of handicap the numbers for whom all data were available were often small. Even greater obstacles would confront an effort by USES to establish the comparability of scores on regular and modified forms of the GATB. The
POTENTIAL EFFECTS OF THE VG-GATB ~FER~L SYSTEM 233 problem of inadequate sample sizes would be even more severe, and the job performance criterion would be more difficult to capture. Conclusions 1. The GATE cannot play the same role in assessing the qualifications of handicapped and nonhandicapped applicants. 2. Because matching people with handicaps to jobs necessitates con- sideration both of their abilities and of their particular disabilities in light of specific job requirements, the VG-GATB Referral System, with its emphasis on test scores and automated file search, is not adequate to the situation. Job counselors are essential to the referral process for handi- capped applicants. Recommendations 1. For applicants with handicapping conditions we recommend the continued use of job counselors to make referrals. 2. The GATE should be used when feasible to assess the abilities of handicapped applicants, but as a supplement to decision making, not to take the place of counseling services. 3. Because special expertise in assessing the capabilities of people with handicaps is necessary and available, we recommend that the Department of Labor encourage closer coordination between state rehabilitation agencies and the State Employment Service Agencies. Consideration should be given to placing state rehabilitation counselors in local Employment Service offices that service a sizable handicapped population. 4. Steps should be taken to ensure that no job order is filled automati- cally and solely through the VG-GATB system. Job counselors who serve handicapped applicants, disabled veterans, or other populations with special needs must have regular access to the daily flow of job orders. Veterans Conclusions 1. The language of the legislation and regulations conferring priority or preference in employment on military veterans consistently uses the terminology "qualified veterans." We infer from this that the intent of Congress was to balance considerations of productivity with preferential treatment for veterans.
234 ASSESSMENT OF THE VG-GATB PROGRAM 2. One of the methods of referral priority in use, the tie-breaker method, provides little advantage to veterans. 3. Absolute priority (ranking all veterans before all nonveterans regardless of test score), however, ignores the goal of promoting produc- tivity. It effectively removes the word "qualified" from the implementa- tion of the law. 4. Although two of the methods of referral priority, variants of the decile method, confer quite a bit of advantage on veterans, adding points to percentile scores has the anomalous effect of helping veterans with high scores substantially more than veterans with moderate test scores. Recommendations 1. If government policy is to strike a balance between maximizing productivity and preference for veterans in employment referral through the VG-GATB Referral System, the Employment Service should adjust veterans' VG-GATB scores by adding a veterans' bonus of some number of points before conversion to percentiles. Unadjusted expectancy scores should also be reported to employers and job seekers. It should be noted on the referral slip that the percentile score has been adjusted for veterans' preference. If the federal rule were followed, the size of the adjustment would range from one-eighth to one-quarter of a standard deviation corresponding to 5 and 10 percentile points, depending on disability status. 2. The Employment Service should continue to meet the needs of disabled veterans through individualized counseling and placement services.