Click for next page ( 192


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 191
10 ; The VG-GATB Program: Concept, Promotion, and Implementation When the U.S. Employment Service (USES) staff became committed to the VG-GATB idea, they envisioned not just a new role for the General Aptitude Test Battery (GATB), but a whole new regimen that would revitalize the Employment Service system and change its reputation as the referral service of last resort. Building on a shift away from the policy of the 1960s and early 1970s, with its emphasis on the needs of targeted groups of job seekers whom the government wanted to draw into the mainstream economy, those who developed the VG-GATB idea sought to reorient the Employment Service to the needs of employers and of the economy as a whole. Believing that service could be optimized for only one of the Employment Service's three clients-employers, applicants, and the nation (or economy) as a whole they chose to concentrate on the employer (Hawk et al., 1986:21. They developed a plan for a thorough- going reorganization of the way local offices operate, a plan designed to rationalize the system, improve the quality of people referred to employ- ers, and thereby attract more and better job orders. In this chapter we present the plan worked out by USES staff for local office operations under the VG-GATB Referral System. We also discuss the sometimes ill-founded claims that have been made for the system and evaluate the results of VG-GATB referral as they have been documented in a handful of pilot studies. 191

OCR for page 191
|92 ASSESSMENT OF THE VG-GATB PROGRAM CONCEPTUAL MODEL OF THE VG-GATB REFERRAL SYSTEM As the details of the plan were worked out with the aid of a number of pilot programs, USES staff put together a conceptual model of the VG-GATB Referral System to assist local offices in transforming their operations. The document is not an official statement of policy, but was used widely in workshops and conferences for state and local Employ- ment Service personnel. Although most of the offices using the VG-GATB Referral System added VG procedures on top of the existing system rather than replace the old procedures, it seems useful to look at the ideal version being promoted by USES. The following discussion is drawn entirely from the document, "Conceptual Model of Full Service Validity Generalization Local Offices', (Hawk et al., 1986~. At the center of the basic principles that inform the conceptual model of a fully implemented VG-GATB Referral System is the idea of high- quality service to employers (and to the economy). Flowing from that overriding goal is the principle that "almost all ES resources" should be devoted to activities designed to provide employers with the most capable available applicants, namely: 1. intake (reception and registration), 2. assessment (testing and evaluation of education, training, and expe- rience), 3. file search, and 4. referral. It follows in this plan that most applicants should be mainstreamed, that is, they should take the GATE and compete on an equal basis for available job openings. The model recognizes that the standard VG procedures are not applicable to some job seekers, including non- English-speaking and illiterate applicants and those with handicapping conditions. It is suggested that local offices maintain close ties with community organizations that are set up to handle people with such special needs so that they can provide efficient referral to the appropriate agencies. The Job Service itself would be reserved largely for the able-bodied and the competitive. Figure 10-1 illustrates the full-service VG-GATB office. For conve- nience, the authors of the document have simplified operations some- what. For example, file search for veterans, which would usually be a separate operation, is not separate in the figure. The standard process moves in a straight line from top to bottom of the flow chart. A job seeker entering the VG-GATB Referral System goes first to reception, proceeds to group orientation and registration, and then on to GATE testing. Scores are entered into the (computerized) applicant

OCR for page 191
CONCEPT, PROMOTION, AND IMPLEMENTATION 193 file. As job orders come in, the best jobs are dealt with first. The files are searched starting with the highest scores so that the best available jobs and the best available applicants are matched. The final stage in the standard process is referral. At each stage in the process, some individuals would be identified who either cannot participate in the standard process or who are unlikely to benefit from it. Listed at the right of Figure 10-1 are in-house and community services to which the local office can refer such individuals to help make them more competitive. The emphasis is on referring noncom- petitive applicants to appropriate external services or assistance-public or private job training organizations, job finding clubs, remedial educa- tion, and self-improvement courses- so that, following remediation, they can reenter the standard referral process. This plan envisions an active role for the Job Service stab in diagnosing the specific problems that prevent applicants from proceeding through the standard process and putting such people in contact with appropriate services. As the diamonds at the left of the figure illustrate, the model also provides counseling services for those in the applicant file who are not being referred because of low scores or lack of marketable skills, or who have a record of multiple unsuccessful referrals. This level of service would require automated file search and so would be possible only in local offices that have computerized files. Group Orientation and Registration One notable characteristic of the VG-GATB Referral System is the attempt to streamline the processing of applicants. Under the old proce- dures, each person entering the local office talks with an interviewer, who assists the individual in filling out an application form. In a busy office, this could entail long waits for the next available counselor. Orientation and application-taking in groups means that one staff member can take care of the preliminaries for many applicants at once and, in the authors' estimation, in about one hour's time. The conceptual plan suggests that orientation sessions be conducted as frequently as the number of people in the reception area warrants, and that the orientation include an overview of the Employment Service, a talk on the benefits of test-based referral, information on other services provided by the local office, and instructions on how to fill out the application form. Part of the orientation is intended to convince applicants that it is in their interest to take the GATE, since most job orders will be filled using VG-GATB procedures. A brochure, "Doing Your Best on Aptitude Tests," is available, and formal pretesting orientation is suggested for those who seem very apprehensive.

OCR for page 191
194 Ul I. o ~ o o ~o' 2 ~ id, 2 ~' 0 2 2 0 ~~ ~ 8 >~-Cot CO ~ US ~ _ ~ ~ _ ~ ~ ~ ~ ~ ~O _ ~ ~ ~ ~ ~ ~ _ G <: m ~ui ~ ~ ~ ' ~ ~ 1~1 \1~' O a) {IS O ~ 2 ~ Q _ O o on credo\: - \~4 J - m c ~ ._ CD

OCR for page 191
195 - 0 cn CO ~ C o _& _ ~c a) In ' Q cow C _ on o ta E E o E O ~ en n <,, ~ - as O ~ s c cat _ 'A ~ In en In a) Q .... CO _ Cal ~ ~ LL CD o ._ ~, - ~n o .O ._ . . .. . . . u, ~ _ c~ co ~ u, co ~ 0 u, n CO ~ ~ ~ = ~= ~ - == ~ ~ 2 ~ ~ ~ == c,, 8 c~ ,q,.o; E C "~"t ~ ~ E ~ ~ ~ ~ ~ o V' o ~ 3 c,, =. s: ~ c~ 0 a~ ct -4 O o ~ ~ D 3 v D~ ~ ~1 Ct C ~ Ct _ ~ 1T ~, __ ~ ~g-~o~ ~2~_o o' 1 ~o cn O q, O_ c

OCR for page 191
]96 ASSESSMENT OF THE VG-GATB PROGRAM All applicants who can take the GATB would be scheduled during the orientation session to do so. Local office staff are reminded that special note should be made of any who require a special administration because of a handicapping condition; individuals whose handicap would interfere with performance on the GATB would be referred to a counselor for a personalized assessment and probable referral to a job development program. Such applicants would be exempted from the VG-GATB system. Applicants who do not take the GATB, the document suggests, should be referred to a counselor for advice on remedial services, alternative assessment methods, or possibly special placement. Their applications would be entered into the file, but they should be informed that file search of tested applicants always precedes search among untested applicants. The files of those who take alternative tests can be placed in the regular VG-GATB file, but they must be flagged to indicate the need for special placement assistance. GATB Testing Testing is, of course, the centerpiece of the referral system. Once again, group processing is recommended; testing is by appointment so that the size of the group and the timing of test administration can be suited to local conditions. The conceptual plan emphasizes the impor- tance of good test administration procedures, pointing out that test scores will have a large impact, whether positive or negative, on an applicant's job prospects, possibly over several years. When the test has been scored, records of applicants with competitive scores are entered into the applicant file; those with very low scores are not likely to be referred under the VG-GATB system and, the document suggests, should probably be sent to a counselor so that the applicant can be directed to remedial services or vocational training available in the community. The counselor might also try a specialized placement such as subsidized employment programs or sheltered workshops. File Search and Referral When an employer places a job order, local office staff assign it a code number from the Dictionary of Occupational Titles (DOT) and, in the VG-GATB system, one of the five job family designations. The files are then searched for the applicants with the highest test scores in that job family; applicants are selected for call-in from the top down. If the job order imposes an experience requirement, then selection will be from the highest test score on down among those with the appropriate experience.

OCR for page 191
CONCEPT, PROMOTION, AND IMPLEMENTATION 197 Veterans enjoy certain rights to priority in file search and/or referral, which must also be built into the file search system. On the basis of the file search, applicants are called (in rank order of test score save for veterans' priority) and those interested in the job are asked to report to the Job Service office to schedule an interview with the employer and pick up a referral card. It is worthy of note that, whatever the other efficiencies of the VG-GATB Referral System, it does take the job seeker three trips to the local office to get a first referral. And the waiting time for some employers is also extended. Under the old system, most job orders tended to be filled from among those individuals who happened to be in the office. For an employer who needs someone on the job the next day, that system would be preferable. However, employers as well as job seekers who want to increase their options would probably benefit by the file-search and call-in method. Office Automation The operational efficiency of the VG-GATB system is heavily depen- dent on computerization of the applicant files. In describing the concep- tual model of the VG-GATB local office, USES devotes a good deal of attention to the experiences of the various states that automated local office operations as part of the VG-GATB experiment. Some have been set up so that a central office receives the answer sheets and maintains the applicant files. In these systems, file search and job matching can be done on a central mainframe computer, and the information can be available across the state. A few states (Arizona, Missouri, Oregon, and South Dakota) have integrated systems; that is, the personal computers in the local office can send test data directly into the mainframe and receive information directly from it. This would seem to offer real advantages in states in which large employers draw from a number of local offices, as is the case with the automobile companies in Michigan. At some time in the future, it could also allow the job seeker to cast a wider net, if job order information for a region or the entire state were available at each local office. An analogy can be drawn to the military selection and classification system, in which 66 Military Entrance Processing Stations (MEPS) across the nation are linked to a central computer. Every potential recruit's test scores and other relevant data are entered directly into the central file, and job counselors at the MEPS have immediate access to information about the jobs available in their Service for which the applicant qualifies. Many of the local offices that have introduced automation have adopted a decentralized system, with each office doing its own scoring and record keeping. Some have a mainframe, and some work strictly with personal computers. In some states, the local offices have their own scoring equip

OCR for page 191
i98 ASSESSMENT OF THE VG-GATB PROD ment but plan to link up with a central mainframe for file maintenance and file search. The development of software for file maintenance and file search activities has been an important part of the VG-GATB experiment (and, more generally, of the modernization of Employment Service operations). Some states are using system-specific software, but over half the states have adopted a package called the Enhanced National Data System (ENDS) for at least some aspects of file management. And, since 1986, some have been exploring a job-match software package called On-line Data Display System (ODDS). The conceptual document explains that automated file search, which it calls "job matching,'' is a key to efficient operation of the VG-GATB Referral System. It allows entry of the employer's selection criteria (e.g., years of experience, education or training requirements, cutoff score on the GATE below which the employer will not accept referrals) and virtually instantaneous identification of the pool of applicants with the highest percentile scores who also meet the employer's criteria. Software systems should be designed to allow keyword matching, the authors point out, so that more precise matching of employers' requirements and applicants' skills can be made than is possible under the very general DOT categories. Within a single DOT code, for example, an applicant's record might specify 46529 secretarial work, 46551 general clerical work, and/or 88595 word processing. Comment on the Conceptual Model There can be little doubt that the automation of Job Service operations offers enormous efficiencies when compared with manual file search. This is true whether or not the local office is using the VG-GATB Referral System. What is less clear, however, is how much the principle of top-down selection, which is the fundamental organizing principle of the VG-GATB concept, actually promotes higher quality in the group of applicants sent to an employer in response to a job order. Given the modest validities of the test, which mean that there is a great deal of error in the rank-ordering of applicants, and given the other important selection criteria already used to winnow the pool of applicants (experience, etc.), the increment of greater productivity contributed by test-based selection could be quite small indeed, particularly for jobs that require specialized training or experience. PROMOTION OF THE VG-GATB REFERRAL SYSTEM Many of the people involved in developing the VG-GATB system have been very enthusiastic about its potential. USES staffin the central office

OCR for page 191
CONCEPT, PROMOTION, AND IMPLEMENTATION 199 and in the regional Test Development Field Centers have advertised the merits of the system to the states and to Employers' National Job Service Committees and have worked hard to promote the VG-GATB idea throughout the Employment Service system. The supposed benefits to employers have been written about by a number of satisfied users in magazine and journal articles, both popular and professional. Although the level of enthusiasm brought to this attempt to improve an important government service cannot but be praised, the committee is concerned by exaggerated and ill-supported claims made for the VG- GATB system, as well as by a noticeable lack of qualifiers and cautions. Our unease begins with the technical reports of the USES test research themselves. Reports No. 43 through No. 47 lay out the scientific basis for the generalizability of the GATB and the estimated economic benefits of the VG-GATB Referral System (U.S. Department of Labor, 1983a,b,c,d,e). These reports, written at a level accessible to the general reader, consistently present the most optimistic interpretation possible. In other chapters of this report we present specific critiques of the various claims made in the technical reports. Here we remark on the tone of the reports because of their influence on Job Service employees and clients who may not be in a good position to judge them critically. For example, Report No. 43, a summary of John Hunter's research on the GATE, prepared by a private consulting firm, begins with the following claim (U.S. Department of Labor, 1983a): The U.S. Employment Service can help to improve the productivity of American industry on the order of 50 to 100 billion dollars in the upcoming year. Sound ridiculous? Not at all. In fact, the impact of the Employment Service on the U.S. economy can be accomplished by a new use of an already established and widely accepted Employment Service device for matching people and jobs the General Aptitude Test Battery in a way that makes the best use of state-of-the-art research evidence. The kindest interpretation of this claim is that it is the hyperbole of an enthusiast. Not only is it, as our analysis in Chapter 12 indicates, the product of a theoretical mode! based on slight empirical evidence; not only does it totally ignore the big difference between theory and likely effects in real-world applications; but it also chooses to express economic gains in a metric (aggregate dollar amounts) that is designed to impress, dazzling all but a small number of macroeconomists who can sensibly interpret numbers of this magnitude. We cannot claim to have conducted a thorough investigation of the way the VG-GATB system is being advertised to employers and job seekers at the state and local levels. Most committee members have visited at least one local office. We have developed impressions based on information

OCR for page 191
200 ASSESSMENT OF THE VG-GATB PROGRAM provided by members of our liaison panel. We have during the past 18 months or so received phone calls and letters, both solicited and unso- licited, from employers, job seekers who have taken the GATB, and people who work in the Employment Service system. And we have frequently found ourselves troubled by an overly optimistic assessment of the benefits and a downplaying of the potential legal problems with the VG-GATB Referral System. For example, a small number of employers told the committee of their shock when the VG-GATB was challenged by the Department of Justice because of its score adjustment procedures. They felt that they had been led to believe that the VG-GATB system would solve potential legal problems by preventing adverse impact in their selection procedures, not make them vulnerable to charges of reverse discrimination. One can see at this distance that this misperception was partly a product of mistaking the Department of Labor's sponsorship of the VG-GATB as a more general stamp of approval by "the Government." But the misperception is also positively encouraged by some of the literature used to promote the system. One state uses a brochure to attract employer-clients that states: The Legal Implications. The legality of the General Aptitude Test Battery has been addressed in court cases. The courts have upheld GATB testing as fair and said the testing system conforms to EEOC guidelines. As a matter of fact, the GATB has been the subject of very little Title VII litigation, too little to talk about "the courts." And the case that was almost certainly the basis for this claim (Pegues v. Mississippi State Employment Service, 699 F.2d 760 [19801) is ambiguous at best. Brought in 1979, it involved a use of the GATB that predates the introduction of the VG-GATB system, with its claim of validity for all 12,000 jobs in the U.S. economy and its use of within-group scoring. In addition, although the trial court accepted arguments for the transportability of validities on the basis of early evidence from the validity generalization research, the appellate court ruled that, in the absence of adverse impact, the testing issue need not be addressed. That record is not sufficient to support the degree of legal acceptability implied by the brochure. Although in these paragraphs we intend to communicate our very real concern with the way the VG-GATB Referral System has been pro- moted and we recommend far more circumspection in the future we must also take note of the severe budget reductions that have forced deep cuts in both staff and research resources at the national, state, and local levels of the Employment Service system in the past decade. The staff members who have become wedded to the new system have done so

OCR for page 191
CONCEPT, PROMOTION, AND IMPLEMENTATION 20 1 partly in response to this serious decline in resources. They have tried to do something about the situation by reorganizing the system to make local operations more efficient, by attracting better job orders through advertising the presumed benefits of the VG-GATB to employers, and by finding ways to handle more job seekers in a given period of time to alleviate the effects of reduced staffing levels. For this they deserve kudos. IMPLEMENTATION OF THE VG-GATB REFERRAL SYSTEM: THE PILOT STUDIES One of the unfortunate consequences of the above-mentioned financial constraints has been that early plans for an ambitious program of pilot studies have gone astray. The empirical evidence of the effects of the VG-GATB Referral System that should have been abuilding in the past eight years has failed for the most part to materialize. States were encouraged to make a gradual implementation, accompanied by careful program evaluation. In most cases, however, these evaluations have been limited to comparing the number of placements made before and after VG-GATB procedures were introduced into an office, a comparison complicated by the downturn of the economy and large staff cuts at the time the experiments were started and by the fact that in most cases there was little disruption of existing practices. Although 40 states have now introduced the VG-GATB system in one or more local offices, either to supplement or to replace traditional operations, only a few formal studies have been conducted to evaluate the effects of the system. The committee evaluated five pilot studies, two growing out of the very early experience with the new program in local offices in North Carolina and three that look at the effects on job performance in three firms. All the studies purport to show improvements attributable to the introduction of the VG-GATB Referral System in one or more of the following areas: staff); 2. market penetration (Employment Service placements as a percent- age of total new hires in the area); and 3. improved worker performance (measured in terms of absenteeism, quality of workmanship, disciplinary actions, training success, or produc- tivity). office effectiveness (number of placements made relative to available However, of the five studies conducted, only one, that by Madigan et al. (1987), satisfies rigorous research standards; the others can be consid- ered no more than suggestive.

OCR for page 191
202 ASSESSMENT OF THE VG-GATB PROGRAM North Carolina Study of Office Electiveness The first study, based on data from 1981-1982, is one of two conducted by M.W. McKinney (1984) as part of the very earliest experiment with the VG-GATB Referral System. It is a study of focal office effectiveness, comparing offices that did and did not use the new system. Data are presented on indices of effectiveness, work load, and two control varia- bles for three groups of local offices: (1) seven VG-GATB offices in the Raleigh-Durham area; (2) the 48 remaining (non-VG-GATB) offices in North Carolina; and (3) a subset of seven from the second group that were matched on a one-to-one basis with the seven in the first group. Matching was on the basis of office staff numbers and insured unemployment rate. Means are presented by group for both 1981 (before VG-GATB was implemented in any North Carolina office) and 1982 (after implementation at the seven pilot sites). These data show that between 1981 and 1982, the seven VG-GATB offices improved more than the other local offices with respect to the number of referrals and job placements made; staff "productivity" (number of placements made relative to the number of staff available); and penetration rate (the number of Job Service place- ments divided by the total number of new hires reported in the area served by the focal office). There was no difference among the groups of offices in the mean increase in salary of the new hires. No data are presented regarding the consistency of effects across the seven matched pairs of sites. Only weighted means are given, without explanation of how the weights were derived. Although McKinney's findings are suggestive, there are several serious threats to their validity. The seven VG-GATB sites are all from the same part of the state, a design factor that, while convenient for operational reasons, could have influenced the results. At baseline (1981), the seven pilot sites differed substantially from the seven "matched'' sites on some key variables. For instance, the seven VG sites were almost one standard deviation lower in productivity, and Raleigh had a disproportionately large demand for secretarial/clerical workers. The interpretability of the results is further blurred by the fact that the VG-GATB system was not even close to fully implemented at the seven VG-GATB sites. Only 18 percent of the applicants at these sites were tested in 1982 instead of the hoped-for 80 percent. North Carolina Employer Survey The second study, conducted by McKinney in 1983, was a face-to-face survey of 295 employers who were asked questions about their hiring practices and experiences and their attitudes toward the Employment

OCR for page 191
CONCEPT, PROMOTION, AND IMPLEMENTATION 203 Service and the VG-GATB Referral System. The participants were among a larger group of employers who had been briefed on the VG-GATB system a year earlier. Of the 173 employers who hired anyone during the 18 months of the pilot program, about 52 percent said that the VG-GATB system made the Employment Service more useful to them; 44 percent said applicant quality improved; and 25 percent said VG-GATB selection reduced training and other personnel costs. These variables appeared to interact with each other and with attitude toward the Employment Service. Again, the results are suggestive, but there are problems with the survey. Employer impressions about hiring experiences were not verified against actual records, although this could have been done with the data base from the first study. Employers who hired hundreds of workers were lumped together with those who hired only one. And employers whose job orders were filled under the VG-GATB procedures were lumped together with those whose referrals may not even have been tested. Above all, the interviews were not neutral. They were conducted by staff working at the Pilot study sites who began each session with the following statement: We have found that Validity Generalization, or VG as we call it, has shown a positive relationship between General Aptitude Test Scores and success on the job. The Philip Morris Study D.L. Warmke (1984) conducted a study to assess the impact of VG-GATB hiring on worker performance. From a pool of 32,000 appli- cants, 1,200 employees were selected at a new Philip Morris plant in Cabarius County, North Carolina, using a four-step screening procedure. At each stage, the percentage of applicants screened out was: Procedure - Application VG-GATB Interview/physical Noncompensated training Screened Out (pro) 40 2 The study compared the performance of the 1,200 employees at the new plant to three other groups: (1) the 32 employees who transferred to the new plant from other plants, (2) employees at another Philip Morris plant, and (3) industry-wide averages. There were also comparisons to quality and production goals set by management. Employees at the new plant performed better than those in the other groups. For instance, compared with the non-VG-GATB plant, the

OCR for page 191
204 ASSESSMENT OF THE VG-GATB PROGRAM workers at the new plant had: an 81 percent lower lost-workday severity rate; a 59 percent lower quality-defect rate; 58 percent fewer disciplinary actions; and 8 percent greater success in training. These improvements are attributed primarily to the VG-GATB because that procedure screened out about as many applicants as the other three hurdles combined. The credibility of this interpretation rests on the comparability of the comparison groups; however, there are no data to indicate the comparability. Were the same standards used in making application screening decisions at the VG-GATB plant and the other plant? Did the other plant screen out 40 percent of its applicants on the basis of the application? Did the other plant also have a 1:27 selection ratio? Conspicuous by their absence are data contrasting the two plants on productivity goals set by management. And no data are provided regard- ing the correlation between GATE scores and performance measures. The author recognizes that locational differences could explain the reported results. He argues against this interpretation because the 32 employees not selected by VG-GATB procedures who transferred from other plants did not perform as well in training as the others at the new plant. However, there is no reason to believe that the transfers were similar to the test-selected employees in other relevant characteristics, nor that the training results would carry over to job performance. The Chrysler Corporation Study The Northern Test Development Field Center (1987) of the USES conducted a study comparing the performance of 246 employees hired by means of VG-GATB scores with two other groups: transfers from other plants (N = 422) and rehires of people from other plants who had used up their recall rights (N = 468~. Workers in the last group were selected on the basis of previous supervisor performance evaluations. Five performance measures were used in the study: a supervisor rating of productivity, quality, tool use, job knowledge, adaptability, and overall ability; a supervisor's rank-ordering on overall quality of all of his or her employees; excused and unexcused absences; supervisor's rating of frequency of visits to the medical department; and recommendation to rehire. On virtually all the measures, the new hires earned better scores than the transfers, who in turn earned better scores than the rehires. However, none of the differences was very large. For example, the VG-GATB group's mean on the first rating scale (productivity, quality, etc.) was only about one-quarter standard deviation above that for all employees. There

OCR for page 191
CONCEPT, PROMOTION, AND IMPLEMENTATION 205 was almost no variation on the recommendation-to-rehire measure 89 percent of the workers received the highest rating. This study provides some support for the hypothesis that selection using the VG-GATB can enhance the quality of worker performance, but the magnitude of observed effects is quite small and could have been produced by factors other than the new referral system. (Were the new hires motivated by probationary status?) The study could be improved substantially by analyses that control for some important but unmeasured factors, such as age, sex, and job mix. It would also be useful to know whether the amount of job experience in the two comparison groups is positively correlated with performance. The Sewing Machine Operator Study By far the most ambitious and most successful study was that undertaken by the State of Virginia and conducted by Madigan et al. (1987~. Its purpose was to assess the validity of VG-GATB selection procedures and to develop estimates of the potential economic benefit to the firm of using such procedures. The study involved sewing machine operators in five plants owned by a manufacturer of casual wear. A group of 751 of a total of 932 new employees was tested with the GATE prior to hire, but their scores were not used in hiring. Within this group, 27 percent were black and 65 percent had no previous experience; all were women. Their mean VG-GATB Job Family V score was 51, with a standard deviation of 26 and a range of 1 to 99. Since the job involves piecework, the performance criterion was actual production records plus supervisor ratings of quantity of output, quality of output, flexibility, dependability, receptiveness to instruction, and an overall rating. Turnover and reason for leaving were also recorded. Employees were paid on a piecework basis and, within the limits of error of industrial engineering studies, employees who worked at the same rate of output but on different operations received the same pay. This allowed the computation of a "time-to-standard" measure (the number of weeks it took the employee to produce at a rate that would earn $3.75 per hour and $4.79 per hour). An analysis of VG-GATB test scores by quarters of the distribution showed that employees in the higher quarters have generally better performance than those in the lower quarters. For example, the average number of weeks it took employees in each quarter to reach the $4.79 standard was:

OCR for page 191
206 ASSESSMENT OF THE VG-GATB PROGRAM Quarter 4 (76~ o) 3 (51-75%) 2 (26-50%) 1 (1-25%) Weeks to Standard 13.6 14.4 14.1 16.4 The percentage of employees in a quarter who were black increased with quartile number; almost half of those above the third quartile (the highest scores) were black workers. Comparison of experienced and inexperienced employees showed that they had similar GATB scores and percentages of black workers (29 percent and 24 percent, respectively). However, the experienced workers tended to reach the $4.79 standard faster, averaging 11.1 weeks to standard compared with 15.4 weeks for the inexperienced workers. Correlations between GATB scores and monthly production averages were statistically significant and ranged from .15 to .24. In addition, turnover rates tended to be higher among those with lower scores. This study is noteworthy because of the relatively large sample (751 employees), the lack of any restriction in range in that sample (the GATB scores ranged from 1 to 99), and the reliability of the criterion (objective measures of output). The results, since they require no corrections or adjustments, pretty well mean what they say: there is a tendency for sewing machine operators with high VG-GATB Job Family V scores to perform better and have less turnover than those with low scores. However, this relationship is not very strong: the average of all validity coefficients is about .20. CONCLUSIONS AND RECOMMENDATIONS Promotion of the VG-GATB Referral System Conclusions 1. The VG-GATB Referral System is frequently oversold, both at the national and at the state levels. Much of the promotional literature that we have seen overstates the psychometric quality and predictive power of the GATB, underestimates the vulnerability of the referral system to legal challenge, and exaggerates the economic impact of preemployment testing. 2. The overselling of the VG-GATB system in Employment Service advertising is encouraged by the tenor of the technical reports describing the research that undergirds the referral program. They provide optimistic projections of the effects of VG-GATB referral for which the empirical evidence is slight.

OCR for page 191
CONCEPT, PROMOTION, AND IMPLEMENTATION 207 Recommendations 1. Given the modest validities of the GATE for the 500 jobs actually studied; given our incomplete knowledge about the relationship be- tween this sample and the remaining 11,500 jobs in the U.S. economy; given the Department of Justice challenge to the legality of within-group scoring and the larger philosophical debates about race-conscious mechanisms and the known problems of using a test with severe adverse impact; given the primitive state of knowledge about the relationship of individual performance and productivity of the firm, we recommend that the claims for the testing program be tempered and that employers as well as job seekers be given a balanced view of the strengths and weaknesses of the GATE and its likely contribution in matching people to jobs. Implementation of the VG-GATB Referral System: The Pilot Studies Findings 1. Although 40 states have introduced some form of the VG-GATB Referral System in a number of local offices, and a small number have experimented with the program statewide, only a few formal studies have been conducted to evaluate the effects of the program. 2. The five pilot studies evaluated by the committee showed some improvements resulting from introduction of the VG-GATB Referral System in one or more of the following areas: a. office effectiveness (number of placements made relative to available stalk; b. market penetration (Employment Service placements as a per- centage of total new hires in the area); and c. improved worker performance (measured in terms of absentee- ism, quality of workmanship, disciplinary actions, training suc- cess, or productivity). 3. However, of the five pilot studies evaluated, only one satisfied rigorous research or evaluation standards. Although some of the other studies were suggestive, the committee cannot place a great deal of confidence in their favorable assessments of the effects of the VG-GATB Referral System, including the evidence of improved job performance.

OCR for page 191
208 ASSESSMENT OF THE VG-GATB PROGRAM Conclusions 1. There is too little evidence based on controlled, rigorous studies of the effects of using the VG-GATB Referral System for the committee to be able to assure policy makers at the Department of Labor that anticipated improvements have indeed occurred. This is not to say that they have not occurred. The evidence simply does not exist to establish the case scientifically. For the moment, policy decisions about the future of the VG-GATB Referral System will have to be made on the basis of more impressionistic and experiential information. Recommendations 1. If USES decides to continue the VG-GATB Referral System, a series of carefully designed studies should be undertaken to establish more scientifically the efficiencies that are believed to result. 2. This research will need to be a cooperative effort, involving federal and state Employment Service personnel and employers. USES should encourage state Employment Security Agencies that deal with large employers (e.g., Michigan) and states that have fully articulated VG- GATB systems in place (e.g., Virginia, Utah, Oklahoma) to take a leading role in conducting studies to demonstrate the efficacy of the VG-GATB Referral System. 3. We also recommend that the employer community, as a potentially major beneficiary of an improved referral system, take an active part in the effort to evaluate the VG-GATB Referral System. The Employers' National Job Service Committee can help to identify appropriate employ- ers who are willing to commit the resources necessary to study the effects of VG-GATB referral.