|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 191
10
·;
The VG-GATB Program: Concept,
Promotion, and Implementation
When the U.S. Employment Service (USES) staff became committed
to the VG-GATB idea, they envisioned not just a new role for the General
Aptitude Test Battery (GATB), but a whole new regimen that would
revitalize the Employment Service system and change its reputation as
the referral service of last resort. Building on a shift away from the policy
of the 1960s and early 1970s, with its emphasis on the needs of targeted
groups of job seekers whom the government wanted to draw into the
mainstream economy, those who developed the VG-GATB idea sought to
reorient the Employment Service to the needs of employers and of the
economy as a whole. Believing that service could be optimized for only
one of the Employment Service's three clients-employers, applicants,
and the nation (or economy) as a whole they chose to concentrate on the
employer (Hawk et al., 1986:21. They developed a plan for a thorough-
going reorganization of the way local offices operate, a plan designed to
rationalize the system, improve the quality of people referred to employ-
ers, and thereby attract more and better job orders.
In this chapter we present the plan worked out by USES staff for local
office operations under the VG-GATB Referral System. We also discuss the
sometimes ill-founded claims that have been made for the system and
evaluate the results of VG-GATB referral as they have been documented in
a handful of pilot studies.
191
OCR for page 192
|92 ASSESSMENT OF THE VG-GATB PROGRAM
CONCEPTUAL MODEL OF THE VG-GATB REFERRAL SYSTEM
As the details of the plan were worked out with the aid of a number of
pilot programs, USES staff put together a conceptual model of the
VG-GATB Referral System to assist local offices in transforming their
operations. The document is not an official statement of policy, but was
used widely in workshops and conferences for state and local Employ-
ment Service personnel. Although most of the offices using the VG-GATB
Referral System added VG procedures on top of the existing system
rather than replace the old procedures, it seems useful to look at the ideal
version being promoted by USES. The following discussion is drawn
entirely from the document, "Conceptual Model of Full Service Validity
Generalization Local Offices', (Hawk et al., 1986~.
At the center of the basic principles that inform the conceptual model
of a fully implemented VG-GATB Referral System is the idea of high-
quality service to employers (and to the economy). Flowing from that
overriding goal is the principle that "almost all ES resources" should be
devoted to activities designed to provide employers with the most capable
available applicants, namely:
1. intake (reception and registration),
2. assessment (testing and evaluation of education, training, and expe-
rience),
3. file search, and
4. referral.
It follows in this plan that most applicants should be mainstreamed, that
is, they should take the GATE and compete on an equal basis for
available job openings. The model recognizes that the standard VG
procedures are not applicable to some job seekers, including non-
English-speaking and illiterate applicants and those with handicapping
conditions. It is suggested that local offices maintain close ties with
community organizations that are set up to handle people with such
special needs so that they can provide efficient referral to the appropriate
agencies. The Job Service itself would be reserved largely for the
able-bodied and the competitive.
Figure 10-1 illustrates the full-service VG-GATB office. For conve-
nience, the authors of the document have simplified operations some-
what. For example, file search for veterans, which would usually be a
separate operation, is not separate in the figure.
The standard process moves in a straight line from top to bottom of the
flow chart. A job seeker entering the VG-GATB Referral System goes
first to reception, proceeds to group orientation and registration, and then
on to GATE testing. Scores are entered into the (computerized) applicant
OCR for page 193
CONCEPT, PROMOTION, AND IMPLEMENTATION 193
file. As job orders come in, the best jobs are dealt with first. The files are
searched starting with the highest scores so that the best available jobs
and the best available applicants are matched. The final stage in the
standard process is referral.
At each stage in the process, some individuals would be identified who
either cannot participate in the standard process or who are unlikely to
benefit from it. Listed at the right of Figure 10-1 are in-house and
community services to which the local office can refer such individuals to
help make them more competitive. The emphasis is on referring noncom-
petitive applicants to appropriate external services or assistance-public
or private job training organizations, job finding clubs, remedial educa-
tion, and self-improvement courses- so that, following remediation, they
can reenter the standard referral process.
This plan envisions an active role for the Job Service stab in diagnosing
the specific problems that prevent applicants from proceeding through the
standard process and putting such people in contact with appropriate
services. As the diamonds at the left of the figure illustrate, the model also
provides counseling services for those in the applicant file who are not
being referred because of low scores or lack of marketable skills, or who
have a record of multiple unsuccessful referrals. This level of service
would require automated file search and so would be possible only in local
offices that have computerized files.
Group Orientation and Registration
One notable characteristic of the VG-GATB Referral System is the
attempt to streamline the processing of applicants. Under the old proce-
dures, each person entering the local office talks with an interviewer, who
assists the individual in filling out an application form. In a busy office,
this could entail long waits for the next available counselor. Orientation
and application-taking in groups means that one staff member can take
care of the preliminaries for many applicants at once and, in the authors'
estimation, in about one hour's time. The conceptual plan suggests that
orientation sessions be conducted as frequently as the number of people
in the reception area warrants, and that the orientation include an
overview of the Employment Service, a talk on the benefits of test-based
referral, information on other services provided by the local office, and
instructions on how to fill out the application form.
Part of the orientation is intended to convince applicants that it is in
their interest to take the GATE, since most job orders will be filled using
VG-GATB procedures. A brochure, "Doing Your Best on Aptitude
Tests," is available, and formal pretesting orientation is suggested for
those who seem very apprehensive.
OCR for page 194
194
Ul I. o ~ o o ~o'
2 ~ id, 2 ~' 0 2 · 2 0 ~~ ~ 8 ·
>~-Cot CO ~ US ~ _ ~ ~ _ ~ ~ ~ ~ ~ ~O _ ~ ~ ~ ~ ~ ~ _
G
<: m ~ui
~ ~ ~ ' ~ ~
1~1 \1~'
O a) {IS O
~ 2 ~ Q _
O o ° on
credo\:
- \~4 J
-
m c
~ ._
CD
OCR for page 195
195
-
0 cn
CO ~
C o
_& _ ~c
a)
In
' Q cow
C _ on o
ta E E o E O
~ en n <,, ~ -°
as O ~ s c cat _
'A ~ In en In
a)
Q ....
CO _ Cal ~ ~
LL CD
o
._
~, -
~n
o .O
°
._ . . .. . . .
u, ~ _ c~ co ~ u, co ~
0
u,
n
CO
~ ~ ~ ·= ~= ~
- == ~ ~
2 ~ ~ ~ ==
c,, 8 c~ ,q,.o;
E C °
"~"t ~ ~ E
~ ~ ~ ~ ~ o V'
o ~ 3 c,, =.° s:
~ c~ 0 a~ ct -4
O o ~ ~ D
3 v D~ ~
~1
Ct C
~ Ct
_ ~ 1T
~, __
~ ~g-~o~ ~2~_o
o'
1
~o
cn O q,
O_ c
OCR for page 196
]96 ASSESSMENT OF THE VG-GATB PROGRAM
All applicants who can take the GATB would be scheduled during the
orientation session to do so. Local office staff are reminded that special
note should be made of any who require a special administration because
of a handicapping condition; individuals whose handicap would interfere
with performance on the GATB would be referred to a counselor for a
personalized assessment and probable referral to a job development
program. Such applicants would be exempted from the VG-GATB
system.
Applicants who do not take the GATB, the document suggests, should
be referred to a counselor for advice on remedial services, alternative
assessment methods, or possibly special placement. Their applications
would be entered into the file, but they should be informed that file search
of tested applicants always precedes search among untested applicants.
The files of those who take alternative tests can be placed in the regular
VG-GATB file, but they must be flagged to indicate the need for special
placement assistance.
GATB Testing
Testing is, of course, the centerpiece of the referral system. Once
again, group processing is recommended; testing is by appointment so
that the size of the group and the timing of test administration can be
suited to local conditions. The conceptual plan emphasizes the impor-
tance of good test administration procedures, pointing out that test scores
will have a large impact, whether positive or negative, on an applicant's
job prospects, possibly over several years.
When the test has been scored, records of applicants with competitive
scores are entered into the applicant file; those with very low scores are
not likely to be referred under the VG-GATB system and, the document
suggests, should probably be sent to a counselor so that the applicant can
be directed to remedial services or vocational training available in the
community. The counselor might also try a specialized placement such as
subsidized employment programs or sheltered workshops.
File Search and Referral
When an employer places a job order, local office staff assign it a code
number from the Dictionary of Occupational Titles (DOT) and, in the
VG-GATB system, one of the five job family designations. The files are
then searched for the applicants with the highest test scores in that job
family; applicants are selected for call-in from the top down. If the job
order imposes an experience requirement, then selection will be from the
highest test score on down among those with the appropriate experience.
OCR for page 197
CONCEPT, PROMOTION, AND IMPLEMENTATION 197
Veterans enjoy certain rights to priority in file search and/or referral,
which must also be built into the file search system.
On the basis of the file search, applicants are called (in rank order of
test score save for veterans' priority) and those interested in the job are
asked to report to the Job Service office to schedule an interview with the
employer and pick up a referral card. It is worthy of note that, whatever
the other efficiencies of the VG-GATB Referral System, it does take the
job seeker three trips to the local office to get a first referral. And the
waiting time for some employers is also extended. Under the old system,
most job orders tended to be filled from among those individuals who
happened to be in the office. For an employer who needs someone on the
job the next day, that system would be preferable. However, employers
as well as job seekers who want to increase their options would probably
benefit by the file-search and call-in method.
Office Automation
The operational efficiency of the VG-GATB system is heavily depen-
dent on computerization of the applicant files. In describing the concep-
tual model of the VG-GATB local office, USES devotes a good deal of
attention to the experiences of the various states that automated local
office operations as part of the VG-GATB experiment. Some have been
set up so that a central office receives the answer sheets and maintains the
applicant files. In these systems, file search and job matching can be done
on a central mainframe computer, and the information can be available
across the state. A few states (Arizona, Missouri, Oregon, and South
Dakota) have integrated systems; that is, the personal computers in the
local office can send test data directly into the mainframe and receive
information directly from it. This would seem to offer real advantages in
states in which large employers draw from a number of local offices, as is
the case with the automobile companies in Michigan. At some time in the
future, it could also allow the job seeker to cast a wider net, if job order
information for a region or the entire state were available at each local
office. An analogy can be drawn to the military selection and classification
system, in which 66 Military Entrance Processing Stations (MEPS) across
the nation are linked to a central computer. Every potential recruit's test
scores and other relevant data are entered directly into the central file,
and job counselors at the MEPS have immediate access to information
about the jobs available in their Service for which the applicant qualifies.
Many of the local offices that have introduced automation have adopted a
decentralized system, with each office doing its own scoring and record
keeping. Some have a mainframe, and some work strictly with personal
computers. In some states, the local offices have their own scoring equip
OCR for page 198
i98 ASSESSMENT OF THE VG-GATB PROD
ment but plan to link up with a central mainframe for file maintenance and file
search.
The development of software for file maintenance and file search
activities has been an important part of the VG-GATB experiment (and,
more generally, of the modernization of Employment Service operations).
Some states are using system-specific software, but over half the states
have adopted a package called the Enhanced National Data System
(ENDS) for at least some aspects of file management. And, since 1986,
some have been exploring a job-match software package called On-line
Data Display System (ODDS).
The conceptual document explains that automated file search, which it
calls "job matching,'' is a key to efficient operation of the VG-GATB
Referral System. It allows entry of the employer's selection criteria (e.g.,
years of experience, education or training requirements, cutoff score on
the GATE below which the employer will not accept referrals) and
virtually instantaneous identification of the pool of applicants with the
highest percentile scores who also meet the employer's criteria. Software
systems should be designed to allow keyword matching, the authors point
out, so that more precise matching of employers' requirements and
applicants' skills can be made than is possible under the very general
DOT categories. Within a single DOT code, for example, an applicant's
record might specify 46529 secretarial work, 46551 general clerical work,
and/or 88595 word processing.
Comment on the Conceptual Model
There can be little doubt that the automation of Job Service operations
offers enormous efficiencies when compared with manual file search. This is
true whether or not the local office is using the VG-GATB Referral System.
What is less clear, however, is how much the principle of top-down
selection, which is the fundamental organizing principle of the VG-GATB
concept, actually promotes higher quality in the group of applicants sent to
an employer in response to a job order. Given the modest validities of the
test, which mean that there is a great deal of error in the rank-ordering of
applicants, and given the other important selection criteria already used to
winnow the pool of applicants (experience, etc.), the increment of greater
productivity contributed by test-based selection could be quite small indeed,
particularly for jobs that require specialized training or experience.
PROMOTION OF THE VG-GATB REFERRAL SYSTEM
Many of the people involved in developing the VG-GATB system have
been very enthusiastic about its potential. USES staffin the central office
OCR for page 199
CONCEPT, PROMOTION, AND IMPLEMENTATION 199
and in the regional Test Development Field Centers have advertised the
merits of the system to the states and to Employers' National Job Service
Committees and have worked hard to promote the VG-GATB idea
throughout the Employment Service system. The supposed benefits to
employers have been written about by a number of satisfied users in
magazine and journal articles, both popular and professional.
Although the level of enthusiasm brought to this attempt to improve an
important government service cannot but be praised, the committee is
concerned by exaggerated and ill-supported claims made for the VG-
GATB system, as well as by a noticeable lack of qualifiers and cautions.
Our unease begins with the technical reports of the USES test research
themselves. Reports No. 43 through No. 47 lay out the scientific basis for
the generalizability of the GATB and the estimated economic benefits of
the VG-GATB Referral System (U.S. Department of Labor,
1983a,b,c,d,e). These reports, written at a level accessible to the general
reader, consistently present the most optimistic interpretation possible.
In other chapters of this report we present specific critiques of the various
claims made in the technical reports. Here we remark on the tone of the
reports because of their influence on Job Service employees and clients
who may not be in a good position to judge them critically.
For example, Report No. 43, a summary of John Hunter's research on
the GATE, prepared by a private consulting firm, begins with the
following claim (U.S. Department of Labor, 1983a):
The U.S. Employment Service can help to improve the productivity of American
industry on the order of 50 to 100 billion dollars in the upcoming year. Sound
ridiculous? Not at all. In fact, the impact of the Employment Service on the U.S.
economy can be accomplished by a new use of an already established and widely
accepted Employment Service device for matching people and jobs the General
Aptitude Test Battery in a way that makes the best use of state-of-the-art
research evidence.
The kindest interpretation of this claim is that it is the hyperbole of an
enthusiast. Not only is it, as our analysis in Chapter 12 indicates, the
product of a theoretical mode! based on slight empirical evidence; not
only does it totally ignore the big difference between theory and likely
effects in real-world applications; but it also chooses to express economic
gains in a metric (aggregate dollar amounts) that is designed to impress,
dazzling all but a small number of macroeconomists who can sensibly
interpret numbers of this magnitude.
We cannot claim to have conducted a thorough investigation of the way
the VG-GATB system is being advertised to employers and job seekers at
the state and local levels. Most committee members have visited at least
one local office. We have developed impressions based on information
OCR for page 200
200 ASSESSMENT OF THE VG-GATB PROGRAM
provided by members of our liaison panel. We have during the past 18
months or so received phone calls and letters, both solicited and unso-
licited, from employers, job seekers who have taken the GATB, and
people who work in the Employment Service system. And we have
frequently found ourselves troubled by an overly optimistic assessment of
the benefits and a downplaying of the potential legal problems with the
VG-GATB Referral System.
For example, a small number of employers told the committee of their
shock when the VG-GATB was challenged by the Department of Justice
because of its score adjustment procedures. They felt that they had been
led to believe that the VG-GATB system would solve potential legal
problems by preventing adverse impact in their selection procedures, not
make them vulnerable to charges of reverse discrimination. One can see
at this distance that this misperception was partly a product of mistaking
the Department of Labor's sponsorship of the VG-GATB as a more
general stamp of approval by "the Government." But the misperception
is also positively encouraged by some of the literature used to promote
the system. One state uses a brochure to attract employer-clients that
states:
The Legal Implications.
The legality of the General Aptitude Test Battery has been addressed in court
cases. The courts have upheld GATB testing as fair and said the testing system
conforms to EEOC guidelines.
As a matter of fact, the GATB has been the subject of very little Title VII
litigation, too little to talk about "the courts." And the case that was
almost certainly the basis for this claim (Pegues v. Mississippi State
Employment Service, 699 F.2d 760 [19801) is ambiguous at best. Brought
in 1979, it involved a use of the GATB that predates the introduction of
the VG-GATB system, with its claim of validity for all 12,000 jobs in the
U.S. economy and its use of within-group scoring. In addition, although
the trial court accepted arguments for the transportability of validities on
the basis of early evidence from the validity generalization research, the
appellate court ruled that, in the absence of adverse impact, the testing
issue need not be addressed. That record is not sufficient to support the
degree of legal acceptability implied by the brochure.
Although in these paragraphs we intend to communicate our very real
concern with the way the VG-GATB Referral System has been pro-
moted and we recommend far more circumspection in the future we
must also take note of the severe budget reductions that have forced deep
cuts in both staff and research resources at the national, state, and local
levels of the Employment Service system in the past decade. The staff
members who have become wedded to the new system have done so
OCR for page 201
CONCEPT, PROMOTION, AND IMPLEMENTATION 20 1
partly in response to this serious decline in resources. They have tried to
do something about the situation by reorganizing the system to make
local operations more efficient, by attracting better job orders through
advertising the presumed benefits of the VG-GATB to employers, and by
finding ways to handle more job seekers in a given period of time to
alleviate the effects of reduced staffing levels. For this they deserve
kudos.
IMPLEMENTATION OF THE VG-GATB REFERRAL SYSTEM:
THE PILOT STUDIES
One of the unfortunate consequences of the above-mentioned financial
constraints has been that early plans for an ambitious program of pilot
studies have gone astray. The empirical evidence of the effects of the
VG-GATB Referral System that should have been abuilding in the past
eight years has failed for the most part to materialize. States were
encouraged to make a gradual implementation, accompanied by careful
program evaluation. In most cases, however, these evaluations have been
limited to comparing the number of placements made before and after
VG-GATB procedures were introduced into an office, a comparison
complicated by the downturn of the economy and large staff cuts at the
time the experiments were started and by the fact that in most cases there
was little disruption of existing practices.
Although 40 states have now introduced the VG-GATB system in one
or more local offices, either to supplement or to replace traditional
operations, only a few formal studies have been conducted to evaluate the
effects of the system. The committee evaluated five pilot studies, two
growing out of the very early experience with the new program in local
offices in North Carolina and three that look at the effects on job
performance in three firms. All the studies purport to show improvements
attributable to the introduction of the VG-GATB Referral System in one
or more of the following areas:
staff);
2. market penetration (Employment Service placements as a percent-
age of total new hires in the area); and
3. improved worker performance (measured in terms of absenteeism,
quality of workmanship, disciplinary actions, training success, or produc-
tivity).
office effectiveness (number of placements made relative to available
However, of the five studies conducted, only one, that by Madigan et
al. (1987), satisfies rigorous research standards; the others can be consid-
ered no more than suggestive.
OCR for page 202
202 ASSESSMENT OF THE VG-GATB PROGRAM
North Carolina Study of Office Electiveness
The first study, based on data from 1981-1982, is one of two conducted
by M.W. McKinney (1984) as part of the very earliest experiment with the
VG-GATB Referral System. It is a study of focal office effectiveness,
comparing offices that did and did not use the new system. Data are
presented on indices of effectiveness, work load, and two control varia-
bles for three groups of local offices: (1) seven VG-GATB offices in the
Raleigh-Durham area; (2) the 48 remaining (non-VG-GATB) offices in
North Carolina; and (3) a subset of seven from the second group that were
matched on a one-to-one basis with the seven in the first group. Matching
was on the basis of office staff numbers and insured unemployment rate.
Means are presented by group for both 1981 (before VG-GATB was
implemented in any North Carolina office) and 1982 (after implementation
at the seven pilot sites). These data show that between 1981 and 1982, the
seven VG-GATB offices improved more than the other local offices with
respect to the number of referrals and job placements made; staff
"productivity" (number of placements made relative to the number of
staff available); and penetration rate (the number of Job Service place-
ments divided by the total number of new hires reported in the area
served by the focal office). There was no difference among the groups of
offices in the mean increase in salary of the new hires.
No data are presented regarding the consistency of effects across the
seven matched pairs of sites. Only weighted means are given, without
explanation of how the weights were derived.
Although McKinney's findings are suggestive, there are several serious
threats to their validity. The seven VG-GATB sites are all from the same
part of the state, a design factor that, while convenient for operational
reasons, could have influenced the results. At baseline (1981), the seven
pilot sites differed substantially from the seven "matched'' sites on some
key variables. For instance, the seven VG sites were almost one standard
deviation lower in productivity, and Raleigh had a disproportionately
large demand for secretarial/clerical workers. The interpretability of the
results is further blurred by the fact that the VG-GATB system was not
even close to fully implemented at the seven VG-GATB sites. Only 18
percent of the applicants at these sites were tested in 1982 instead of the
hoped-for 80 percent.
North Carolina Employer Survey
The second study, conducted by McKinney in 1983, was a face-to-face
survey of 295 employers who were asked questions about their hiring
practices and experiences and their attitudes toward the Employment
OCR for page 203
CONCEPT, PROMOTION, AND IMPLEMENTATION 203
Service and the VG-GATB Referral System. The participants were
among a larger group of employers who had been briefed on the
VG-GATB system a year earlier.
Of the 173 employers who hired anyone during the 18 months of the
pilot program, about 52 percent said that the VG-GATB system made the
Employment Service more useful to them; 44 percent said applicant
quality improved; and 25 percent said VG-GATB selection reduced
training and other personnel costs. These variables appeared to interact
with each other and with attitude toward the Employment Service.
Again, the results are suggestive, but there are problems with the
survey. Employer impressions about hiring experiences were not verified
against actual records, although this could have been done with the data
base from the first study. Employers who hired hundreds of workers were
lumped together with those who hired only one. And employers whose
job orders were filled under the VG-GATB procedures were lumped
together with those whose referrals may not even have been tested.
Above all, the interviews were not neutral. They were conducted by staff
working at the Pilot study sites who began each session with the following
statement:
We have found that Validity Generalization, or VG as we call it, has shown a
positive relationship between General Aptitude Test Scores and success on the
job.
The Philip Morris Study
D.L. Warmke (1984) conducted a study to assess the impact of
VG-GATB hiring on worker performance. From a pool of 32,000 appli-
cants, 1,200 employees were selected at a new Philip Morris plant in
Cabarius County, North Carolina, using a four-step screening procedure.
At each stage, the percentage of applicants screened out was:
Procedure
-
Application
VG-GATB
Interview/physical
Noncompensated training
Screened Out (pro)
40
2
The study compared the performance of the 1,200 employees at the new
plant to three other groups: (1) the 32 employees who transferred to the
new plant from other plants, (2) employees at another Philip Morris plant,
and (3) industry-wide averages. There were also comparisons to quality
and production goals set by management.
Employees at the new plant performed better than those in the other
groups. For instance, compared with the non-VG-GATB plant, the
OCR for page 204
204 ASSESSMENT OF THE VG-GATB PROGRAM
workers at the new plant had: an 81 percent lower lost-workday severity
rate; a 59 percent lower quality-defect rate; 58 percent fewer disciplinary
actions; and 8 percent greater success in training.
These improvements are attributed primarily to the VG-GATB because
that procedure screened out about as many applicants as the other three
hurdles combined. The credibility of this interpretation rests on the
comparability of the comparison groups; however, there are no data to
indicate the comparability. Were the same standards used in making
application screening decisions at the VG-GATB plant and the other
plant? Did the other plant screen out 40 percent of its applicants on the
basis of the application? Did the other plant also have a 1:27 selection
ratio?
Conspicuous by their absence are data contrasting the two plants on
productivity goals set by management. And no data are provided regard-
ing the correlation between GATE scores and performance measures.
The author recognizes that locational differences could explain the
reported results. He argues against this interpretation because the 32
employees not selected by VG-GATB procedures who transferred from
other plants did not perform as well in training as the others at the new
plant. However, there is no reason to believe that the transfers were
similar to the test-selected employees in other relevant characteristics,
nor that the training results would carry over to job performance.
The Chrysler Corporation Study
The Northern Test Development Field Center (1987) of the USES
conducted a study comparing the performance of 246 employees hired by
means of VG-GATB scores with two other groups: transfers from other
plants (N = 422) and rehires of people from other plants who had used up
their recall rights (N = 468~. Workers in the last group were selected on
the basis of previous supervisor performance evaluations.
Five performance measures were used in the study: a supervisor rating
of productivity, quality, tool use, job knowledge, adaptability, and overall
ability; a supervisor's rank-ordering on overall quality of all of his or her
employees; excused and unexcused absences; supervisor's rating of
frequency of visits to the medical department; and recommendation to
rehire.
On virtually all the measures, the new hires earned better scores than
the transfers, who in turn earned better scores than the rehires. However,
none of the differences was very large. For example, the VG-GATB
group's mean on the first rating scale (productivity, quality, etc.) was only
about one-quarter standard deviation above that for all employees. There
OCR for page 205
CONCEPT, PROMOTION, AND IMPLEMENTATION 205
was almost no variation on the recommendation-to-rehire measure 89
percent of the workers received the highest rating.
This study provides some support for the hypothesis that selection
using the VG-GATB can enhance the quality of worker performance, but
the magnitude of observed effects is quite small and could have been
produced by factors other than the new referral system. (Were the new
hires motivated by probationary status?) The study could be improved
substantially by analyses that control for some important but unmeasured
factors, such as age, sex, and job mix. It would also be useful to know
whether the amount of job experience in the two comparison groups is
positively correlated with performance.
The Sewing Machine Operator Study
By far the most ambitious and most successful study was that
undertaken by the State of Virginia and conducted by Madigan et al.
(1987~. Its purpose was to assess the validity of VG-GATB selection
procedures and to develop estimates of the potential economic benefit
to the firm of using such procedures.
The study involved sewing machine operators in five plants owned by
a manufacturer of casual wear. A group of 751 of a total of 932 new
employees was tested with the GATE prior to hire, but their scores were
not used in hiring. Within this group, 27 percent were black and 65
percent had no previous experience; all were women. Their mean
VG-GATB Job Family V score was 51, with a standard deviation of 26
and a range of 1 to 99.
Since the job involves piecework, the performance criterion was actual
production records plus supervisor ratings of quantity of output, quality
of output, flexibility, dependability, receptiveness to instruction, and an
overall rating. Turnover and reason for leaving were also recorded.
Employees were paid on a piecework basis and, within the limits of
error of industrial engineering studies, employees who worked at the
same rate of output but on different operations received the same pay.
This allowed the computation of a "time-to-standard" measure (the
number of weeks it took the employee to produce at a rate that would earn
$3.75 per hour and $4.79 per hour).
An analysis of VG-GATB test scores by quarters of the distribution
showed that employees in the higher quarters have generally better
performance than those in the lower quarters. For example, the average
number of weeks it took employees in each quarter to reach the $4.79
standard was:
OCR for page 206
206 ASSESSMENT OF THE VG-GATB PROGRAM
Quarter
4 (76~ o)
3 (51-75%)
2 (26-50%)
1 (1-25%)
Weeks to Standard
13.6
14.4
14.1
16.4
The percentage of employees in a quarter who were black increased
with quartile number; almost half of those above the third quartile (the
highest scores) were black workers.
Comparison of experienced and inexperienced employees showed that
they had similar GATB scores and percentages of black workers (29
percent and 24 percent, respectively). However, the experienced workers
tended to reach the $4.79 standard faster, averaging 11.1 weeks to
standard compared with 15.4 weeks for the inexperienced workers.
Correlations between GATB scores and monthly production averages
were statistically significant and ranged from .15 to .24. In addition,
turnover rates tended to be higher among those with lower scores.
This study is noteworthy because of the relatively large sample (751
employees), the lack of any restriction in range in that sample (the GATB
scores ranged from 1 to 99), and the reliability of the criterion (objective
measures of output). The results, since they require no corrections or
adjustments, pretty well mean what they say: there is a tendency for
sewing machine operators with high VG-GATB Job Family V scores to
perform better and have less turnover than those with low scores.
However, this relationship is not very strong: the average of all validity
coefficients is about .20.
CONCLUSIONS AND RECOMMENDATIONS
Promotion of the VG-GATB Referral System
Conclusions
1. The VG-GATB Referral System is frequently oversold, both at the
national and at the state levels. Much of the promotional literature that we
have seen overstates the psychometric quality and predictive power of
the GATB, underestimates the vulnerability of the referral system to legal
challenge, and exaggerates the economic impact of preemployment
testing.
2. The overselling of the VG-GATB system in Employment Service
advertising is encouraged by the tenor of the technical reports describing
the research that undergirds the referral program. They provide optimistic
projections of the effects of VG-GATB referral for which the empirical
evidence is slight.
OCR for page 207
CONCEPT, PROMOTION, AND IMPLEMENTATION 207
Recommendations
1. Given the modest validities of the GATE for the 500 jobs actually
studied; given our incomplete knowledge about the relationship be-
tween this sample and the remaining 11,500 jobs in the U.S. economy;
given the Department of Justice challenge to the legality of within-group
scoring and the larger philosophical debates about race-conscious
mechanisms and the known problems of using a test with severe
adverse impact; given the primitive state of knowledge about the
relationship of individual performance and productivity of the firm, we
recommend that the claims for the testing program be tempered and that
employers as well as job seekers be given a balanced view of the
strengths and weaknesses of the GATE and its likely contribution in
matching people to jobs.
Implementation of the VG-GATB Referral System:
The Pilot Studies
Findings
1. Although 40 states have introduced some form of the VG-GATB
Referral System in a number of local offices, and a small number have
experimented with the program statewide, only a few formal studies have
been conducted to evaluate the effects of the program.
2. The five pilot studies evaluated by the committee showed some
improvements resulting from introduction of the VG-GATB Referral
System in one or more of the following areas:
a. office effectiveness (number of placements made relative to
available stalk;
b. market penetration (Employment Service placements as a per-
centage of total new hires in the area); and
c. improved worker performance (measured in terms of absentee-
ism, quality of workmanship, disciplinary actions, training suc-
cess, or productivity).
3. However, of the five pilot studies evaluated, only one satisfied
rigorous research or evaluation standards. Although some of the other
studies were suggestive, the committee cannot place a great deal of
confidence in their favorable assessments of the effects of the VG-GATB
Referral System, including the evidence of improved job performance.
OCR for page 208
208 ASSESSMENT OF THE VG-GATB PROGRAM
Conclusions
1. There is too little evidence based on controlled, rigorous studies of
the effects of using the VG-GATB Referral System for the committee to
be able to assure policy makers at the Department of Labor that
anticipated improvements have indeed occurred. This is not to say that
they have not occurred. The evidence simply does not exist to establish
the case scientifically. For the moment, policy decisions about the future
of the VG-GATB Referral System will have to be made on the basis of
more impressionistic and experiential information.
Recommendations
1. If USES decides to continue the VG-GATB Referral System, a
series of carefully designed studies should be undertaken to establish
more scientifically the efficiencies that are believed to result.
2. This research will need to be a cooperative effort, involving federal
and state Employment Service personnel and employers. USES should
encourage state Employment Security Agencies that deal with large
employers (e.g., Michigan) and states that have fully articulated VG-
GATB systems in place (e.g., Virginia, Utah, Oklahoma) to take a leading
role in conducting studies to demonstrate the efficacy of the VG-GATB
Referral System.
3. We also recommend that the employer community, as a potentially
major beneficiary of an improved referral system, take an active part in
the effort to evaluate the VG-GATB Referral System. The Employers'
National Job Service Committee can help to identify appropriate employ-
ers who are willing to commit the resources necessary to study the effects
of VG-GATB referral.
Representative terms from entire chapter:
file search
