Click for next page ( 24

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 23
23 Many high-performing motor carriers have selection ratios of 20% or even lower. Schneider National (Osterberg 2004; Knipling 2009a) uses a multilayer selection process that hires only about 13% of applicants and a much lower percentage (3%) of total driver inquiries. Figure 7 shows the Schneider numbers for 2004 at six steps of their process, beginning with recruiting calls received and ending with new hires. FIGURE 7 Selectivity of Schneider National driver hiring. (Source : Based on Osterberg 2004.) TEST CHARACTERISTICS AND REQUIREMENTS This section covers basic testing concepts, federal require- ments, and principles for carriers to better assess their current and planned use of testing to hire safer drivers. Information on employment testing can also be found in var- ious textbooks on industrial psychology and management. These include texts by Sonnentag (2001), Cascio (2003), and Spector (2008). The following information is intended to provide a basic understanding of employment testing and how it might be improved. It is not sufficient as a guide for conducting test validation studies or implementing major new selection procedures. For these, companies are advised FIGURE 6 Systematic hiring process. to consult HR specialists in staffing and employment law. Key Assessment Terms and Concepts For any job, the selection ratio is the proportion (or per- The following are some key terms and concepts in employee centage) of job applicants actually hired (Cascio 2004). assessment: Stated as a ratio, this is Job analysis Predictor(s) Job performance criterion (criteria) Test reliability Test validity: A low selection ratio leads to higher quality employees Content validity because a lower percentage of applicants have been chosen. Construct validity

OCR for page 23
24 Criterion-based validity: Measurement reliability is a concern both for selection Predictive validity measures and for job performance measures. To the extent Concurrent validity that measures are unreliable, they are confounded by ran- Success ratio. dom error. Random error in either a predictor measure or a criterion measure means that neither can be perfectly accu- One must fully understand a job to be able to accurately rate (valid) as a measure. select the best employees for that job. Job analysis is the delineation of specific tasks and performance involved in Validity is the accuracy of a measure, or the degree to a job. They may include job function or duties, work tasks, which it actually measures what it purports to measure. skills or competencies, work-related knowledge, work Selection test validity can be assessed in various ways. Con- environment factors, decision-making authority, educa- tent validity is the degree to which the content of a test corre- tional requirements, communication, training, and physi- sponds to the content of a job. A road/range driving test, for cal abilities. A job analysis is often necessary to validate example, has obvious content validity in relation to a com- the use of a selection procedure. This is especially true mercial driver's job. Construct validity is more conceptual. for any procedure that disadvantages groups of potential Construct validation usually involves showing that a test employees that are protected under the employment dis- measures specific personal characteristics that are known to crimination laws. be relevant to the performance of the job. Chapter two, for example, cites extensive evidence linking aggressive/hostile Appendix B provides a commercial driver job description personalities to high crash rates. If a test provides reliable developed and used by a medium-sized regional TL carrier in measures of personal aggressiveness that correlate well with Canada. This carrier uses this document to provide its appli- other measures, then it could has construct validity in rela- cants and employed drivers with full information on their tion to this element of safe driving. driver jobs and performance expectations. Although it is not a formal job analysis, it contains many of the same elements. Criterion-based validity is the degree to which test scores A job analysis document helps a carrier to identify the most correlate with actual job performance criteria. For a person- important and valid elements of its selection process. These ality measure of aggressiveness, this might be its correlation selection elements are predictors of job performance. A job with future crash rates of new hires ( predictive validity) or performance criterion is a measure of employee success on with current/past crash rates of existing employees (con- the job. Generally, any job has multiple performance crite- current validity). Criterion-based validity is the practical, ria. Of particular concern are job safety performance crite- "bottom line" validity of a test; that is, how well it actually ria, which may include such measures as crash rate, rate of performs as a test. Thus, a well-conducted criterion-based preventable crashes, violation rate, and rapid decelerations validation study is generally the strongest method to demon- captured in onboard recordings. More sophisticated compa- strate the value of a test. Evidence attesting to content or con- nies use driving behavior criteria such as hard-braking rate, struct validity is generally supportive rather than definitive. speed compliance, and fuel economy. Criterion-based validity also is expressed as a correlation The reliability of a test or measure is the degree to which coefficient. The term v-score is sometimes used. V-scores it provides consistent measurements. Measurements like are almost always lower than test reliabilities because so height and weight are almost perfectly reliable because many factors contribute to job performance and because per- repeated measures will provide exactly (or almost exactly) formance is hard to measure. The U.S. DOL (2000) assesses the same result. The test-retest reliability of height, expressed v-scores as follows: as a correlation between two successive measurements, is a perfect +1.00 or nearly so. Some tests of psychological 0.35 or higher: test is "very beneficial" in assessing traits, such as IQ and aptitude tests, often have test-retest likely employee success. or split-half reliabilities of +0.90 or more (Associated Con- 0.210.34: test "is likely to be useful" to the employer tent, 2010), but the reliability of tests of personality traits 0.110.20: test may be useful, depending on circum- like impulsivity and sensation-seeking is lower. Subjectively stances and whether other assessments are also used. scored interviews are likely to have even lower reliabilities. 0.11 or less: test is "unlikely to be useful." A test with a reliability of 0.00 would be worthless because it would reflect random answers or scoring. A U.S. Depart- Understanding the overall concept of prediction is as ment of Labor publication (DOL 2000) classifies coefficients important as familiarity with prediction statistics. Figure 8 is of +0.90 or above as excellent, +0.80 to +0.89 as good, and a simplistic model of employee selection (Cascio 2004). The +0.70 to +0.79 as adequate. If multiple assessments are used, horizontal x-axis represents the predictor (i.e., selection test) their combined reliability may be greater than any one test score and the vertical y-axis is the job performance criterion. because multiple assessments capture more elements of per- Assume that higher scores are "good" for both scales. The formance and behavior. slanted oval represents a hypothetical population of appli-

OCR for page 23
25 cants. It is slanted upward because of the assumption that applies: The employer is trying to maximize zones A and C the selection test has moderate validity as a predictor (i.e., in the model and minimize zones B and D. a moderate positive correlation with job performance). The vertical line down the middle is the cutoff hiring score for In the model, how would a highly valid selection test look predictor test scores. The horizontal line across the middle is different than one that was less valid? Other factors being the minimum satisfactory job performance. Assume, for the equal, the difference would be in the shape of the oval. A sake of the model, that everyone is hired so that their predic- highly valid selection test would generate a "skinny" oval. tor scores can be compared with their job performance. A poor test would generate a "fat" oval, and a completely worthless one would generate a circle or other shape in which the sum of A + C was no greater than the sum of B + D. One can further break down a test's performance by its "hit" rate for identifying unsafe drivers and its "false alarm" rate for rejecting safe drivers. A valid test would have a high hit rate and a low false alarm rate. In the context of Figure 8, these two measures can be defined as follows: Correct Rejection (C) Unsafe Driver "Hit " Rate Total Number of Unsafe Drivers (BC) Erroneous Rejections (D) Safe Driver "False Alarm" Rate Total Number of Safe Drivers (AD) None of these statistics can be calculated based on an actual selection process where some candidates are hired FIGURE 8 Simplistic model of the relation between predictor test scores and employee job performance. In this figure, and others are not hired. Nonhired drivers would have no higher is better for both dimensions. (Source : Based on job performance criterion data with which to classify them Cascio 2004.) as "safe" or "unsafe." A company could, however, norm a test against its existing driver force (i.e., determine concur- The four areas of the oval are as follows: rent validity). For example, the worst 15% of drivers in terms of crash rates, violation rates, complaint rates, or other met- A. Correct acceptances (high test score, and satisfactory rics could be compared with the best 85%. A high hit rate job performance) for unsafe drivers and a low false alarm rate for safe drivers B. Erroneous acceptances (high test score, but unsatis- would indicate a valid and useful test. factory job performance) C. Correct rejections (low test score, and unsatisfactory Federal Requirements for Employment Tests job performance) D. Erroneous rejections (low test scores, but satisfactory All employers have an ethical and a legal duty to treat appli- job performance). cants for employment fairly. Employers also have the same duty with respect to assessing current employees. Nongov- A successful selection system would have a high propor- ernmental employers have a particular legal duty, but most tion of correct decisions; for example, hiring good drivers state and local agencies and all federal agencies have similar and rejecting bad drivers. In the model, zones A and C repre- requirements. The primary sources for this section are Chee- sent correct decisions, whereas zones B and D are bad deci- seman (2006) and Mann and Roberts (2006). sions. The percent of correct decisions or selection success ratio is given by the following equation: Several laws shape this legal duty, the most important of which is the Fair Employment Practices Act, or Title VII of the Civil Rights Act of 1964 as amended by the Equal Employment Opportunity Act of 1972. The basic require- ment of this law is that employers shall not discriminate in This model illustrates selection concepts rather than actual hiring, promotion, wages, training, or any other term, con- practice. It is simplistic because it considers only one predic- dition, or privilege of employment, according to the race, tor and one job criterion, and because it assumes that there is a color, religion, sex, or national origin of the affected per- sharp cutoff score for each. In the real world, almost every job sons. The italicized categories are the "protected classes" of involves multiple selection factors and multiple measures of individuals under the act. The text box lists this and four job performance. Yet the same conceptual model of selection other laws that further define the duty of fair treatment.

OCR for page 23
26 Laws Defining the Fair Treatment of Four-Fifths Rule Example Applicants and Employees If a given procedure selects 91% of male applicants 1. The Fair Employment Practices Act (also known (screening out 9%), then the four-fifths guideline says that as Title VII of the Civil Rights Act of 1964, as females, or any other protected class, must be selected for amended by the Equal Employment Opportunity hiring at a rate of no less than 80% of that 91%, which is Act of 1972) 73% (0.8 0.91 = 0.73). Or to put it the other way around, no more than 27% of females may be screened out by a 2. The Equal Pay Act of 1963 (EPA) method that screens out only 9% of men. 3. Title I of the Americans with Disabilities Act of 1990 (ADA) If a method of selection discriminates numerically 4. The Age Discrimination in Employment Act of because it does not satisfy the four-fifths rule, it may still 1967 be legal to use if it can be shown to be valid with respect to the job for which applicants are applying. Validity means 4. The Genetic Information Nondiscrimination Act that the employer can show, with specific statistical evi- of 2008 (GINA). dence, that the selection method generates measurements that are demonstrably correlated with job performance. Or, Cheeseman (2006), Mann and Roberts (2006) the employer can show (typically by job analysis) that the selection method has content that is demonstrably repre- sentative of important parts of the job. There is also a third way to show validity, by showing that the method mea- sures a related set of personal characteristics (a construct) Along with the other laws listed in the text box, the Fair that is important in successful job performance. However, Employment Practices Act is administered by a stand- the EEOC regulations note that this approach is less well alone federal agency, the Equal Employment Opportu- documented in the academic literature. An employer tak- nity Commission (EEOC). The EEOC issues regulations ing this route might take some extra care in meeting the that spell out the meaning of fair treatment, and accepts regulatory requirements. complaints from individuals who believe they have been unfairly treated in employment settings owing to their There is a limitation to using validity to defend a selection membership in one of the protected classes. The EEOC method that is otherwise desirable because it selects safer can bring suit in federal court to enforce its regulations or drivers, but that has the side effect of numerically discrimi- resolve complaints. nating against a protected class. The employer must obtain and keep current statistical evidence of the method's impact Very small trucking fleets (10 or fewer trucks) are not on its own applicant pool. Generic information provided by covered by EEOC regulations and enforcement because the the vendor of a test, for instance, will not be sufficient. Also, regulations apply only to private employers who have had if the employer uses either the criterion or construct meth- 15 or more employees. Further, to be covered, employees ods of showing the validity of a selection procedure, extra must have worked for at least 20 calendar weeks in the cur- specific statistical evidence is required. The employer must rent or preceding year. For age discrimination issues only, show by specific statistical evidence from the job behavior the threshold is 20 employees. But this still leaves a large of its employees that the criterion or construct used to select number of trucking firms covered. among applicants is statistically linked to safer driving per- formances. However, record keeping is permitted to be sim- The EEOC has issued uniform regulations governing pler if the employer has fewer than 100 employees. the use of selection tests in hiring and promotion. These appear in the Code of Federal Regulations, items 29 CFR The fact that a selection method has an adverse impact 1607.1 through 1607.18. The regulations provide uniform on a protected group may not by itself be a sufficient reason guidelines defining what constitutes an adverse impact on a for not using it, if it is valid. Firefighters in Connecticut sued protected class. Adverse impact would trigger federal scru- their employer, the city of New Haven, in 2006 on this issue. tiny and a presumption that unfair treatment may be taking White and Hispanic firefighters who were selected for pro- place. An adverse impact occurs when a protected group is motion by an exam that appeared to be valid objected when selected at less than 80% of the rate at which nonprotected the city dropped the use of the exam because it discrimi- applicants are selected (29 CFR 1607.4). Thus, this trigger is nated against African American firefighters according to the called the "four-fifths rule." four-fifths rule. In two connected cases on this issue, in 2009

OCR for page 23
27 the Supreme Court ruled that the city had illegally discrimi- Principles for Improved Employee Assessment nated against the whites and Hispanics, and that because the test was valid, it should use the results in deciding who to Based largely on the previously mentioned concepts and laws promote (Ricci v. DeStefano 2009). related to employee selection and other assessments, the U.S. DOL has produced Testing and Assessment: An Employer's Guide to Good Practices (DOL 2000). The report describes EEOC Links and explains basic principles that employers should follow when considering and designing employee assessments. 1. The EEOC home page: They include selection-related assessments of candidates as well as assessments of current employees for promotion, 2. An overview of the EEOC and its regulations placement, or other actions. for employers: index.cfm The DOL guide is designed to help managers and HR professionals use tests and other assessments to improve 3. A clear statement of the duty not to discriminate, employee and organizational performance. It helps employ- and prohibited practices generally: http://www. ers to-- Evaluate and select assessment tools that maximize 4. Laws and regulations enforced by the EEOC: chances for getting the right fit between jobs and employers. Correct administer and score assessment tools. 5. Regulations governing selection procedure impact Accurately interpret assessment results. and validity: Understand and follow professional and legal standards. waisidx_10/29cfr1607_10.html The guide presents and explains 13 principles for Four other laws may be relevant to selecting drivers. improved and legal employee assessment. These have also The Equal Pay Act of 1963 (EPA) prohibits paying different been summarized by Kahle (2010). Almost all of the prin- wages to men and women if they do the same work in the ciples follow from the testing concepts and laws discussed same workplace. Title I of the Americans with Disabilities previously: Act of 1990 (ADA) prohibits discrimination against quali- fied individuals who have disabilities. Further, it requires Use assessment tools in a purposeful manner; that is, that employers reasonably accommodate the known physi- for the purpose for which they are designed. Misuse or cal or mental limitations of an otherwise qualified indi- improper use could be harmful or possibly illegal. vidual with a disability, unless doing so would impose an Use the whole-person approach to testing; that is, con- undue hardship on the operation of the employer's busi- sider all the information you have about the candidate. ness. Of potential relevance for selecting drivers for safe No test is perfect. Use a combination of assessments driving performance is this exception to the ADA: "The that give you as much information as possible about ADA permits an employer to require that an individual not behaviors of greatest importance. pose a direct threat to the health and safety of the individual Use tests that are unbiased and fair to all groups. Tests or others in the work-place." Discrimination on the basis of that deliberately or inadvertently discriminate prevent the age is prohibited by the Age Discrimination in Employ- employer from achieving the most qualified work group. ment Act of 1967. The law applies only to discrimination Use tests that are reliable. Will the same person get the against older workers, not younger ones, and the threshold same results each time they take the test? for coverage begins at age 40. Finally, the Genetic Infor- Tests must be valid for the purpose they are being used. mation Nondiscrimination Act of 2008 (GINA) prohibits Validity is the most important criterion for selection of discrimination against applicants based on genetic infor- a proper test instrument. Validity determination may mation about them or their family. All of these laws are be based on content, criterion prediction, or constructs administered by the EEOC, and in general the regulations captured by the test. Criterion-based validity is the issued by the EEOC with respect to the separate potential definitive test. ways in which selection and hiring may be unfair are simi- Tests must be appropriate (e.g., content and difficulty) lar to those for the Fair Employment Practices Act. The for the target population. EEOC offers a comprehensive set of web pages that pro- Test instructions and other documentation must be vide clear linkage to the different issues and questions that comprehensive and easy to understand. may be of interest to employers who are concerned about Test proctors, administrators, and scorers must be selection procedures. The text box contains a few of the properly trained. Some instruments require an exten- more useful links. sive certification process for these roles.