cant behind a screen that prevents the judges from seeing the musician. In doing so, the music that the applicant plays—the relevant part of the audition—is separated from the irrelevant characteristics of the applicant's sex, race, and age. Goldin and Rouse find that the use of screen increases—by 50 percent—the probability that a female musician will be advanced from certain preliminary rounds and increases by severalfold the likelihood that a female musician will be selected in the final round.24

Effectiveness of Assessment Techniques and the Role of Job Analysis

How effective are current methods for identifying highly productive workers among applicants from underrepresented groups? As noted above, IT employers commonly use unstructured assessment methods and fail to evaluate applicants against the same criteria. One primary reason may be that many hiring managers are simply unaware that there are alternatives. Another reason may be that the alternatives are more expensive. On the other hand, the appropriate cost comparison must consider not only the immediate cost of unstructured informal methods versus the alternatives, but also the likelihood that one may find a larger number of productive workers from within a given applicant pool.

A variety of structured assessment methods are described in Box 6.1. Several of the methods have a solid empirical basis, a long and consistent record of validity,25 and demonstrated cost-effectiveness.26 Furthermore,

24  

Goldin, Claudia, and Cecilia Rouse. 2000. “Orchestrating Impartiality: The Impact of “Blind” Auditions on Female Musicians,” American Economic Review 90(September):715-742.

25  

A 1999 review of 85 years of research into alternative methods of assessment found the following average correlation with measures of job performance: cognitive ability tests (0.51), work samples (0.54), structured interviews (0.51), and assessment centers (0.37). See Schmidt, F.L., and J.E. Hunter. 1998. “The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings,” Psychological Bulletin 124:262-274. However, note that these correlation figures are based on a number of statistical and psychometric corrections that are controversial (Hartigan and Wigdor, 1989; Murphy and DeShon, in press) and that probably lead to inflated estimates. Nevertheless, there does seem to be clear and compelling evidence that selection tests can show substantial validity as predictors of performance, and conclusions about the relative validity of these tests (i.e., rank-ordering these tests by validity) appear reasonable.

26  

Numerous studies have shown that structured assessment methods can be highly cost-effective. For example, Campion et al. conclude that structured interviews have substantially greater validity than unstructured interviews. See Campion, M., D. Palmer, and S. Campion. 1997. “A Review of Structure in the Selection Interview,” Personnel Psychology 50(3):655-703. Three factors contribute to the cost-effectiveness of certain structured assessment methods. First, these methods are good at predicting job performance. Second, the quality and productivity of IT workers vary tremendously, so that good selection methods have a huge potential payoff. Finally, the costs of most assessment methods are minuscule in comparison to the benefits of making good hiring decisions (i.e., minimizing false positives and false negatives). If structured assessment can result in the identification of the most productive IT workers, the benefits are potentially huge. Because the productivity of individual systems analysts and programmers varies tremendously, companies that can identify the top-performing candidates are likely to have greater revenues and profits than will companies that do not use structured assessments.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement