Measuring Human Capabilities An Agenda for Basic Research on the Assessment of Individual and Group Performance Potential for Military Accession (2015) / Chapter Skim
Currently Skimming:
9 Situations and Situational Judgment Tests
9 Situations and Situational Judgment Tests
Pages 187-202
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 187... ...
Situational judgment tests (SJTs) are psychological measures that present test takers with hypothetical situations that often reflect constructs that may be interpersonal (e.g., communication, teamwork)
|
From page 188... ...
In fact, the specific test items for measures of practical intelligence look a lot like SJT items (e.g., the Wagner and Sternberg, 1991, measure of the practical intelligence of managers) .1 (For a useful overview of critical SJT characteristics, such as constructs assessed, situational content, scoring methods, instructions and testing medium, see Campion et al., 2014.)
|
From page 189... ...
It also improves understanding of the nature of the situation being tested by a particular SJT. In a related innovative approach, existing Multidimensional Item Response Theory models can be applied to situational judgment measures that are designed in a similar manner.
|
From page 190... ...
. Incremental Validity Testing situational judgment has maintained its decades-long prominence in the research and practice of employment testing because, even though the test formats and the constructs they assess vary widely, SJTs have demonstrated persistent incremental validity above cognitive ability and personality measures when predicting job performance of either a technical or interpersonal nature.
|
From page 191... ...
found no such mean differences for its SJT measuring interpersonal skills. Video versus Written SJTs Traditionally, SJTs have been administered in a written format, although there are some notable instances of using video and computerized formats of the SJT, in which the nontraditional format has served at least four important purposes.
|
From page 192... ...
found smaller ethnic-group differences with video SJTs than written SJTs, Weekley and Jones (1997) found that video SJTs still exhibited considerable subgroup mean differences (0.3 to 0.6 standard deviations)
|
From page 193... ...
SUBGROUP DIFFERENCES AND ADVERSE IMPACT A meta-analysis of subgroup differences by Whetzel and colleagues (2008) found that SJTs collectively exhibit smaller white-black effect size differences than traditional cognitive ability tests (standard deviation of
|
From page 194... ...
More specifically, "should do" or knowledge instructions, which ask respondents to choose the best/worst or most/least effective option(s) from a series of alternatives, tend to increase correlations with cognitive ability measures and thus increase subgroup differences.
|
From page 195... ...
. Thus, alpha reliability coefficients are inappropriate indices of reliability for SJTs because the situational stems and item responses reflect complex situations and thus are not internally consistent.
|
From page 196... ...
used stratified random assignment by assigning SJT items randomly within each of 12 dimensions within intellectual, interpersonal, and intrapersonal domains. This assignment method, in conjunction with traditional scale construction and evaluation practices, produced 144 forms of an SJT to predict different types of college student performance.
|
From page 197... ...
(2) Alternative SJT forms can be produced in large numbers by using computationally intensive methods that are similar in spirit to adaptive testing and automated test assembly algorithms (e.g., van der Linden, 1998; van der Linden and Glas, 2010)
|
From page 198... ...
Advances in SJT Item Development and Scoring Given current SJT construction and scoring practices and the emergent nature of IRT methodologies, the committee believes the military testing process would benefit from future research and thinking on how psychometric technology, which has been used to improve precision and efficiency in large-scale testing programs, might be adapted for testing situational judgment. Additionally, the committee encourages applied researchers to think creatively about SJT item writing and answer formats; if changes can be made to increase the suitability of prevailing psychometric models without destroying the realism of SJT items, then an array of new possibilities for test construction and evaluation can be expected to follow.
|
From page 199... ...
RESEARCH RECOMMENDATION The U.S. Army Research Institute for the Behavioral and Social Sci ences should support research to understand constructs and assessment methods specific to the domain of situational judgment, including but not limited to the following lines of inquiry: A. Develop situational judgment tests with items reflecting constructs that are otherwise difficult to assess using other tests, that are im portant, and that show promise for validity (e.g., prosocial knowl edge, team effectiveness)
|
From page 200... ...
. Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions.
|
From page 201... ...
. Video-based versus written situational judgment tests: A comparison in terms of predictive validity.
|
From page 202... ...
. Assessing Leader Cognitive Skills with Situational Judgment Tests: Construct Validity Results (Research Product 2010-04)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.