New Directions in Assessing Performance Potential of Individuals and Groups Workshop Summary (2013) / Chapter Skim
Currently Skimming:
5 New Approaches and Capabilities in Assessment
5 New Approaches and Capabilities in Assessment
Pages 79-100
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 79... ...
Paul Sackett, who spoke during a panel on the sec ond day, categorized the various approaches to improve assessment and gave a broad overview of current thinking on the subject. The workshop's keynote speaker, Alina von Davier, discussed current and potential future research at the Educational Testing Service relevant to the next generation of assessments.
|
From page 80... ...
To encourage the workshop participants to think about how one might begin to improve existing predictor constructs, he offered three specific exam ples: contextualized personality items, narrower dimensions of personal ity measures, and use of real-time faking warnings. Contextualized Personality Items To begin, Sackett addressed personality assessments.
|
From page 81... ...
Narrower Dimensions of Personality Measures A variety of researchers have discussed the idea of dividing the Big Five personality traits into smaller, more focused traits. In 2007, for exam ple, DeYoung and colleagues suggested splitting each of the Big Five traits into two: • neuroticism [emotional stability]
|
From page 82... ...
"It's not useful." Furthermore, the two facets predicted different aspects of work suc cess. "The achievement piece receives the dominant weight if you're predicting task performance, while dependability receives the domi nant weight in predicting job dedication and the avoidance of counter productive work behaviors," he said.
|
From page 83... ...
. Table 5-1 shows validity scores for three measures -- general cognitive ability, need for achieve ment, and dependability -- and three performance domains -- task perfor mance, citizenship, and counterproductive work behavior.
|
From page 84... ...
Predictors General Cognitive Need for Criteria Ability Achievement Dependability Task Performance .43 .11 .11 Citizenship .22 .30 .22 Counterproductive Work Behavior .11 .18 .30 SOURCE: Sackett presentation. tributing extra effort, helping others, and supporting the organization.
|
From page 85... ...
Profiles of Predictors A set of predictors, such as the Big Five personality traits, can be thought of as a collection of numerical values, Sackett suggested, and one can then ask, what about that set of predictors is most predictive of a certain criterion? In 2002, Davison and Davenport developed a tech nique that focuses explicitly on criterion-related profiles.
|
From page 86... ...
" When Sackett and his colleague examined how the Big Five personal ity traits predicted citizenship and counterproductive behavior in a group of 900 university employees, they found that the predictive power for citizenship came completely from the profile level. For counterproductive behavior, however, both the level and the pattern were important predic tors (Shen and Sackett, 2012)
|
From page 87... ...
Using raw data on 117 samples of Big Five personality trait performance relationships, the student is examining the relationships between the personality traits and various criteria, such as job perfor mance. One example is the relationship between conscientiousness and an overall job performance measure.
|
From page 88... ...
Some of the most interesting assessments in this second category, von Davier said, use various complex tasks, such as simulations and collaborative tasks. People are working on using serious games as a context for the assessment and, eventually, as an unobtrusive way to test.
|
From page 89... ...
She imagines tests in the future that will produce data from fre quent or continuous test paradigms. There will be outcome data from complex tasks, simulations, and collaborations, all of which will produce very different types of data than tests based on classic item response theory, she explained.
|
From page 90... ...
In her work, she said, she has used the former definition of a successful collaboration. With a diagram, von Davier sketched out a framework for how one might assess cognitive skills using collaborative problem-solving tasks (see Figure 5-3)
|
From page 91... ...
I think that matters, given that we don't have anything taking skillsexcept the use of process data for the measurement of skills." hological traits Knowledge Problem al skills Data and Scores Solving A collaborative problem-solving assessment can produce both process and outcome data, von Davier noted. The process data offer an insight into the interaction dynamics of the team members, which is important Collaborative Individual
|
From page 92... ...
The modeling strategies include dynamic factor analysis, multilevel modeling, dynamic linear models, differential equation models, nonparametric exploratory models such as social networks analysis, intravariability models, hidden Markov models and Bayes nets, machine learning methods, latent class analysis and neural networks, and point processes -- which are tochastic s processes for discrete events. The last of these strategies is what von Davier is using now.
|
From page 93... ...
"We are trying to integrate the stochastic processes modeling approach with the outcome data approach." Among the models they are considering applying to outcome data are what von Davier referred to as item response theory–based models. She explained that she meant models that use the same type of rationale that is used in response patterns.
|
From page 94... ...
Possible Approaches for Outcome Data In the final part of her address to the workshop, von Davier said that she is seeking ways to integrate the models for process data with the out come data. One idea her team is developing goes further with the Hawkes processes to assign an assessment of correct or incorrect outcome to each event.
|
From page 95... ...
He then asked von Davier which of the approaches might be more fruitful for coding the results of the collaborative problemsolving tasks. Would it be the data mining approaches, which have beat out the expert judgment approaches in fields from Google's Internet searches to playing chess, or would it be the more evaluative, theoretical approach?
|
From page 96... ...
For more information on Little's ideas BOX 5-1 Missing Data In his presentation during the workshop's first panel, Todd Little of the Univer sity of Kansas in Lawrence, mentioned the value of missing data techniques in various types of analyses, although he did not go into detail at that point. Later, at various points during the two-day workshop, Little and other attendees offered more detailed suggestions for how missing data techniques could be put to work in research and assessment.
|
From page 97... ...
Different individuals would be tested on different constructs, he explained, and one could have a set of variables that could be used to examine all constructs in one big multivariate model, to see if incremental validity is obtained from any of the constructs, above and beyond what is already being measured. Little also suggested that planned missing data designs could help in studies such as those described by Alina von Davier that collect a great deal of intensive data through the use of observers to code what they observe.
|
From page 98... ...
. Applied Missing Data Analysis.
|
From page 99... ...
. The Relationship of Big Five Personality Profiles to Job Perfor mance.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.