3

New Constructs for Assessing Individuals

Recent psychological research points toward a variety of constructs that can be used in assessments to provide greater accuracy or additional valuable information about the individuals being assessed. The workshop presenters described a number of such cutting-edge constructs, primarily during the first panel: Emerging Constructs and Theory. In this chapter, invited presentations during the first panel from Christopher Patrick, Michael Kane, and Todd Little are described, as well as the related presentation by James Rounds on interests, during the workshop’s second day.

NEUROBEHAVIORAL CONSTRUCTS

Christopher Patrick, a professor of clinical psychology at Florida State University, discussed one approach to revising existing constructs and developing new constructs: the psychoneurometric approach. He defined psychoneurometrics as “the systematic development of neurobiologically based measures of individual difference constructs, using psychometric operationalizations as referents.” In essence, it is a way of developing measures of individual differences by combining information and insights from neurobiology (the study of the biological aspects of the brain and nervous system) with what has been learned from psychological studies of individual differences. An associated aim of the psychoneurometric approach, Patrick continued, is to refine the individual difference constructs themselves through the incorporation of physiological data—such



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 21
3 New Constructs for Assessing Individuals R ecent psychological research points toward a variety of constructs that can be used in assessments to provide greater accuracy or additional valuable information about the individuals being assessed. The workshop presenters described a number of such cutting- edge constructs, primarily during the first panel: Emerging Constructs and Theory. In this chapter, invited presentations during the first panel from Christopher Patrick, Michael Kane, and Todd Little are described, as well as the related presentation by James Rounds on interests, during the workshop’s second day. NEUROBEHAVIORAL CONSTRUCTS Christopher Patrick, a professor of clinical psychology at Florida State University, discussed one approach to revising existing constructs and developing new constructs: the psychoneurometric approach. He defined psychoneurometrics as “the systematic development of neurobiologically based measures of individual difference constructs, using psychometric operationalizations as referents.” In essence, it is a way of developing measures of individual differences by combining information and insights from neurobiology (the study of the biological aspects of the brain and nervous system) with what has been learned from psychological studies of individual differences. An associated aim of the psychoneurometric approach, Patrick continued, is to refine the individual difference con­ structs themselves through the incorporation of physiological data—such 21

OCR for page 21
22 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL things as brain activity, the levels of hormones and other biological mol­ ecules, and measurements of various reflexes. “I want to work back and forth between the physiological data and the starting constructs and come to things [new individual difference conceptions] that would make more sense from a neurobiological standpoint,” he said. There are several reasons for incorporating physiology into individ­ ual differences assessments, Patrick said. First, many of the major con­ temporary trait-dispositional models refer to neurobiology, but the neuro­ biological referents tend to be added after the fact. That is, the traits used in these models were initially developed on the basis of self-report data, and it was only afterward that researchers sought to identify their neuro­ biological counterparts. By contrast, the psychoneurometric approach seeks to incorporate neurobiological indicators from the beginning so that the trait conceptions themselves are shaped by neurobiological data. Patrick gave two other reasons for incorporating physiology into the assessment of individual differences: (1) to help address the issue of response bias (which refers to the tendency of people answering ques­ tions to be influenced by what they believe the questioner expects), and (2) to gain insight into the processes involved in how people confront and cope with a given situation. The old model of understanding behav­ ior in a situation, the stimulus–response model, was superseded by the stimulus–organism–response model, in which processes occurring within the organism are considered crucial for understanding the connection between the stimulus and the response. Biology is an important part of understanding such relationships. Finally, Patrick noted that understand­ ing the physiological basis of capabilities is important to the design of optimal training performance methods. Before introducing the two main constructs that he studies, ­ atrick P offered two key points in thinking about the psychoneurometric approach. First, neurobiological indicators of any type are complex and multi­ determined. In particular, the reliable person-variance in any physiologi­ cal indicator will reflect sources other than just the performance capability of interest. And second, linking the domains of physiology and adap­ tive performance requires a bridge of some sort. The bridging approach he employs is the use of neurobehavioral constructs, by which he means “constructs that have clear referents in both neurobiology and behavior.” Constructs of this type can serve as referents for combining physiological indicators with indicators from other domains (e.g., self-report or overt behavioral responses) to form composite measures that have meaning both psychologically and physiologically.

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 23 Defensive Reactivity Patrick’s work in psychoneurometrics to date has focused on two neuro­ ehavioral constructs. The first is defensive reactivity, which he b defined as “proneness to negative emotional reactivity in the face of threat.” It is a cue-specific negative response, he emphasized, in contrast to, for instance, a free-floating anxiety or neuroticism—that is, the sort of worry or negative emotion that has no obvious immediate cause or trigger. The presumed neural basis of defensive reactivity is “individual dif­ ferences in the sensitivity or responsiveness of the brain’s defensive sys­ tem, including the amygdala and affiliated structures.” The amygdala is a part of the brain that plays a key role in the processing of cues signal­ ing uncertainty or danger and in the formation of memories of events that have emotional content, such as the memory of a frightening event. Defensive reactivity may also involve interactions with frontal cortical systems of the brain, he said, in terms of what we think of as emotion regulation or inhibition. Frontal cortical brain regions are involved with, among other things, the representation of complex emotional cues or contexts, the formation of long-term memories associated with emotions, and control (regulation) of affective responses. Patrick’s operational model of defensive reactivity—that is, the specific way in which he measures it in human subjects—is based on measures that have been developed to index variations in fear versus fearlessness (or boldness). The model was based on an analysis of data from 2,500 twin participants who completed self-report questionnaires whose scores have been shown in experimental studies to be related to fear-potentiated startle. This fear-potentiated startle is a standard physiological indica­ tor of fear that is often, in practice, the observation of an eye blink in response to some unexpected stimulus, such as a loud noise (Kramer et al., 2012). ­ igure 3-1 shows how the model represents dispositional fear F versus ­ oldness as the common individual difference dimension indexed b by differing scale measures of fear/fearlessness. “We found evidence for a general factor [or dimension] on which all of these measures either loaded positively or negatively,” Patrick said. (Two measures are positively correlated when increases in one are associ­ ated with increases in the other; they are negatively correlated when an increase in one is associated with a decrease in the other.) Some of the measures Patrick discussed reflect the expression of fear versus boldness in the social domain; others reflect such expression in the activity prefer­ ence or the sensation-seeking domain, and still others reflect expression in the perceived experience (feeling) domain. “From the standpoint of this model,” he said, “we think of neurobiological fear as the core of expres­

OCR for page 21
19 item Boldness scale: r = -.8 w/ gen factor of this model Fear vs. Boldness - N = 2,511 twins - scale var’s all relate to fear-potentiated startle FIGURE 3-1  Operational model: trait fear versus boldness. NOTE: EAS-F = emotionality–activity–sociability fearfulness scale; FSS = fear survey schedule-III; PPI = psychopathic personality inventory (F = fearlessness subscale, SI = stress immunity subscale, SP = social potency subscale); SSS-TAS = sensation seeking Figure 3-1 scale, thrill and adventure seeking subscale; TPQ-HA = tridimensional personality questionnaire, harm avoidance scale (HA1 = R02494 anticipatory worry and pessimism subscale, HA2 = fear of uncertainty subscale, HA3 = shyness with strangers subscale, HA4 = 24 fatigability and asthenia subscale). partly bitmapped, partially vector editable, (type at top) SOURCE: Adapted from Kramer et al. (2012).

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 25 sion of fear in different domains, as manifested in self-report, that might hang together separately for other reasons than the physiology of those characteristics.” That is, this “dispositional fear” can be thought of as the degree of defensive reactivity exhibited in different psychological con­ texts; these contexts may seem separate in personality models based on self-report, but from a neurobiological standpoint, the behavior in each context is influenced by variations in dispositional fear. Patrick added that he and his colleagues have followed up on this quantitative modeling work to develop fine-grained scales for measuring the general fear versus boldness dimension of the model using self-reports. They are also inter­ ested in measuring the construct behaviorally with various tasks as well as with physiological measures. “Again,” he reiterated, “we are choosing these indicators because of their relationship to a neurophysiological indi­ cator,” that is, to the fear-potentiated startle response. Thus assessment techniques framed around this model are grounded in neurophysiology. Applying his ideas to real-world scenarios, Patrick said the trait char­ acteristic of low dispositional fear or “boldness” was well illustrated by the recent article, “Fearless Dominance and the U.S. Presidency: Implica­ tions of Psychopathic Personality Traits for Successful and Unsuccessful Leadership” (Lilienfeld et al., 2012). The authors had expert presidential biographers rate the presidents on facet-level traits of the Big Five person­ ality traits model. Based on prior work linking Big Five personality traits to measures of psychopathy, the authors then used these facet ratings to estimate scores for the presidents on factors of psychopathy, one of which, “fearless dominance,” is very similar to his own concept of bold­ ness, ­ atrick said. Among the U.S. presidents considered in the article, P Theodore Roosevelt was rated highest in boldness. In his book The Antisocial Personalities, David Lykken writes, “The hero and the psychopath are twigs on the same genetic branch” (Lykken, 1995). The idea, Patrick explained, is that some people may possess the temperament of a psychopath, but because of other factors they do great things. “Lykken talked about the bold tendencies of Winston Churchill as an example of someone who had what he saw as the temperament of a psychopathic individual, but expressed in a more benign sort of adaptive way.” What is the difference between someone who expresses this characteristic of boldness adaptively versus maladaptively? Part of the answer, Patrick said, may be found in the second trait he has been studying—inhibitory control.

OCR for page 21
26 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL Inhibitory Control The second construct on which Patrick’s research focuses is inhibitory control, which is defined as “the ability to restrain or modulate impulses.” This trait relates to the degree to which people can modulate their tenden­ cies and their behavior under conditions that require a certain amount of flexibility and foresight. The presumed neural basis for inhibitory control lies in individual differences in the functioning of anterior brain circuitry, including the prefrontal cortex subdivisions and the anterior cingulate cortex. The operational model for inhibitory control is what he refers to as the “externalizing spectrum,” or “trait inhibition–disinhibition” model (Krueger et al., 2007). The sample that Patrick and his colleagues used to develop the model consisted of students as well as prisoners “because we wanted to make sure that we mapped the full range of the continuum by including representations of individuals with very extreme externalizing tendencies.” Again, as with defensive reactivity, the goals of this work were to develop an individual difference measure that had physiological corre­ lates and to develop effective and efficient scales for measuring this con­ struct through self-reports. The model they developed contains 23 facet scales that represent proclivities toward impulsiveness, aggression, rebel­ liousness, risk taking, and use and abuse of substances, all of which cor­ relate with a broad inhibition–disinhibition factor (Krueger et al., 2007). In this case, the physiological correlates of the inhibition–disinhibition factor include the P300 response and the error-related negativity response, which are two types of brain reactions that can be detected and recorded using the technique of electroencephalography that records brain electri­ cal activity from the scalp surface. The Value of the Constructs Patrick gave three reasons why the constructs of defensive reactivity and inhibitory control may be of interest to those developing assessments for military personnel: (1) adaptive performance, (2) direct brain referents, and (3) insights from psychoneurometrics. Importance to Adaptive Performance The first reason why these constructs may be of interest for military assessments is that they are important to adaptive performance. For exam­ ple, a 2009 study found that individuals who were high in boldness were better able to maintain their focus on a task under threat of a shock (Dvorak- Bertscha et al., 2009). Thus it is reasonable to predict that high boldness would predict enhanced adaptive flexibility in a threatening situation.

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 27 Responding to a question regarding the distribution of boldness across the population, Patrick indicated that it shows a normal distribu­ tion very similar to the levels of intelligence within the population. He also noted that lack of inhibitory control has been shown to correlate with a propensity to develop post-traumatic stress disorder (PTSD) (Miller et al., 2006). He also speculated that, while individuals with high inhibitory control may still develop PTSD, the presence of inhibitory control is likely to produce a more focused and contained reaction to the specific event, rather than manifesting as generalized fear. Similar to boldness, inhibitory control—which can be assessed via self-report—also correlates with performance in a variety of contexts. Miyake and Friedman (2012) showed, for example, that variations in gen­ eral executive function contribute to success on a range of different cogni­ tive performance tasks. A 2009 study of twins by Young and colleagues showed performance on tasks indicative of general executive function correlated with scores on the aforementioned inhibition-disinhibition fac­ tor (c.f., Krueger et al., 2007), and operationalized by the presence or absence of tendencies toward antisocial behavior and substance abuse. That is, the less inhibited participants were (as evidenced by clinical problems), the poorer they performed on the cognitive tasks related to executive function. The Constructs Have Direct Brain Referents The second point Patrick made about the constructs of defensive reactivity and inhibitory control is that they have direct brain referents. A major physiological indicator of boldness, for example, is fear-potentiated startle, which is defined as the increase in the magnitude of the natural defensive startle reflex (generally to a loud noise) that occurs when a person is doing something or viewing something that is scary. People who score high on boldness measures exhibit a reduced fear-potentiated startle, which indicates that they are not so likely to automatically mobi­ lize their defenses in the face of a threat—a characteristic that may be valuable in contexts involving stress or uncertainty. Similarly, there are various established neurophysiological indica­ tors of disinhibition. One of these is the P300 response, which occurs in response to certain types of stimuli during tasks when a subject is asked to respond selectively to certain stimuli within a series—that is, to respond to some and not to others. The P300 has been studied since the 1980s as an indicator of alcohol problems, Patrick said, and more recently it has been shown to be an indicator of the inhibition–disinhibition dimension that undergirds impulse problems more broadly. Another brain ­ ctivity– a based indicator of inhibition–disinhibition is error-related negativity, a

OCR for page 21
28 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL negative deflection in brain activity that occurs when the subject realizes a performance error has been made. Highly disinhibited individuals have smaller responses of this sort, which suggests that they may not be as aware of errors as they occur—a factor that likely contributes to repeti­ tion of mistakes. Patrick’s main message in describing this research is that it is possible to investigate—and in the process, clarify—the nature of these individual difference constructs through use of indicators that are purely neuro­ physiological or by using physiological indicators together with indica­ tors from other domains, such as self-reports or behavioral responses. The Constructs Can Be Sharpened with Psychoneurometrics The third reason that constructs of these types are interesting and valuable, Patrick said, is because they have one foot in the domain of psychometrics, the field of psychological measurements, and the other in the domain of neurobiology. This makes it possible to work back and forth between the two domains to sharpen the operationalizations of the con­ structs and to gain greater insight into their structures and relationships. In closing, Patrick described a general research strategy that could be useful to achieve this sharpening of individual difference constructs, con­ sisting of the following steps: (1) identify replicable neuro­ hysiological p indicators of psychometric measures of target constructs (e.g., disinhi­ bition and trait fear measures); (2) evaluate the covariation among the neuro­ hysiological indicators, that is, identify coherent neurophysiologi­ p cal factors; and (3) revise the psychometric measures and trait conceptions to cohere better with the neurophysiological factors. “The construct and the way we think about it are movable, as a function of what we learn about the convergence of the physiological indicators,” he explained. “This allows for psychological trait conceptions to be reshaped by physi­ ological data. Then the revisions to the psychological trait conceptions in turn can help to reshape [one’s] conceptions of performance capacities to better accommodate physiological data.” The process can be carried out iteratively, sharpening both the psychometric measures and the cor­ responding physiological indicators. WORKING MEMORY CAPACITY AND EXECUTIVE ATTENTION While Christopher Patrick, the first speaker of the Emerging Con­ structs and Theory panel, focused on the use of neurobiology in measur­ ing and refining constructs, Michael Kane focused on constructs derived from psychological theory. Kane, a professor of psychology at the Uni­ versity of North Carolina at Greensboro, described two such constructs—

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 29 working memory capacity and executive attention—that are measurable and predictive of a number of outcomes relevant to the military. Working Memory Capacity The construct of working memory capacity, Kane said, is derived from basic cognitive theory and, in particular, from Baddeley’s theory of working memory as a complex system that has several storage structures that hold specific types of short-term memories (the phonological loop for the sounds of language, the visuospatial sketchpad for visual and spatial information, and the episodic buffer for various other short-term memories) combined with a “central executive” that controls where atten­ tion is focused and coordinates different cognitive processes. The short- term memory structures are closely associated with the corresponding structures for long-term or secondary memory, which consequently have implications for performance. Figure 3-2 illustrates Baddeley’s model of working memory. “The theory is very functionally based,” Kane said. “The idea is that working memory evolved for a purpose, which is to help us maintain access to memory representations in the service of ongoing cognitive activities, like comprehending language or solving multistep problems.” Central Executive Visuospatial Episodic Phonological Sketchpad Buffer Loop Visual Semantics FIGURE 3-2  Baddeley model of working memory. NOTE: LTM = long-term memory. SOURCE: Reprinted from Baddeley, A.D. (2000). The episodic buffer: A new component of working memory? Trends of Cognitive Science, 4(11):417-423. With permission from Elsevier. Figure 3-2 R02494 some type replaced, otherwise bitmapped

OCR for page 21
30 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL According to Kane, the most compelling evidence for the functional importance of working memory capacity has come from research into individual differences. For example, research by Daneman and Carpenter (1980) demon­ strated that language comprehension capabilities were strongly predicted by students’ working memory span scores. A decade later, broad working memory capacity was shown to be an almost perfect predictor of Air Force recruits’ general reasoning capabilities (Kyllonen and Christal, 1990). “Most of the modern research on working memory capacity uses some version of a working memory span task,” Kane continued. These are variations on the traditional short-term memory span tasks, which ask subjects to recall item lists in serial order, with the added feature that the items to be remembered are presented alternatively with a secondary processing task, such as judging sentences or verifying equations. The distinguishing feature of the working memory capacity test is that the subjects must “maintain ready access to the goal-relevant information— the memory items—in the face of massive proactive interference from prior trials and attention shifts away from those memoranda as they shift to the processing tasks.” Executive Attention Not surprisingly, there is a great deal of overlap between measures of working memory capacity and measures of short-term memory and various cognitive abilities. For example, a span task that involves equa­ tions will leverage mathematical ability in addition to working memory capacity, and a test that involves mental rotations will reflect spatial abil­ ity as well as working memory. By using a series of different types of tests, along with latent-variable analyses, it is possible to tease out working memory capacity from other factors, Kane explained. In one key example of this type of study, Engle and colleagues (1999) looked at the relationships between memory and reasoning by having their subjects complete three verbal measures of working memory capac­ ity, three span measures of short-term memory capacity, and two non­ verbal measures of general fluid intelligence. In this way they could assess commonalities between working memory and short-term memory as well as distinctions between the two. They discovered that memory storage in itself was not correlated with general intelligence. “Rather, it was what working memory did independently, over and above the memory storage demands, that was the strongest predictor of general fluid intelligence.” From the point of view of Baddeley’s model, it was not the storage system that was key to general intelligence but rather the “attentional executive capability”—the attention-directing part of the system that came to the

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 31 fore in the memory tests involving multiple tasks—that was most closely associated with fluid intelligence. Kane referred to this capacity as execu- tive attention. Kane went on to describe several studies that demonstrate the roles that executive attention plays, including its relationship to working mem­ ory. In one study, he asked subjects to learn and then recall three lists of words, all from the same category, such as “animals” (see Figure 3-3). In some cases they were required to divide their attention by tapping a novel finger sequence on a keyboard over and over again, either while they studied each list (encoding load) or while they tried to recall each list (retrieval load) (Kane and Engle, 2000). Part a of Figure 3-3 shows the results for subjects whose attention was not divided. Subjects with high- working memory capacity (high span) and subjects with low-working memory capacity (low span) recalled approximately the same number of words on the first list, but a clear difference emerged on recall of words on the second and third lists. Both the high span and low span subjects recalled fewer words from the second list and even fewer from the third, but the drop-off was much more dramatic among the low span subjects. The explanation, Kane said, is that those with higher working memory capacity were better able to deal with the “interference” of having to memorize and retain the earlier groups of words. Interestingly, when the high span subjects were asked to memorize or recall the lists of words while having to divide their attention, their performance (shown in part b of Figure 3-3) dropped to the level of the low span subjects, when not required to divide their attention (shown in part a of Figure 3-3). “Essentially,” Kane said, “dividing attention turns high-working memory subjects into functional low-working memory sub­ jects.” Thus the test is not just assessing memory. “It is attention that really seems to matter,” Kane observed. This difference in ability to focus attention between subjects with high or low working memory was reinforced in a second study performed by Kane and colleagues (2001) that examined something quite different from memory. In this case high- and low-working memory subjects were tested on how quickly they could move their eyes in the proper direction after a stimulus. It is a standard assessment of executive control referred to as the antisaccade/prosaccade task. (A saccade is a quick movement of the eye.) “In the prosaccade task,” Kane explained, “you stare at a computer screen, wait for a flash on one side of the screen, and look at it. Right there, there is going to be a letter that you have to identify. This is easy. The flash pulls attention toward the cue.” The antisaccade version is more difficult: When the light flashes, the subject is instructed to direct his or her vision to the opposite side of the screen to see the target letter. Because this antisaccade movement goes against natural tendencies, everyone is

OCR for page 21
40 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL The theoretical definition that Little provided for “agency beliefs” is “Agent (A) has means (M) that is relevant to end (E).” The operational definition is “the person’s belief that he or she personally has access to, can use, can implement, or possesses a specific means that is relevant to achieve the outcome.” This is where action-control beliefs have an advantage over concepts like efficacy, Little said. “It is a very easy system to operationalize.” It is also straightforward to develop specific items to measure agency beliefs, he said. Once the specific context is known, one determines the kinds of means that would be useful in the particular context—being as inclusive as possible—and develops items to assess a person’s beliefs about those particular means. Examples of the sorts of items he has used in his own assessments include, “I can try hard,” “I am smart enough to do it,” “I am unlucky at it,” and “I can get others to help me” (see Little and Wanner, 1997). It is important to note that these are all intrapersonal beliefs—effort, ability, and even luck. “When it comes to the side of agency, luck is an intrapersonal thing,” Little said. “It is something that I possess and own, just like my effort and my ability.” However, he noted, when an indi­ vidual evaluates others, luck tends to be interpreted as a factor external to the individual. Little has studied the relationship between these beliefs and an indi­ vidual’s actual performance in various tasks. He emphasized the striking result that quite often there is no added value to knowing an individual’s control expectancy belief—the degree to which an individual believes he or she can achieve the specific goal—versus simply knowing an indi­ vidual’s abilities (Little et al., 1995). “If I know what you possess, in terms of your effort and ability, those will always outperform just whether or not you think you will get it done. The ‘I think I can, I think I can, I think I can’ doesn’t buy us anything in terms of predictive ability.” Similarly, there is generally relatively little correlation between an individual’s score on a self-efficacy measure and actual performance (Multon et al., 1991). The correlations between traditional measures of self-efficacy and performance are generally around 0.3, Little said. By contrast, in certain contexts he has seen correlations between action- control beliefs and performance that are greater than 0.7. The key, he said, is refining the measures to be very specific about the goal structure and the specific means that can be used to achieve the goal. The traditional measures of self-efficacy are generally at too high a level of aggregation to be very predictive. Finally, Little described the causality beliefs, or means–ends beliefs in his model. “These are really contingent belief operations: my understand­ ing of how the world works; my understanding of, will effort get me to this goal, will ability get me to this goal, will my looks, my personality,

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 41 t ­eachers, parents, whatever, get me to that goal; and to what degree are they important for achieving that particular goal?” These beliefs develop in a variety of ways, he noted, and they are relatively trainable. “In the school setting, we teach kids what it takes to do well in school. In a mili­ tary context, we can teach people what it takes to do well in this assign­ ment.” Once a person knows what the assignment is, he or she can look for a match between agency beliefs and the causality beliefs. The important feature here, Little said, is that it is generally possible to provide multiple means to an end, so if a person does not have the tools to do it one way, it can be done another way. For example, a person might think of it this way: “I am not really the sharpest pencil in the box, but I will work . . . and I will just go and go and go and go. I can get to the same goal that you can because you are smart. I can get there by effort.” There are various ways that action-control beliefs are acquired, ­ ittle L said. Some of them come from direct experiences with success and failure, while others are taught by parents, teachers, peers, and others (­ ittle, L 1998). Feedback on one’s own performance helps refine the various beliefs, as do vicarious observations: watching and seeing how others do things, in person or virtually. One important influence on action-control beliefs is social compari­ sons: a person learning how he or she measures up to others. But the comparisons need to be accurate. In the United States, Little said, “when we ask kids about their agency beliefs for doing well in school, everybody believes they are above average because the teachers keep telling them they are great and wonderful.” By contrast, in Germany honest feedback is a “valued cultural aspect,” and agency beliefs are much more closely cor­ related with performance scores (Little et al., 1995; Oettingen et al., 1994). According to Little, another important influence on action-control beliefs comes from symbolic actions—that is, by thinking through and rehearsing what it would take to carry out a certain task, without actu­ ally doing it (Boesch, 1991; Brandstädter, 1998). Using symbolic actions, people can develop action-control beliefs in a completely new context, one with which they have no previous experience. Little closed by describing some of the differences that have been reported in the literature between agentic and nonagentic people (­ awley H and Little, 2002; Little et al., 2006); that is, between those who tend to believe in their ability to control things in their lives and those who do not (see Box 3-1). Nonagentic people have little sense of personal empower­ ment, feel helpless when they are challenged, and tend to accept failures. They have low aspirations and perform poorly on most tasks. Agentic individuals are just the opposite. They have a greater sense of personal empowerment, they persist in the face of obstacles, and they learn from failures. They have high aspirations and perform well on tasks.

OCR for page 21
42 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL BOX 3-1 Nonagentic Versus Agentic Profiles Nonagentic Profile Agentic Profile • Has low aspirations • Has high aspirations • Feels helpless when challenged • Persists in the face of • Is hindered by problem-solving obstacles blinders • Sees more and varied options • Performs poorly • Performs well • Accepts failures • Learns from failures • Has greater ill-being • Has greater well-being • Has little sense of personal • Has a greater sense of empowerment personal empowerment SOURCE: Little presentation. DISCUSSION Following the three panel presentations, committee members, the presenters, and other participants engaged in a roundtable discussion that included specific questions for the panelists as well as thought-provoking brainstorming on the future of measurements and assessments. The fol­ lowing section captures some of the more salient ideas expressed during the group discussion. The first several questions to panelists addressed the distribution of various traits across the population and when certain traits might be more desirable than others. Participants debated in detail the importance of the context of the task in judging desirable traits. For example, when might it be better to be more likely to be distracted from a task (for example, if an emergency happens outside your area of attention) versus being highly focused on the task? In response, Patrick clarified the idea of fear versus boldness as a normal individual difference continuum: “To have a lot of fear is non-normative, but potentially adaptive in certain contexts,” he said. Likewise, he continued, to have very limited fear is non-normative but potentially adaptive in other contexts. That is, the existence of indi­ vidual differences in fearfulness reflects an adaptive trade-off between exploratory types of behaviors and tendencies toward defensive with­ drawal. Little agreed with this point, noting that it is not necessarily always the leader who survives; “followers survive, too, when they hook up with the right leaders” (see Hawley 1999; Hawley and Little, 2002; see also Buss and Hawley, 2010, for an edited volume on the evolution of individual differences).

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 43 The discussion of the many potential factors contributing to success at a specific task led Little to emphasize that any assessment of potential performance is a “multivariate problem.” “Working memory is clearly an important factor,” Little said, “but boldness is there, too. How does boldness work with working memory, under which contexts do we see an optimal one? We might find that boldness is selected for in some con­ texts, whereas memory is in others.” Several of the participants suggested that creating performance context, affective context, and other relevant contexts may be an interesting approach to assess the relationships and interactions of many different constructs in different situations. Later, Patrick took the discussion further by suggesting that not only should some things be assessed together but other things may need to be dissoci­ ated. Assessments may thus need to strategically separate constructs, to determine how they respond as indicators are manipulated. For example, Patrick added, “you are still holding onto something that is, say, callous­ ness, but you are moving it away from disinhibition, so aggression is no longer an indicator.” Further proof of the need to both combine and disentangle constructs, according to Patrick, is the existence of suppressor relationships between correlated constructs: predictions improve as clear discriminate relationships emerge. Stephen Stark, a committee member, asked Little about the context for the beliefs and traits assessed by his model. “There have been studies that ­ suggest that contextualizing personality items seems to increase the pre­ dictive validity of the measures,” Stark noted, adding that one of the c ­ ommon concerns with generalized contextualization is that contextualiz­ ing the items too much may ultimately require different types of items for different situations. From a practical standpoint, this could be a challenge. Stark then asked Little to explain context with regard to his measures and the level of specificity necessary to achieve high validities relative to the broader, higher-level self-efficacy constructs and locus of control. In par­ ticular, Stark said that there are different occupational specialties in the military and that recent work suggests that different personality profiles are more or less predictive of performance across different jobs. He asked how specific the items would have to be in Little’s tool to be useful for at least families of jobs that are fairly similar in terms of their goals, charac­ teristics, and environments. Little responded that he believed his tool could be effective at the level of job families such as military occupation specialties. “I could see a level of aggregation for these beliefs, that you would still have predic­ tive capability, but wouldn’t necessarily have to get down to specific job title.” He also thought it may be possible to have more refined tools that could determine which of two jobs in a particular job family a candidate might be best suited for. “You do the big sweep, you get your low-hanging

OCR for page 21
44 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL fruit first,” he said. “Then, you start doing follow-up assessments, after you can start moving people in the right direction, and then you can get to specific components.” INTERESTS In his presentation during the workshop’s second-day panel of individual differences and performance, James Rounds, a professor of psychology and educational psychology at the University of Illinois at Urbana-Champaign, further contributed to the workshop’s theme of emerging constructs and theory. Rounds began by suggesting that the future of individual performance prediction should revisit its history. In the 1920s and 1930s, psychologists paid a great deal of attention to per­ sonal interests in the belief that they would be predictive of many things, including how successful people would be in their careers. However, over time that attention faded as evidence seemed to indicate that interests had relatively little predictive power. However, Rounds argued that it is time to give interests a second look. Not only are interests surprisingly stable over the course of a person’s life, he said, but, when analyzed properly, they can also be used to predict performance and achievement. Rounds then offered some background on how interests are gener­ ally approached by psychologists, describing two approaches to studying interests. One approach considers interests in terms of situations, that is, as context-specific “emotional states, curiosity, and momentary motiva­ tion” (Schraw and Lehman, 2001). This approach is associated more with educational psychology and with an experimental approach to interests. It is generally referred to as “situational interests” or sometimes “individual interests.” The other approach, generally referred to as “dispositional interests,” considers interests more in terms of personal traits that reflect a person’s “preferences for behaviors, situations, contexts in which activities occur, and/or the outcomes associated with the preferred activities” (Rounds, 1995). In other words, interests are seen as expressions of underlying personality traits. Generally speaking, Rounds said, researchers who study situational interests do not collaborate with those who study dispositional interests, to the extent that they might even be considered two separate scientific disciplines. They do not share data or ideas, and each group is generally unaware of the other group’s work. Thus, according to Rounds, some things have fallen between the cracks. Some of the earliest work in the field was done by people in the dispositional interest camp. In the 1920s, Walter Bingham set in motion a program that eventually led to the development of nine different inter­

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 45 est inventories—sets of questions designed to identify interests in vari­ ous areas. One of these inventories, developed by Edward K. Strong, Jr. (1943) is still used today. The underlying assumption in much of this work, according to Rounds, is that understanding interests would lead to a ­ etter understanding of performance. b However, in recent years the more predominant assumption has been that interests actually have relatively little to do with performance. Much of the change, Rounds said, can be traced to a paper published in 1984 by Hunter and Hunter, who reported that there was very little correla­ tion between interests and performance—generally no more than about a 0.1 correlation. Other studies used the evidence from that paper to suggest that interests are not effective predictors of performance (for example, B ­ arrick and Mount, 2005). The result, Rounds said, is that “you will not find anyone talking about performance and interests, period.” Further­ more, “you hardly find interests in textbooks anymore.” But Rounds believes that there is a great deal to learn from studying interests, and he offered three lines of evidence, summarized below, to support his contention. Interests and Performance Much of what has been written about interests, Rounds said, dis­ cusses interests as a motivational type of variable. It provides some sort of direction, it energizes a person, and it increases persistence (Nye et al., 2012). There is also a parallel literature on person–environment fit (see, for example, Kristof-Brown et al., 2005, and Verquer et al., 2003). That literature suggests that the extent of compatibility between an individual and his or her environment can influence performance outcomes. Unfor­ tunately, Rounds said, much of that literature ignores interests. Yet he believes that the literature on person–environment fit has the potential to show interests in a different, more compelling light. To show why, Rounds described a meta-analysis that he and col­ leagues conducted on 60 studies published since 1934 (Nye et al., 2012). Of those 60 studies, 45 percent were published after the Hunter and Hunter paper appeared in 1984. The analysis included both studies that used interest scale scores and studies that used congruence indices reflect­ ing the fit between a person’s interest profile and either the person’s job or the person’s occupational profile. The authors tested for correlations between the interest scores or congruence and several measures of perfor­ mance, including task performance, organizational citizenship behavior, persistence in the workplace, and persistence and grades in an academic setting.

OCR for page 21
46 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL The analysis found a clear correlation between interests and perfor­ mance, no matter how performance was measured. The overall correlation was 0.20—about double what Hunter and Hunter had reported nearly three decades earlier. What was most striking, however, was the over­ all correlation between performance and the congruence indices—about 0.36, which was very significant (Nye et al., 2012). Thus, while interests are moderate predictors of performance criteria, the correlations increase substantially when the congruence between the individual’s interests and the environment is considered. The takeaway message, Rounds said, is that fit, particularly with regard to interests, really matters. Interests and Career Success A second study from Rounds’ laboratory looked at the incremental validity of vocational interests beyond personality and cognitive abili­ ties in predicting academic achievement and career success (Su, 2012). It involved the analysis of data from a large-scale longitudinal study, Project Talent, which began in 1960. Five percent of American high school stu­ dents in grades 9 through 12 participated in a full 2 days of testing, which included a comprehensive set of cognitive ability measures, 10 personality scales, and a large collection of interest items. After the original testing, the participants were surveyed 3 additional times—at 1, 5, and 11 years after their high school graduation—about their educational, occupational, and personal development. Rounds’ graduate student performing the study, Rong Su, used regression analysis to determine the relative importance of interests, personality, and ability for various types of achievements, such as col­ lege degrees attained and income. Su found that ability was clearly important and that, indeed, it was the most important predictor of every achievement except income. Personality also played a role, but interests played a larger role than personality in every area, and interests were by far the biggest predictor of income (Su, 2012). The results are shown in Figure 3-7. It is natural to assume, Rounds noted, that the correlation between interests and income can be explained by people selecting jobs in better- paying fields, such as science or business, but the data indicate it is not that simple. “The predictive power of interests for income does not just come from its influence on career choices,” he said, “but also comes from its influence on advancement in a career.”

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 47 100% 90% 21.7 26.6 32.9 35.4 80% 11.9 70% 8.2 23.1 Interests 60% 7.4 83.3 Percentage 50% Personality 40% 66.4 Ability 30% 58.9 57.2 50.4 20% 4.7 10% 12.0 0% Income Occupational Degree College Grades in Prestige Attainment Persistence College FIGURE 3-7  Relative importance of interests, personality, and ability for educa­ tional and career success. SOURCE: Adapted from Su (2012, p. 126). Figure 3-7 The Stability of Interests R02494 vector editable Interests have always been considered fairly stable, Rounds said, but no one had looked carefully at exactly when they become stable and how they might change, for example, from adolescence to young adulthood. To investigate the stability of interests, Rounds and his colleagues carried out a meta-analysis of 66 studies (Low et al., 2005). They found there were few studies that examined the stability of interests beyond age 40. How­ ever, they were able to access a significant amount of data concerning the stability of vocational interests from ages 12 to 40. The analysis found that there was a big jump in stability of interests at about age 18, at which point they stabilize and stay about the same through age 40 (Low et al., 2005). This finding is important, Rounds said, because it indicates interests stabilize much earlier in life than had previ­ ously been thought. It is also useful information for predictive purposes, since one can reasonably assume that whatever measures of interest are obtained for subjects after age 18 are likely to remain fairly stable. One surprising result of the study, Rounds said, concerned the stabil­ ity of interests versus the stability of personality. Most researchers tend to think of things like interests and values as deriving from basic personality traits, and so it would seem that interests should be less stable than per­

OCR for page 21
48 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL sonality traits. However, the data indicated that interests are substantially more stable than personality traits (Low et al., 2005). REFERENCES Baddeley, A.D. (2000). The episodic buffer: A new component of working memory? Trends of Cognitive Science, 4(11):417-423. Barrick, M.B., and M.K. Mount. (2005). Yes, personality matters: Moving on to more impor­ tant matters. Human Performance, 18(4):359-372. Boesch, E.E. (1991). Symbolic Action Theory and Cultural Psychology. Berlin, Germany: Springer-Verlag. Brandstädter, J. (1998). Action perspectives on human development. In W. Damon (Series Ed.) and R. Lerner (Vol. Ed.), Handbook of Child Psychology: Vol. 1. Theoretical Models of Human Development (pp. 1029-1144). New York: Wiley. Buss, D.M., and P.H. Hawley (Eds.). (2010). The Evolution of Personality and Individual Differ- ences. New York: Oxford University Press. Daneman, M., and P.A. Carpenter. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19(4):450-466. Dvorak-Bertscha, J.D., J.J. Curtin, T.J. Rubinstein, and J.P. Newman. (2009). Psychopathic traits moderate the interaction between cognitive and affective processing. Psycho- physiology, 46(5):913-921. Engle, R.W., S.W. Tuholski, J.E. Laughlin, and A.R.A. Conway. (1999). Working memory, short-term memory, and general fluid intelligence: A latent variable model approach. Journal of Experimental Psychology: General, 128(3):309-331. Furley, P.A., and D. Memmert. (2012). Working memory capacity as controlled attention in tactical decision making. Journal of Sport and Exercise Psychology, 34(3):322-344. Hawley, P.H. (1999). The ontogenesis of social dominance: A strategy-based evolutionary perspective. Developmental Review, 19(1):97-132. Hawley, P.H., and T.D. Little. (2002). Evolutionary and developmental perspectives on the agentic self. In D. Cervone and W. Mischel (Eds.), Advances in Personality Science (vol. 1, pp. 177-195). New York: Guilford Press. Hunter, J., and R.F. Hunter. (1984). Validity and utility of alternative predictors of job per­ formance. Psychological Bulletin, 96(1):72-98. Kane, M.J., and R.W. Engle. (2000). Working memory capacity, proactive interference, and divided attention: Limits on long-term memory retrieval. Journal of Experimental ­ sychology: Learning, Memory, and Cognition, 26(2):336-358. P Kane, M.J., M.K. Bleckley, A.R.A. Conway, and R.W. Engle. (2001). A controlled-­ ttention view a of working-memory capacity. Journal of Experimental Psychology: General, 130(2):169-183. Kane, M.J., L.H. Brown, J.C. McVay, I. Myin-Germeys, P.J. Silvia, and T.R. Kwapil. (2007). For whom the mind wanders, and when: An experience-sampling study of working memory and executive control in daily life. Psychological Science, 18(7):614-621. Kane, M.J., T.K. Kwapil, and P.J. Silvia. (unpublished). Executive Control and Schizotypy in the Laboratory and Daily Life. National Institute of Mental Health-funded R15 project (MH 093771-01). Kleider, H.M., D.J. Parrott, and T.Z. King. (2010). Shooting behaviour: How working memory and negative emotionality influence police officer shoot decisions. Applied Cognitive Psychology, 24(5):707-717. Kramer, M.D., C.J. Patrick, R.F. Krueger, and M. Gasperi. (2012). Delineating physiologic defensive reactivity in the domain of self-report: Phenotypic and etiologic structure of dispositional fear. Psychological Medicine, 42(6):1305-1320.

OCR for page 21
NEW CONSTRUCTS FOR ASSESSING INDIVIDUALS 49 Kristof-Brown, A.L., R.D. Zimmerman, and E.C. Johnson. (2005). Consequences of indi­ viduals’ fit at work: A meta-analysis of person-job, person-organization, person-group, and person-supervisor fit. Personnel Psychology, 58(2):281-342. Krueger, R.F., K.E. Markon, C.J. Patrick, S.D. Benning, and M. Kramer. (2007). Linking anti­ social behavior, substance use, and personality: An integrative quantitative model of the adult externalizing spectrum. Journal of Abnormal Psychology, 116(4):645-666.  Kyllonen, P.C., and R.E. Christal. (1990). Reasoning ability is (little more than) working- memory capacity?! Intelligence, 14(4):389-433. Lilienfeld, S.O., I.D. Waldman, K. Landfield, A.L. Watts, S. Rubenzer, and T.R. ­ aschingbauer. F (2012). Fearless dominance and the U.S. presidency: Implications of psychopathic per­ sonality traits for successful and unsuccessful political leadership. Journal of Personality and Social Psychology, 103(3):489-505. Little, T.D. (1998). Sociocultural influences on the development of children’s action-control beliefs. In J. Heckhausen and C.S. Dweck (Eds.), Motivation and Self-Regulation Across the Life Span (pp. 281-315). New York: Cambridge University Press. Little, T.D., and B. Wanner. (1997). The Multi-CAM: A Multidimensional Instrument to Assess Children’s Action-Control Motives, Beliefs, and Behaviors (Materialen aus der Bildungs­ forschung, Nr. 59, ISBN #3-87985-064-x). Berlin, Germany: Max Planck Institute for Human Development. Little, T.D., G. Oettingen, A. Stetsenko, and P.B. Baltes. (1995). Children’s action-control b ­ eliefs about school performance: How do American children compare with German and Russian children? Journal of Personality and Social Psychology, 69(4):686-700. Little, T.D., C.R. Snyder, and M. Wehmeyer. (2006). The agentic self: On the nature and origins of personal agency across the lifespan. In D.K. Mroczek and T.D. Little (Eds.), Handbook of Personality Development (pp. 61-79). Mahwah, NJ: Lawrence Erlbaum Associates. Low, K.S.D., M. Yoon, B.W. Roberts, and J. Rounds. (2005). The stability of vocational inter­ ests from early adolescence to middle adulthood: A quantitative review of longitudinal studies. Psychological Bulletin, 131(5):713-737. Lykken, D.T. (1995). The Antisocial Personalities. Hillsdale, NJ: Lawrence Erlbaum Associates. McVay, J.C., and M.J. Kane. (2009). Conducting the train of thought: Working memory capacity, goal neglect, and mind wandering in an executive-control task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(1):196-204. McVay, J.C., and M.J. Kane. (2012). Drifting from slow to “d’oh!”: Working memory ­capacity and mind wandering predict extreme reaction times and executive control errors. Jour- nal of Experimental Psychology: Learning, Memory, and Cognition, 38(3):525-549. McVay, J.C., M.J. Kane, and T.R. Kwapil. (2009). Tracking the train of thought from the laboratory into everyday life: An experience-sampling study of mind wandering across controlled and ecological contexts. Psychonomic Bulletin and Review, 16(5): 857-863. Miller, M.W., D.S. Vogt, S.L. Mozley, D.G. Kaloupek, and T.M. Keane. (2006). PTSD and substance-related problems: The mediating roles of disconstraint and negative emo­ tionality. Journal of Abnormal Psychology, 115(2):369-379. Miyake, A., and N.P. Friedman. (2012). The nature and organization of individual differences in executive functions: Four general conclusions. Current Directions in Psychological Science, 21(1):8-14. Multon, K.D., S.D. Brown, and R.W. Lent. (1991). Relation of self-efficacy beliefs to academic outcomes: A meta-analytic investigation. Journal of Counseling Psychology, 38(1):30-38. Nye, C.D., R. Su, J. Rounds, and F. Drasgow. (2012). Vocational interests and performance: A quantitative summary of over 60 years of research. Perspectives on Psychological ­ cience, S 7(4):384-403.

OCR for page 21
50 NEW DIRECTIONS IN ASSESSING PERFORMANCE POTENTIAL Oettingen, G., T.D. Little, U. Lindenberger, and P.B. Baltes. (1994). Causality, agency, and control beliefs in East versus West Berlin children: A natural experiment on the role of context. Journal of Personality and Social Psychology, 66(3):579-595. Piper, W. (1930). The Little Engine That Could. New York: Platt and Munk. Redick, T., Z. Shipstead, M. Meier, J. Evans, K. Hicks, N. Unsworth, M. Kane, Z. Hambrick, and R. Engle. (unpublished). Understanding the Role of Working Memory Capacity in Complex Task Performance to Improve Sailor and Marine Selection, Classification, and Training. Office of Naval Research project. Rounds, J.B. (1995). Vocational interests: Evaluation of structural hypotheses. In D. Lubinski and R.V. Dawis (Eds.), Assessing Individual Differences in Human Behavior: New Concepts, Methods, and Findings (pp. 177-232). Palo Alto, CA: Consulting Psychologists Press. Schraw, G., and S. Lehman. (2001). Situational interest: A review of the literature and direc­ tions for future research. Educational Psychology Review, 13(1):23-52. Strong, E.K., Jr. (1943). Vocational Interests of Men and Women. Palo Alto, CA: Stanford Uni­ versity Press. Su, R. (2012). The Power of Vocational Interests and Interest Congruence in Predicting Career Success. Dissertation, University of Illinois at Urbana-Champaign. Available: https:// www.ideals.illinois.edu/bitstream/handle/2142/34329/Su_Rong.pdf?sequence=1 [June 2013]. Verquer, M.L., T.A. Beehr, and S.H. Wagner. (2003). A meta-analysis of relations between person-organization fit and work attitudes. Journal of Vocational Behavior, 63(3):473-489. Young, S.E., N.P. Friedman, A. Miyake, E.G. Willcutt, R.P. Corley, B.C. Haberstick, and J.K. Hewitt. (2009). Behavioral disinhibition: Liability for externalizing spectrum disorders and its genetic and environmental relation to response inhibition across adolescence. Journal of Abnormal Psychology, 118(1):117-130.