4

Assessing Intrapersonal Skills

The third cluster of skills—intrapersonal skills—are talents or abilities that reside within the individual and aid him or her in problem solving. The previous workshop report that defined a set of 21st century skills (National Research Council, 2010) identified two broad skills that fall within this cluster:

Adaptability: The ability and willingness to cope with uncertain, new, and rapidly changing conditions on the job, including responding effectively to emergencies or crisis situations and learning new tasks, technologies, and procedures. Adaptability also includes handling work stress; adapting to different personalities, communication styles, and cultures; and physical adaptability to various indoor or outdoor work environments (Houston, 2007; Pulakos et al., 2000).

Self-management/self-development: The ability to work remotely, in virtual teams; to work autonomously; and to be self-motivating and self-monitoring. One aspect of self-management is the willingness and ability to acquire new information and skills related to work (Houston, 2007).

These kinds of skills operate across contexts, as Rick Hoyle, professor of psychology and neuroscience at Duke University, who presented findings from a paper about them and how they might be assessed, pointed out (Hoyle and Davisson, 2011).1 They are “transportable,” he explained,

________________

1See http://www7.national-academies.org/bota/21st_Century_Workshop_Hoyle_Paper. pdf [August 2011].



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 63
4 Assessing Intrapersonal Skills T he third cluster of skills—intrapersonal skills—are talents or abili- ties that reside within the individual and aid him or her in problem solving. The previous workshop report that defined a set of 21st century skills (National Research Council, 2010) identified two broad skills that fall within this cluster: Adaptability: The ability and willingness to cope with uncertain, new, and rapidly changing conditions on the job, including responding effectively to emergencies or crisis situations and learning new tasks, technologies, and procedures. Adaptability also includes handling work stress; adapting to different personalities, communication styles, and cul - tures; and physical adaptability to various indoor or outdoor work envi - ronments (Houston, 2007; Pulakos et al., 2000). Self-management/self-development: The ability to work remotely, in virtual teams; to work autonomously; and to be self-motivating and self- monitoring. One aspect of self-management is the willingness and ability to acquire new information and skills related to work (Houston, 2007). These kinds of skills operate across contexts, as Rick Hoyle, professor of psychology and neuroscience at Duke University, who presented find - ings from a paper about them and how they might be assessed, pointed out (Hoyle and Davisson, 2011).1 They are “transportable,” he explained, 1 See http://www7.national-academies.org/bota/21st_Century_Workshop_Hoyle_Paper. pdf [August 2011]. 63

OCR for page 63
64 ASSESSING 21ST CENTURY SKILLS automatically transferred from one context to the next so that the very same skills that serve a person well in the social arena, for example, serve the person well in health decisions and in schooling and academics. Fur- thermore, he added, these skills ultimately contribute to adaptive behav - ior and productivity in that they counteract undesired influences that may arise from within the person or from the environment. Intrapersonal skills support volitional behavior, which Hoyle defined as discretionary behavior aimed at accomplishing the goals an individual sets for himself or herself. Examples of intrapersonal skills include attributes such as plan- fulness, self-discipline, delay of gratification, the ability to deal with and overcome distractions, and the ability to adjust one’s strategy or approach as needed. In Hoyle’s view, the common thread among these attributes is a skill called self-regulation. Even though the field of psychology has studied self-regulation since the late 1960s, Hoyle said, disagreement about how to define it remains. To provide the audience with the broad spectrum of definitions, he pre - sented varying points of view that four prominent researchers have put forth: • “The capacity of individuals to guide themselves, in any way pos- sible, toward important goal states” (Fitzsimons and Bargh, 2004) • “The capacity to plan, guide, and monitor one’s behavior flexibly in the face of changing circumstances” (Brown, 1998) • “Self-generated thoughts, feelings, and actions that are planned and cyclically adapted to the attainment of personal goals” (Zimmerman, 2000) • “The process by which one monitors, directs attention, maintains, and modifies behaviors to approach a desirable goal” (Ilkowska and Engle, 2010) Hoyle identified some common threads among the definitions. They all recognize that people need to monitor their behavior and that they are doing this in the service of goal pursuit. In addition, they all acknowledge that flexibility is needed. Most importantly, they all involve affect. Hoyle emphasized that self-regulation does not just involve cognition but also involves feelings and emotions. Hoyle prefers the following definition: the processes by which people remain on course in their pursuit of the goals they have adopted. In some cases, such as a school setting, these goals may not be the student’s own, but they are put before students. The question is if they capable and ready to do the things that need to be done to pursue those goals and to move forward on them.

OCR for page 63
65 ASSESSING INTRAPERSONAL SKILLS WHY IS SELF-REGULATION IMPORTANT? Invariably the goals we adopt or are given to us are challenged in a number of ways. We may have counterproductive impulses, such as eat- ing dessert even though we have a goal to lose weight. We may encounter situational hurdles, obstacles that interfere with the ongoing pursuit of some goal. We may have competing goals, so that satisfying one goal detracts from accomplishing another. Thus, people must manage the con - flict between goals. And, in some cases, progress may be so slow that it is difficult to sustain motivation. Remaining on course toward goal pursuit requires a set of strategies that, collectively, constitute self-regulation. Not every good behavior involves self-regulation, Hoyle clarified; self-regulation is behavior over which the individual exercises some level of discretion. Self-regulation requires considerable cognitive energy and effort. If the individual is constantly self-regulating, it is impossible to sustain momentum toward accomplishing a goal. It is most effective for the individual to move many behaviors outside the realm of the processes that require self-regulation. For example, some behaviors are contingent on cues in the environ- ment and are simply habits. The individual performs a habit when he or she links a behavior with some cue in the environment, Hoyle explained, and thus can accomplish the behavior without having to draw on self- regulation. Also, many behaviors are attributable to impulse. Impulses may be productive or they may detract from goal pursuit in some way, but they occur without self-regulation. Other behaviors are strongly influ- enced by normative pressure. This is frequently seen among adolescents, who experience a critical push/pull between the normative environment and their own individual goals. Finally, there are behaviors that are deter- mined by social, political, or religious systems within which people live. In some cases those systems serve the role of regulating behavior, thereby circumventing the need for self-regulation. Trends in society demonstrate some of the consequences that result from a lack of self-regulation. As Hoyle put it, “I don’t think we need much convincing that a lot of what we see around us seems to involve a failure of self-control at a fairly large level.” For example, Hoyle noted, U.S. consum- ers have not exerted much self-regulation when it comes to debt levels. In the late 1970s, consumer revolving credit debt was $54 billion. By the end of the 1990s, it rose to more than $600 billion, and now approaches $1 trillion.2 Likewise, Hoyle observed, obesity rates are at crisis levels. In 1990, no state had an obesity prevalence rate above 15 percent. By 2007, only one state had an obesity prevalence rate less than 20 percent, and 30 states had a preva- 2 See http://www.federalreserve.gov/releases/g19/hist/cc_hist_sa.html [July 2011].

OCR for page 63
66 ASSESSING 21ST CENTURY SKILLS lence rate of 25 percent or more.3 In addition, a notable number of deaths can be attributed to failure of self-regulation. According to a report by the Centers for Disease Control and Prevention (Anderson, 2002), 33 percent of deaths were attributable to obesity, physical inactivity, and tobacco use. In addition, 8 percent of deaths were attributable to a cluster of behavioral causes including alcohol consumption, motor vehicle crashes, incidents involving firearms, sexual behaviors, and use of illicit drugs. Furthermore, in Hoyle’s view, the current economic crisis can be con- sidered as a failure of self-regulation on a grand scale. Systemic, circum - stantial, and societal issues all contributed to the crisis, but the excessive borrowing and lending and high-risk investments made with little or no concern for potential long-term consequences are all hallmarks of a lack of self-regulation. Hoyle emphasized that these examples all provide evi - dence of the importance of equipping children to be better self-regulated citizens as they approach adulthood. Hoyle also noted that considerable evidence in the literature underscores the value and importance of self- regulation. He focused on three studies. One longitudinal study, conducted by Walter Mischel, began in the late 1950s and focused on delay of gratification (Mischel, 1958; Mischel et al., in press). Mischel used a variety of paradigms to study delay of gratification, and Hoyle described one that involved a set of preschool - ers. The children were presented with an object they desired (e.g., a piece of candy or a marshmallow) but were told that they must wait until the experimenter returned to the room before they could have it. The experi - menter left the room, closed the door, and intentionally did not return. Mischel collected data on how long each child waited before reaching for the object. After 10 to 12 years, Mischel contacted the parents of the par- ticipants and gathered information about their academic and social com- petence. He found that adolescent behavior was significantly predicted by the duration of the self-imposed delay in gratification. That is, the longer the preschooler was able to delay gratification, the better he or she fared as an adolescent in terms of a variety of self-regulation characteristics, such as attentiveness, planfulness, and reasoning ability. A second longitudinal study by Caspi, Moffitt, and others (Caspi et al., 1997) with youngsters in Dunedin, New Zealand, is currently under- way. The researchers are studying an entire birth cohort, collecting data every 2 to 3 years. At age 3, the children’s temperament was evaluated, with some classified as “under-controlled.” At age 18, the children who fell into the under-controlled classification rated high on a number of qualities that indicate poor self-regulation, including impulsivity and danger-seeking behavior, aggression, and interpersonal alienation. At 3 See http://www.cdc.gov/obesity/data/trends.html [July 2011].

OCR for page 63
67 ASSESSING INTRAPERSONAL SKILLS age 21, these individuals were more likely to be engaged in activities that show evidence of a failure to control their behavior, such as alcohol dependence, dangerous driving, violent behavior, and having unsafe sex. The likelihood of engaging in these behaviors was double that of the other children in the birth cohort. Third, Hoyle cited work by economist Jim Heckman, who argues that noncognitive abilities are equally, if not more, important than traditional cognitive abilities when it comes to predicting educational and socioeco- nomic outcomes. The noncognitive attributes that Heckman refers to— attentiveness, persistence, impulse control, and social competence—are all evidence of self-regulation from a psychological perspective. Heckman’s work shows that a gap between the disadvantaged and the advantaged begins to emerge very early. The point he makes is that there is a window of opportunity during which we can invest in those children. They can be taught to self-regulate, which Heckman finds will eventually result in significant dividends in terms of economic productivity, life success, and the like. Heckman (2006) reported findings from a study of children who participated in the Perry Preschool Program, which included a significant component of training of self-regulatory skills. He found students who participated in this program were less likely to drop out of school, spend time in jail, smoke, and participate in other self-destructive behaviors. In terms of economic productivity, the Perry Preschool Program partici - pants were 15 to 17 percent higher than children who did not participate. Heckman argues that from an economic perspective there was a nine-fold payoff in what it costs to operate the Perry Preschool Program versus the payoff in economic productivity down the line. DEFINING SELF-REGULATION Although there has been considerable work on the topic of self-regu- lation in the field—with 114 chapters in edited volumes between 1998 and 20104 and about 120 published articles each year—Hoyle said the field has no current consensus regarding a single definition of self-regulation. His review of the body of work revealed a definition is sometimes, but not always, provided. He finds no evidence of even minimal acceptance of a common definition, and even the same authors sometimes use differ- ent definitions. Furthermore, he thinks self-regulation has been applied 4 Such as Motivation and Self-Regulation Across the Life Span published in 1998; the Handbook of Self-Regulation published in 2000; Self-Regulation of Health and Illness Behaviour published in 2003; the Handbook of Self-Regulation: Research, Theory, and Applications published in 2004; Self-Regulation in Health Behavior published in 2006; and the Handbook of Personality and Self- Regulation published in 2010.

OCR for page 63
68 ASSESSING 21ST CENTURY SKILLS far too broadly and, in many cases, inappropriately. Hoyle believes the current state of the conceptualization of self-regulation is the primary obstacle to producing assessments of it. Hoyle laid out a conceptualization of self-regulation, which he empha- sized was not really a model or a theory, but a framework that might help move forward in developing assessments. This conceptualization is pre - sented in Figure 4-1. Understanding these components of self-regulation helps to provide a basis for defining constructs that might be assessed. Hoyle explained each of the components. In the leftmost column (“Foundations“) are a series of variables or traits the individual “brings to the table.” These include (1) executive function, (2) temperament, and (3) personality characteristics. Hoyle added it is not clear whether these foundations are susceptible to change, but they are the “raw materials” that self-regulation draws upon. Executive function is a set of cognitive processes and propensities that originate early in life (Goldman-Rakic, 1987; for a review, see Best and Miller, 2010). Three core functions underlie the processes involved in most acts of self-regulation (Miyake et al., 2000). Inhibition involves stopping ongoing thoughts and actions either when prompted by an external sig- nal or upon determining that continuation would lead to an error (Logan and Cowan, 1984). Working memory involves keeping information active in primary memory while searching and retrieving information stored in secondary memory (Unsworth and Engle, 2007). Because keeping relevant information active while ignoring or suppressing competing information that is not relevant involves inhibition, inhibition and working memory FIGURE 4-1 A conceptualization of self-regulation. Figure 4-1.eps SOURCE: Adapted from Rick Hoyle’s presentation. Used with permission. bitmap

OCR for page 63
69 ASSESSING INTRAPERSONAL SKILLS are related. Complex tasks require the coordination of information rel- evant to multiple task components, requiring working memory to be flexible and controlled. Finally, shifting involves moving back and forth between mental states, rules, or tasks (Miyake et al., 2000). The importance of these basic capacities is evident in a cornerstone of self-regulation, the delay of gratification, which requires the inhibition of an impulse to act in response to a temptation in the immediate environment in favor of one or more longer-term goals or priorities (Mischel et al., in press). Variability in executive function is expressed as individual dif - ferences in temperament, which Hoyle said is defined as individual differences in emotional and motor reactivity and in the attentional capacities that support self-regulation (Rothbart and Hwang, 2002, p. 113). One of the most important capacities is referred to as effortful control, which Hoyle explained is apparent when the child “is able to say no to that thing in front of them in service of some other thing that needs attention at that moment.” A related dimension of temperament is reactive control, which Hoyle described as the “relatively involuntary influence of approach and avoidance motives.” Extreme forms of reac - tive control can result in overcontrolled reactivity, such as shyness, or undercontrolled reactivity, such as impulsivity. Hoyle defines personality as tendencies of thought, feeling, and action that are moderately stable across the lifespan (Roberts and DelVecchio, 2000), and he noted they can be separated into higher-order dimensions and lower-order dimensions. Research has shown that there are between three and seven higher-order dimensions (depending on the model and classification strategy) into which all personality traits fall. The dimension most relevant for self-regulation is conscientiousness, which generally concerns the ways people manage their behavior. Individuals who are high on conscientiousness tend to be confident, disciplined, orderly, and planful (Costa and McCrae, 1992). A large number of narrower (lower-order) personality constructs also tend to facilitate or impede self-regulation. One of the most impor- tant is impulsivity, which Hoyle said might be viewed as the absence of self-regulation. Other lower-order personality constructs are relevant to self-regulation—those that concern self-regulatory style—and how (rather than whether) self-regulation is accomplished. They are foundational in their provision of the basic capacities and tendencies on which the pro- cesses involved directly in self-regulation draw. In the middle column of Figure 4-1, Hoyle provided a list of the processes individuals go through as they try to accomplish a goal, although he cautioned that there is no agreement in his field on the exact nature of these processes. He noted the list helps to understand what

OCR for page 63
70 ASSESSING 21ST CENTURY SKILLS would be involved in an assessment of how effective an individual is at self-regulation. The process generally begins with forethought, when the individual receives information, evaluates it, considers options, sets goals, and for- mulates a plan to achieve these goals. This is followed by performance, in which the individual implements the plan. From a self-regulation per- spective, performance involves exercising self-control for the purpose of engaging in goal-relevant behaviors while avoiding behaviors irrelevant to or in conflict with the goal. Hoyle said a critical aspect of performance is self-observation or self-reflection, when the individual assesses the effectiveness of his or her performance and re-engages the process for subsequent attempts at goal pursuit. This model assumes a cyclical pro - cess whereby the individual repeatedly moves from forethought to per- formance to self-reflection, realizing progress toward the goal with each successive cycle. The rightmost column of Figure 4-1 is labeled “Consequences,” which Hoyle maintains is probably the quickest approach to getting at a person’s skill level at self-regulation. What observable evidence is there that an individual is skilled or unskilled at self-regulation? He classified conse - quences into three categories. One type of consequence is normative: that is, certain behaviors are evidence of a well-regulated individual regardless of the context or the particular population. Examples include academic success as evidenced by regularly completing assignments as instructed on schedule; social success in the form of routine relationship maintenance behaviors; and good health as evidenced by proper diet and exercise and general avoid - ance of health-risk behaviors. Another type of consequence is domain-specific, such as self-regula - tion in the context of health behavior. For instance, hypertension patients often are prescribed a regimen that includes control of diet and regular intake of medications. Certain forms of psychotherapy might prescribe goals and behavioral evidence of their pursuit. In such instances, self- regulation is necessary and evidence of successful self-regulation is con - crete and specific. The final category is the idiosyncratic goals that each person decides on his or her own to pursue. APPROACHES FOR ASSESSING SELF-REGULATION Hoyle described a number of approaches for assessing self-regulation. One frequently used approach is self-report. In the typical use of this strategy, the respondent is given a set of statements and asked to select one of the provided response options to indicate extent of agreement or

OCR for page 63
71 ASSESSING INTRAPERSONAL SKILLS disagreement with the statement or the degree to which the statement accurately describes him or her. There are advantages and disadvantages to this strategy, and Hoyle described several. It is often the least expensive approach in terms of materials as well as time and space requirements. There is also an implicit assumption that an individual is uniquely posi - tioned to report on his or her standing on statements about the constructs and may well be the best source for the information. On the other hand, Hoyle noted, individuals are biased in both how they think about their own behavior and what they think is the task before them when they are responding to questionnaire items. There is evidence that people often do not have access to higher-order processes and therefore are unable to report about them accurately (Nisbett and Wilson, 1977). Hoyle said that there is also an age issue in that young children may lack the cognitive skills and reading ability to understand the statements they are asked to rate and the use of rating scales to do so. Another approach is informant reports, which, Hoyle said, share many of the qualities of self-reports and address some of the limitations of the self-report strategy. One advantage of informant reports is that they eliminate the self-referential biases that may undermine the valid - ity of self-reports. That is, Hoyle explained, well-trained informants who observe the target across time and situations may be able to infer and accurately report on characteristics of the target that the target is unable to accurately report about himself or herself. Another advantage Hoyle cited is that the informant report strategy allows for assessment of preverbal children, as well as of individuals who for other reasons may be unable to read and understand the statements on which they are to be rated. A clear drawback of the strategy, Hoyle noted, is the lim - ited access most informants have to the individuals they are rating. For example, teachers only observe children in academic settings, parents see them primarily in the home, and peers are privy to behavior only in selected settings. Further, Hoyle stated, it may be difficult to extract information about specific skills and abilities from complex behavior sequences. That is, sometimes it is difficult to know, even after extensive observation, what is actually going on in the head of the person one is observing. A third approach is behavioral task performances, which, Hoyle said, are designed so that they require only the capacity or skill of interest. Hoyle noted that these tasks are most often used to assess constructs in the foundations (see Figure 4-1), generally those capacities that consti- tute executive function. Speed and efficiency in completing these tasks is assumed to measure strength of the capacity being assessed. According to Hoyle, the tasks are tailored to the age group being assessed, and they generally do not require verbal skills or awareness by the individual of

OCR for page 63
72 ASSESSING 21ST CENTURY SKILLS his/her use of the capacity. The tasks are typically scored in terms of objective characteristics of performance (e.g., time to completion, number of mistakes). The positive features of assessments based on behavioral task performance are offset somewhat by two shortcomings, Hoyle cau - tioned. First, behavioral tasks tend to be tailored to the age group being assessed, which interferes with the ability to track performance over time. A second shortcoming concerns the purity of capacities assessed by the tasks. Complex tasks likely require multiple, interdependent capaci- ties, thereby producing scores that cannot be used to pinpoint standing on specific capacities (Garon, Bryson, and Smith, 2008). They have the advantages of not requiring verbal skills, they do not require the person to report on higher order mental activity, and the scores tend to be objec- tive (e.g., time to completion, number of mistakes). The measures tend to be things like Mischel’s delay of gratification, which was the amount of time before the individual reached for the tempting object on the table. The disadvantages of this approach, Hoyle said, are that the tasks must be tailored to the age of the respondent and they often tap more than one skill or ability. Hoyle described some examples of behavioral tasks performances intended to measure each of the foundational skills (see Figure 4-1). One task, referred to as the “stop signal measure” (see Box 4-1), is designed to measure executive function. In fairly rapid succession, the subject is presented with a series of cards. When the greater sign appears on the card, the subject is to press the right key, and when the lesser sign appears, the subject is to press the left key. At variable intervals, an audible sound occurs at which point the subject is not to press any key. The assessment measures how well they are able to inhibit and not press the key. Another example, the star counting task, measures working memory. As shown in Box 4-2, the task begins with the number 15. In this case, when the subject reaches a plus sign, he/she is to count in the forward direction (16, 17, 18, etc.); when the subject reaches a minus sign, he/she is to change and count downward (18, 17, 16, etc.). The task is to get the right answer within a minute. A series of these is presented, and then the rules change so that a plus sign indicates to count in the backward direc- tion and a minus sign indicates to count in the forward direction. This task measures the ability to change rules and hold the new rule in memory while overriding the old one. Hoyle also showed examples of assessments intended to measure self-regulation through process and consequences (see Figure 4-1). The first is a self-report instrument on which the candidate rates him/herself on statements about processes, such as the ones that appear below: • “I usually keep track of my progress toward my goals.” • “I have personal standards, and try to live up to them.”

OCR for page 63
73 ASSESSING INTRAPERSONAL SKILLS BOX 4-1 Example of a Stop Signal Task Designed to Measure Executive Function SOURCE: Adapted from Rick Hoyle’s presentation. From Chamberlain, S.R. (2006). Neuro- chemical modulation of response inhibition and probabilistic learning in humans. Science, 311, 861. Reprinted with permission of American Association for the Advancement of Science. • “I am willing to consider other ways of doing things.” • “I have sought out advice or information about changing.” • “Once I have a goal, I can usually plan how to reach it.” • “I get easily distracted from my plans.” (reverse-scored) • “I don’t seem to learn from my mistakes.” (reverse-scored) The second is a self-report instrument that includes measures of behavior indicators of conscientiousness. The assumption is that the rou - tine production of these behaviors is a sign of an individual who is either capable or not capable at self-regulation. The test taker rates him/herself on statements such as those shown below: 4-21 • “Play sick to avoid doing something” (avoid work) • “Make a grocery list before going to the store” (organization) • “Buy something on the spur of the moment” (impulsivity) • “Clean the inside of the microwave oven” (cleanliness) • “Work or study on a Friday or Saturday evening” (industriousness) • “Clean up right after company leaves” (appearance) • “Allow extra time for getting lost when going to new places” (punctuality)

OCR for page 63
82 ASSESSING 21ST CENTURY SKILLS The syndrome scales were derived via factor analysis and were normed on large population-based and clinical samples. Reliabilities, based on test-retest estimates and coefficient alpha, are in the .90 range. The assessment takes about 15 minutes to complete. It is also compat - ible with the Diagnostic and Statistical Manual’s definition of conduct disorder and antisocial behavior disorder, which allows people to talk across disciplines. Other psychometric information about the assessment is available on the ASEBA website (see footnote 7). For young children, parents and teachers complete the checklist. For adolescents and adults, the checklist is self-reported, although there is usually an attempt to gather information from another informant (parents and teachers for adolescents; spouse or significant other for adults). Odgers has been involved in the longitudinal study in Dunedin, New Zealand, that Hoyle described. The study has followed 1,000 individuals born in 1972 and 1973, and the researchers have just finished the age-38 assessment. Assessments were done at birth and every couple of years thereafter, thus providing a longitudinal perspective of when these skills emerge (or when problems emerge) and how they relate to other skills and deficits. The study has yielded considerable information about the relation- ships between these skills and life outcomes. Odgers said that they are find- ing that conduct disorder, particularly persistent conduct disorder across childhood, is one of the most accurate signals of future problems across a wide array of domains, including mental health, physical health, economic functioning, and job prospects. Odgers presented the graph shown in Figure 4-2 that displays the inci- dence of conduct problems for the males in the sample, following them from ages 7 to 26. The researchers identified four patterns of behavior: (1) individuals who were consistently low in conduct problems (solid line); (2) individuals who exhibited conduct problems in childhood, but the problems diminished over time (line with triangles); (3) individuals who began exhibiting conduct problems during adolescent years (line with circles); and (4) individuals who persistently exhibited conduct disorders from childhood on into adulthood (line with squares). The researchers have compared outcomes for these four groups. Odgers said the first finding from this analysis is that antisocial behav- ior in childhood does not necessarily signal poor outcomes in adulthood. Some children may exhibit conduct problems early on, but these problems are dealt with or as Odgers put it “socialized out.” Through the influences of family, school, peers, and other factors, these children develop effec- tive self-regulation skills, and the conduct problems diminish over time. However, this does not happen for all children with early-onset conduct problems, and individuals whose problems persist into adulthood experi- ence difficulties in a number of areas of life.

OCR for page 63
83 ASSESSING INTRAPERSONAL SKILLS (10.5%) Life-course persistent 6 (19.6%) Adolescent-onset (24.3%) Childhood-limited 5 (45.6%) Low Conduct Problems Scale 3 2 1 0 7 9 11 13 15 18 21 26 Age FIGURE 4-2 Incidence of conduct problems between ages 7 and 26 for longitudi- nal sample of individuals in Dunedin, New Zealand. Figure 4-2.eps SOURCE: Odgers et al. (2007). Reprinted from Odgers, C.L., Milne, B.J., et al. bitmap w some vector type (2007). Predicting prognosis for the conduct-problem boy: Can family history help? Journal of American Academy of Child and Adolescent Psychiatry, 46 (10), 1240- 1249, Copyright 2007, with permission from Elsevier. And from Archives of General Psychiatry, 64, 476-484, Copyright 2007, American Medical Association, All rights reserved. For example, Figures 4-3 and 4-4 use effect sizes to compare health outcomes for males in the different groups. Figure 4-3 compares health outcomes for males with life-course-persistent conduct disorders versus those who scored low on conduct disorders. Figure 4-4 compares health outcomes for males with childhood-limited conduct disorders ver- sus those who scored low on conduct disorders. The figures show the health outcomes for males with childhood-limited conduct disorders are quite similar to the health outcomes for individuals who scored low in conduct problems. On the other hand, the males with life-course persis -

OCR for page 63
84 ASSESSING 21ST CENTURY SKILLS VIOLENCE TOWARD OTHERS Official convictions (lifetime) Self-report violence Controlling/stalking Partner abuse MENTAL HEALTH DIAGNOSIS Anxiety Major depressive disorder Cannabis dependency Other drug dependency Alcohol dependency Post-traumatic stress disorder PHYSICAL HEALTH PROBLEMS Cardiovascular risk C-reactive protein Type 2 herpes Smoker Nicotine dependency Lung functioning Chronic bronchitis Gum disease Decayed tooth surfaces Serious injury -0.5 0 0.5 1 1.5 2 Effect-size FIGURE 4-3 Health outcomes for males with life-course persistent conduct disor- ders compared to those who scored low4-3.eps disorders. Figure on conduct SOURCE: Data from Odgers et al. (2008). Used with permission. tent problems tended to be violent toward others and have convictions for this activity. They tended to suffer from anxiety and depression; were more likely to be dependent on alcohol, drugs, and tobacco; and had a greater incidence of health issues associated with these activities. More - over, by age 32, 59 percent of this group had no educational qualifications8 as compared to an average of about 7 percent in the population at large. Only 24 percent of the males with childhood-limited conduct disorders had no educational qualifications, which Odgers noted was higher than average but half that for the males with life-course persistent conduct disorders. 8 Forthe study, “No Educational Qualifications” was defined as ending secondary education prior to receiving qualifications (i.e., a diploma) and not having pursued further education.

OCR for page 63
85 ASSESSING INTRAPERSONAL SKILLS VIOLENCE TOWARD OTHERS Official convictions (lifetime) Self-report violence Controlling/stalking Partner abuse MENTAL HEALTH DIAGNOSIS Anxiety Major depressive disorder Cannabis dependency Other drug dependency Alcohol dependency Post-traumatic stress disorder PHYSICAL HEALTH PROBLEMS Cardiovascular risk C-reactive protein Type 2 herpes Smoker Nicotine dependency Lung functioning Chronic bronchitis Gum disease Decayed tooth surfaces Serious Injury -0.5 0 0.5 1 1.5 2 Effect-size FIGURE 4-4 Health outcomes for males with childhood-limited conduct disorders compared to those who scored low on conduct disorders. SOURCE: Data from Odgers et al. (2008). Used with permission. Odgers closed by highlighting some new issues being pursued in her field. Bullying is a topic being intensely explored, including bullying Figure 4-4.eps in school and in the workplace, as well as cyber-bullying in all contexts. Assessment strategies are emerging, which are focusing on the traits of being callous and unemotional as a subtype of antisocial behavior in which the person lacks empathy and the ability to read and relate to others. These traits are being considered as a precursor to psychopathy. Odgers noted that children who have both antisocial behavior and this lack of empathy seem to have particularly poor outcomes. There are con - siderations to adding this characteristic to the conduct disorder diagnosis to help improve prediction of outcomes. Odgers said that the field is quickly realizing the importance of col- lecting family history information about children, much in the way that

OCR for page 63
86 ASSESSING 21ST CENTURY SKILLS Performance Phase Self-Control Self-Instruction Imagery Attention Focusing Task Strategies Self-Observation Self-Recording Metacognitive Monitoring Forethought Phase Self-Reflection Phase Task Analysis Self-Judgment Goal Setting Self-Evaluation Strategic Planning Causal Attributions Self-Motivational Beliefs Self-Reaction Self-Efficacy Self-Satisfaction/Affect Outcome Expectations Adaptive Inferences Intrinsic Interest Goal Orientation FIGURE 4-5 Three-phase model of self-regulated thought and action. Figure 4-5.eps SOURCE: Adapted from Zimmerman (2000). Used with permission. it is done by medicine. Knowing about the parents’ levels of antisocial behavior can help considerably in the diagnosis and prediction of long- term outcomes. Microanalysis of Self-Regulated Learning A self-regulated learner, Tim Cleary explained,9 is an individual who • sets goals and develops/uses strategic plans; • is highly self-motivated and proactive; • engages in forms of self-control; • monitors strategies, performance, and cognition; and • frequently participates in self-reflection and analysis. Cleary presented a three-phase model of self-regulated thought and action, as shown in Figure 4-5, which was developed by Zimmerman (2000) and referred to as a Cyclical Feedback Loop. The three phases of the model are forethought, performance, and self-reflection. The idea is that an individual approaches a task by considering what is involved, what it 9 Cleary’s presentation is available at http://www7.national-academies.org/bota/21st_ Century_Workshop_Cleary.pdf [August 2011].

OCR for page 63
87 ASSESSING INTRAPERSONAL SKILLS would take to complete the task, and how he or she should approach it. At the same time, the individual’s approach to the task is influenced by how motivated he or she is to do the task, how important or valuable it is, and how confident he or she feels about successfully completing it. These ideas and thoughts impact the person’s performance. During the perfor- mance phase, the individual uses self-regulation in order to complete the task. That is, he or she uses self-control strategies to stay on task and to learn what is being taught; he or she uses self-observation strategies to remain motivated and monitor learning. After a performance—typically after the individual receives some outcome such as a test grade, a quiz grade, or feedback on homework—he or she engages in reflection. At this phase, the individual evaluates the extent to which the goal has been reached, the factors that interfered with or helped with goal attain- ment, and considers his or her reaction to the performance (good or bad). Reflection is hypothesized to have an impact on subsequent attempts or subsequent strategies and modification of goals before the next learning attempt. Cleary noted this model forms the basis for microanalysis, which essentially focuses on diagnosis or the assessment of self-regulated learn - ing and diagnosis of problems. Cleary distinguished between two approaches toward measuring self-regulated learning based on work by Winne and Perry (2000): apti - tude measures and event measures. The differences are in part related to the conceptualization of the construct. Aptitude measures, Cleary explained, are assessment tools that target self-regulated learning as a relatively global and enduring attribute of a person that predicts future behavior. They typically include self-report scales that rely on retrospec - tive accounts of student behaviors and thoughts in terms of frequency, typicality, and usefulness. They generally capture the characteristics of self-regulated learning but they do so in a decontextualized manner. Some examples of aptitude scales are the (1) Motivated Strategies for Learning Questionnaire, (2) Learning and Study Strategies Inventory, (3) School Motivation and Learning Strategy Inventory, and (4) Self-Regulation Strat- egy Inventory. He highlighted two potential problems with this approach to measuring self-regulated learning. First, there are validity issues that relate to context-specificity. Research has shown that students’ self-reports of self-regulated learning behaviors vary across different content areas as well as across tasks within a course. Second, student self-reports are often not consistent with their actual behaviors (Hadwin et al., 2001; Winne and Jamieson-Noel, 2002). Event measures are assessment tools that target self-regulated learn - ing as an event, behavior, or cognition that may vary across contexts and tasks. They involve direct assessment of self-regulatory processes as they occur in real time and in authentic contexts (as opposed to self-reports

OCR for page 63
88 ASSESSING 21ST CENTURY SKILLS about past events or behaviors). In Cleary’s view, these measures are well equipped to capture the process of self-regulated learning. There are four approaches to obtaining event measures: (1) direct observations of students’ actual behaviors in an authentic environment; (2) trace mea - sures, which are overt indicators of student cognition created during task engagement (such as underlining or highlighting text while reading); (3) personal diaries in which students record their study behaviors at home or the types of thoughts they had and actions they took when performing specific tasks; and (4) verbal reports or “think-aloud protocols,” which are records of students’ thought as they complete authentic activities. Self-regulated learning microanalysis is an event measure that uses a structured interview approach to measure students’ beliefs, attitudes, and cognitive regulatory processes before, during, and after some task or activity. There are essentially four steps to the microanalysis approach that Cleary has studied. The first is to select a task with a clear beginning, middle, and end, such as studying for an exam or writing an essay. The second step is to identify the cyclical phase process that is of interest (see Figure 4-5), and the third step is to develop context-specific assessment questions to target the specific phase and process. Finally, and the most important element to Cleary’s approach, is to link the three-phase cycle processes to temporal dimensions of the task: that is, to identify the ques - tions to ask in the forethought phase, the performance phase, and the self- reflection phase. Cleary said it is the matching of the questions and the task in temporal terms that is the most important aspect of this approach. Cleary and his colleagues have developed a bank of questions that can be adapted to a variety of contexts and tasks. They have administered these questions to school-aged and college samples in order to gather data on their reliability and validity. The reliability estimates, which are coef - ficient Alpha estimates for metric variables and inter-rater agreement for categorical variables tend to run in the .80 to .90 range. Cleary said they have developed coding manuals and scoring rubrics for training the rat - ers, which helps to produce these high reliability coefficients. In terms of validity, all of the questions are derived from opera- tional definitions of theoretical constructs from social cognitive theory and expert consensus, which Cleary noted helps to provide evidence of content validity. The researchers have also collected evidence on the dif - ferential and predictive validity of self-regulated learning microanalysis. In one recent study with college students, the authors examined the extent to which the microanalytic self-regulation questions accounted for unique variance in student course grades over and above that accounted for by the most commonly used self-report measure of self-regulation, the Motivated Strategies Learning Questionnaire (MSLQ; Cleary et al.,

OCR for page 63
89 ASSESSING INTRAPERSONAL SKILLS 2010). The analyses indicated that the microanalytic questions Cleary and colleagues developed, which included “attribution” (the reasons why students thought they had received the grade) and “adaptive differences” (the ways that they thought they should do differently), accounted for approximately 30 percent of the variance in final course grades over and above that accounted for by scores on the MSLQ along with several back- ground variables.10 Cleary and his colleagues have also conducted differential validity studies in the context of motor tasks and physical activities that dem - onstrate that goal-setting, strategic planning, attributions, and adaptive inferences reliably differentiate low and high achievers (Cleary and Zimmerman, 2001; Cleary, Zimmerman, and Keating, 2006; Kitsantas and Zimmerman, 2002). Different groups of students who had different levels of achievement (novices or experts) showed distinct profiles of regulatory processes. Cleary closed by stressing that attribution and adaptive differences play an important role in how engaged students are in their studies and the extent to which they have effective strategies to identify their weak - nesses and improve their performance. Assessing Emotional Intelligence Gerald Matthews began by cautioning the audience that the field of psychology is still in its infancy in terms of defining and assessing emotional intelligence.11 On one hand, no one would want to be referred to as low on emotional intelligence. As he put it, “Saying that somebody has low emotional intelligence is now a pretty standard insult in various public domains.” On the other hand, research on emotional intelligence has not yet yielded a single conception of what it entails or how best to assess it. Thus, he advised, he would provide a “wide-angle” view of the state of the field, but he said there is no basis for coming to clear-cut conclusions about the construct. In its broadest sense, Matthews explained, emotional intelligence includes abilities, competencies, and skills in perceiving, understanding, and managing emotion; however, there are a multitude of conceptualiza - tions of the construct. One conception considers it as a set of abilities for 10 The R square (variance explained) for the regression equation when background variables and MSLQ scores were included was .082. When information on responses to the attributions and adaptive inferences questions were added to the model, the R square increased to .373. This .291 change in the R square value was statistically significant at p < .000. 11 Matthews’ presentation is available at http://www7.nationalacademies.org/bota/21st_ Century_Workshop_Matthews.pdf [August 2011].

OCR for page 63
90 ASSESSING 21ST CENTURY SKILLS processing emotional stimuli (Mayer et al., 2000) and treats the construct as a standard intelligence, having the kind of properties that other forms of intelligence and ability have. Another conception views emotional intelligence as part of the personality domain (Petrides and Furnham, 2003). In both cases, the assumption is that there is a general emotional intelligence factor that can be broken down into a number of more distinct competencies or skills. Matthews thinks that neither conceptualization is useful. In his view, “emotional intelligence” is too vague a term to be of much use in either theory or practice (Roberts et al., 2007). He thinks it has become an “umbrella term” for a variety of separate attitudes, competencies, and skills that are only loosely interrelated, including basic temperament (e.g., positive and negative emotionality), information processing (e.g., emotion recognition), emotion-regulation (e.g., mood repair), and miscellaneous kinds of implicit and explicit skills. Matthews talked about two commonly used strategies for assessing emotional intelligence—trait questionnaires and ability tests—though he cautioned that each strategy has drawbacks. He said many trait ques - tionnaires are available and most are personality-like scales that provide scores for various emotional intelligence traits. He noted these are self- report assessments, which he thinks raises a paradox that undermines their validity. As Matthews put it, “If having good self-awareness of your emotional functioning is central to emotional intelligence, then if you lack emotional intelligence, how can your questionnaire responses be very meaningful?” The Mayer-Salovey-Caruso Emotional Intelligence Test (the MSCEIT) is an example of an ability test for measuring emotional intelligence. The MSCEIT assesses the respondent’s ability to perceive, use, understand, and regulate emotions. The assessment uses scenarios drawn from every - day life situations to measure how well people perform tasks and solve emotional problems. For instance, the assessment includes the “Faces Subtest,” in which the test taker is presented with the face of a person showing an emotion, and the test taker rates the extent to which certain emotions are being expressed. Matthews showed an example of the face of a woman smiling. The test taker is asked to rate on a 5-point scale of “definitely not present” to “definitely present” the extent to which the face shows anger, disgust, sadness, happiness, fear, surprise, etc. Matthews said one issue with the MSCEIT and other assessments like it is determining the “correct” response to an item. For the MSCEIT, the correct answers are determined through use of an expert panel and through collecting data from a normative sample. In Matthews’ view, nei- ther approach is ideal, although he said that the assessment shows modest

OCR for page 63
91 ASSESSING INTRAPERSONAL SKILLS correlations (.1 to .3) with a variety of criteria including life satisfaction, social skills and relationships, and coping. Matthews and his colleagues Richard Roberts and others at ETS have been working on another assessment strategy that relies on situational judgment tests. The researchers are exploring the use of both text-based and video-based scenarios designed to evaluate how well individuals can judge the emotions of a situation. An example of a text-based scenario follows: Clayton has been overseas for a long time and returns to visit his family. So much has changed that Clayton feels left out. What action would be the most effective for Clayton? In the video-based format, a clip of an emotive situation is shown, and the test taker is presented with several response options. Matthews presented an example in which a person in a work situation is upset because her office is being moved around, and this has disrupted her work activities. The test taker is presented with four possible responses that the boss might make to address the employee’s complaint. In one response, the boss becomes angry, tells her that the move is important for the firm’s functioning, and that she should simply put up with it. In another, the boss is more empathetic with the employee, recognizes that the employee has some grounds for being upset, and explains the ratio - nale behind the office move. The test taker is instructed to choose the best response. Matthews said that the work is in its early stages, but there seems to be some evidence that the results are predictive of high school GPA, well-being, and social support, even controlling for other factors. Matthews closed by restating that emotional intelligence remains a nebulous and ill-defined construct. The field has not yet come to consen- sus on a definition or conceptualization of the construct, and findings from research examining its malleability—that is, the extent to which is it trainable—are inconclusive. While there are multiple strategies for assess- ing the construct, he thinks they are better suited for research than for any form of high-stakes testing.

OCR for page 63