National Academies Press: OpenBook

Assessing 21st Century Skills: Summary of a Workshop (2011)

Chapter: 4 Assessing Intrapersonal Skills

« Previous: 3 Assessing Interpersonal Skills
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

4

Assessing Intrapersonal Skills

The third cluster of skills—intrapersonal skills—are talents or abilities that reside within the individual and aid him or her in problem solving. The previous workshop report that defined a set of 21st century skills (National Research Council, 2010) identified two broad skills that fall within this cluster:

Adaptability: The ability and willingness to cope with uncertain, new, and rapidly changing conditions on the job, including responding effectively to emergencies or crisis situations and learning new tasks, technologies, and procedures. Adaptability also includes handling work stress; adapting to different personalities, communication styles, and cultures; and physical adaptability to various indoor or outdoor work environments (Houston, 2007; Pulakos et al., 2000).

Self-management/self-development: The ability to work remotely, in virtual teams; to work autonomously; and to be self-motivating and self-monitoring. One aspect of self-management is the willingness and ability to acquire new information and skills related to work (Houston, 2007).

These kinds of skills operate across contexts, as Rick Hoyle, professor of psychology and neuroscience at Duke University, who presented findings from a paper about them and how they might be assessed, pointed out (Hoyle and Davisson, 2011).1 They are “transportable,” he explained,

________________

1See http://www7.national-academies.org/bota/21st_Century_Workshop_Hoyle_Paper. pdf [August 2011].

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

automatically transferred from one context to the next so that the very same skills that serve a person well in the social arena, for example, serve the person well in health decisions and in schooling and academics. Furthermore, he added, these skills ultimately contribute to adaptive behavior and productivity in that they counteract undesired influences that may arise from within the person or from the environment. Intrapersonal skills support volitional behavior, which Hoyle defined as discretionary behavior aimed at accomplishing the goals an individual sets for himself or herself. Examples of intrapersonal skills include attributes such as planfulness, self-discipline, delay of gratification, the ability to deal with and overcome distractions, and the ability to adjust one’s strategy or approach as needed. In Hoyle’s view, the common thread among these attributes is a skill called self-regulation.

Even though the field of psychology has studied self-regulation since the late 1960s, Hoyle said, disagreement about how to define it remains. To provide the audience with the broad spectrum of definitions, he presented varying points of view that four prominent researchers have put forth:

 

  • “The capacity of individuals to guide themselves, in any way possible, toward important goal states” (Fitzsimons and Bargh, 2004)
  • “The capacity to plan, guide, and monitor one’s behavior flexibly in the face of changing circumstances” (Brown, 1998)
  • “Self-generated thoughts, feelings, and actions that are planned and cyclically adapted to the attainment of personal goals” (Zimmerman, 2000)
  • “The process by which one monitors, directs attention, maintains, and modifies behaviors to approach a desirable goal” (Ilkowska and Engle, 2010)

Hoyle identified some common threads among the definitions. They all recognize that people need to monitor their behavior and that they are doing this in the service of goal pursuit. In addition, they all acknowledge that flexibility is needed. Most importantly, they all involve affect. Hoyle emphasized that self-regulation does not just involve cognition but also involves feelings and emotions.

Hoyle prefers the following definition: the processes by which people remain on course in their pursuit of the goals they have adopted. In some cases, such as a school setting, these goals may not be the student’s own, but they are put before students. The question is if they capable and ready to do the things that need to be done to pursue those goals and to move forward on them.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

WHY IS SELF-REGULATION IMPORTANT?

Invariably the goals we adopt or are given to us are challenged in a number of ways. We may have counterproductive impulses, such as eating dessert even though we have a goal to lose weight. We may encounter situational hurdles, obstacles that interfere with the ongoing pursuit of some goal. We may have competing goals, so that satisfying one goal detracts from accomplishing another. Thus, people must manage the conflict between goals. And, in some cases, progress may be so slow that it is difficult to sustain motivation. Remaining on course toward goal pursuit requires a set of strategies that, collectively, constitute self-regulation.

Not every good behavior involves self-regulation, Hoyle clarified; self-regulation is behavior over which the individual exercises some level of discretion. Self-regulation requires considerable cognitive energy and effort. If the individual is constantly self-regulating, it is impossible to sustain momentum toward accomplishing a goal. It is most effective for the individual to move many behaviors outside the realm of the processes that require self-regulation.

For example, some behaviors are contingent on cues in the environment and are simply habits. The individual performs a habit when he or she links a behavior with some cue in the environment, Hoyle explained, and thus can accomplish the behavior without having to draw on self-regulation. Also, many behaviors are attributable to impulse. Impulses may be productive or they may detract from goal pursuit in some way, but they occur without self-regulation. Other behaviors are strongly influenced by normative pressure. This is frequently seen among adolescents, who experience a critical push/pull between the normative environment and their own individual goals. Finally, there are behaviors that are determined by social, political, or religious systems within which people live. In some cases those systems serve the role of regulating behavior, thereby circumventing the need for self-regulation.

Trends in society demonstrate some of the consequences that result from a lack of self-regulation. As Hoyle put it, “I don’t think we need much convincing that a lot of what we see around us seems to involve a failure of self-control at a fairly large level.” For example, Hoyle noted, U.S. consumers have not exerted much self-regulation when it comes to debt levels. In the late 1970s, consumer revolving credit debt was $54 billion. By the end of the 1990s, it rose to more than $600 billion, and now approaches $1 trillion.2 Likewise, Hoyle observed, obesity rates are at crisis levels. In 1990, no state had an obesity prevalence rate above 15 percent. By 2007, only one state had an obesity prevalence rate less than 20 percent, and 30 states had a preva-

________________

2See http://www.federalreserve.gov/releases/g19/hist/cc_hist_sa.html [July 2011].

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

lence rate of 25 percent or more.3 In addition, a notable number of deaths can be attributed to failure of self-regulation. According to a report by the Centers for Disease Control and Prevention (Anderson, 2002), 33 percent of deaths were attributable to obesity, physical inactivity, and tobacco use. In addition, 8 percent of deaths were attributable to a cluster of behavioral causes including alcohol consumption, motor vehicle crashes, incidents involving firearms, sexual behaviors, and use of illicit drugs.

Furthermore, in Hoyle’s view, the current economic crisis can be considered as a failure of self-regulation on a grand scale. Systemic, circumstantial, and societal issues all contributed to the crisis, but the excessive borrowing and lending and high-risk investments made with little or no concern for potential long-term consequences are all hallmarks of a lack of self-regulation. Hoyle emphasized that these examples all provide evidence of the importance of equipping children to be better self-regulated citizens as they approach adulthood. Hoyle also noted that considerable evidence in the literature underscores the value and importance of self-regulation. He focused on three studies.

One longitudinal study, conducted by Walter Mischel, began in the late 1950s and focused on delay of gratification (Mischel, 1958; Mischel et al., in press). Mischel used a variety of paradigms to study delay of gratification, and Hoyle described one that involved a set of preschoolers. The children were presented with an object they desired (e.g., a piece of candy or a marshmallow) but were told that they must wait until the experimenter returned to the room before they could have it. The experimenter left the room, closed the door, and intentionally did not return. Mischel collected data on how long each child waited before reaching for the object. After 10 to 12 years, Mischel contacted the parents of the participants and gathered information about their academic and social competence. He found that adolescent behavior was significantly predicted by the duration of the self-imposed delay in gratification. That is, the longer the preschooler was able to delay gratification, the better he or she fared as an adolescent in terms of a variety of self-regulation characteristics, such as attentiveness, planfulness, and reasoning ability.

A second longitudinal study by Caspi, Moffitt, and others (Caspi et al., 1997) with youngsters in Dunedin, New Zealand, is currently underway. The researchers are studying an entire birth cohort, collecting data every 2 to 3 years. At age 3, the children’s temperament was evaluated, with some classified as “under-controlled.” At age 18, the children who fell into the under-controlled classification rated high on a number of qualities that indicate poor self-regulation, including impulsivity and danger-seeking behavior, aggression, and interpersonal alienation. At

________________

3See http://www.cdc.gov/obesity/data/trends.html [July 2011].

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

age 21, these individuals were more likely to be engaged in activities that show evidence of a failure to control their behavior, such as alcohol dependence, dangerous driving, violent behavior, and having unsafe sex. The likelihood of engaging in these behaviors was double that of the other children in the birth cohort.

Third, Hoyle cited work by economist Jim Heckman, who argues that noncognitive abilities are equally, if not more, important than traditional cognitive abilities when it comes to predicting educational and socioeconomic outcomes. The noncognitive attributes that Heckman refers to—attentiveness, persistence, impulse control, and social competence—are all evidence of self-regulation from a psychological perspective. Heckman’s work shows that a gap between the disadvantaged and the advantaged begins to emerge very early. The point he makes is that there is a window of opportunity during which we can invest in those children. They can be taught to self-regulate, which Heckman finds will eventually result in significant dividends in terms of economic productivity, life success, and the like. Heckman (2006) reported findings from a study of children who participated in the Perry Preschool Program, which included a significant component of training of self-regulatory skills. He found students who participated in this program were less likely to drop out of school, spend time in jail, smoke, and participate in other self-destructive behaviors. In terms of economic productivity, the Perry Preschool Program participants were 15 to 17 percent higher than children who did not participate. Heckman argues that from an economic perspective there was a nine-fold payoff in what it costs to operate the Perry Preschool Program versus the payoff in economic productivity down the line.

DEFINING SELF-REGULATION

Although there has been considerable work on the topic of self-regulation in the field—with 114 chapters in edited volumes between 1998 and 20104 and about 120 published articles each year—Hoyle said the field has no current consensus regarding a single definition of self-regulation. His review of the body of work revealed a definition is sometimes, but not always, provided. He finds no evidence of even minimal acceptance of a common definition, and even the same authors sometimes use different definitions. Furthermore, he thinks self-regulation has been applied

________________

4Such as Motivation and Self-Regulation Across the Life Span published in 1998; the Handbook of Self-Regulation published in 2000; Self-Regulation of Health and Illness Behaviour published in 2003; the Handbook of Self-Regulation: Research, Theory, and Applications published in 2004; Self-Regulation in Health Behavior published in 2006; and the Handbook of Personality and Self-Regulation published in 2010.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

far too broadly and, in many cases, inappropriately. Hoyle believes the current state of the conceptualization of self-regulation is the primary obstacle to producing assessments of it.

Hoyle laid out a conceptualization of self-regulation, which he emphasized was not really a model or a theory, but a framework that might help move forward in developing assessments. This conceptualization is presented in Figure 4-1. Understanding these components of self-regulation helps to provide a basis for defining constructs that might be assessed. Hoyle explained each of the components.

In the leftmost column (“Foundations“) are a series of variables or traits the individual “brings to the table.” These include (1) executive function, (2) temperament, and (3) personality characteristics. Hoyle added it is not clear whether these foundations are susceptible to change, but they are the “raw materials” that self-regulation draws upon.

Executive function is a set of cognitive processes and propensities that originate early in life (Goldman-Rakic, 1987; for a review, see Best and Miller, 2010). Three core functions underlie the processes involved in most acts of self-regulation (Miyake et al., 2000). Inhibition involves stopping ongoing thoughts and actions either when prompted by an external signal or upon determining that continuation would lead to an error (Logan and Cowan, 1984). Working memory involves keeping information active in primary memory while searching and retrieving information stored in secondary memory (Unsworth and Engle, 2007). Because keeping relevant information active while ignoring or suppressing competing information that is not relevant involves inhibition, inhibition and working memory

images

FIGURE 4-1 A conceptualization of self-regulation.

SOURCE: Adapted from Rick Hoyle’s presentation. Used with permission.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

are related. Complex tasks require the coordination of information relevant to multiple task components, requiring working memory to be flexible and controlled. Finally, shifting involves moving back and forth between mental states, rules, or tasks (Miyake et al., 2000). The importance of these basic capacities is evident in a cornerstone of self-regulation, the delay of gratification, which requires the inhibition of an impulse to act in response to a temptation in the immediate environment in favor of one or more longer-term goals or priorities (Mischel et al., in press).

Variability in executive function is expressed as individual differences in temperament, which Hoyle said is defined as individual differences in emotional and motor reactivity and in the attentional capacities that support self-regulation (Rothbart and Hwang, 2002, p. 113). One of the most important capacities is referred to as effortful control, which Hoyle explained is apparent when the child “is able to say no to that thing in front of them in service of some other thing that needs attention at that moment.” A related dimension of temperament is reactive control, which Hoyle described as the “relatively involuntary influence of approach and avoidance motives.” Extreme forms of reactive control can result in overcontrolled reactivity, such as shyness, or undercontrolled reactivity, such as impulsivity.

Hoyle defines personality as tendencies of thought, feeling, and action that are moderately stable across the lifespan (Roberts and DelVecchio, 2000), and he noted they can be separated into higher-order dimensions and lower-order dimensions. Research has shown that there are between three and seven higher-order dimensions (depending on the model and classification strategy) into which all personality traits fall. The dimension most relevant for self-regulation is conscientiousness, which generally concerns the ways people manage their behavior. Individuals who are high on conscientiousness tend to be confident, disciplined, orderly, and planful (Costa and McCrae, 1992).

A large number of narrower (lower-order) personality constructs also tend to facilitate or impede self-regulation. One of the most important is impulsivity, which Hoyle said might be viewed as the absence of self-regulation. Other lower-order personality constructs are relevant to self-regulation—those that concern self-regulatory style—and how (rather than whether) self-regulation is accomplished. They are foundational in their provision of the basic capacities and tendencies on which the processes involved directly in self-regulation draw.

In the middle column of Figure 4-1, Hoyle provided a list of the processes individuals go through as they try to accomplish a goal, although he cautioned that there is no agreement in his field on the exact nature of these processes. He noted the list helps to understand what

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

would be involved in an assessment of how effective an individual is at self-regulation.

The process generally begins with forethought, when the individual receives information, evaluates it, considers options, sets goals, and formulates a plan to achieve these goals. This is followed by performance, in which the individual implements the plan. From a self-regulation perspective, performance involves exercising self-control for the purpose of engaging in goal-relevant behaviors while avoiding behaviors irrelevant to or in conflict with the goal. Hoyle said a critical aspect of performance is self-observation or self-reflection, when the individual assesses the effectiveness of his or her performance and re-engages the process for subsequent attempts at goal pursuit. This model assumes a cyclical process whereby the individual repeatedly moves from forethought to performance to self-reflection, realizing progress toward the goal with each successive cycle.

The rightmost column of Figure 4-1 is labeled “Consequences,” which Hoyle maintains is probably the quickest approach to getting at a person’s skill level at self-regulation. What observable evidence is there that an individual is skilled or unskilled at self-regulation? He classified consequences into three categories.

One type of consequence is normative: that is, certain behaviors are evidence of a well-regulated individual regardless of the context or the particular population. Examples include academic success as evidenced by regularly completing assignments as instructed on schedule; social success in the form of routine relationship maintenance behaviors; and good health as evidenced by proper diet and exercise and general avoidance of health-risk behaviors.

Another type of consequence is domain-specific, such as self-regulation in the context of health behavior. For instance, hypertension patients often are prescribed a regimen that includes control of diet and regular intake of medications. Certain forms of psychotherapy might prescribe goals and behavioral evidence of their pursuit. In such instances, self-regulation is necessary and evidence of successful self-regulation is concrete and specific.

The final category is the idiosyncratic goals that each person decides on his or her own to pursue.

APPROACHES FOR ASSESSING SELF-REGULATION

Hoyle described a number of approaches for assessing self-regulation. One frequently used approach is self-report. In the typical use of this strategy, the respondent is given a set of statements and asked to select one of the provided response options to indicate extent of agreement or

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

disagreement with the statement or the degree to which the statement accurately describes him or her. There are advantages and disadvantages to this strategy, and Hoyle described several. It is often the least expensive approach in terms of materials as well as time and space requirements. There is also an implicit assumption that an individual is uniquely positioned to report on his or her standing on statements about the constructs and may well be the best source for the information. On the other hand, Hoyle noted, individuals are biased in both how they think about their own behavior and what they think is the task before them when they are responding to questionnaire items. There is evidence that people often do not have access to higher-order processes and therefore are unable to report about them accurately (Nisbett and Wilson, 1977). Hoyle said that there is also an age issue in that young children may lack the cognitive skills and reading ability to understand the statements they are asked to rate and the use of rating scales to do so.

Another approach is informant reports, which, Hoyle said, share many of the qualities of self-reports and address some of the limitations of the self-report strategy. One advantage of informant reports is that they eliminate the self-referential biases that may undermine the validity of self-reports. That is, Hoyle explained, well-trained informants who observe the target across time and situations may be able to infer and accurately report on characteristics of the target that the target is unable to accurately report about himself or herself. Another advantage Hoyle cited is that the informant report strategy allows for assessment of preverbal children, as well as of individuals who for other reasons may be unable to read and understand the statements on which they are to be rated. A clear drawback of the strategy, Hoyle noted, is the limited access most informants have to the individuals they are rating. For example, teachers only observe children in academic settings, parents see them primarily in the home, and peers are privy to behavior only in selected settings. Further, Hoyle stated, it may be difficult to extract information about specific skills and abilities from complex behavior sequences. That is, sometimes it is difficult to know, even after extensive observation, what is actually going on in the head of the person one is observing.

A third approach is behavioral task performances, which, Hoyle said, are designed so that they require only the capacity or skill of interest. Hoyle noted that these tasks are most often used to assess constructs in the foundations (see Figure 4-1), generally those capacities that constitute executive function. Speed and efficiency in completing these tasks is assumed to measure strength of the capacity being assessed. According to Hoyle, the tasks are tailored to the age group being assessed, and they generally do not require verbal skills or awareness by the individual of

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

his/her use of the capacity. The tasks are typically scored in terms of objective characteristics of performance (e.g., time to completion, number of mistakes). The positive features of assessments based on behavioral task performance are offset somewhat by two shortcomings, Hoyle cautioned. First, behavioral tasks tend to be tailored to the age group being assessed, which interferes with the ability to track performance over time. A second shortcoming concerns the purity of capacities assessed by the tasks. Complex tasks likely require multiple, interdependent capacities, thereby producing scores that cannot be used to pinpoint standing on specific capacities (Garon, Bryson, and Smith, 2008). They have the advantages of not requiring verbal skills, they do not require the person to report on higher order mental activity, and the scores tend to be objective (e.g., time to completion, number of mistakes). The measures tend to be things like Mischel’s delay of gratification, which was the amount of time before the individual reached for the tempting object on the table. The disadvantages of this approach, Hoyle said, are that the tasks must be tailored to the age of the respondent and they often tap more than one skill or ability.

Hoyle described some examples of behavioral tasks performances intended to measure each of the foundational skills (see Figure 4-1). One task, referred to as the “stop signal measure” (see Box 4-1), is designed to measure executive function. In fairly rapid succession, the subject is presented with a series of cards. When the greater sign appears on the card, the subject is to press the right key, and when the lesser sign appears, the subject is to press the left key. At variable intervals, an audible sound occurs at which point the subject is not to press any key. The assessment measures how well they are able to inhibit and not press the key.

Another example, the star counting task, measures working memory. As shown in Box 4-2, the task begins with the number 15. In this case, when the subject reaches a plus sign, he/she is to count in the forward direction (16, 17, 18, etc.); when the subject reaches a minus sign, he/she is to change and count downward (18, 17, 16, etc.). The task is to get the right answer within a minute. A series of these is presented, and then the rules change so that a plus sign indicates to count in the backward direction and a minus sign indicates to count in the forward direction. This task measures the ability to change rules and hold the new rule in memory while overriding the old one.

Hoyle also showed examples of assessments intended to measure self-regulation through process and consequences (see Figure 4-1). The first is a self-report instrument on which the candidate rates him/herself on statements about processes, such as the ones that appear below:

 

  • “I usually keep track of my progress toward my goals.”
  • “I have personal standards, and try to live up to them.”
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

BOX 4-1
Example of a Stop Signal Task Designed
to Measure Executive Function

images

SOURCE: Adapted from Rick Hoyle’s presentation. From Chamberlain, S.R. (2006). Neurochemical modulation of response inhibition and probabilistic learning in humans. Science, 311, 861. Reprinted with permission of American Association for the Advancement of Science.

  • “I am willing to consider other ways of doing things.”
  • “I have sought out advice or information about changing.”
  • “Once I have a goal, I can usually plan how to reach it.”
  • “I get easily distracted from my plans.” (reverse-scored)
  • “I don’t seem to learn from my mistakes.” (reverse-scored)

The second is a self-report instrument that includes measures of behavior indicators of conscientiousness. The assumption is that the routine production of these behaviors is a sign of an individual who is either capable or not capable at self-regulation. The test taker rates him/herself on statements such as those shown below:

 

  • “Play sick to avoid doing something” (avoid work)
  • “Make a grocery list before going to the store” (organization)
  • “Buy something on the spur of the moment” (impulsivity)
  • “Clean the inside of the microwave oven” (cleanliness)
  • “Work or study on a Friday or Saturday evening” (industriousness)
  • “Clean up right after company leaves” (appearance)
  • “Allow extra time for getting lost when going to new places” (punctuality)
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

BOX 4-2
Example of a Star Counting Task Designed
to Measure Working Memory

images

SOURCE: Adapted from Rick Hoyle’s presentation. Reprinted from De Jong, P.F., and Das-Smaal, E.A. (1990). The star counting test: An attention test for children. Personality and Individual Differences, 11(6), 597-604. Copyright 1990, with permission from Elsevier.

In concluding, Hoyle noted that measures of foundational constructs are well established and, in many cases, have been adapted for use with infants and children. Measures of the self-regulation process are few and generally have not been adapted for use outside the research context. Behavioral consequences of the skill at self-regulating have not been considered in efforts at conceptualization and assessment.

ASSESSMENT EXAMPLES

At the workshop, four speakers discussed other examples of assessments of intrapersonal skills. Paul Sackett, professor of psychology with the University of Minnesota, made the first presentation and covered a variety of strategies for assessing integrity in employee selection settings. The second presentation, made by Candice Odgers, assistant professor of psychology, social behavior, and education with the University of California at Irvine, focused on strategies for assessing antisocial behaviors and conduct disorders in K-12 and counseling settings. Both of these types of assessments have been used operationally for some time. The remaining

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

two presenters discussed assessment strategies that are currently under research. Tim Cleary, associate professor of psychology with the University of Wisconsin–Milwaukee, discussed research on assessments of self-regulated learning. Gerald Matthews, professor of psychology with the University of Cincinnati, discussed research on assessing emotional intelligence.

Assessing Integrity in Job Applicants

Sackett began by talking about the origin of assessments like tests of integrity.5 He noted that for employers, the goal has always been to hire people likely to be good job performers. More recently, however, there has been a move to consider employees’ contribution to an organization beyond simple task completion—not only what they do, but what they do not do. For instance, in a retail setting, the employer wants to hire sales clerks who perform their job well but who also do not pilfer money from the cash drawer. A trucking firm wants to hire drivers who deliver the products on time but who also obey traffic laws and drive safely. Sackett said that there are a host of behaviors, which he referred to as “counterproductive work behaviors,” that employers want to avoid in the people they hire, such as drinking or using drugs on the job, stealing, sexually abusing coworkers, lying, and cheating.

In many work settings, people work untended and have access to cash, money, and merchandise. Employers can take a number of preemptive steps to reduce the prevalence of these behaviors, such as installing cameras or other kinds of monitoring and control systems. But, there are limits to how many cameras can be installed and where they can be installed, and many settings are not easily monitored. Thus, Sackett explained, employers have moved in the direction of trying to screen potential employees and eliminate those likely to participate in counterproductive work behaviors. In response to this demand from employers, a set of commercial products emerged generically referred to as integrity tests.

Integrity tests are designed to predict theft and other forms of counterproductive work behavior. The measures are used internally by organizations, so the test taker never receives a score or feedback of any kind. The basis for using the test is predictive validity at the aggregate level. From an organization’s point of view, it is not necessary to be able to precisely pinpoint which individuals will lie, cheat, or steal. The focus is on reducing these behaviors in the aggregate. As Sackett put it, “if I use this

________________

5Sackett’s presentation is available at http://www7.national-academies.org/bota/21st_Century_Workshop_Sackett.pdf [August 2011].

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

instrument to hire a workforce of 200 people, will I have fewer incidents of wrongdoing than if I did not use it?”

To help the audience understand the construct of integrity, Sackett presented them with a set of scenarios, all of which involved $20 in cash that clearly belongs to someone else. He asked them to consider what they might do in each situation.

Scenario 1: You go to the gym with a friend for your weekly game of racquetball. After you finish your game, your friend heads for the shower before you, and you are left alone in the locker room. Your friend has not locked his locker, and you see his wallet peaking out of his jacket pocket. Would you reach in and take $20 out of his wallet?

Scenario 2: You go to a “big box” store, such as Walmart or Target. You are checking out, and you ask the cashier a question. The cashier does not know the answer, leaves the cash drawer open, and leaves the counter to see if a coworker knows the answer. You see a $20 bill within easy reach. Would you reach in and take the $20 out of the cash drawer?

Scenario 3: You go to the ATM and follow the procedures to withdraw $100. Your withdrawal arrives as six $20 bills. Would you contact the bank to notify them that you received $20 more than you had requested?

Scenario 4: You are making a purchase at your local grocery store. The cashier totals your purchase, and you pay in cash. When the cashier counts out your change, you are given $20 more than you are due. You realize this on the spot. Would you let the cashier know of the mistake?

Scenario 5: You are walking down the street and find a wallet. You pick it up and discover that it is full of credit cards, along with an I.D. and $20 in cash. You use the I.D. to locate the owner. When you return the wallet to the owner, do you keep the $20 in cash or do you return it?

Sackett said that one of the readily apparent points is that integrity can depend on the situation. Many people would never take $20 from a friend’s wallet but might take the extra $20 from a bank or a cashier, justifying it as “the ATM screwed up” or “if the sales clerk cannot count, it’s not my problem.” There are some situations that virtually no one reports that he or she would do, and others that many people report they would do. Situational features affect the percentage of individuals who say they would engage in the counterproductive behavior. But within any one situation, individual differences influence who does and does not engage in these behaviors.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

Sackett described several different types of integrity measures. He noted that integrity tests were originally developed within the polygraph industry during the 1960s and 1970s. The practice of administering pre-employment polygraphs to prospective employees was banned by many states during this period (and nationally in 1988), and thus the polygraph industry began searching for alternatives methods for prescreening job applicants. The measures were initially developed to predict theft, out of a demand by employers for pre-employment screenings of job candidates who would have access to cash. Eventually, they were expanded for use with a full range of behaviors. Over time, Sackett said three types of measures have emerged, all self-report.

He described the first as an “overt integrity test,” noting that it is overt in the sense that the intent of the assessment is clear to the job applicant. The assessment consists of a series of questions probing the candidate’s beliefs about the frequency and extent of theft and other forms of wrongdoing and their attitudes about punishment. The test consists of questions that assess seven categories of beliefs and attitudes, as shown below:

 

  • Beliefs about the frequency and extent of theft (e.g., what percentage of people do you think cheat on taxes?)
  • Punitiveness toward theft (e.g., an employee is caught taking $100 from his organization. What punishment should the employee receive?)
  • Ruminations about theft
  • Perceived ease of theft
  • Rationalizations about theft
  • Assessments of one’s own honesty
  • Admissions

Sackett explained the final category, “Admissions,” is actually a constructed-response item that asks the job candidate about his or her own theft behaviors, such as “what is the dollar value of cash and merchandise you have taken from your previous employer in the last six months?” Sackett said many are surprised to see this type of question and, more so, that anyone responds to it with an actual dollar amount. He hypothesized that people who steal from their employers believe everyone does it; therefore, they believe the employer would think they were lying if they said they took nothing (since everyone does it).

A second type of measure includes “personality-oriented tests,” which have their roots in the psychology discipline rather than the polygraph industry. He described three different commercial products:

 

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
  • The Personnel Reaction Blank: designed to measure wayward impulses. The items focus on dependability, conscientiousness, and social conformity.
  • The Employment Inventory: intended to measure employee deviance. The items deal with trouble with authority, thrill seeking, hostility, unhappy home life, and lack of work motivation.
  • The Hogan Personality Inventory Reliability Scale: a measure of organizational delinquency. The items assess levels of hostility, impulse control, and attachment.

A third type of measure, called “conditional reasoning tests,” has emerged only recently. Sackett said the theory underlying these tests is that a person’s standing on a trait affects the justification mechanisms the person uses to explain his or her own behavior. Developed by Larry James at Georgia Institute of Technology, the theory is that people who are prone to engage in counterproductive work behavior will tend to be also high on a construct called “hostile attribution bias.” A sample item appears below:

American cars are now more reliable than they used to be 15 to 20 years ago. Why?

Option A: American car makers knew less about building reliable cars 15 to 20 years ago.

Option B: Prior to the introduction of high-quality foreign cars, American car makers purposely built cars badly in order to sell more repair parts.

According to James’ theory, endorsing option B is a manifestation of hostile attribution bias. The purpose of the assessment is disguised to the candidate. The candidate is told that it is a reasoning test, but, in fact, the focus is on the frequency with which he or she chooses the option with the aggressive or hostile undertone.

Sackett then turned to empirical evidence documenting the validity of integrity tests. He has conducted several literature reviews on this topic. In the first review (Sackett and Decker, 1979), he found six studies of tests of honesty. In subsequent reviews, he found 40 (Sackett and Harris, 1984), 70 (Sackett, Burris, and Callahan, 1989), and most recently 665 (Sackett and Wanek, 1996). At this point, the field of employment testing considers the validity of integrity tests to be well established.

Generally, the findings show validity coefficients in the .20 to .30 range. He said the three types of tests appear to have similar levels of validity. While most of the tests originally focused on identifying job candidates likely to steal, the tests predict a wide array of counterproductive

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

work behaviors. Some behaviors (e.g., absence from work) are more predictable than others (e.g., theft). He commented that theft, in particular, is difficult to predict because it tends to occur rarely, and detection of it is rare. The detection rate is much lower than the rate of engaging in the behavior, which complicates attempts to study the behavior.

The studies also show the tests predict overall job performance. Sackett believes this relationship is attributable to underlying constructs of conscientiousness and other forms of self-regulation that cause people to perform well at work as well as to avoid wrong-doing.

The studies show minimal subgroup differences on the tests, suggesting that employers do not need to worry about fairness or adverse impact in using them. Generally, women perform higher than men, but the performance differences follow the gender patterns seen in other forms of deviant and criminal behavior.

The tests tend to have a low relation with measures of cognitive ability, indicating they provide information that is not redundant with the other kinds of measures employers often use. There is evidence they are valid for both high- and low-complexity jobs.

One concern with these tests is their reliance on self-reports of attitudes and behaviors, which raises concerns about fakeability. Studies have been conducted to investigate this using a strategy called “instructed faking.” In the classic instructed faking paradigm, subjects are randomly assigned to two conditions. One group is told to try to score as high as possible and not to worry about responding honestly; the other group is told to respond honestly. The results show those in the first group score higher. Sackett said this result demonstrates the tests are conceptually fakeable, but he thinks it is important to evaluate this finding in light of results from the validity studies discussed above. In his view, if faking were prevalent in live applicant contexts, the validity coefficients would be diminished. Sackett added that faking is currently a concern for the first two types of integrity tests (overt and personality oriented), but not for conditional reasoning tests because their purpose is disguised. These tests would become fakeable if test takers were to discern their true purpose.

Sackett closed by noting that integrity testing developed from an applied standpoint, but the field has now shifted to a theoretical orientation. Initially, employers simply sought a device to identify job candidates likely to participate in wrong-doing on the job; they simply wanted a method that worked. The objective of more recent research has been to investigate why integrity tests work and to understand the underlying mechanisms by which they predict counterproductive work behaviors. Much of this work has centered on the self-regulation literature.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

Assessing Antisocial Behavior and Conduct Disorders

Candice Odgers’ work focuses on antisocial behavior, sometimes referred to as conduct disorders in children.6 A list of typical antisocial behaviors appears below:

 

  • Aggression (e.g., fights, is physically cruel to people or animals, bullies, uses weapon, makes threats)
  • Theft (e.g., steals, shoplifts, takes things)
  • Deceitful (e.g., lies, cheats, blames others)
  • Personality problems (e.g., is irritable, loud, jealous, hostile, annoying or demanding, brags and boasts)
  • Rule-breaking (e.g., is disobedient, is truant, runs away from home)
  • Oppositional (e.g., argues, swears, is stubborn, has tantrums)
  • Destructive (e.g., commits vandalism, sets fires)

Odgers explained these are the children who have no ability to delay gratification. As she put it, “in the study by Mischel that Hoyle described, these are the children who would eat the candy or marshmallow before the interviewer left the room and then give the interviewer a defiant look.” They are the children who become adults who steal from their employers. Antisocial behavior is apparently quite prevalent, Odgers reported, with an estimated lifetime prevalence of nearly 10 percent (Nock et al., 2006). Research shows antisocial behavior in children is a robust predictor of a number of problematic behaviors in adults, including poor physical health, school failure, and economic problems (Moffitt et al., 2002; Odgers et al., 2008). It is closely linked to difficulties with self-regulation and deficits in executive functioning (Dishion and Connell, 2006; Ellis et al., 2004; Moffitt, 1993). As Odgers put it, “antisocial behavior is clearly a marker of bad things to come.”

She pointed out that these behavior problems can be costly. One study showed that they result in an additional expense of about $70,000 per child in terms of the services used over the course of the 7 years of adolescence (Foster and Jones, 2005). The behaviors translate into unique challenges for families and schools, mental health and justice-related settings, and employers and social welfare systems.

Odgers addressed one question that arose repeatedly during the workshop—whether these kinds of intrapersonal skills are malleable, or specifically, whether these self-regulation skills can be changed. Odgers

________________

6Odgers’ presentation is available at http://www7.national-academies.org/bota/21st_Century_Workshop_Odgers.pdf [August 2011].

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

BOX 4-3
Example Items from the Childhood
Behavior Checklist in the ASEBA

Below is a list of items that describe children and youths. For each item that describes your child now or within the past 6 months, please circle the 2 if the item is very true or often true of your child. Circle the 1 if the item is somewhat or sometimes true of your child. If the item is not true of your child, circle the 0. Please answer all items as well as you can, even if some do not seem to apply to your child.

 

0 = Not true (as far as you know)

1 = Somewhat or sometimes true

2 = Very true or often true

 

0 1 2 1. Argues a lot
0 1 2 2. Has temper tantrums or a hot temper
0 1 2 3. Is stubborn, sullen or irritable
0 1 2 4. Doesn’t get along well with other kids
0 1 2 5. Destroys his/her own things
0 1 2 6. Lying or cheating
0 1 2 7. Runs away from home
0 1 2 8. Physically attacks people
0 1 2 9. Cruelty, bullying, or meanness to others

SOURCE: Copyright T.M. Achenbach. Reproduced with permission.

said the answer depends in part on when interventions are attempted. Her field of developmental psychology has established optimal timing for the development of some of these skills, and a payoff associated with early identification and intervention. Early intervention ultimately reduces the persistence of antisocial behaviors and subsequent involvement with the juvenile justice system.

Fortunately, Odgers noted, antisocial behavior is relatively easy to diagnose. It is assessed in a semi-structured interview setting, and the most widely used instrument is the Achenbach System of Empirically Based Assessments (ASEBA).7 The ASEBA has been translated into 85 languages and reported in more than 7,000 articles. It is designed to assess children’s academic performance, adaptive functioning, and behavioral/ emotional problems. The assessment system uses behavior checklists designed for different age groups. A sample of the checklist items for school-aged children appears in Box 4-3.

________________

7See http://www.aseba.org [August2011] for further information about the assessment.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

The syndrome scales were derived via factor analysis and were normed on large population-based and clinical samples. Reliabilities, based on test-retest estimates and coefficient alpha, are in the .90 range. The assessment takes about 15 minutes to complete. It is also compatible with the Diagnostic and Statistical Manual’s definition of conduct disorder and antisocial behavior disorder, which allows people to talk across disciplines. Other psychometric information about the assessment is available on the ASEBA website (see footnote 7).

For young children, parents and teachers complete the checklist. For adolescents and adults, the checklist is self-reported, although there is usually an attempt to gather information from another informant (parents and teachers for adolescents; spouse or significant other for adults).

Odgers has been involved in the longitudinal study in Dunedin, New Zealand, that Hoyle described. The study has followed 1,000 individuals born in 1972 and 1973, and the researchers have just finished the age-38 assessment. Assessments were done at birth and every couple of years thereafter, thus providing a longitudinal perspective of when these skills emerge (or when problems emerge) and how they relate to other skills and deficits. The study has yielded considerable information about the relationships between these skills and life outcomes. Odgers said that they are finding that conduct disorder, particularly persistent conduct disorder across childhood, is one of the most accurate signals of future problems across a wide array of domains, including mental health, physical health, economic functioning, and job prospects.

Odgers presented the graph shown in Figure 4-2 that displays the incidence of conduct problems for the males in the sample, following them from ages 7 to 26. The researchers identified four patterns of behavior: (1) individuals who were consistently low in conduct problems (solid line); (2) individuals who exhibited conduct problems in childhood, but the problems diminished over time (line with triangles); (3) individuals who began exhibiting conduct problems during adolescent years (line with circles); and (4) individuals who persistently exhibited conduct disorders from childhood on into adulthood (line with squares). The researchers have compared outcomes for these four groups.

Odgers said the first finding from this analysis is that antisocial behavior in childhood does not necessarily signal poor outcomes in adulthood. Some children may exhibit conduct problems early on, but these problems are dealt with or as Odgers put it “socialized out.” Through the influences of family, school, peers, and other factors, these children develop effective self-regulation skills, and the conduct problems diminish over time. However, this does not happen for all children with early-onset conduct problems, and individuals whose problems persist into adulthood experience difficulties in a number of areas of life.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

images

FIGURE 4-2 Incidence of conduct problems between ages 7 and 26 for longitudinal sample of individuals in Dunedin, New Zealand.

SOURCE: Odgers et al. (2007). Reprinted from Odgers, C.L., Milne, B.J., et al. (2007). Predicting prognosis for the conduct-problem boy: Can family history help? Journal of American Academy of Child and Adolescent Psychiatry, 46(10), 1240-1249, Copyright 2007, with permission from Elsevier. And from Archives of General Psychiatry, 64, 476-484, Copyright 2007, American Medical Association, All rights reserved.

For example, Figures 4-3 and 4-4 use effect sizes to compare health outcomes for males in the different groups. Figure 4-3 compares health outcomes for males with life-course-persistent conduct disorders versus those who scored low on conduct disorders. Figure 4-4 compares health outcomes for males with childhood-limited conduct disorders versus those who scored low on conduct disorders. The figures show the health outcomes for males with childhood-limited conduct disorders are quite similar to the health outcomes for individuals who scored low in conduct problems. On the other hand, the males with life-course persis-

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

images

FIGURE 4-3 Health outcomes for males with life-course persistent conduct disorders compared to those who scored low on conduct disorders.

SOURCE: Data from Odgers et al. (2008). Used with permission.

tent problems tended to be violent toward others and have convictions for this activity. They tended to suffer from anxiety and depression; were more likely to be dependent on alcohol, drugs, and tobacco; and had a greater incidence of health issues associated with these activities. Moreover, by age 32, 59 percent of this group had no educational qualifications8 as compared to an average of about 7 percent in the population at large. Only 24 percent of the males with childhood-limited conduct disorders had no educational qualifications, which Odgers noted was higher than average but half that for the males with life-course persistent conduct disorders.

________________

8For the study, “No Educational Qualifications” was defined as ending secondary education prior to receiving qualifications (i.e., a diploma) and not having pursued further education.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

images

FIGURE 4-4 Health outcomes for males with childhood-limited conduct disorders compared to those who scored low on conduct disorders.

SOURCE: Data from Odgers et al. (2008). Used with permission.

Odgers closed by highlighting some new issues being pursued in her field. Bullying is a topic being intensely explored, including bullying in school and in the workplace, as well as cyber-bullying in all contexts. Assessment strategies are emerging, which are focusing on the traits of being callous and unemotional as a subtype of antisocial behavior in which the person lacks empathy and the ability to read and relate to others. These traits are being considered as a precursor to psychopathy. Odgers noted that children who have both antisocial behavior and this lack of empathy seem to have particularly poor outcomes. There are considerations to adding this characteristic to the conduct disorder diagnosis to help improve prediction of outcomes.

Odgers said that the field is quickly realizing the importance of collecting family history information about children, much in the way that

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

images

FIGURE 4-5 Three-phase model of self-regulated thought and action.

SOURCE: Adapted from Zimmerman (2000). Used with permission.

it is done by medicine. Knowing about the parents’ levels of antisocial behavior can help considerably in the diagnosis and prediction of long-term outcomes.

Microanalysis of Self-Regulated Learning

A self-regulated learner, Tim Cleary explained,9 is an individual who

 

  • sets goals and develops/uses strategic plans;
  • is highly self-motivated and proactive;
  • engages in forms of self-control;
  • monitors strategies, performance, and cognition; and
  • frequently participates in self-reflection and analysis.

Cleary presented a three-phase model of self-regulated thought and action, as shown in Figure 4-5, which was developed by Zimmerman (2000) and referred to as a Cyclical Feedback Loop. The three phases of the model are forethought, performance, and self-reflection. The idea is that an individual approaches a task by considering what is involved, what it

________________

9Cleary’s presentation is available at http://www7.national-academies.org/bota/21st_Century_Workshop_Cleary.pdf [August 2011].

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

would take to complete the task, and how he or she should approach it. At the same time, the individual’s approach to the task is influenced by how motivated he or she is to do the task, how important or valuable it is, and how confident he or she feels about successfully completing it. These ideas and thoughts impact the person’s performance. During the performance phase, the individual uses self-regulation in order to complete the task. That is, he or she uses self-control strategies to stay on task and to learn what is being taught; he or she uses self-observation strategies to remain motivated and monitor learning. After a performance—typically after the individual receives some outcome such as a test grade, a quiz grade, or feedback on homework—he or she engages in reflection. At this phase, the individual evaluates the extent to which the goal has been reached, the factors that interfered with or helped with goal attainment, and considers his or her reaction to the performance (good or bad). Reflection is hypothesized to have an impact on subsequent attempts or subsequent strategies and modification of goals before the next learning attempt. Cleary noted this model forms the basis for microanalysis, which essentially focuses on diagnosis or the assessment of self-regulated learning and diagnosis of problems.

Cleary distinguished between two approaches toward measuring self-regulated learning based on work by Winne and Perry (2000): aptitude measures and event measures. The differences are in part related to the conceptualization of the construct. Aptitude measures, Cleary explained, are assessment tools that target self-regulated learning as a relatively global and enduring attribute of a person that predicts future behavior. They typically include self-report scales that rely on retrospective accounts of student behaviors and thoughts in terms of frequency, typicality, and usefulness. They generally capture the characteristics of self-regulated learning but they do so in a decontextualized manner. Some examples of aptitude scales are the (1) Motivated Strategies for Learning Questionnaire, (2) Learning and Study Strategies Inventory, (3) School Motivation and Learning Strategy Inventory, and (4) Self-Regulation Strategy Inventory. He highlighted two potential problems with this approach to measuring self-regulated learning. First, there are validity issues that relate to context-specificity. Research has shown that students’ self-reports of self-regulated learning behaviors vary across different content areas as well as across tasks within a course. Second, student self-reports are often not consistent with their actual behaviors (Hadwin et al., 2001; Winne and Jamieson-Noel, 2002).

Event measures are assessment tools that target self-regulated learning as an event, behavior, or cognition that may vary across contexts and tasks. They involve direct assessment of self-regulatory processes as they occur in real time and in authentic contexts (as opposed to self-reports

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

about past events or behaviors). In Cleary’s view, these measures are well equipped to capture the process of self-regulated learning. There are four approaches to obtaining event measures: (1) direct observations of students’ actual behaviors in an authentic environment; (2) trace measures, which are overt indicators of student cognition created during task engagement (such as underlining or highlighting text while reading); (3) personal diaries in which students record their study behaviors at home or the types of thoughts they had and actions they took when performing specific tasks; and (4) verbal reports or “think-aloud protocols,” which are records of students’ thought as they complete authentic activities. Self-regulated learning microanalysis is an event measure that uses a structured interview approach to measure students’ beliefs, attitudes, and cognitive regulatory processes before, during, and after some task or activity.

There are essentially four steps to the microanalysis approach that Cleary has studied. The first is to select a task with a clear beginning, middle, and end, such as studying for an exam or writing an essay. The second step is to identify the cyclical phase process that is of interest (see Figure 4-5), and the third step is to develop context-specific assessment questions to target the specific phase and process. Finally, and the most important element to Cleary’s approach, is to link the three-phase cycle processes to temporal dimensions of the task: that is, to identify the questions to ask in the forethought phase, the performance phase, and the self-reflection phase. Cleary said it is the matching of the questions and the task in temporal terms that is the most important aspect of this approach.

Cleary and his colleagues have developed a bank of questions that can be adapted to a variety of contexts and tasks. They have administered these questions to school-aged and college samples in order to gather data on their reliability and validity. The reliability estimates, which are coefficient Alpha estimates for metric variables and inter-rater agreement for categorical variables tend to run in the .80 to .90 range. Cleary said they have developed coding manuals and scoring rubrics for training the raters, which helps to produce these high reliability coefficients.

In terms of validity, all of the questions are derived from operational definitions of theoretical constructs from social cognitive theory and expert consensus, which Cleary noted helps to provide evidence of content validity. The researchers have also collected evidence on the differential and predictive validity of self-regulated learning microanalysis. In one recent study with college students, the authors examined the extent to which the microanalytic self-regulation questions accounted for unique variance in student course grades over and above that accounted for by the most commonly used self-report measure of self-regulation, the Motivated Strategies Learning Questionnaire (MSLQ; Cleary et al.,

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

2010). The analyses indicated that the microanalytic questions Cleary and colleagues developed, which included “attribution” (the reasons why students thought they had received the grade) and “adaptive differences” (the ways that they thought they should do differently), accounted for approximately 30 percent of the variance in final course grades over and above that accounted for by scores on the MSLQ along with several background variables.10

Cleary and his colleagues have also conducted differential validity studies in the context of motor tasks and physical activities that demonstrate that goal-setting, strategic planning, attributions, and adaptive inferences reliably differentiate low and high achievers (Cleary and Zimmerman, 2001; Cleary, Zimmerman, and Keating, 2006; Kitsantas and Zimmerman, 2002). Different groups of students who had different levels of achievement (novices or experts) showed distinct profiles of regulatory processes.

Cleary closed by stressing that attribution and adaptive differences play an important role in how engaged students are in their studies and the extent to which they have effective strategies to identify their weaknesses and improve their performance.

Assessing Emotional Intelligence

Gerald Matthews began by cautioning the audience that the field of psychology is still in its infancy in terms of defining and assessing emotional intelligence.11 On one hand, no one would want to be referred to as low on emotional intelligence. As he put it, “Saying that somebody has low emotional intelligence is now a pretty standard insult in various public domains.” On the other hand, research on emotional intelligence has not yet yielded a single conception of what it entails or how best to assess it. Thus, he advised, he would provide a “wide-angle” view of the state of the field, but he said there is no basis for coming to clear-cut conclusions about the construct.

In its broadest sense, Matthews explained, emotional intelligence includes abilities, competencies, and skills in perceiving, understanding, and managing emotion; however, there are a multitude of conceptualizations of the construct. One conception considers it as a set of abilities for

________________

10The R square (variance explained) for the regression equation when background variables and MSLQ scores were included was .082. When information on responses to the attributions and adaptive inferences questions were added to the model, the R square increased to .373. This .291 change in the R square value was statistically significant at p < .000.

11Matthews’ presentation is available at http://www7.nationalacademies.org/bota/21st_Century_Workshop_Matthews.pdf [August 2011].

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

processing emotional stimuli (Mayer et al., 2000) and treats the construct as a standard intelligence, having the kind of properties that other forms of intelligence and ability have. Another conception views emotional intelligence as part of the personality domain (Petrides and Furnham, 2003). In both cases, the assumption is that there is a general emotional intelligence factor that can be broken down into a number of more distinct competencies or skills.

Matthews thinks that neither conceptualization is useful. In his view, “emotional intelligence” is too vague a term to be of much use in either theory or practice (Roberts et al., 2007). He thinks it has become an “umbrella term” for a variety of separate attitudes, competencies, and skills that are only loosely interrelated, including basic temperament (e.g., positive and negative emotionality), information processing (e.g., emotion recognition), emotion-regulation (e.g., mood repair), and miscellaneous kinds of implicit and explicit skills.

Matthews talked about two commonly used strategies for assessing emotional intelligence—trait questionnaires and ability tests—though he cautioned that each strategy has drawbacks. He said many trait questionnaires are available and most are personality-like scales that provide scores for various emotional intelligence traits. He noted these are self-report assessments, which he thinks raises a paradox that undermines their validity. As Matthews put it, “If having good self-awareness of your emotional functioning is central to emotional intelligence, then if you lack emotional intelligence, how can your questionnaire responses be very meaningful?”

The Mayer-Salovey-Caruso Emotional Intelligence Test (the MSCEIT) is an example of an ability test for measuring emotional intelligence. The MSCEIT assesses the respondent’s ability to perceive, use, understand, and regulate emotions. The assessment uses scenarios drawn from everyday life situations to measure how well people perform tasks and solve emotional problems. For instance, the assessment includes the “Faces Subtest,” in which the test taker is presented with the face of a person showing an emotion, and the test taker rates the extent to which certain emotions are being expressed. Matthews showed an example of the face of a woman smiling. The test taker is asked to rate on a 5-point scale of “definitely not present” to “definitely present” the extent to which the face shows anger, disgust, sadness, happiness, fear, surprise, etc.

Matthews said one issue with the MSCEIT and other assessments like it is determining the “correct” response to an item. For the MSCEIT, the correct answers are determined through use of an expert panel and through collecting data from a normative sample. In Matthews’ view, neither approach is ideal, although he said that the assessment shows modest

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

correlations (.1 to .3) with a variety of criteria including life satisfaction, social skills and relationships, and coping.

Matthews and his colleagues Richard Roberts and others at ETS have been working on another assessment strategy that relies on situational judgment tests. The researchers are exploring the use of both text-based and video-based scenarios designed to evaluate how well individuals can judge the emotions of a situation. An example of a text-based scenario follows:

Clayton has been overseas for a long time and returns to visit his family. So much has changed that Clayton feels left out. What action would be the most effective for Clayton?

In the video-based format, a clip of an emotive situation is shown, and the test taker is presented with several response options. Matthews presented an example in which a person in a work situation is upset because her office is being moved around, and this has disrupted her work activities. The test taker is presented with four possible responses that the boss might make to address the employee’s complaint. In one response, the boss becomes angry, tells her that the move is important for the firm’s functioning, and that she should simply put up with it. In another, the boss is more empathetic with the employee, recognizes that the employee has some grounds for being upset, and explains the rationale behind the office move. The test taker is instructed to choose the best response. Matthews said that the work is in its early stages, but there seems to be some evidence that the results are predictive of high school GPA, well-being, and social support, even controlling for other factors.

Matthews closed by restating that emotional intelligence remains a nebulous and ill-defined construct. The field has not yet come to consensus on a definition or conceptualization of the construct, and findings from research examining its malleability—that is, the extent to which is it trainable—are inconclusive. While there are multiple strategies for assessing the construct, he thinks they are better suited for research than for any form of high-stakes testing.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

This page intentionally left blank.

Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 63
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 64
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 65
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 66
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 67
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 68
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 69
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 70
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 71
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 72
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 73
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 74
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 75
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 76
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 77
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 78
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 79
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 80
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 81
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 82
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 83
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 84
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 85
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 86
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 87
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 88
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 89
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 90
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 91
Suggested Citation:"4 Assessing Intrapersonal Skills." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 92
Next: 5 Measurement Considerations »
Assessing 21st Century Skills: Summary of a Workshop Get This Book
×
Buy Paperback | $37.00 Buy Ebook | $29.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The routine jobs of yesterday are being replaced by technology and/or shipped off-shore. In their place, job categories that require knowledge management, abstract reasoning, and personal services seem to be growing. The modern workplace requires workers to have broad cognitive and affective skills. Often referred to as "21st century skills," these skills include being able to solve complex problems, to think critically about tasks, to effectively communicate with people from a variety of different cultures and using a variety of different techniques, to work in collaboration with others, to adapt to rapidly changing environments and conditions for performing tasks, to effectively manage one's work, and to acquire new skills and information on one's own.

The National Research Council (NRC) has convened two prior workshops on the topic of 21st century skills. The first, held in 2007, was designed to examine research on the skills required for the 21st century workplace and the extent to which they are meaningfully different from earlier eras and require corresponding changes in educational experiences. The second workshop, held in 2009, was designed to explore demand for these types of skills, consider intersections between science education reform goals and 21st century skills, examine models of high-quality science instruction that may develop the skills, and consider science teacher readiness for 21st century skills. The third workshop was intended to delve more deeply into the topic of assessment. The goal for this workshop was to capitalize on the prior efforts and explore strategies for assessing the five skills identified earlier. The Committee on the Assessment of 21st Century Skills was asked to organize a workshop that reviewed the assessments and related research for each of the five skills identified at the previous workshops, with special attention to recent developments in technology-enabled assessment of critical thinking and problem-solving skills. In designing the workshop, the committee collapsed the five skills into three broad clusters as shown below:

  • Cognitive skills: nonroutine problem solving, critical thinking, systems thinking
  • Interpersonal skills: complex communication, social skills, team-work, cultural sensitivity, dealing with diversity
  • Intrapersonal skills: self-management, time management, self-development, self-regulation, adaptability, executive functioning

Assessing 21st Century Skills provides an integrated summary of the presentations and discussions from both parts of the third workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!