Summary

The importance of higher education has never been clearer. Educational attainment—the number of years a person spends in school—strongly predicts adult earnings, as well as health and civic engagement (National Research Council, 2012a). Yet relative to other developed nations, educational attainment in the United States is lagging, with young Americans who heretofore led the world in completing postsecondary degrees now falling behind their global peers (OECD, 2013b). As part of a broader national college completion agenda aimed at increasing college graduation rates, higher education researchers and policy makers are exploring the role of intrapersonal and interpersonal competencies in supporting student success. These sets of competencies represent two of three domains of competence outlined in a previous study (National Research Council, 2012b):

• Intrapersonal competencies involve self-management and the ability to regulate one’s behavior and emotions to reach goals.
• Interpersonal competencies involve expressing information to others as well as interpreting others’ messages and responding appropriately.
• Cognitive competencies involve thinking, reasoning, and related skills.

That study found that these competencies across all three domains were closely intertwined with—and supported the acquisition of—core knowledge and skills in science, mathematics, and English language arts. Because

it also found more available research to guide the teaching and assessment of cognitive competencies relative to the other two domains, the committee that carried out that study recommended additional research on the assessment of intrapersonal and interpersonal competencies that traditionally have not been targeted as educational goals (National Research Council, 2012b).

To address this recommendation and the national imperative to increase college completion, the National Science Foundation (NSF) commissioned the National Academies of Sciences, Engineering, and Medicine (the National Academies) to convene an ad hoc committee to conduct a study of the assessment of intrapersonal and interpersonal competencies in higher education, with the following charge:

In fulfilling this charge, the committee was tasked with undertaking three principal analytical tasks:

FRAMING THE STUDY

When interpreting this charge, the committee defined “postsecondary persistence and success” as “persistence and success in undergraduate education,” as is clearly stated in Task 1. Thus defined, persistence and

success are reflected in such measures as retention from one semester or 1 year to the following semester or year, retention and/or success in STEM, grade point average (GPA), and graduation. This definition followed on a prior, related report, which found that educational attainment was strongly predictive of labor market success, a finding that was upheld even with rigorous research approaches designed to approximate random assignment (National Research Council, 2012b). The committee interpreted “especially in STEM” in Task 1 to mean that it should give special attention to research on the role of intra- and interpersonal competencies in supporting persistence and completion within STEM majors. However, the committee found insufficient evidence to reach any conclusions specifically about these majors.

The committee adopted a broad definition of “competency” to include a range of attitudes, behaviors, beliefs, and dispositions that reside within the individual student and that may also be influenced by college environments and contexts. These competencies incorporate both intrapersonal and interpersonal dimensions, as well as cognitive dimensions. In light of the growing diversity of the undergraduate student population and its charge to focus on persistence and success especially in STEM, the committee viewed diversity and inclusion as central themes of its work. The committee gave special attention to research on student groups that have historically experienced lower persistence and success than other groups, both in postsecondary education generally and in STEM specifically. These include three racial/ethnic minority groups (black, Hispanic, and American Indian), along with first-generation college students, students from low-income families, and, in certain STEM disciplines, women. In this report, the committee refers to these diverse student groups as “underrepresented groups” and to black, Hispanic, and American Indian student groups as “underrepresented minorities.”

When considering the meaning of Task 2, the committee defined “examine available assessments” primarily as a charge to examine available assessment methods that are currently being used or could potentially be used to assess the competencies identified in this report. The committee also reviewed prominent existing assessment instruments. Finally, while focusing primarily on competencies related to persistence and success in undergraduate education, the committee recognized that higher education leaders have identified certain intrapersonal and interpersonal competencies (e.g., teamwork, ethics, and intercultural sensitivity) as desired learning outcomes for graduates. In response to this development, the committee briefly explored research on competencies identified as important learning outcomes.

Identifying Competencies for College Success

To address its charge to identify competencies that are related to persistence and success in undergraduate education and can be enhanced through intervention, the committee conducted an extensive search of the literature. It found that the research base is limited, especially in the area of relationships between interpersonal competencies and college success. There were also gaps related to other important topics.

Conclusion: Only limited research has been conducted to date on the potential relationships between various intra- and interpersonal competencies and students’ college success. There are major gaps in the research evidence:

• Little research is available on the possible relationship between interpersonal competencies and students’ college success.
• The available research has been conducted almost entirely in 4-year institutions; very little experimental evidence is available on the possible relationship between intra- and interpersonal competencies and students’ success in community colleges.
• There is a paucity of evidence on the possible relationships between intra- and interpersonal competencies and the success of students intending to major in science, technology, engineering, and mathematics fields.

RECOMMENDATION 1: Federal agencies and foundations should invest in research exploring the possible relationships between various intra- and interpersonal competencies and students’ college success. To address gaps in the research base, these investments should include support for research examining

• how interpersonal competencies may be related to student success in 4-year colleges;
• how intra- and interpersonal competencies may be related to student success in community colleges; and
• how intra- and interpersonal competencies may be related to student success in 2- and 4-year science, technology, engineering, and mathematics programs and majors.

Promising Competencies

Based on its review of the limited available research, the committee identified promising competencies that appear to be related to college

success. Correlational research suggests that, among the competencies reviewed by the committee, the most robust predictor of college success is dispositional conscientiousness—the tendency to be self-controlled, responsible to others, hardworking, persevering, rule abiding, and achievement oriented. Conscientiousness is closely related to other constructs, such as self-control, self-discipline, persistence, and grit. Indeed, scores on measures of conscientiousness are nearly as predictive of college success as are measures of general cognitive ability. Yet evidence supports conscientiousness as a deeply ingrained dispositional trait that is difficult to change, at least in the short term. Nevertheless, a few interventions have targeted specific behaviors associated with conscientiousness, to date yielding significant but small effects on college success.

Conclusion: Beyond cognitive factors, correlational research has shown that individual differences in intrapersonal competencies predict college success and completion. These competencies include the broad personality trait of conscientiousness. Although an individual’s relative standing on conscientiousness tends to be highly stable over time, some interventions have successfully targeted task management and other specific manifestations of this trait.

Moving beyond dispositions and traits, the committee reviewed the available research on the relationships between other intra- and interpersonal competencies and college success. Through this process, the committee identified eight intrapersonal competencies. (Reflecting a lack of research evidence, this list includes no purely interpersonal competencies.) These eight competencies have been studied using a range of methods, and the committee identified them based on correlational and experimental research. However, the committee judged the strength of the evidence related to each competency based exclusively on research that has developed and tested them through interventions using random assignment.

Conclusion: The limited intervention studies conducted to date have generated promising evidence that the competencies of sense of belonging, growth mindset, and utility goals and values are related to college success and are malleable in response to interventions. Available intervention studies provide more modest evidence that five other competencies are similarly related to college success and malleable, yielding a total of eight identified competencies:

1. Behaviors related to conscientiousness—behaviors related to self-control, hard work, persistence, and achievement orientation.

2. Sense of belonging—a student’s sense that he or she belongs at a college, fits in well, and is socially integrated.
3. Academic self-efficacy—a student’s belief that he or she can succeed in academic tasks.
4. Growth mindset—a student’s belief that his or her own intelligence is not a fixed entity, but a malleable quality that can grow and improve.
5. Utility goals and values—personal goals and values that a student perceives to be directly linked to the achievement of a future, desired end.
6. Intrinsic goals and interest—personal goals that a student experiences as rewarding in and of themselves, linked to strong interest.
7. Prosocial goals and values—the desire to promote the well-being or development of other people or of domains that transcend the self.
8. Positive future self—a positive image or personal narrative constructed by a student to represent what kind of person he or she will be in the future.

Interventions that often required very little time and money to implement have helped students develop these eight competencies. Some of these interventions have been particularly effective for underrepresented student groups that are most at risk for academic failure.

Conclusion: Notably, evidence shows that low-cost interventions aimed at developing sense of belonging, growth mindset, and utility goals and values have sometimes generated the largest benefits for underrepresented student groups that are most at risk for academic failure. Although encouraging, this evidence is limited and recent, and further research is needed to replicate and extend it.

RECOMMENDATION 2: Federal agencies and foundations should invest in intervention research using random assignment and research employing a range of other methods to understand better the competencies identified in this report, their relationship to college success, and the mechanisms through which they operate to improve college success. Research focused on supporting the college success of underrepresented student groups should be a priority.

RECOMMENDATION 3: Colleges and universities should support the intervention research proposed in Recommendation 2 by facilitating the implementation and evaluation of random-assignment interventions, thereby gaining valuable information about their students

and building the knowledge base on effective interventions needed to increase student retention and success.

Sensitivity to Context and Subgroup Effects

The committee’s review of the research indicates that issues of race, ethnicity, gender, social class, and culture need to be carefully considered when educators, administrators, researchers, and policy makers think about competencies and their contribution to college success. Certain competencies, and the problems they might help address or solve, may be more salient or useful for certain groups of students than others. Underrepresented minority students, for example, may bring to college such competencies as a strong racial or cultural identity that may help them navigate academic or social environments. More research also is needed to examine how particular educational and cultural contexts currently influence the development of motivation, intrinsic goals and interest, and other intra- and interpersonal competencies among underrepresented groups (minority students, first-generation college students, students from low-income families, and women).

Conclusion: Certain competencies develop and function differently for different groups and within different cultural and educational contexts. For example, although a strong sense of belonging in college is an important factor for success among underrepresented student groups, members of these groups may find it difficult to develop this competency if they experience campus environments that are discriminatory, negative, or unwelcoming.

RECOMMENDATION 4: To help reduce disparities in college success among student groups, institutions of higher education should evaluate and improve their social and learning environments to support the development of the eight identified competencies, especially among underrepresented student groups.

ASSESSMENT METHODS FOR THE IDENTIFIED COMPETENCIES

The committee reviewed the nature and quality of existing competency assessments, focusing particularly on the eight identified competencies, together with research and professional standards related to the overall process of developing, validating, and implementing assessments, and interpreting, evaluating, and using the assessment results. The test development practices used to create assessments of cognitive knowledge and skills that meet these professional standards are equally applicable to intra- and interpersonal competency assessments.

Current Assessments

The committee examined the assessments used in the intervention studies targeting the eight identified competencies and commissioned a literature search on measurement of these competencies. Drawing on both sources, the committee also identified and closely analyzed a small sample of established assessment instruments targeting one or more of the eight competencies. Overall, the review revealed that self-report methods, with their known limitations, predominated in the assessments of the eight competencies. Analysis of the quality of the assessments used in the intervention studies revealed spotty attention to reliability and almost no reported evidence of validity or fairness. However, more evidence of assessment quality was found for some established assessment instruments used in higher education research, particularly those that have received funding for assessment research and development. These instruments provide evidence on reliability and validity but lack evidence on fairness. Assessments developed by professional testing companies provide even more evidence of quality, including fairness data; however, these assessments target a wider range of competencies, only partially addressing some of the eight competencies.

Conclusion: Most current assessments of the eight identified competencies are uneven in quality, providing only limited evidence to date that they meet professional standards of reliability, validity, and fairness.

Assessments for High-Stakes Purposes

Developers of all types of assessments, whether they aim to measure cognitive, intrapersonal, or interpersonal competencies, must exercise particular care when an assessment will serve a high-stakes purpose. Assessments are considered high stakes when their results carry serious consequences for individuals or institutions.

Conclusion: The development and validation of assessments of intra- and interpersonal competencies for high-stakes purposes is a rigorous, time-consuming, and expensive process that depends critically on expertise in assessment and psychometrics. Validity, reliability, and fairness are essential considerations in evaluating assessment quality.

RECOMMENDATION 5: When developing and validating intra- and interpersonal competency assessments to be used for high-stakes purposes, stakeholders in higher education (e.g., faculty, administrators, student services offices) should comply with professional standards,

legal guidelines, and best practices to enable appropriate interpretations of the assessment results for particular uses.

RECOMMENDATION 6: Institutions of higher education should not make high-stakes decisions based solely on current assessments of the eight identified competencies, given the relatively limited research to date demonstrating their validity for predicting college success.

Assessments for Low-Stakes Purposes

Researchers and practitioners in higher education also use assessments for low-stakes purposes, such as to evaluate the quality of interventions, policies, and instructional practices or simply to monitor student change over time. When used for these low-stakes purposes, assessments need not meet the high evidentiary requirements of individual high-stakes student assessments, such as college admissions tests. Professional testing standards clearly state that the amount and type of evidence needed to support a test’s validity may vary depending on the use or interpretation of the test scores. At the same time, even when assessments are not used for high-stakes purposes, they need to be sensitive to the competencies they are intended to measure.

Conclusion: Even low-stakes uses of intra- and interpersonal competency assessments require attention to validity, reliability, and fairness, although they need not meet the high evidentiary requirements of high-stakes assessments.

RECOMMENDATION 7: Those who develop, select, or use intra- and interpersonal competency assessments should pay heed to, and collect evidence of, validity, reliability, and fairness as appropriate for the intended high-stakes or low-stakes uses.

Definition of Constructs Being Assessed

After reviewing both general principles for assessment development and use and recent research on measurement of the eight identified competencies, the committee concluded that defining each competency clearly and comprehensively is a critical first step in developing high-quality assessments. Clear definitions are especially important in light of the wide variety of terms used for these competencies. For example, conscientiousness, grit, and persistence are closely related constructs, despite being named differently. In fact, assessments of all of these constructs may contain very similar

item content. Conversely, assessments bearing the same name may in fact comprise items measuring different competencies.

Conclusion: High-quality assessment begins with a clear definition of the competency to be measured, and identifies how the assessment will be used and what kinds of inferences it will support.

Competency definitions guide assessment development and selection by making it possible to evaluate how well the assessment represents the competency it is intended to measure, thereby supporting appropriate inferences about the construct for particular uses. High-quality assessments avoid construct underrepresentation, represent the breadth and depth of the competency, and minimize any distortions caused by competency-irrelevant influences.

Innovative Methods and Technologies for Assessment

Self-report measures, such as those frequently used to assess the eight identified competencies, have several limitations. First, individuals responding to both high- and low-stakes assessments may be motivated to present themselves in a favorable light. In addition, people often express themselves on a response scale in habitual or characteristic ways, such as tending to mark the extremes (e.g., “strongly agree” or “strongly disagree”) or to agree or respond positively regardless of the question. In addition, respondents’ tendency to compare themselves with those around them can compromise the use of the responses to measure growth or to compare groups of individuals because such comparisons depend on an absolute rather than a relative standard. Because self-report measures are widely used, these limitations affect a broad swath of current intra- and interpersonal competency assessments.

Recent research has identified various methods that can mitigate these limitations. For example, the use of forced-choice and ranking methods for collecting self-evaluations avoids response-style bias by circumventing traditional rating scales altogether. The use of anchoring vignettes also addresses response-style bias by having raters make use of detailed objective anchors, and may potentially deal with reference group effects as well. Other nontraditional measures include situational judgment tests, as well as games or simulations, which avoid many of the documented limitations of self-ratings. Further research is needed to develop, extend, and refine these and other promising new approaches.

Conclusion: Most existing assessments of the eight identified competencies, as well as many existing assessments of other intra- and interpersonal competencies, use self-report measures, which have well-

documented limitations. These limitations may constrain or preclude certain uses of the results. Innovative approaches for assessing intra- and interpersonal competencies can address these limitations.

RECOMMENDATION 8: Federal agencies and foundations should support additional research, development, and validation of new intra- and interpersonal competency assessments that address the shortcomings of existing measures.

Fairness in Assessment

The 2014 Standards for Educational and Psychological Testing make clear that fairness to all individuals for whom an assessment is intended should be a driving concern throughout the development, validation, and use of all types of assessments. Assessment development should minimize construct-irrelevant characteristics that would interfere with the ability of some individuals or subgroups to show their standing on a competency or lead to individual or subgroup differences in the meaning of test scores. Whenever differences in subgroup scores are observed, follow-up research may be needed to examine the reasons, the potential sources of bias, and the comparability of score interpretations across individuals and subgroups in light of the intended uses of the assessment results. The committee applied these fairness principles in its review of current assessments of the eight identified competencies:

Conclusion: Despite the ever-increasing diversity of undergraduate student populations, attention to fairness for diverse populations is often inadequate in the development, validation, and use of current assessments of the eight identified competencies.

Because these fairness principles apply broadly to all types of assessments, the committee recommends:

RECOMMENDATION 9: Researchers and practitioners in higher education should consider evidence on fairness during the development, selection, and validation of intra- and interpersonal competency assessments.

Consideration of Contextual Factors

Self-, peer, or instructor ratings of such an intrapersonal competency as conscientiousness or such an interpersonal competency as teamwork may vary depending on local norms (e.g., reference group effects). In ad-

dition, contextual variables may mediate or moderate the relationships between intra- and interpersonal competencies and educational outcomes. For example, an intervention intended to develop sense of belonging may be effective only for underrepresented student groups.

Conclusion: Appropriate interpretation of the results of intra- and interpersonal competency assessments requires consideration of contextual factors such as student background, college climate, and department or discipline.

RECOMMENDATION 10: Higher education researchers and assessment experts should incorporate data on context (e.g., culture, climate, discipline) into their analyses and interpretations of the results of intra- and interpersonal competency assessments.

Implementing this recommendation will require that higher education researchers use appropriate statistical analyses that incorporate data on context when examining assessment results. Such analyses include use of multilevel statistical models, measurement invariance analyses, application of differential item functioning, and mediator and moderator analyses. These analyses can enhance understanding of the complex interactions and processes entailed in students’ individual competencies and of features of higher education contexts that contribute to students’ persistence and success. Multiple measures also can be used to minimize the possibility that inferences about a student’s intra- and interpersonal competencies are due to a particular measurement approach.

ASSESSMENT USES AND STAKEHOLDERS

Addressing its charge to prioritize the use of assessments of intra- and interpersonal competencies, the committee reviewed research on how institutions of higher education are using assessments of cognitive, intrapersonal, and interpersonal competencies. This review revealed four major uses of such assessments in higher education:

1. selection and placement of individual students;
2. formative improvement of local educational processes, practices, and programs;
3. research and evaluation supporting knowledge generation; and
4. accountability.

Assessments for these four purposes are carried out by a variety of stakeholders, including families, K-12 schools, faculty members, college admin-

istrators, accreditors, and state and federal policy makers. To understand how these stakeholders presently and potentially could use data resulting from these assessments, the committee reviewed relevant higher education literature and reports on current practice.

Assessment Processes Supporting Student Success

Different higher education stakeholders may have different needs for assessments of intra- and interpersonal competencies, depending on the immediacy of those needs, the purposes to be served by the data, and stakeholders’ assessment-related knowledge and skills. Variations in assessment uses necessitate different assessment instruments, different levels of evidence and aggregation, and different kinds of buy-in for the assessment process and its uses. It is important to consider these contextual aspects of the assessment process when implementing an intra- or interpersonal competency assessment in practice.

Conclusion: Assessments of intra- and interpersonal competencies in higher education are most valuable for supporting student success when their selection, design, analysis, and interpretation are guided by stakeholder information needs, intended uses, and users.

RECOMMENDATION 11: Leaders in higher education should select, design, analyze, and interpret data from assessments of intra- and interpersonal competencies based on stakeholder information needs, intended uses, and users.

The research literature contains convincing evidence that institutions of higher education can benefit from using assessments for both institutional improvement and accountability purposes, and these uses can ultimately be mutually reinforcing. However, assessment processes that emphasize improvement tend to garner more institutional support, including faculty buy-in, relative to those emphasizing accountability. Indeed, some administrators are concerned that external accountability mandates may focus institutional conversations about assessment on bottom-line compliance rather than institutional improvement, especially given limited assessment resources. College stakeholders also tend to be more receptive to assessment processes when they are internally derived, sensitive to specific institutional and disciplinary contexts, and driven by a belief that the assessment process can serve the goal of improving student learning outcomes. Therefore, institutional improvement requires planning for needed resources and putting systems in place to support moving assessments from data collection to improvement processes.

Conclusion: Assessments are more likely to be implemented and used by stakeholders to improve student success when they are motivated by internal institutional improvement purposes than when they are motivated by accountability purposes.

In one example, assessment results were used to catalyze multiple stakeholders’ collaborative work toward the shared goal of improving students’ leadership abilities. In another example, university advisers and student affairs staff used assessment data individually with students to tailor support services, while central administrators found the data useful to support strategic initiatives aimed at retaining diverse and underprepared students through graduation. Overall, research has highlighted the need for multiple stakeholders at various levels (students, staff, faculty members, administrators) to work together in an assessment process if they wish to effect pervasive change on a college campus.

Conclusion: Assessments are more likely to contribute to student retention and completion if efforts to use their results involve stakeholders at multiple levels of the organization (e.g., student support services, faculty, diversity officers, administrators) as opposed to involving individual stakeholders acting alone.

Support for Stakeholders’ Assessment Capacity

Administrators and faculty in institutions of higher education may not have specialized training or expertise in educational assessment with regard to instrument design and selection, test administration, data analysis, or the best uses of assessment data. Yet while some stakeholders on campus, such as institutional researchers and assessment experts, can help with educating the broader campus community about assessment, they may not be familiar enough with the specific issues involved in intra- and interpersonal competency assessment. Therefore, training targeted at specific stakeholders may be necessary for the full value of these assessments to be realized. In addition, although data on intra- and interpersonal competencies can potentially add substantial value to efforts to enhance the success of underrepresented student groups, faculty may not be familiar with this particular use of the data.

Conclusion: Some stakeholders in higher education will require support and training to develop the knowledge and skills needed to select, use, and interpret data from assessments to improve student success in higher education. Such training can also help stakeholders understand how these assessments can contribute to the success of underrepre-

sented student groups in particular and how to engage stakeholders who are resistant to assessment in general.

Research Needs

As noted above, it is important when implementing an assessment in a college or university to consider the contextual aspects of the assessment as a process. Research has yielded preliminary evidence of the importance of the eight identified competencies to success in college, and case studies of the use of cognitive assessment data by institutions of higher education for purposes of institutional and instructional improvement also are widely available. By contrast, evidence on how data from assessments of the eight identified competencies, or other intra- or interpersonal competencies, can be used for these purposes is relatively sparse. As additional assessments of these competencies take place on college campuses, they may yield a more robust understanding of how such assessments can lead to improvement within specific institutional, disciplinary, and student contexts.

Conclusion: Limited evidence is available from an organizational science perspective on how stakeholders in higher education can use data on intra- and interpersonal competencies for improvement and evaluation purposes.

RECOMMENDATION 12: To broaden understanding of how assessments of intra- and interpersonal competencies can lead to greater student retention and success, institutions of higher education should study and report on their use of these assessments for improvement purposes (e.g., enhancing student support services, developing underrepresented students’ sense of belonging, improving courses, identifying effective programs).

INTRA- AND INTERPERSONAL COMPETENCIES AS COLLEGE OUTCOMES

The intra- and interpersonal competencies of ethics, lifelong learning/career orientation, intercultural/diversity competence, civic engagement/citizenship, communication, and teamwork have been identified as valued outcomes of college education. Although it might seem intuitive that these competencies would predict academic success, there is little evidence to date that these desired outcomes for graduates are actually improving in college itself and also contribute to persistence, GPA, and graduation. There simply are too many large gaps in the research literature and in the available data

to say with any certainty that these competencies do or do not matter for students’ college success.

Conclusion: To date, only limited research has been conducted on the intra- and interpersonal competencies that have been identified as important learning outcomes for college graduates. Therefore, little is known about whether and under what conditions these competencies are related to persistence and success in college.

RECOMMENDATION 13: Federal agencies and foundations should invest in research examining whether, and under what conditions, the intra- and interpersonal competencies identified as outcomes for college graduates may also be related to students’ persistence and success in college.

When considering the research needs, the committee identified three issues. First, the state of measurement of most of these competencies is still markedly underdeveloped. This issue echoes the committee’s conclusion that most current assessments of the eight competencies showing some evidence of a relationship with students’ college success are uneven in quality and reinforces the need for further assessment research and development, as called for in Recommendation 8. Second, much theoretical and conceptual work remains to be done before statistical analysis is undertaken to explore potential areas of overlap between competencies identified as college outcomes and predictors of college persistence. Finally, the committee endorses a broad research agenda incorporating multiple methods to better understand the role of these competencies in students’ college success.