Measuring Human Capabilities: An Agenda for Basic Research on the Assessment of Individual and Group Performance Potential for Military Accession (2015)

11

The Research Agenda

IMPLEMENTATION

The charge to the committee requested the recommendation of a basic research agenda, for implementation by the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI), that might in time lead to improvements in the Army enlisted soldier selection process. In developing the recommended research agenda and considering an implementation strategy that includes the necessary funding level, the committee excluded possible methods of improving selection that were, in the committee’s judgment, beyond the basic research stage. However, the committee recognizes that aspects of the research topics identified in this report are already under investigation by ARI and other entities to varying degrees, while other aspects may be in the process of being developed and implemented.

As described in the report’s first chapter, throughout its work the committee recognized the importance of developing selection systems based on criteria of interest to organizational values. However, the Army’s currently used selection tools and systems apply to multiple missions, environments, and criteria that represent its organizational values, and the Army is also forward-looking in considering jobs, environments, and selection in the future. Thus the committee was instructed to think broadly about the selection of military personnel across all occupational specialties rather than to consider selection issues that might be unique to any specific outcome or function. The research recommendations, as compiled and restated in this chapter, reflect the committee’s requirement that a conceptual or empirical link could be identified between an attribute under consideration and one

or more outcomes that constitute a component of overall individual or team effectiveness. (The reader is referred back to Table 1-1 in Chapter 1 for the grid presenting the links between the research domains included in this report and many of the outcomes identified by the committee as potentially of importance to the Army.)

In considering the implementation of the research agenda, the question of the necessary funding level for future ARI basic research is of key import. Because the committee recognized that it lacked critical expertise and insight into the Army organization and missions, this report was developed on the basis that the Army would need to identify the outcomes strategically of greatest value to its mission(s), then basic research domains linked to those outcomes would become higher in priority. Funding allocations would be impacted by such a priority scheme.

Absent priorities assigned to the 10 substantive recommendations made in this report, the committee sees each of the areas as independently worthy of pursuit. The research topics have been grouped into relevant sections in the report, based upon the taxonomic system described in Chapter 1, and interrelated topics could be developed into integrative research programs. However, to produce findings that have the potential to improve the quality of Army selection decisions in the relatively near term, the committee believes all topics identified in this report should be pursued at levels commensurate with the outcomes of greatest import to the Army.

If all research topics could be pursued, a modest start would be to fund one project in each of these 10 areas. A reasonable average funding level for these projects might be $350,000 per year. We note that this funding level is consistent with the typical current funding level for basic research projects supported from ARI’s Personnel Performance and Training budget line. This funding would be exclusive to the basic research program and would not include formal validity studies or applied programs of research prior to implementation. Note that the per-project funding cited above is an average value; work in some domains can be expected to be more costly than in others, and different research strategies within a domain may be more costly than others. Equipment needs and participant payment costs are among the features that are likely to vary across domains and across projects.

Thus, a research budget of $3.5 million would support this initial plan of one project per substantive area per year. One project certainly reflects progress. But each substantive research domain is multifaceted, and multiple projects per area would permit quicker progress and potential synergies across projects. So a more ambitious plan would be to fund two projects per year in each of these 10 areas, thus suggesting a research budget of $7 million. To be clear, this represents funding for basic research. Follow-up

research moving toward operational use of new measures (e.g., field validation studies) will be necessary but is beyond the committee’s charge.

In the committee’s opinion, to implement such a program effectively and expeditiously would require a funding commitment in the range of $3.5 million per year (supporting one project per substantive area) to $7 million per year (supporting two projects per substantive area) in order to support research on potential enhancement of enlisted soldier selection.

RESEARCH TOPICS: COMMITTEE CONCLUSIONS AND RECOMMENDATIONS

For the convenience of the reader, this section of the report’s final chapter restates the conclusions and recommendations that were originally presented in each of the relevant research topic chapters (Chapters 2 through 10) and that, combined, make up the committee’s recommended research agenda for ARI to take its basic research program to the next leap forward in identifying, assessing, and assigning quality personnel.

Fluid Intelligence, Working Memory Capacity, Executive Attention, and Inhibitory Control (Chapter 2)

Committee Conclusion

The constructs of fluid intelligence (novel reasoning), working memory capacity, executive attention, and inhibitory control are important to a wide range of situations relevant to the military, from initial selection, selection for a particular job, and training regimes to issues having to do with emotional, behavioral, and impulse control in individuals after accession. These constructs reflect a range of cognitive, personality, and physiological dimensions that are largely unused in current assessment regimes. The committee concludes that these topics merit inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation: Fluid Intelligence, Working Memory Capacity, and Executive Attention

1. Research should be conducted to ascertain whether these constructs reflect a common mechanism or are highly related but distinct mechanisms.
2. Assessments reflecting the results of research into the commonality versus distinctness of these constructs should be developed for purposes of validity investigations.
3. Ultimately, the basic research results from items A and B above should be used to inform research into time-efficient, computer-automated assessment(s).

Research Recommendation: Inhibitory Control

1. Develop time-efficient, computer-automated self-report and behavioral assessments of inhibitory control capacity that demonstrate convergence with neurophysiological indices, as well as differentiation from constructs considered distinct from inhibitory control.
2. Examine the extent to which inhibitory control—as assessed through self-report, task-behavioral, and physiological response measures—predicts performance outcomes of interest (e.g., accidents, disciplinary incidents) and understand the common and unique aspects of the different assessment approaches in terms of underlying processes tapped by each and how these processes relate to performance.

Cognitive Biases (Chapter 3)

Committee Conclusion

Cognitive biases, such as confirmation bias, anchoring, overconfidence, sunk cost, availability, and others, appear broadly relevant to the military because of findings, from both the analysis of large-scale disasters and the broader literature on cognitive biases, that show how irrational decision making results from failing to reflect on choices. Research on a tendency to engage in cognitive biases as a stable individual-differences measure is limited, and there are measurement challenges that must be dealt with before operational cognitive bias assessment could be implemented. The conceptual relevance of this topic, paired with the limited research to date, which takes an individual-differences orientation, leads the committee to

conclude that cognitive biases merit inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Research should be conducted to ascertain whether various cognitive biases and heuristics are accounted for by common bias susceptibility factors or whether various biases reflect distinct constructs (e.g., confirmation bias, fundamental attribution error).
2. A battery of cognitive bias and heuristics assessments should be developed for purposes of validity investigations.
3. Research should be conducted to examine the cognitive, personality, and experiential correlates of susceptibility to cognitive biases. This should include both traditional measures of personality and cognitive abilities (e.g., the Armed Services Vocational Aptitude Battery), and information-processing measures of factors such as working memory, executive attention, and inhibitory control.
4. Research should be conducted to identify contextual factors, that is, situations in which cognitive biases and heuristics may affect thought and action, and then to develop measures of performance in such situations, for use as criteria in studies aimed at understanding how cognitive biases affect performance. The research should consider the differentiating characteristics of contexts that determine when the use of heuristics for “fast and frugal” decision making might be beneficial, and when such thinking is better thought of as biased and resulting in poor decision making.

Spatial Abilities (Chapter 4)

Committee Conclusion

A spatial ability measure, Assembling Objects (AO), is included in the Armed Services Vocational Aptitude Battery (ASVAB). Research suggests incremental validity for spatial measures over general mental ability measures in predicting important military outcomes. Research also suggests that sex differences vary across different operationalizations of spatial ability. Together, these findings suggest exploring varying approaches to the measurement of spatial abilities to ascertain whether the AO test is the best measure of spatial ability for military selection and classification. The

committee concludes that spatial ability merits inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Identify or develop measures of various facets of spatial ability, with particular attention to the role of technology to overcome prior limitations in test-item formats.
2. Examine the interrelationships among various facets of spatial ability, including but not limited to spatial relations, spatial orientation, and spatial visualization.
3. Examine sex differences on the various facets of spatial ability, as well as the degree to which sex differences are mitigated or accentuated by various forms of training on the facets of spatial ability.
4. Develop measures reflecting various work outcomes that can be used as criterion measures in evaluating the validity of various measures of spatial ability.

Teamwork Behavior (Chapter 5)

Committee Conclusion

Research has identified a number of individual-differences attributes that are broadly predictive of success in a team environment. There has also been progress in identifying attributes that when aggregated across team members (e.g., mean level of cognitive ability, minimum agreeableness), are predictive of team effectiveness. More research is needed to expand and amplify this work in the context of potential utility in military accession. The committee concludes that the teamwork knowledge, skills, abilities, and other characteristics (KSAO) domain merits inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Develop a better understanding of, and new metrics to operationalize, team outcomes and effectiveness. In addition, new technologies should be explored to better assess teamwork behaviors beyond paper-and-pencil measures.
2. Identify individual and team cognitions, affect/motivation, and behaviors that are linked to successful team outcomes and effectiveness. Essential to this is developing methods of team task analysis.
3. Identify optimal within-individual profiles that are linked to team effectiveness. This research should also consider types of team structures, tasks, and environmental conditions that moderate relationships between profile attributes and their combined influence on team processes and outcomes.
4. Investigate the effects of teamwork training and team experiences on the predictive power of individual-differences measures.

Hot Cognition: Defensive Reactivity, Emotional Regulation, and Performance under Stress (Chapter 6)

Committee Conclusion

“Hot cognition” includes the topics of defensive reactivity, emotional regulation, and performance under stress. Research and military experience suggest that the ability to perform well in situations that elicit emotional responses is important in many contexts that are relevant to the military. Research on performance has tended to underplay the role emotions can play in governing behavior, whether for good or bad. The committee concludes that the hot cognition domain merits inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Research should explore behavioral performance measures and also physiological measures of dispositional defensive reactivity, such as the eye-blink startle measure and other biological indica-

1. tors (biomarkers) of fear activation, and more generally other traits conceived as “biobehavioral.” Research should examine how biobehavioral dispositions like defensive reactivity relate to and are distinct from other personality constructs such as the Big Five (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism). In addition, research should compare the predictive validity of trait dispositions as assessed by physiological or behavioral measures in relation to survey assessments and examine how traits affect performance outcomes in differing situational contexts (e.g., impact of dispositional boldness on behavioral effectiveness in social versus affective versus workplace versus battlefield context).
2. Research should clarify how emotions and cognitions together affect human capability and performance and should expand understanding of the physiological bases for emotional regulation. Key themes include neural mechanisms of inhibition, the role of the prefrontal cortex in higher cognitive control including affective processing, and the role of the dorsal region of the anterior cingulate cortex in monitoring conflicts (e.g., conflict between emotional and cognitive influences on moral dilemma tasks).
3. Research should explore measuring emotional regulation with established forms of assessment such as rating scales, situational judgment tests, and performance measures (e.g., delay-of-gratification measures, emotional conflict tests, cooperation versus competition tasks).
4. Research should examine the conditions that improve or diminish cognition and performance under stress, in order to develop measures of susceptibility to stress.
5. Research should evaluate whether susceptibility to stress is contingent on the type of stressor (e.g., time pressure, peer pressure, fatigue) and whether there are cognitive, personality, and experiential correlates of susceptibility.

Adaptability and Inventiveness (Chapter 7)

Committee Conclusion

The military has a strong interest in adaptive behavior, expressed in terms of assessing novel problems and solving them or acting upon them effectively. Research indicates two promising lines of inquiry. The first would use measures of frequency and quality of ideas generated in open-ended tasks, which have demonstrated incremental validity over and above measures of general cognitive ability for predicting important outcomes related to work performance. The second line of inquiry would use narrow

personality constructs to predict adaptive behavior and inventive/creative problem solving. Thus, the committee concludes that idea generation measures and narrow personality measures specific to adaptability and inventiveness merit inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Compare alternative approaches to the measurement and scoring of idea generation as a cognitive measure of adaptability/inventiveness.
2. Use existing literature, theory, and empirical research to identify and develop narrow personality measures as candidates for predicting adaptive performance.
3. Develop a range of measures of relevant work criteria that reflect adaptive performance in research studies.
4. Examine the use of these personality and idea generation measures in predicting the above adaptive performance criteria.

Psychometrics and Technology (Chapter 8)

Committee Conclusion

The military has long been in the forefront of modernized operational adaptive testing. Recent research offers promise for improvements in measurement in a variety of areas, including the application and modeling of forced-choice measurement methods; development of serious gaming; and pursuing Multidimensional Item Response Theory (MIRT), Big Data analytics, and other modern statistical tools for estimating applicant standing on attributes of interest with greater efficiency. Efficiency is a key issue, as the wide range of substantive topics recommended for research in this report may result in proposed additions to the current battery of measures administered for accession purposes. The committee concludes that such advances in measurement and statistical models merit inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Potential topics of research on Item Response Theory (IRT) include the use of multidimensional IRT models, the application of rank and preference methods, and the estimation of applicant standing on the attributes of interest with greater efficiency (e.g., via automatic item generation, automated test assembly, detecting item pool compromise, multidimensional test equating, using background information in trait estimation).
2. Ecological momentary assessments (e.g., experience sampling) and dynamic interactive assessments (e.g., team interaction, gaming, and simulation) yield vast amounts of examinee data, and future research should explore the new challenges and opportunities for innovation in psychometric and Big Data analytics.
3. Big Data analytics also may play an increasingly important role as candidate data from multiple diverse sources becomes increasingly available. Big Data methods designed to find structure in datasets with many more columns (variables) than rows (candidates) might help identify robust variables, important new constructs, interactions between constructs, and nonlinear relationships between those constructs and candidate outcomes.

Situations and Situational Judgment Tests (Chapter 9)

Committee Conclusion

The ability to use judgment to interpret, evaluate, and weigh alternate courses of action appropriately and effectively is relevant to a wide variety of situations within the military. Various streams of research, including new conceptual and measurement developments in assessing situational judgment, as well as evidence of consistent incremental validity of situational judgment measures over cognitive ability and personality measures for predicting performance in various work settings, lead the committee to conclude that measures of situational judgment merit inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Develop situational judgment tests with items reflecting constructs that are otherwise difficult to assess using other tests, that are important, and that show promise for validity (e.g., prosocial knowledge, team effectiveness).
2. Consider innovative formats for presenting situations (e.g., ranging from simple text-based scenarios to dynamic and immersive computer-generated graphics), capturing examinee responses (e.g., open-ended, voice, gestures, facial expressions, eye movements, reaction times), and evaluating examinee responses (e.g., advanced natural language processing, automated reasoning, machine learning).
3. Develop and explore psychometric models and methods that can accommodate the rich array of data that innovative assessment methods for situational judgment may yield, facilitating the development of psychometrically and practically equivalent assessments, and improving reliability and testing efficiency.

Assessment of Individual Differences Through Neuroscience Measures (Chapter 10)

Committee Conclusion

A wide variety of measures fall within the domain of neuroscience (e.g., direct neuroscience measures such as electroencephalography [EEG], positron emission tomography, magnetic resonance imagery [MRI], or functional MRI [fMRI] and indirect biomarkers of neural activity such as heart rate or eye blink). These measures may take multiple roles in the Army accession process including (a) monitoring test takers for constructs such as anxiety, attention, and motivation during other assessments; (b) use in research settings as criteria for evaluating other potential assessments; and (c) use as direct selection and classification assessments. Although the third role may be well in the future in terms of technically feasible and cost-effective assessment, the first two uses have near-term promise. The committee concludes that the neuroscience domain merits inclusion in a program of basic research with the long-term goal of improving the Army’s enlisted accession system.

Research Recommendation

1. Research seeking refinement of current and future Army assessments (e.g., the Armed Services Vocational Aptitude Battery and Tailored Adaptive Personality Assessment System) through a deeper understanding of the constructs measured in selection and classification testing. In this role, biomarkers may reveal underlying neurophysiological correlates of constructs of interest (e.g., cortisol as a biomarker for anxiety). Deeper understanding of physiology has the potential to differentiate complex constructs or alternatively to reveal the relative strength of measures.
2. Independent use of biomarkers as direct selection instruments or their use in combination with traditional assessments. Research should identify biomarker correlates (e.g., consistent gaze, pupilometry, reaction time in a vigilance test) of abilities and outcomes. Test stimuli or conditions for eliciting biological responses from test takers (e.g., simulated rifle drill or other novel muscle coordination task to assess parietal-dominant brain) should be developed.

1. Monitoring candidates for attributes such as anxiety during assessment and offering training on mitigation strategies for applicants not selected on the basis of their test scores. Such attributes can contribute to bias in test scores, and success in controlling for the effects of these attributes can result in more valid assessment. The committee expects challenges in determining whether applicants’ observed performance reflects their true ability (e.g., whether appli-

1. cants are experiencing normal performance stress or an interfering level of anxiety). Additionally, we expect challenges in designing a simple and effective mitigation program.
2. Basic research to apply modern neurophysiological tools to model test-taker response data (e.g., response time distributions, answer patterns that may suggest unmotivated responding or intentional distortion).

This page intentionally left blank.