Measuring
Human Capabilities
AN AGENDA FOR BASIC RESEARCH ON THE ASSESSMENT
OF INDIVIDUAL AND GROUP PERFORMANCE POTENTIAL
FOR MILITARY ACCESSION
Committee on Measuring Human Capabilities:
Performance Potential of Individuals and Collectives
Board on Behavioral, Cognitive, and Sensory Sciences
Division of Behavioral and Social Sciences and Education
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract/Grant No. W-911NF-12-1-0504 between the National Academy of Sciences and the Department of the Army. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-31717-7
International Standard Book Number-10: 0-309-31717-7
Library of Congress Control Number: 2015934162
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2015 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Research Council. (2015). Measuring Human Capabilities: An Agenda for Basic Research on the Assessment of Individual and Group Performance Potential for Military Accession. Committee on Measuring Human Capabilities: Performance Potential of Individuals and Collectives, Board on Behavioral, Cognitive, and Sensory Sciences, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
COMMITTEE ON MEASURING HUMAN CAPABILIITES:
PERFORMANCE POTENTIAL OF
INDIVIDUALS AND COLLECTIVES
PAUL R. SACKETT (Chair), Department of Psychology, University of Minnesota, Minneapolis
GEORGIA T. CHAO, Eli Broad Graduate School of Management, Michigan State University
ANN DOUCETTE, The Evaluators’ Institute, The George Washington University
RANDALL W. ENGLE, School of Psychology, Georgia Institute of Technology
RICHARD J. GENIK II, Department of Biomedical Engineering, Wayne State University College of Engineering, and School of Medicine, Detroit, MI
LEAETTA HOUGH, Dunnette Group, Ltd., Saint Paul, MN
PATRICK C. KYLLONEN, Center for Academic and Workforce Readiness and Success, Educational Testing Service, Princeton, NJ
JOHN J. MCARDLE, Department of Psychology, University of Southern California
FREDERICK L. OSWALD, Department of Psychology, Rice University
STEPHEN STARK, Department of Psychology, University of South Florida
WILLIAM J. STRICKLAND, Human Resources Research Organization, Alexandria, VA
CHERIE CHAUVIN, Study Director
TINA M. WINTERS, Associate Program Officer
RENÉE L. WILSON GAINES, Senior Program Assistant
BOARD ON BEHAVIORAL, COGNITIVE, AND SENSORY SCIENCES
SUSAN T. FISKE (Chair), Department of Psychology and Woodrow Wilson School of Public and International Affairs, Princeton University
LAURA L. CARSTENSEN, Department of Psychology, Stanford University
JENNIFER S. COLE, Department of Linguistics, University of Illinois at Urbana–Champaign
JUDY DUBNO, Department of Otolaryngology-Head and Neck Surgery, Medical University of South Carolina
ROBERT L. GOLDSTONE, Department of Psychological and Brain Sciences, Indiana University
DANIEL R. ILGEN, Department of Psychology, Michigan State University
NINA G. JABLONSKI, Department of Anthropology, Pennsylvania State University
JAMES S. JACKSON, Institute for Social Research, University of Michigan
NANCY G. KANWISHER, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
JANICE KIECOLT-GLASER, Department of Psychology, Ohio State University College of Medicine
BILL C. MAURER, School of Social Sciences, University of California, Irvine
JOHN MONAHAN, School of Law, University of Virginia
STEVEN E. PETERSEN, Department of Neurology and Neurological Surgery, Washington University in St. Louis School of Medicine
DANA M. SMALL, Department of Psychiatry, Yale Medical School
TIMOTHY J. STRAUMAN, Department of Psychology and Neuroscience, Duke University
ALLAN R. WAGNER, Department of Psychology, Yale University
JEREMY M. WOLFE, Brigham and Women’s Hospital, Departments of Ophthalmology and Radiology, Harvard Medical School
BARBARA A. WANCHISEN, Director
TENEE DAVENPORT, Program Coordinator
Preface
In the face of increasing pressure to improve the productivity of the Army’s workforce, the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) approached the National Research Council (NRC) to develop an agenda for basic research on effective measurement of human capability with the goal of enhancing the military’s selection and assignment process. ARI requested a research agenda to guide policy, procedures, and research related to the measurement of individual capability and the combination of individual capabilities to create collective capacity to perform.
In response to the request from ARI, the NRC established the Committee on Measuring Human Capabilities: Performance Potential of Individuals and Collectives, under the oversight of the Board on Behavioral, Cognitive, and Sensory Sciences. This report is the work of that committee and presents the committee’s final conclusions and recommendations.
Members of the committee were volunteers carefully selected by the NRC to cover a spectrum of relevant academic specialties and to bring expertise in both basic research and practical applications. Several committee members have had significant experience with historical and current assessment programs utilized in the military enlistment process as well as outside the military.
The study was conducted in two phases over a 30-month period, during which the committee met a total of five times and hosted a public workshop, the summary of which was published in 2013. The study’s first phase focused entirely on planning and hosting the public workshop, as well as the subsequent summary publication. The study’s second phase was designed to allow the committee to consider specific research areas
presented at the original workshop (as well as some areas that were not included due to time constraints or the availability of key presenters) in order to develop consensus findings and recommendations in accordance with the study’s statement of task. In considering the most promising areas of research presented during the workshop, the committee membership was altered and expanded during the second phase to supplement the expertise of the original workshop planning committee members.
The recommendations presented in this report focus on an agenda for basic research that is likely to develop into a viable applied research program. In the course of preparing this report, each committee member took an active role in drafting chapters, leading discussions, and reading and commenting on successive drafts. The committee deliberated all aspects of this report, and its final content is the result of the members’ tremendous effort, dedication, and interest in developing improved assessments of performance potential that are specifically relevant to the U.S. military services’ selection and assignment processes.
Paul R. Sackett, Chair
Cherie Chauvin, Study Director
Committee on Measuring Human Capabilities:
Performance Potential of Individuals and Collectives
Acknowledgments
This study was sponsored by the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI). The committee wishes to thank Gerald Goodwin, chief of foundational science at ARI, and his entire research team for their support and enthusiasm for this project. The committee also wishes to thank Paul Gade, George Washington University and formerly at ARI, for his continued interest in ARI’s work and in this study in particular. The active participation of ARI representatives (current and former) in the committee’s data gathering sessions played a crucial role in the committee’s understanding of ARI’s needs and interests. Specifically, these discussions aided the committee’s awareness of the challenges and opportunities in the process of selection and assignment across the armed services.
In addition to the insightful presentations of invited experts during the study’s public workshop held April 3-4, 2013, which are summarized in the National Research Council (NRC) report, New Directions in Assessing Performance Potential of Individuals and Groups: Workshop Summary, the committee benefited from several subsequent presentations that explored certain topics in more depth and detail. The committee thanks the following individuals for their expert presentations during the study’s second phase: Christopher Codella, IBM; Mica Endsley, U.S. Air Force; Richard Landers, Old Dominion University; David Lubinski, Vanderbilt University; and Wim Van der Linden, CTB/McGraw-Hill. And finally, Dr. Michael I. Posner, Department of Psychology, University of Oregon (Emeritus), was appointed as an agent of the committee to provide subject matter expertise to assist
the committee in verifying the scientific integrity of technical aspects of the final report.
Among the NRC staff, special thanks are due to Barbara A. Wanchisen, director, Board on Behavioral, Cognitive, and Sensory Sciences, who provided oversight to the study process. Additionally, special thanks to Tina Winters, associate program officer, who was instrumental in organizing data gathering opportunities for the committee and facilitating the development of the committee’s final report. Renée Wilson Gaines, senior program assistant, also provided critical support to the study process and answered the committee’s administrative and logistical needs. Ellen Kimmel and Rebecca Morgan, of the NRC Research Center, were an invaluable resource over the duration of the project as they identified relevant research to answer questions on historical precedent, current developments, and future forecasts. We also thank NRC consultant Robert Katt for final editing of the manuscript. And finally we thank the executive office reports staff of the Division of Behavioral and Social Sciences and Education, especially Kirsten Sampson Snyder, who managed the review process, and Yvonne Wise, who oversaw the final publication process.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the summary meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this summary: Janis Cannon-Bowers, Institute for Simulation and Training, University of Central Florida; Scott T. Grafton, Department of Psychological and Brain Sciences, University of California at Santa Barbara; Michael A. McDaniel, Department of Management, Virginia Commonwealth University; James L. Mohler, College of Technology, Purdue University; James L. Outtz, President, Outtz and Associates, Washington, DC; Christopher Patrick, Psychology Department, Florida State University; Thomas Redick, Department of Psychological Sciences, Purdue University; Michael G. Rumsey, retired, Chief Personnel Assessment Research Unit, U.S. Army Research Institute for the Behavioral and Social Sciences; Ghebre (Gabe) E. Tzeghai, Corporate Research and Development and Innovation, Procter & Gamble Company; Howard M. Weiss, School of Psychology, Georgia Institute of Technology; and Carl E. Wieman, Department of Physics, Stanford University.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the
report nor did they see the final draft of the report before its release. The review of this report was overseen by Daniel R. Ilgen, Department of Psychology and Management, Michigan State University, and Floyd E. Bloom, Molecular and Integrative Neuroscience Department, The Scripps Research Institute (Emeritus). Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Paul R. Sackett, Chair
Cherie Chauvin, Study Director
Committee on Measuring Human Capabilities:
Performance Potential of Individuals and Collectives
This page intentionally left blank.
Contents
Setting the Stage: The Current Army Enlisted Soldier Accession System
An Overview of the Research Agenda
SECTION 2: IDENTIFICATION AND MEASUREMENT OF NEW PREDICTOR CONSTRUCTS
2 Fluid Intelligence, Working Memory Capacity, Executive Attention, and Inhibitory Control
Working Memory Capacity and Executive Attention
IARPA’s Cognitive Bias Mitigation Program
Testing Spatial Abilities for Military Entrance
SECTION 3: IDENTIFICATION AND PREDICTION OF NEW OUTCOMES
Team Outcomes: Defining Team Effectiveness
Teamwork Processes and Emergent States of Teams
Team Member Inputs: Selecting and Classifying Individuals for Effective Teams
SECTION 4: HYBRID TOPICS WITH JOINT FOCUS ON NEW PREDICTOR CONSTRUCTS AND PREDICTION OF NEW OUTCOMES
6 Hot Cognition: Defensive Reactivity, Emotional Regulation, and Performance under Stress
Defensive Reactivity/Fearfulness versus Fearlessness/Boldness
7 Adaptability and Inventiveness
Adaptability/Inventiveness as an Individual-Difference Noncognitive Variable
SECTION 5: METHODS AND METHODOLOGY
8 Psychometrics and Technology
9 Situations and Situational Judgment Tests
Subgroup Differences and Adverse Impact
10 Assessment of Individual Differences Through Neuroscience Measures
Designing Better Tests and Testing Environments