Human-AI Teaming State-of-the-Art and Research Needs (2022) / Chapter Skim
Currently Skimming:
10 HSI Processes and Measures of Human-AI Team Collaboration and Performance
10 HSI Processes and Measures of Human-AI Team Collaboration and Performance
Pages 69-84
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 69... ...
TAKING AN HSI PERSPECTIVE IN HUMAN-AI TEAM DESIGN AND IMPLEMENTATION The committee notes that, to date, HSI methods have had limited application to the design of human-AI teams. This is largely attributable to the fact that AI systems are currently being developed primarily in a research-anddevelopment context and for non-military applications, in which HSI methods are not commonly applied.
|
From page 70... ...
• There is limited research and guidance to support analysis, design, and evaluation of human-AI teams to ensure resilient performance under challenging conditions at the boundaries of an AI system's capabilities. Research Needs The committee recommends addressing the following research objective for improved HSI practice relevant to human-AI teaming.
|
From page 71... ...
Key Challenges and Research Gaps The committee finds that an improved ability to determine requirements for human-AI teams, particular those that involve ML-based AI, is needed. Research Needs The committee recommends addressing the following research objective for improved HSI requirements relevant to human-AI teaming.
|
From page 72... ...
How can competency boundaries of both humans and AI systems be mapped so that degraded and potentially dangerous phases of operational systems can be avoided? RESEARCH TEAM COMPETENCIES To address the gap in understanding how AI systems could and should influence requirements and the design of systems that support human work, particularly in settings that are high in uncertainty, the committee finds that a new approach is needed for the formation of research teams to tackle such problems.
|
From page 73... ...
A systems perspective is needed to create successful human-AI teams that will be effective in future multidisciplinary operations, and this will require synergistic work across multiple disciplines that cannot be achieved through a siloed approach. Exploration and evaluation of mechanisms for achieving successful team collaboration in human-AI development teams are needed.
|
From page 74... ...
In the committee's judgment, as there are for aircraft, there will need to be an AI maintenance workforce whose jobs entail database curation, continual model accuracy and applicability assessment, model retraining thresholds, and coordination with testing personnel. In the committee's judgment, the USAF should create an AI maintenance workforce, which, if done correctly, could be the model for both other military branches and commercial entities.
|
From page 75... ...
. Such scenarios, though predominantly occurring in the civilian domain, have clear relevance for military operations, and occur not only in computer vision applications of AI but also in natural language processing (Morris et al., 2020)
|
From page 76... ...
. Although such a mistake seems relatively benign, there have also been several high-profile incidents in which a Tesla crashed broadside into a tractor trailer or hit a barrier head on, killing the drivers; so the combination of significant AI blind spots and human inattention can be deadly (NTSB, 2020)
|
From page 77... ...
in AI systems, how to identify, measure, and mitigate concept drift is still very much an open research question. Living labs involving disaster management may form suitable surrogates for research on multi-domain operations human-AI teams.
|
From page 78... ...
Research Needs The committee recommends addressing the following research objective for developing testbeds to support human-AI teaming research-and-development activities. Research Objective 10-7: Human-AI Team Testbeds.
|
From page 79... ...
. Figure 10-4 shows pertinent measures for evaluating human-AI teams, including overall team performance, team knowledge structures, team processes, team efficiency measures, and team sustainability considerations.
|
From page 80... ...
• Although emerging measures of trust, mental models, and explanation quality are important additions for evaluation of people's understanding and level of trust in AI systems, there is a growing proliferation of alternative methods for measuring each of these constructs. The reliability and validity of these alternative methods need to be determined.
|
From page 81... ...
These findings highlight that, if not conducted in a thoughtful manner, agile software processes may limit the ability to produce coherent, innovative software solutions that depend on a comprehensive understanding of mission and performance requirements. By emphasizing rapid sprints without the benefit of a big-picture understanding of the larger problem space, there is a real risk of missing important mission requirements or opportunities to dramatically improve performance.
|
From page 82... ...
This can lead to a failure to systematically gather user performance requirements, develop coherent innovative solutions that support human performance, and conduct comprehensive evaluations to ensure effective performance across a range of normal and off-normal conditions. Research Needs The committee recommends following research objective to address the incorporation of HSI into agile software development, particularly as it relates to human-AI teaming and MDO.
|
From page 83... ...
These include the following: • Adopting DOD HSI practices in development and evaluation; • Adopting human readiness levels in evaluating and communicating the maturity of AI systems; • Conducting human-centered, context-of-use research and evaluations, prior to and alongside more formal technical performance evaluations; • Including a focus on systems engineering of human-AI teams within the USAF HSI program; • Establishing an AI maintenance workforce; • Establishing an AI TEVV capability that can address human use of AI, and that would feed into existing development and operational test efforts; • Documenting and assessing the data and models used in developing AI systems to reveal possible biases and concept drift; • Continuing to monitor performance of the human-AI team after implementation and throughout the life cycle, to identify any bias or concept drift that may emerge from changes to the environment, the human, or the AI system; • Incorporating and analyzing real-time audit logs of system performance failures throughout the lifecycle of an AI system, to identify and correct performance deficiencies; and • Assessing the state-of-the-art in agile software-development practices in the DOD and in industry, and developing recommendations for more effective processes for incorporating agile software methods into the DOD HSI and acquisition process.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.