C

Workshop Statement of Task

To address quality concerns raised during the 2017 ODNI workshop on “Challenges in Machine Generation of Analytic Products from Multi-Source Data,” a 2-day National Academies’ workshop will explore methods for assessing the accuracy and veracity of machine-generated analytic intelligence products and techniques for addressing the potential impacts of adversarial manipulation of analytical inputs.

A planning committee will organize a workshop to discuss:

1. The current state of machine-driven approaches such as machine learning and natural language processing that can be used to generate and evaluate analytic products from disparate structured and unstructured data types and to detect anomalies;
2. Approaches for ensuring that machine-generated products compare favorably with those of trained human analysts;
3. Statistical methods that can be used to establish confidence hierarchies, model uncertainty, and error propagation, and manage risk as a function of time and complexity and;
4. Techniques for responding to adversarial manipulation of input data to influence analytical products by exploiting weaknesses in machine learning and other AI algorithms and vulnerabilities in their implementation.

A rapporteur-authored workshop proceedings will be prepared.

The planning committee will consider the following research questions:

• What are the technical objectives and metrics needed for success?
• What are the primary issues?
• What are the current and “next level” key performance metrics? What is the “level after next” of expected research and development performance?
• What is the research knowledge base?
• How can the government best prepare the scientific workforce to enhance discovery in this area?