Learning from the Science of Cognition and Perception for Decision Making Proceedings of a Workshop (2018) / Chapter Skim
Currently Skimming:
3 Forecasting and Anticipatory Thinking
3 Forecasting and Anticipatory Thinking
Pages 11-22
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 11... ...
more directly applicable data are often available for physical models; (3) engineered systems are often designed to operate so that the various subprocesses behave linearly, while social systems do not (Keller noted that "humans simply do not have design specifications")
|
From page 12... ...
For these tournaments, IARPA defined accuracy using the Brier scoring rule.1 Mellers' research team, whose Good Judgment Project was a participant in the IARPA forecasting program,2 studied more than 1 million forecasts made by thousands of participants from around the world who attached numerical probabilities to events ranging from pandemics and UN negotiations to leadership change and economic trends. Mellers explained that in making forecasts, intelligence analysts generally use such verbal phrases as "we believe," "we assess," and "we judge" or such terms as "low," "medium," and "high confidence" that can be difficult to interpret and can result in serious miscommunication.
|
From page 13... ...
The Wall Street Journal written by Bill Gates in 20163 about a book by William Rosen, The Most Powerful Idea in the World, which tells the story of the steam engine and the technological innovations needed to develop it. One of these innovations was a micrometer that allowed inventors to see whether incremental design changes led to engine improvements.
|
From page 14... ...
The three-category rounding consisted of confidence ratings of low, medium, or high, and two seven-category roundings consisted of either equal or unequal intervals of the terms shown in Figure 3-1. Mellers explained that the new Brier scores computed after rounding were compared with the initial Brier scores for the thousands of forecasters and for the superforecasters.
|
From page 15... ...
." He observed, moreover, that encouraging analysts to follow strict procedures can send the wrong message that errors can be tolerated as long as they do so. Klein cited evaluating analysts solely on forecasting accuracy as another misguided practice because being accurate is only part of the forecasting process; it is also important, he said, for forecasts to be useful.
|
From page 16... ...
He defined speculative thinking as using abilities necessary to manage complexity, uncertainty, anomalies, and ambiguity. These abilities, according to Klein, are the natural strengths of analysts and are weakened when analysts are forced into procedural mindsets.
|
From page 17... ...
LAS, Wilson explained, is an academic–industry–government partnership that works at the intersection of technology and the tradecraft of intelligence analysis to enable analytic innovation for the IC. The impetus for the creation of LAS was the need to address questions not unique to the IC, such as how to use the massive amount of data from social media for strategic advantage, which could be asked by almost every company in the United States.
|
From page 18... ...
The second LAS project Wilson described involves studying a straightforward question about transliteration of Korean names in a way that would encourage consistent pronunciation. The research team assembled for this task includes experts in design of experiments and data collection and language analysts.
|
From page 19... ...
To conclude, Wilson emphasized the importance of shifting the focus of the IC from monitoring current events to anticipating future events, noticing patterns, and staying ahead of the actions of adversaries. To facilitate this shift, she highlighted the need to understand how to train analysts so they can more effectively look forward, generate insights, and use those insights to make accurate predictions about actions that might be taken by adversaries.
|
From page 20... ...
Mellers therefore believes that these superforecaster strategies are teachable skills. A workshop attendee asked the panelists for thoughts on the value of 7 National Intelligence Council.
|
From page 21... ...
She noted that not every analyst can be expected to be a data scientist or have the ability to carefully tune machine learning models. Therefore, she argued, to make the partnership work, the development of collaborative computing would need to consider who does what best and how to integrate machine output with the intelligence analysis process.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.