familiarity of a range of analytic approaches, they are unlikely to identify the basic kinds of interdependency between actors’ decisions inherent in game theoretic situations.


Often, analysts are left to reach conclusions by applying their own expert judgment to situations about which they have deep knowledge. Indeed, many analysts spend years, even decades, developing substantive expertise on specific countries or geographic regions, cultures, languages, religions, terrorist organizations, political movements, weapons systems, or industrial processes. This expertise will always be the primary resource in intelligence analysis.

Taking full advantage of domain-specific knowledge requires being able to apply it to new situations, to combine it with other forms of expertise, and to assess the definitiveness of the result. As discussed in Chapter 2, evidence from other areas finds that even knowledgeable individuals may make poor inferences and have unwarranted confidence in them (for reviews, see Arkes and Kajdasz, 2011; Spellman, 2011). For example, experienced stock analysts often do little better than chance in selecting profitable stock portfolios (Malkiel, 1999). The same has been found for doctors’ predictions of how faithfully individual HIV-infected drug users would adhere to antiretroviral therapy (Tchetgen et al., 2001). Foreign policy subject-matter experts do little better than well-informed lay people (or simple extrapolation from recent events) when predicting future political scenarios (Tetlock, 2006).

One condition that contributes to such overconfidence is the lack of task structure. Experts outperform novices (and chance) when tasks have well-structured cues, but when tasks are ill structured—as occurs with the ambiguous cues that often confront intelligence analysts—experts perform no better than novices (Devine and Kozlowski, 1995). A second condition that contributes to overconfidence is hindsight bias, which leads even experts to exaggerate how much they know or would have known if they had had to make others’ predictions (Fischhoff, 1975; Wohlstetter, 1962). A third condition is the ambiguity of many forecasts, allowing people to give themselves the benefit of the doubt when interpreting their predictions (Erev and Cohen, 1990).

A cornerstone of the behavioral and social sciences is a suite of analytical methods designed to address these conditions by structuring tasks, reducing their ambiguity, and providing evaluative criteria. The committee believes that all analysts should have basic familiarity with these analytical methods, taking advantage of the rigorous evaluation that they have undergone. Analysts’ familiarity should be minimally sufficient to identify the

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement