Panel moderator David Matsumoto, San Francisco State University, opened this session by observing that various social and behavioral science disciplines often focus on different units of analysis, ranging from individuals and small groups to societies or cultures. These differences, he asserted, pose challenges to researchers because concepts or findings at one level of analysis may not translate to other levels. “When we are [looking] at the broad spectrum of social science research related to the question of changing social cultural dynamics and its implications for people’s behaviors,” he said, “one question then becomes, What is the relationship between what you find at a macro-social level [and] predicting people’s behaviors and vice versa? We can understand people’s behaviors, and we may or may not understand what the macro-social dynamics are like.” Following Matsumoto’s introductory remarks, the session included presentations on levels of influence, patterns of cultural variation, and levels of analysis and linguistics. These presentations were followed by discussion and concluding remarks for the workshop.
Gwyneth Sutherlin, Geographic Services, Inc., suggested that methods ranging from ethnographic fieldwork to the use of artificial intelligence could allow social science research to provide significant advances in understanding patterns of human behavior and culture. In her view, however, social science methods can be improved to increase the richness of data
and possibilities for collaboration with other disciplines and also to better address national security questions.
Although trained as a mechanical engineer, Sutherlin noted, she now researches how discrete technological advances can allow forward progression in political science and conflict analysis. Thus, her work has focused on issues of conflict among communities around the world. She has sought to understand the difficult-to-define characteristics of the cultures of communities based on how their members understand themselves, as well as the characteristics that define these communities in relation to others with which they are in conflict. Among the many characteristics of a culture, she focuses only on those that are the source of conflict, then operationalizes them so that they are observable and discrete and can inform decision making with respect to conflict and peace.
Increasingly, Sutherlin noted, new technologies are being applied to collect these cultural data. However, she has observed a disconnect between the data she collects on the ground and the analytic output to inform decision making. She highlighted the need to enrich understanding of places outside the United States “linguistically in terms of how other cultures think and understand their environment, problem solve, strategize, [and] have different time event horizons” based on what is encountered in the field and in the data collected.
In her role at Geographic Services, Inc., Sutherlin directs the development of technology that enables examination of large datasets following the same rigorous processes she applies in her fieldwork. The technology uses interactive graph databases with multiple variables, she explained, enabling her and other researchers to scale rich, robustly validated data from the neighborhood level to the country and regional levels. According to Sutherlin, this capability can provide a diverse picture of a particular scenario, and, she added, these rich sociocultural data can also provide a training set for machine learning.
Sutherlin then described ways in which these data could be further enriched. For example, she pointed out, languages spoken in a particular area can be depicted graphically on a map. She argued that maps illustrating the many other primary languages around the world underscore the need to better understand these cultures on their own terms, especially given the dominant focus on English and closely related European languages in academic research. This focus, she asserted, has the limited understanding of linguistics and how other cultures function and also limited the usefulness of software and analytic systems that rely on English-language data sources for training.
Sutherlin continued by observing that applying principles and theories from one culture to similar phenomena in another can be problematic. She cited the example of the application of lessons learned about how people
joined gangs in Chicago during the 1930s to understanding ISIS recruitment. “I think it is reasonable to question if that should be applied outside the U.S. in the 21st century to non-Americans,” she cautioned. “There are many theories like this that people do not [examine] critically. . . .” In other words, she asserted, theories applied to solving problems need to be culturally adapted.
According to Sutherlin, knowledge of how language works around the world is quite limited, and language universals have yet to be discovered. In addition, she observed, researchers know little about language on online platforms, which presents both a problem and an opportunity. She explained that cultures can differ greatly in how they think, including their concepts of time; concepts of location, number, and counting; and whether and how they use categorization. In addition, she said, they may differ in how they conceptualize events in ways that are important to understanding their narratives. She gave the example of the way in which members of a culture explain who was part of an event and the causes and effects of the event, all of which can be highly culturally specific. “That construction of how people perceive, remember, and recall an event is completely different,” she argued. “Not language by language, but culture by culture. We are not able to contend with that with our current [data] collection and analysis. That is a huge area for development,” she asserted.
Whether intentionally programmed or unintentionally encoded, Sutherlin continued, algorithms and analytic software encode social theories and assumptions. Often, she explained, these systems separate information based only on language, and that process is frequently imperfect. At other times, she noted, proxy variables for culture are used, but these variables are difficult for software developers to identify. She argued that concepts derived from the way cultures understand themselves should drive software models so that analytics are specific to each area researchers and others want to understand.
According to Sutherlin, promising directions for addressing the limitations of how cultures around the world are understood include starting small with well-researched and field-based cultural concepts before pairing these approaches with other methodologies. She emphasized the need for more research on the particular languages and dialects used in various places. She cited the example of Iraq, where three separate Kurdish dialects are used, yet many groups working to understand Iraq remain unaware of such differences. She believes that ultimately, interactive, culturally based models aided by technology can help address such gaps in understanding of different cultures around the world. One promising approach, she suggested, is “putting a lot of these social and culture factors in a graph database, which allows you to take very small data and scale them out in a large network.”
Michele J. Gelfand, University of Maryland, College Park, discussed her approaches to integrating multiple levels of analysis across different disciplines in her work as a cross-cultural psychologist studying the tightness and looseness of controls and norms of behavior. She looks for patterns across these levels while recognizing the distinct causes and consequences of phenomena within each level.
Gelfand has observed differences across cultures in how tightly behavior is constricted by the rules and norms of the society. For example, she pointed out, Singapore has a culture with many rules, and its citizens can be fined for flying a kite or spitting in a public place; in New Zealand, by contrast, people walk into public buildings barefoot, and rules are more relaxed and more eccentric behavior tolerated. She added that similar contrasts can be observed among cities around the world in, for example, observance of traffic laws, control and acceptance of marijuana use, and even patterns of how parents choose to name their children. “What ties these together, in my view, are that they are all related to social norms, these standards of behavior that we share in certain human groups,” she said.
Gelfand explained that these norms are the “glue” that keeps cultures together and that having such norms is universal. Humans are expert at developing, following, and enforcing social norms, she observed, especially across generations. However, she added, cultures differ greatly in the strength of that social “glue”—how tight or loose the social norms are in any particular group across different levels of analysis. Tight groups have strong norms and strong punishments for deviance, she explained, while loose groups have weaker norms and a higher degree of permissiveness. She noted that such differences have been observed worldwide even since ancient times.
Gelfand has studied this phenomenon of tightness and looseness at the level of the individual, within social classes, in organizations, and across modern nations and states. Ecological factors and historical events influence the social organization of a country, she observed, which in turn affects the characteristics of social situations and associated psychological processes.1 She suggested that this is one way of modeling culture across different levels of analysis. Her research has focused on how the broad historical, ecological, social, and political institutions of a society affect cultural concerns about norms and how these norms then influence everyday behavior (see Figure 4-1). For this research, she has identified 33 countries across six continents based on her theoretical predictions and recruited a sample of nearly 7,000 participants, who spoke 22 different languages. She gathered
1 Triandis, H.C. (1972). The Analysis of Subjective Culture. New York: Wiley.
survey data from these participants and collected ecological and historical data. In addition, her research team conducted unobtrusive observations of behavior in public settings.
As Figure 4-1 illustrates, Gelfand’s tightness–looseness system can be understood as part of a larger complex, loosely integrated system involving processes across multiple levels of analysis. Her research indicated that people can agree on the degree of tightness or looseness of the norms in their society when asked. Further, she observed, the tightness–looseness construct was distinct from participants’ ideas about collectivism and other factors related to the economic state of their society. She added that both tight and loose societies enable multiple levels of analysis that convey the trade-offs people perceive within each type.
According to Gelfand, tight cultures have greater order and predictability. They have more security and cleaning personnel and have less crime, generally speaking. Tighter cultures display greater uniformity in cars and clothing, and have more synchronous clocks and even more synchronous stock markets,2 she noted. By contrast, loose cultures are more disorganized, have more crime, and have less synchrony. However, loose cultures are characterized by more openness, less cultural superiority, less ethnocentrism, and more acceptance of immigrants and people who tend to be stigmatized (e.g., immigrants, homosexuals, people with tattoos).
Gelfand explained that her research is aimed at identifying factors that predict this tightness or looseness. Gross domestic product, common religion, language, and geographic location do not serve this purpose, she stated. One the other hand, common threats experienced by countries and groups—such as territorial invasions, natural disasters, food scarcity, dense population, and prevalent pathogens—can serve as a strong impetus for coordination and organization that help societies survive. Accordingly, her data show a strong connection between these threats and tightness or looseness, controlling for gross national product per capita. Gelfand cited the example of Singapore, a very densely populated country with three dominant groups, noting that the tightness of its culture helps prevent conflict among these groups living so closely together. She explained that people in Singapore agree to tighter controls because they feel secure in return. “In the context of high threat,” she said, “you can sacrifice liberty for some security. In a context where you have very little threat or less threat, then you prioritize freedom over security.”
At another level of analysis, Gelfand examines the strength or weakness of the controls on behavior in certain social situations. Situations with strong controls on behavior, such as being in a library or attending
2 Eun, C.S., Wang, L., and Xiao, S.C. (2015). Culture and R2. Journal of Financial Economics, 115(2), 283–303.
a funeral, allow for little variation in behavior, and behavior that violates norms is likely to be censured. On the other hand, Gelfand explained, situations with weak controls on behavior, such as being in a public park or in one’s home, allow for a wider range of behavior. She noted that people negotiate both types of situations throughout daily life, but that people in tight cultures experience a greater preponderance of situations with strong controls. These cultural effects can be observed at the individual level, she added. People who live in tight cultures focus more on prevention, regulate their own behavior, and desire structure to a greater extent relative to people in loose cultures, she elaborated. By contrast, people in loose cultures focus more on promotion and have a higher tolerance for ambiguity compared with people in tight cultures. Gelfand added that, as opposed to thinking of individuals as tight or loose, she sees people as adapting individually to situations with strong or weak controls on behavior.
Gelfand has also explored variability within countries, within organizations, and within social classes with respect to tightness and looseness by examining survey and archival data. She reported that findings from this research in the United States indicate that southern states are tightest, coastal states are loosest, and the remaining states vary in this regard.3 Tightness, she noted, is predicted by amount of food insecurity, proneness to natural disaster, disease stress, and how rural the state is. The consequences of this tightness, she said, include more organization, more law enforcement, less homelessness, and less divorce, as well as more self-control, less drug abuse, and less debt. Looser states are on the other end of the spectrum on those same factors, she added, and their openness is also associated with greater creativity. Thus these states are characterized by more patents and fine artists per capita, more equality and fewer equal employment opportunity claims, and more minority businesses.
Gelfand and her collaborator, Jesse Harrington, have also examined differences in tightness and looseness among people from different social classes. In this research, they were seeking to determine whether members of a lower social class perceived themselves to be under threat in a pattern similar to that of people in countries threatened by conflict or natural disasters. According to Gelfand, working-class families worry about falling into poverty or being the victim of crime. They also, she noted, typically work in more dangerous professions that require following rules to a greater extent relative to people in middle-class environments, which display more emphasis on creativity. She added that families often raise their children to fit into their environments.
Gelfand described results she and Harrington derived from a survey
3 Harrington, J.R., and Gelfand, M.J. (2014). Tightness–looseness across the 50 united states. Proceedings of the National Academy of Sciences, 111(22), 7990–7995.
of 300 working- and middle-class adults, on which working-class adults reported greater tightness and more positive attitudes toward rules compared with middle-class adults. She and Harrington also measured threat by linking zip codes to indexes of poverty and unemployment. Patterns of psychological characteristics of working-class families (e.g., need for structure, conscientiousness) were consistent with national-level data. Additional studies with larger, representative samples have replicated these findings, Gelfand noted. She added that differences in reactions to norm violations enacted by puppets in a laboratory setting could be observed in children as young as 3 years old. Middle-class children tended to find the puppets’ rule breaking funny, whereas working-class children asked the puppets to stop and more often saw the behavior as wrong.
Gelfand and her colleagues also examined neural data to determine whether cultural differences and norm violations produce any visible changes in the brain. They explored whether any such changes were related to outcomes they had observed at other levels of analysis, such as self-control, ethnocentrism, or creativity. In this research, they examined the brain responses of individuals in the United States and China 400 milliseconds after being presented with a stimulus (referred to in neuroscience as the N400 response). Some of these stimuli were strong norm violations (e.g., a person dancing in an art museum), but others violated norms weakly or not at all (e.g., a person dancing in the park or during a tango lesson). As expected, Gelfand reported, the brain activity of people from both cultures showed stronger reactions to strong norm violations than to other stimuli. However, she added, the two groups showed differing responses in the frontal area of the brain when asked such questions as “Why did someone do this?” or “What should we do about this behavior?” She suggested that these differences, in turn, were related to self-control, creativity, and feelings of cultural superiority.
Another line of Gelfand’s research is beginning to examine linkages between perceived societal threat and the speed of brain synchrony between people. Theory predicts, she explained, that people who perceive a threat develop strong norms needed to coordinate social action. In this research, she and her team measured the coordination of brain waves of pairs of subjects under three different conditions representing ingroup threat, out-group threat, and no threat. They then measured pairs’ coordination on a counting task while in separate rooms. Gelfand reported that under high ingroup threat, the pairs showed more behavioral coordination, related in part to the synchrony of their brain wave activity.
Gelfand explained further that the threat people perceive can also be manipulated and activated subjectively. Before the November 2016 election, she and her colleagues measured the perceived threat from such groups as ISIS, North Korea, and immigrants; belief about whether the country was
too loose; the desire for greater tightness; and whether these perceptions and feelings predicted voting for Donald Trump among a representative sample of 500 Americans. According to Gelfand, the results of these measurements were in line with predictions: perceived threat was positively related to desire for greater tightness, which in turn was positively related to voting for Trump. She and her team replicated this study in France to predict voting for Le Pen, a far-right presidential candidate. This dynamic is not a modern phenomenon, she added.
Gelfand concluded by reporting research findings indicating that neither extreme looseness nor extreme tightness is healthy,4 noting that both of these extremes are related to higher depression and suicidality, higher blood pressure, and lower happiness. “I think what this suggests to us,” she said, “is that we need to be thinking about this dimension in terms of what is happening around the world, when groups [have] these pendulum shifts from very tight to very loose and back to tight again. This happened in Egypt in the Arab Spring,” she added, “where you went from very tight culture where you got rid of this top-down control. Without any ability to self-organize and synchronize, we could see that people desire the same kind of autocracy that they had in the first place, what we call autocratic recidivism. We can show that with data.” She noted that her team’s research in Egypt after Mubarak was overthrown confirmed this pattern.
Computer algorithms provide an important way to learn about language, explained Jesse Egbert, Northern Arizona University, but they must remain interpretable. At times, he observed, maintaining interpretability means omitting variables he cannot explain, which reduces to some extent the ability to predict outcomes of interest. However, he said, any predictive accuracy that is lost is more than made up for by an increase in the ability to explain the patterns identified by computers. Without this interpretability, he added, computational linguistic research often raises more questions than it answers, especially with large datasets. Referencing Nate Silver’s book The Signal and the Noise,5 Egbert stated that the ability to distinguish “signal,” or meaningful patterns, from “noise” may be declining, despite the fact that people have access to more data than ever before via the Internet. He proposed that preserving linguistic interpretability in computational linguistics is one way researchers can counter this trend.
4 Harrington, J.R., Boski, P., and Gelfand, M.J. (2015). Culture and national well-being: Should societies emphasize freedom or constraint? PLOS ONE, 10(6), e0127173.
5 Silver, N. (2012). The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t. New York: Penguin.
Egbert suggested that a corpus, or large sample of texts, is like a forest. This forest, he said, can be thought of as a single unit, but it also comprises individual “trees” or texts, independent units of naturally occurring language (e.g., a conversation or a workshop presentation) that have a beginning and an end and serve a function. Like individual trees in a forest, he continued, these texts are made up of smaller parts—linguistic characteristics such as words or syntax. Just as no single tree can represent the whole forest, he added, no individual text can represent an entire body of language. In his view, texts with definable boundaries (e.g., a complete conversation) are an ideal unit of observation in corpus research because they preserve explanatory information, such as the purpose and function of the text. He described these texts as fundamental units of discourse that constitute meaningful and valid social constructs and have integrity as a unit both situationally and linguistically.6,7
Egbert further explained that he seeks meaningful levels of analysis that fall between the individual text level and the complete body of language he is analyzing to enable classifying texts in useful ways. Traditionally, he noted, researchers have examined text in terms of geography, gender, age, or race, for example. In his view, however, the most important variable for explaining variation in texts across speakers and writers is register. He explained that registers are varieties of language defined by the situations in which they are used.8 “There is a functional link between the language that is used and the situation that it is used in,” he asserted, adding that research, including his own, has indicated that register can often explain why people make the language choices they do.9 Thus, his research focuses on understanding the functional reasons for the language choices people make (e.g., using more or fewer verbs or nouns). Evidence suggests, he observed, that differences in register are detectable from the discourse level down to the level of speech sounds.
Egbert has been involved with a research project examining Internet language, which traditionally does not have clear categories of registers.10 To assign categories of registers to this vast body of linguistic data, he
6 Egbert, J., and Biber, D. (2016). Do all roads lead to Rome?: Modeling register variation with factor analysis and discriminant analysis. Corpus Linguistics and Linguistic Theory. doi: https://doi.org/10.1515/cllt-2016-0016.
7 Biber, D., and Conrad, S. (2009). Register, Genre, and Style (Cambridge Textbooks in Linguistics). Cambridge, UK: Cambridge University Press. doi:10.1017/CBO9780511814358.
8 Biber, D., and Conrad, S. (2009). Register, Genre, and Style (Cambridge Textbooks in Linguistics). Cambridge, UK: Cambridge University Press. doi:10.1017/CBO9780511814358.
9 Biber, D. (2012). Register as a predictor of linguistic variation. Corpus Linguistics and Linguistic Theory, 8(1), 9–37.
10 Biber, D., and Egbert, J. (2016). Register variation on the searchable web: A multidimensional analysis. Journal of English Linguistics, 44(2), 95–137.
elaborated, the researcher can predefine and sample from categories of interest, such as Twitter texts or YouTube video comments. An alternative method of sampling Internet texts, he explained, is to gather a random sample of texts and assign the texts to register categories in a bottom-up fashion. Using the latter method, Egbert and his colleagues collected a random sample of 50,000 texts, consisting of approximately 50 million words, from Google searches. Next, nearly 1,000 MTurk raters with minimal training coded each text for situational features, such as how interactive the text was, who its audience was, and what its purpose was. The researchers found that these nonexpert raters were unable to code the texts into register categories reliably. That being said, these nonexpert raters were able to code the texts for their situational features, which could then be used to generate categories of registers. Using this method, Egbert explained, the researchers were able to identify 8 categories of registers and 33 subregisters. They also found that although 4 raters rated each text and achieved nearly 70 percent agreement on the 8 register categories, 3 of 4 raters agreed just over half of the time (51.4%) on 33 subregister categories.
According to Egbert, the results of this research indicate that by using a bottom-up approach to identify registers in Internet language, one can detect variation in language use in register and subregister categories (e.g., use of first-person pronouns). He elaborated on these findings by showing how blogs, which are considered to be a meaningful text type, incorporate many different situations and registers. He then presented five major categories of blogs (see Table 4-1). Personal blogs, he noted, often contain first-personal pronouns because the author is writing about his or her own personal experiences. He contrasted this with informational blogs, where the use of first-person pronouns decreases because the focus of the writing is no longer the author. However, he cautioned, blogs do not constitute a
TABLE 4-1 Examples of Blog Registers
|Personal Blog||Travel Blog||Religious Blog||Opinion Blog||Informational Blog|
|Subject||Author’s life||Travel||Religion||Author’s stance||Topic to be explained|
SOURCE: Presentation by Jesse Egbert at the workshop.
register in themselves, even though they are often analyzed as a meaningful category in computational linguistic research. He explained further that mode of language (or the spoken or written method of communication) could also be considered as a unit of analysis. Spoken texts include transcribed interviews or song lyrics, for example.
Egbert reiterated that he views register as an important “signal” amid the “noise” of online linguistic information, and asserted that its use improves the usefulness and accuracy of computer algorithms employed in examining texts. “I think that we need to keep the linguistics in computational linguistics,” he said. “This is critical considering the rapid increase in computational power in terms of speed [without] a corresponding increase in the availability of sound linguistic theory,” he concluded.
Following the panel presentations, Matsumoto moderated a discussion. The discussion addressed (1) levels of analysis, (2) limitations of public opinion polling for learning about culture, and (3) the role of prediction.
Levels of Analysis
A member of the Intelligence Community (IC) shared his response to the issue of multiple levels of analysis in practical terms. A key concern of intelligence analysts, he explained, is how to convey essential information to decision makers effectively in a way that is compelling and trust building. “The time of who we are trying to convey information to is limited enough that they are going to need to quickly trust whatever is happening with [the methodology],” he noted, asserting that the methods described during the panel presentations would likely raise questions about the reliability of the interpretations they had generated. He emphasized that the ability to communicate clearly is at least as important as the rigor of the methods and the accuracy of the information. Another workshop participant, a former member of the IC, underscored the importance of communicating to decision makers. She explained that analysts seek to be informed by social science but not to use it overtly in their presentation of findings to decision makers.
The IC member explained that analysts rarely are able to choose the unit of analysis for the particular problem at hand. However, he observed that the various methods discussed during the panel presentations afforded flexibility that could be used to address gaps in explanatory narratives, and that presenters had related how various methodologies might be useful in building confidence in conclusions reached by researchers and analysts. Egbert noted that in his work, he examines text in context, and
that this text represents people and the choices they have made within an exchange. Without context, he argued, interpretations will be incomplete or inaccurate.
Sutherlin asserted that the rich data now available should be used to improve predictions. No longer should people use maps that broadly label groups, she argued, because this approach oversimplifies conditions in an area and impedes informed decision making. Instead, she suggested that units of analysis should be as fine-grained as possible and then aggregated, given that, like weather forecasts, models and predictions are improved with the use of more fine-grained data. She suggested further that methods are needed with which to understand cultural differences because there are significant, foundational differences in the ways people from different cultures relay information. Without these understandings, she argued, interpretations will be very distorted, particularly if misunderstandings are used to train computer algorithms or are aggregated.
Gelfand suggested that it is also important to understand differences among the cultures of academics, the IC, and policy makers. “I think we need to understand the culture of your community,” she said, adding that bridges between communities are needed. She also has observed that the strategy of trying to tell stories quickly and take action as rapidly as possible has backfired in cultures outside of the United States. Finally, she suggested that rather than identifying which levels of analysis are best, it is more important to identify how principles from one level of analysis can help inform analyses on other levels.
Limitations of Public Opinion Polling for Learning about Culture
Workshop participants discussed the benefits and drawbacks of public opinion polling as a means of understanding populations or individual actors. The IC often uses this approach, explained a former IC member. Gelfand, Egbert, and Sutherlin all emphasized the importance of triangulation.
Gelfand explained that “I do not trust any of my research until I see it with another method.” In addition, she observed, polling can be problematic in cultures other than the United States where there is less openness and trust in talking to people conducting surveys. She also noted another limitation of public opinion polls that she has observed in her research. In her work interviewing former members of ISIS, she has found that they are unwilling to explain why they joined ISIS, but framing the question around why other people joined made them more willing to respond. She noted that few surveys ask questions about these descriptive norms.
Egbert emphasized that sampling is an important consideration in polling. Many polls are not representative, he observed, and many poll results are misreported or omit important information, such as the margin of error.
Sutherlin added that polls often parse their findings by gender, race, or age, but usually not by meaningful groups or communities that may be more relevant to understanding behavior and informing decision making. Thus, she said, she works to relate survey data to meaningful units within a particular society, such as families, tribes, or castes. She also related her experience of learning from other cultures during times of disaster or crisis, when people share their views through unstructured polling on important issues they are facing.
Dan Kahan, Yale University, explained his concerns with relying heavily on public opinion polls for decision making. First, he said, the opinions they describe often are statistical artifacts because people have limited information or opinions on the topics of many polls. Therefore, the opinion data are not revealing true sentiments on the topic as much as being a product of being asked about the topic in a poll. He illustrated the point by noting that most members of the public have no opinion about genetically modified organisms and eat them regularly, yet they will express an opinion on the subject if asked in a poll. Second, Kahan observed that polls are generally not constructed to measure or create coherent models of factors that explain people’s behavior. Therefore, he argued, reliance on public polling data to understand how and why people behave as they do can be misleading.
Egbert also suggested that members of the IC may be especially interested in understanding outliers and anomalies in the data they analyze. Although he acknowledged that identifying such cases is difficult because of the “noise” in the data, he stressed that identifying cases that stand out from the baseline norms (e.g., threatening people) is important, and the stakes are high for being accurate. Another participant suggested that it is also important to understand the context and factors in the environment that may improve the prediction or understanding of outlier cases rather than waiting for outlier cases to occur.
Despite these concerns about relying on public opinion polls to achieve some of the analysis goals of the IC, Susan Weller, University of Texas Medical Branch, observed that it is important to distinguish such polls from federal surveys, which can be very useful data sources. Citing an example in health care, she noted that the National Health and Nutrition Examination Survey is carefully developed, pilot tested, and useful. She argued that although surveys are considered “old technology,” professional and federal surveys remain extremely important.
The Role of Prediction
Presenters and panelists also discussed the meaning and importance of prediction in social and behavioral research and in the IC. A participant
from the IC explained that when decision makers in the IC are presented with information related to prediction, they need to know what to do with that information. In addition, he said, they need to know important caveats about how the information should and should not be used, which may include information about the sampling or about the inferences that can be drawn from the data. “It is really important to do everything that we can to make sure that we recognize that what we are conveying will be used in that context,” he stressed. “That is the difference between an applied and a theoretical world,” he added. Sutherlin noted that she often seeks to avoid using the term “prediction,” instead preferring to use such terminology as “to infer” or “to anticipate possibilities.”
Asked by Matsumoto to convey how the social science community could help the IC over the next decade, a participant from the IC explained that the most important advance would be establishing a two-way dialogue between the two communities, which would allow the social science community to better understand the practicalities facing the IC. Another participant observed that the IC has an important deliberative process for weighing the evidence and alternative explanations provided by analysts. He added that this process includes communicating across agencies about outlier cases. Issues related to outliers are regular considerations of analysts and the IC more broadly when contending with risk and uncertainty, he said. One participant said, “What is needed is continued communication where government brings the notion of what the applied world looks like and academia brings theoretical insights, tools, and techniques. We are able to have a conversation with candor so that we can each learn from that. That is what I would like to have or that is the mechanism by which I would like to work over the next 10 years from my perspective.”
Jeffrey Johnson, chair of the workshop steering committee, described several ways the social science community might be more helpful to the national security community. First, he noted that the IC gathers many types of data from various sources. The challenge for the social science community, he said, is to consider how it can communicate the insights derived from the methods for learning about culture, language, and behavior described during the workshop and help the IC integrate these insights with the data it is collecting on its own, such as polling data, survey data, and text data.
Second, Johnson suggested that the social science community consider the IC’s needs for prediction. Although many social scientists are uncomfortable with that term, he said, providing the IC with probabilities of potential outcomes may still be useful.
Third, Johnson suggested that the social science community should consider how it can help build the trust and confidence of the national security comunity and the IC in its methodologies.
Finally, Johnson underscored the importance of understanding the various contexts of individuals. He emphasized the value of drawing on the different methodologies across many social science disciplines for that understanding.