This panel explored the methods of forecasting through discussion of four topics in which forecasting is particularly important: democratic backsliding, military conflict and violence, epidemics, and environmental security. Suzanne Fry, National Intelligence Council, then commented on these presentations and suggested topics and methodological approaches for future research. The panel closed with an open discussion between the presenters and audience members.
FROM PREDICTION TO PRACTICE: INTEGRATING FORECASTING MODELS INTO PUBLIC HEALTH EDUCATION AND RESPONSE
Kacey Ernst, University of Arizona, focused on the importance of incorporating the social and behavioral sciences into predictive models of vectorborne diseases. She explained that traditional predictive research on such diseases tends to focus on environmental factors, such as temperature, rainfall, and vegetation cover. But while monitoring environmental factors is helpful in identifying baseline risk, she argued, human factors, such as risk behaviors, mobility patterns, and social and political infrastructure, also affect how diseases are transmitted and how those transmissions are detected.
It is well documented, Ernst noted, that temperature and precipitation are drivers of mosquito development. Thus, she said, a basic predictive model using only temperature and precipitation inputs has historically provided accurate predictions of dengue transmission. Researchers have
speculated, she continued, that the unexpected peak in dengue transmission experienced during drier years in San Juan, Puerto Rico, occurred at least in part because residents were keeping containers of standing water (an excellent mosquito breeding ground) to water plants.1 Seeing how this change in human behavior played a role, Ernst and her colleagues began to incorporate demographic factors into model projections. She stressed, however, that demographic factors are not enough; behaviors must also be accounted for in attempting to predict transmission. For example, she explained, the transmission of some vectorborne diseases, such as malaria, is more likely to be predicted by the availability of treatments or behavioral interventions, such as the use of bed nets. For interventions to be effective, she added, it is critical that researchers study the drivers of behavior change.
According to Ernst, social factors are particularly useful when incorporated in early warning systems, which make projections 3–6 months ahead. She explained that early warning systems can be divided into three categories: watch, warning, and emergency. “Watch” systems develop an early assessment of the risk of an emergency event, “warning” systems are used when human disease has been detected, and “emergency” systems come into play when an epidemic or outbreak is under way. Ernst observed that “early warning systems for epidemics face several key challenges, including integration of disparate data streams, changing risk landscapes as new controls, human behavior, and response capacity shift.” Thus, she stressed, multiple components must be monitored for early warning system forecasts to be accurate.
As an example of an early warning system, Ernst cited a projection she worked on with colleagues to identify the risk of transmission of the Zika virus in the continental United States.2 She explained that she and the team, led by Dr. Monaghan at the National Center for Atmospheric Research (NCAR), “incorporated Aedes aegypti seasonality, socioeconomic status, travel from endemic countries, and prior dengue infection” to develop accurate predictions of where Zika would most likely be transmitted (see Figure 4-1). She noted that this one-time assessment was developed with the use of climate and travel patterns. However, she argued, systems that process real-time data, such as those used for drought monitoring and early warning of famine, would be better at predicting disease transmission. She
1 Morin, C.W., Monaghan, A.J., Hayden, M.H., Barrera, R., and Ernst, K. (2015) Meteorologically driven simulations of Dengue epidemics in San Juan, PR. PLOS Neglected Tropical Diseases, 9(8), e0004002. doi:10.1371/journal.pntd.0004002.
2 Monaghan, A.J., Morin, C.W., Steinhoff, D.F., Wilhelmi, O., Hayden, M., Quattrochi, D.A., Reiskind, M., Lloyd, A.L., Smith, K., Schmidt, C.A., Scalf, P.E., and Ernst, K. (2016). On the seasonal occurrence and abundance of the Zika virus vector mosquito Aedes aegypti in the Contiguous United States. PLOS Currents, 8. doi:10.1371/currents.outbreaks.50dfc7f 46798675fc63e7d7da563da76.
added that early identification can help control an outbreak by making it possible to reduce contact, treat cases quickly to reduce the duration of infectiousness, target interventions, and increase their uptake.
Ernst then pointed to social listening as a potential tool to aid in the early detection of disease outbreaks. By developing algorithms for monitoring keywords across such social media platforms as Twitter, Facebook, and Instagram, and even Internet search engines, she noted, researchers can identify trends in symptoms or illness. However, she cautioned, there are drawbacks to this method. For example, she said, data are often biased by age and geography, and noisy data can make identifying large-scale trends difficult.
Ernst cited community-based participatory surveillance (CBPS) as another method that has shown promise in early detection of disease outbreaks. One example of a successful CBPS system, she noted, is Flu Near You, which has a greater than 90 percent correlation with the Centers for Disease Control and Prevention’s influenza surveillance data. She did acknowledge, however, that research is needed to increase participation across demographic groups to reduce bias in the data.
Ernst and her colleagues recently developed a system for symptom monitoring to target mosquito activity and mosquitoborne diseases across regions on the U.S.–Mexico border. To reduce difficulties with recruitment, Ernst plans to leverage Kidenga as a resource. Similar to Flu Near You, Kidenga is an online app that allows members of the community to self-report illness that has been transmitted by mosquito bites. Ernst and her colleagues have been investigating behavior change models with an eye to increasing use of the app. As a result of this research, they have discovered that people are motivated to participate because they consider mosquitoes to be an annoyance, not because they are concerned about the risk of disease.
Ernst closed by suggesting that research is needed to understand how much uncertainty stakeholders are willing to accept before they are willing to invest in early warning systems. Research is also needed, she asserted, to understand what motivates public health partners and the public to take action. She argued further for investment in systems, not just research grants, so that validated early warning systems can be maintained and provide sustainable forecasts.
Afreen Siddiqi, Massachusetts Institute of Technology and Harvard University, discussed methods of forecasting water availability in arid regions. Because water is essential to human existence, she observed, limited access to water can have a devastating effect on human welfare, societal welfare, and public health.
Siddiqi and her colleagues have been studying countries that are water-stressed. Water is not just a critical resource, Siddiqi explained, but also a finite one. Thus, she said, no additional water supply is available to accommodate an increase in population; when a population increases, the amount of water per capita decreases. Figure 4-2 shows the decrease in per capita water supply in water-stressed regions. Siddiqi pointed out that the red line at the top of the figure represents the water poverty line defined by the United Nations. Thus it is clear that many of these countries are already in an extreme situation.
Siddiqi emphasized that solving the problem of water scarcity requires
examining what the future may look like years in advance because any investment made now must be sustainable in the long term, and also must consider infrastructures that rely on water, such as energy and agriculture. Because experience has shown that water forecasting methods using historical data are not helpful, she and her colleagues initiated a research program to find a method that would work. They began by identifying large-scale infrastructure projects under consideration in arid regions. Next, Siddiqi explained, she and her colleagues assessed the likelihood of these projects being implemented and how much water they could be expected to add to the system.
When considering the likelihood of implementation, Siddiqi and her colleagues studied regional decision makers and identified the factors likely to be significant in their decision-making process. They then used multi-criteria decision analysis methods to organize and code the collected data. According to Siddiqi, it was also necessary to identify several key assumptions:
- The resource is being developed through a discrete set of infrastructure projects.
- The resource is being developed for direct use and cannot be substituted for through trade and imports.
- The decisions are being made by a specific group of key actors.
- The decision makers are evaluating each project using a discrete set of factors.
The research team then organized the various projects and evaluated them based on the decision criteria, Siddiqi continued. These data were then placed in a performance matrix, and the projects were compared. Siddiqi explained that the next step was to compare the results of the team’s analysis with information about past projects known to have been commissioned and implemented. Finally, the team assessed the likelihood of future project selection by evaluating performance on different preference sets and conducted an evaluation in which the probabilities of importance for different criteria were varied. Once they had decided that a project was likely to be implemented, Siddiqi explained, they assessed the amount of water the project would be expected to add to the system.
As an example, Siddiqi shared results of the team’s methodology after it had been applied to identifying projects in Jordan.3 She noted that the projects being evaluated ranged in size and scope: some were large desalination projects, while others were smaller wastewater treatment projects. She pointed out that desalination systems, which remove salt from seawater so that it is safe for human consumption, are widely employed in countries where water is scarce.
According to Siddiqi, decision makers use a least-cost approach on large infrastructure projects, meaning that they often start with the least expensive system and move on to other systems from there. She explained that she and her colleagues began by developing a detailed analysis of the stakeholders involved in project selection. They then held structured interviews with these stakeholders to understand their decision-making process. Table 4-1 shows the decision criteria and their ranking as provided by the decision makers.
Reviewing the decision criteria in Table 4-1, Siddiqi noted that, because many of the large-scale infrastructure projects in Jordan are financed by foreign financiers, foreign investment potential is a significant consideration for stakeholders. She observed that political feasibility is another important factor for decision makers. She pointed out that, because of the sociopolitical aspects of many water infrastructure projects and the resulting
3 Siddiqi, A., Ereiqat, F., and Anadon, L.D. (2016). Formulating expectations for future water availability through infrastructure development decisions in arid regions. Systems Engineering, 19(2), 101–110.
TABLE 4-1 Decision Criteria and Their Ranking as Provided by Decision Makers in Jordan
|Decision Maker 1a||Decision Maker 2a||Decision Maker 3b||Decision Maker 4c||Decision Maker 5c|
|2||Cost||Sectoral Priorities||Cost||Foreign Investment Potential||Foreign Investment Potential|
|3||Political Feasibility||Geographic Distribution||Annual Supply||Annual Supply||Annual Supply|
|4||Foreign Investment Potential||Environmental Impact||Environmental Impact||Political Feasibility
|5||Resource Sustainability||Foreign Investment Potential||Social Priority|
aMinistry of Water and Irrigation
bMinistry of Planning and International Cooperation
cMinistry of Environment
SOURCE: Adapted from Siddiqi, A., Ereiqat, F., and Anadon, L.D. (2016). Formulating expectations for future water availability through infrastructure development decisions in arid regions. Systems Engineering, 19(2), 101–110. Used with permission.
political implications, it can be difficult for such projects to be approved for development.
After performing their initial analysis, Siddiqi and her colleagues determined that the Waste Water Treatment Expansion (WWTE) project had an 83 percent chance of being implemented. They then considered which project would most likely be implemented if WWTE were taken out of the equation. From that analysis, they predicted that the Disi Pipeline would be next in line for implementation.
Siddiqi and her colleagues also considered scenarios in which it was possible that a dominant decision maker set the priorities for water projects. This analysis gave them a chance to see how changing priorities affected the outcomes.
Siddiqi closed by delineating a few of the limitations of this study. First, she noted that, although her team’s method successfully predicted the outcome of interest, it should not be assumed that the team has developed a unique set of predictive factors. She added that it is not always easy to identify the decision makers involved or their priorities. Furthermore, decision makers’ priorities in a country may change because of changes occurring internationally or within the country.
Jennifer Dresden, Georgetown University, focused on the measurement and forecasting of what is termed “authoritarian backsliding,” also called in the literature “democratic backsliding,” “democratic reversal,” “autocratization,” and “democratic deconsolidation.” Authoritarian backsliding, she explained, refers to “actions taken by government or state actors to degrade democratic institutions and procedures that impart horizontal or vertical constraints on government power.” She added that “the imparting of constraints refers to the institutions and the procedures, not the backsliding itself. Backsliding is the removal of constraints on power, typically by the executive.”
Dresden explained further that use of the term “democratic” does not necessarily mean a country is already democratic. Many democratic institutions exist (e.g., elections, political parties), she observed, but not many democracies in practice. However, she elaborated, backsliding can happen in full democracies—a phenomenon that has been on the rise since 2014—as well as in hybrid regimes. She clarified that the latter regimes may appear democratic in that they usually have democratic institutions, such as elections, but they are also likely to have elements found in authoritarian regimes. She cited Turkey as an example of a country that had been moving toward becoming a democracy but has been backsliding in recent years.
Although backsliding is not new, Dresden explained, it began to gain attention from the policy community only in 2007 after the failure of the color revolutions (a collection of nonviolent uprisings in the 2000s). However, she asserted, the phenomenon is important to international security in two ways. First, she said, “many of the international institutions that we think of as being stabilizing in the international community, like NATO and the European Union, for example, have as part of their core foundation an expectation that there will be adherence to a certain set of democratic norms and practices. When those start to erode, you start to see countries getting a little uncomfortable with one another.” For example, she suggested, recent backsliding in Hungary and Poland has caused some to question the stability of these countries. She added that security may also be compromised when backsliding encourages citizens to flee their country, sometimes without formally applying for asylum, as is happening in Venezuela.
Backsliding, Dresden continued, tends to come in three different forms: coups d’état (the overthrow of executive powers by security forces), expansion of executive power (by removing term limits or making changes to the legal structure), and electoral manipulation (which can include the jailing of opposition leaders). However, she noted, consensus is lacking on what causes backsliding to happen, and existing research on this question using forecasting models is not directly relevant to current circumstances. She suggested that part of the problem may be that researchers have “measure[ed] backslides in terms of the collapse of a government regime” but that backsliding is not always that clear-cut: it may also be a relevant way to analyze gray areas in terms of what can be considered democratic.
Table 4-2 shows a matrix developed by Dresden, designed to clarify various causes of backsliding identified in prior studies. There is some agreement, she explained, on backsliding that falls in the structural/institutional category. “The higher-quality democracy you have, the stronger your institutions are,” she observed, “the less likely you are to experience backsliding.” One risk factor on which researchers do agree as significant, she pointed out, is polarized political competition. Referring to the presentations on the strategic use of information summarized in Chapter 3, she noted that polarization is a method often employed in cyberwarfare.
According to Dresden, the literature often suggests that democracies are especially vulnerable to backsliding in the months leading up to an election. She argued, however, that this view probably is no longer accurate given that researchers are beginning to see backsliding behavior begin well before an election is scheduled. She noted that backsliding strategies are often used when incumbents worry that their position in the current system is no longer secure. To protect their status, these incumbents may take steps—such as eliminating term limits—to maintain their position in the future.
TABLE 4-2 Factors That Increase or Decrease the Risk of Authoritarian Backsliding
|Risk Increasing||Risk Reducing|
|Robust civil liberties
|AGENT-CENTRIC||Polarized political competition
Violent acts by opposition
International “autocracy promotion” (maybe)
|High margins of victory in free and fair elections
SOURCE: Adapted from Jennifer Dresden presentation at workshop.
Dresden closed with a few final thoughts on backsliding. First, she observed that most forms of backsliding are not as dramatic and abrupt as one might expect. Backsliding is often a collection of a series of subtle actions that slowly erode democracy. Furthermore, she noted, “the background conditions that put a country at risk [of backsliding] are not the same as the proximate conditions that trigger it.” “Institutional weakness can persist for a very long time before the opposition actually becomes threatening and motivates a response from the incumbent,” she elaborated. She also argued that more research is needed to understand what motivates an incumbent to use one strategy over another in a given situation. Finally, she said, because backsliding is often reactive, the “interaction between the incumbent and opposition actors is central to backsliding processes.”
According to Christopher Gelpi, Ohio State University, quantitative research in international relations is often dismissed as not being policy-relevant. In his presentation, he challenged this assumption by arguing that some forecasting methods can actually bridge theory and policy.
Gelpi began by identifying two ways in which researchers attempt
to make policy-relevant predictions. The first is the traditional method of collecting data, examining the data using statistical analysis, and then making a prediction based on that analysis. The second method, according to Gelpi, is to use forecasting models, three types of which are frequently used in the fields of international relations and comparative politics: game theoretic models, which study interactions between decision makers in competitive situations; time series models, which involve the coding and analysis of event data over a period of time; and structural models, which make a prediction by using an estimated set of coefficients to measure data that have been collected on a set of covariates. Gelpi asserted that structural models are the best method for connecting theory and policy because—unlike game theoretic and time series models—they are both theoretical and generalizable.
According to Gelpi, another good method for bridging theory and policy is the random forest model. Using the prediction of civil wars as an example, he explained that the random forest model takes the variables provided by researchers and uses an algorithm to divide that dataset into “civil war” and “not civil war” cases.4 Gelpi explained that, because the random forest model yields a relatively small number of false positives, it is a good choice for providing a policy-relevant forecast.
According to Gelpi, several methodological challenges can be addressed with the use of forecasting methods. The first such challenge he identified was “overfitting or making models excessively complex.” For example, he explained, if a researcher applies a theory against a set of cases, such as alliances in the Cold War era, but the theory does not work, applying a new theory to predict the same cases can make it difficult to know whether the researcher is “crafting an explanation that generalizes” or “just overfitting the model to the data that we happen to observe.”
Another challenge Gelpi has observed is that both quantitative and qualitative researchers will often overgeneralize models. For example, he noted, a quantitative scholar may “run an analysis on data from the Cold War and say democracy is correlated with peace,” and then, based on that information, make “a very general claim [that] democracy causes peace across all space and time,” which likely cannot be supported by the data.
Gelpi has also found that researchers will choose theories with effects that are statistically significant, but it is unclear which effects are substantively significant. This may occur if a researcher has generated marginal effects from models in which all variables but one have been held constant. However, this is not realistic, Gelpi argued. “It’s like imagining an
4 Muchlinski, D., Siroky, D., He, J., and Kocher, M. (2015). Comparing random forest with logistic regression for predicting class-imbalanced civil war onset data. Political Analysis, 24(1), 87–103.
authoritarian Canada or something like that,” he said. “You imagine what Canada’s foreign policy would be like if they were authoritarian. Authoritarian Canada is not a thing. There’s a whole combination of other things that are correlated with a democracy.”
To demonstrate how forecasting can help correct overgeneralization, Gelpi shared the results of a study he recently conducted with a former graduate student.5Figure 4-3 depicts a model of military conflict over a period of time. Gelpi explained that he and his colleague “found that there were different eras of international politics where the causal model of war was really different.” In Figure 4-3, Gelpi noted, the receiver operating characteristic (ROC) curves and the x-axis are flipped so that the scale can be read to show that “the vertical axis is the true-positive rate and the horizontal axis is the false-positive rate.” He added that the bottom line represents both forecasts “trying to predict militarized disputes in the Cold War era.” The top model, he continued, which has been calibrated based on Cold War–era data, has a strong ROC curve (the top line curved toward the upper left), whereas the bottom model, which was calibrated “on the interwar period and then forecasted onto the Cold War period,” has a much weaker ROC curve, indicating that this model is a much weaker fit.
Gelpi then shared results of a study he conducted to forecast transnational terrorism.6 One finding of this study, he reported, was “that democratic states are likely to be targets of terrorism” (see Figure 4-4). He and a colleague then carried out a forecasting analysis to “see how much… including or excluding democracy from our forecasting model actually changes our ability to forecast terrorist attacks.” According to Gelpi, they found that democracy as a variable contributed very little to their model’s ability to predict terrorist attacks.
Gelpi closed by reiterating that structural models can be a useful tool in making policy-relevant predictions. However, he stressed, structural models should not be used to identify a causal connection; rather, the forecasting models described in his presentation should be supplemented with a causal inference model, such as statistical matching.
5 Jenke, L., and Gelpi, C. (2016). Theme and variations: Historical contingencies in the causal model of inter-state conflict. Journal of Conflict Resolution, 61(10), 2262–2284. doi:10.1177/0022002715615190.
6 Gelpi, C., and Avdan. N. (2015). Democracies at risk? A forecasting analysis of regime type and the risk of terrorist attack. Conflict Management and Peace Science, 35(1), 18–42.
Fry began by stating that the Intelligence Community (IC) values and utilizes forecasts from both the natural and the social sciences. However, she said, only a small portion of the analytic community understands how to use them. She pointed out that the majority of the IC’s customers are policy makers, who are more comfortable making decisions when quantitative data are translated into a narrative. She emphasized the importance of Gelpi’s discussion of connecting forecasting with theory, because if an analysis begins with a well-developed theory, the analyst will find it easier to communicate the information in narrative form.
According to Fry, it would also be helpful if analysts were given guidance as to when history is no longer a helpful guide. Advances in technology, for example, may cause historical data to be irrelevant in some situations, she argued.
Fry added that disciplinary boundaries can also create forecasting challenges. Scholars in comparative politics and international relations often
have limited knowledge of theories and methodologies outside of their area of expertise, she noted, suggesting that it would be beneficial if scholars could overcome these limitations.
Fry also suggested three types of data that she believes should be represented more commonly in forecasting models. The first is demographic data. According to Fry, much of what is known about political behavior is based on demographic data. Furthermore, she asserted, it is important for the IC to maintain access to a variety of demographic data. Equally important, she suggested, are data on subnational structural phenomena and the associated behavioral phenomena. Finally, she highlighted systematic data on policy treatments as important to include in forecasting models. Such data are often not available when needed, she argued, hampering an analyst’s ability to provide timely information.
According to Fry, detecting changes in the balance of power or influence of major powers internationally is a topic of particular interest to the IC. She added that identifying core systemic risks in the international community could also be useful to the IC. Systemic risk analysis is a well-developed methodology in the natural sciences, she observed, but is not applied as often as it could be in the social sciences.
Turning to regime types, Fry asserted that, in addition to hybrid regimes, administrative capacity—whether things are improving or deteriorating—is another dimension of governance that requires more research. Moreover, “when you think about outcomes of interest to forecast,” she suggested, “make sure that you scrub them for the types of political or ideological biases they might be containing because we want to make sure that these tools are useful over time.”
Fry closed by responding to Ernst’s question about what motivates the IC to act upon warning statements and forecasts. She explained that one of the most important factors in motivating action is the ability to explain a forecast. To give an example, she noted that when the Ebola crisis occurred the IC was flooded with forecasts anticipating incredibly high death tolls, but analysts were so inundated with these reports that they began not to take them seriously. In such a situation, Fry suggested, it would be helpful if forecasts were provided with interpretive guidance that clearly explains the research design and data analyzed.
Addressing all panelists, a participant opened the discussion session by asking them to provide their thoughts on “the mix of strong or weak theory and mix of inductive or deductive elements” in forecasting with respect to their discipline.
Ernst responded that when thinking in terms of theoretical models
it is helpful to have an initial layer that is rooted in biological processes, which provides well-established parameters that dictate the baseline risk. However, she suggested, epidemiologists have not done a great job of incorporating elements from the social sciences. She said that while listening to the panel presentations she began to realize just how relevant public health and the sociopolitical sciences are to one another; for example, an epidemic could be a factor pushing a state to the point of instability. She argued that more work is needed to develop a theoretical model that reflects this connection between pandemics and political stability.
Siddiqi noted that much of the work discussed in her presentation related to decision analysis. Decision analysis, she said, is rooted in the assumption that options are being reviewed and weighed by rational and logical decision makers. Looking at cases in this way, she suggested, is a rather simple approach to developing simulations that provide a number of future possibilities. She also pointed out that decision analysis is well established, theoretical, and practical.
Dresden asserted that theory is crucial to understanding backsliding. She stated that such questions as What are the areas we should be paying attention to? and What changes would meaningfully degrade the accountability in a system? are theoretically driven. She also pointed out that theoretical considerations are relevant to answering questions about the strategic interactions between incumbents and opposition groups. Finally, she suggested that inductive work is needed to increase the accuracy of models so they can be used to consider such questions as Who counts as the opposition? and What is the set of possible actions or strategies or tactics in any given context?
Gelpi suggested that international relations theory is strongest in the middle range, which involves examining well-established theories on such questions as how democracy and trade affect the likelihood that force will be used. However, he added, more work is needed on how to integrate these questions. He asserted that linearity is often imposed when it should not be, and that “the random forest model is good because it does not impose linearity.” Gelpi also suggested that more work is needed on scope conditions. “I think we have a bunch of tools that we can apply where we know these things are associated with violence in various ways,” he said, “but exactly how they fit together I think is weaker.”
Another participant asked whether the IC agrees with the view that predictions are better if they are cast in specific quantitative terms. Fry responded by pointing out that not all people think in terms of numbers and probabilities. Furthermore, she observed, predicted percentages are often presented using such estimative phrases as “roughly 10 percent” or “70 percent likelihood.” To understand what these numbers mean, it is important that the IC have access to two different types of forecasters to aid in the
decision-making process: individuals with deep expertise in a specific area and people that know “a lot about lots of different things.” She added that, although deep country experts or deep functional experts may work with many different models, “they tend to be variants on one phenomenon.” On the other hand, she continued, those that know “a lot about lots of different things” are often very good at forecasting because they have access to a variety of models that allow them to “think about different ways of envisioning an outcome.” It is also important, she stressed, to know which type of forecaster is needed in a given situation.
Gelpi added that quantitative models are helpful because they force people to “be specific and concrete” in defining the variables and measurements being used. He also seconded Fry’s comments, and suggested that scholars need to be careful about being so committed to a particular set of theories that they become unwilling to change or develop other methods.
This page intentionally left blank.