Computational modeling of social processes has been used for many years in numerous disciplines for a variety of purposes, including assisting in public policy decisions. Computational models can be used to inform the regulatory process and can “range from single parameter linear relationship models to models with thousands of separate components and many billions of calculations” (NRC, 2007, p. 36). Models have been used to forecast the health effects associated with risk behaviors, including tobacco use. For example, several population dynamic models have been used to simulate the dynamics of smoking and smoking-attributed deaths in a state or nation and the effects of policies on those outcomes (HHS, 2014). Since 2009 the U.S. Food and Drug Administration (FDA) has had broad regulatory authority over tobacco products and has used models as one tool to inform its policy decision-making activities. Recently, FDA has been exploring the usefulness of a particular computational modeling approach—agent-based modeling (ABM)—to inform its policy decisions.
Thus, the FDA Center for Tobacco Products (CTP) asked the Institute of Medicine (IOM) to review an ABM developed for use by FDA; to comment on its strengths, weaknesses, and usefulness for examining various tobacco regulatory policies; and to provide recommendations on strategies to improve the model and for using ABM to inform decision making in the future. To address that request, the IOM created the Committee on the Assessment of Agent-Based Models to Inform Tobacco Product Regulation (see Box S-1 for the full committee statement of task).
CTP has several reasons for its interest in using ABMs to inform tobacco control policy, including their potential for exploring individual-
BOX S-1
Committee on the Assessment of Agent-Based Models to Inform Tobacco Product Regulation Statement of Task
The Institute of Medicine (IOM) shall convene a committee to assess the applicability of agent-based models of tobacco use and public health as a guide to inform regulators and improve the effect of tobacco regulation policies on public health. The committee shall:
- comment on implications of using agent-based models to examine various tobacco regulatory policies
- assess the strengths and weaknesses of an agent-based model developed for the U.S. Food and Drug Administration (FDA) (to be provided by the Center for Tobacco Products [CTP]) and models currently available in the literature that have been used for similar purposes (to be identified by CTP)
- make recommendations on future directions and strategies to improve the usefulness of the model developed for or to be used by FDA, if needed
level factors that dictate tobacco use and their ability to simulate potential use patterns in an evolving market (Fultz, 2014). It is important to note that the committee formally assessed only one ABM in this report, and although lessons from the development of that model may be applied to the development of future ABMs, this report’s conclusions are not indicative of the strengths or limitations of other tobacco control ABMs or of tobacco control models using other modeling approaches. This report is meant to build on the large body of work on tobacco use modeling by exploring how ABMs might be a helpful tool to add to the existing tobacco control modeling toolkit.
BACKGROUND
What Are Agent-Based Models?
An ABM is a type of computational model that is used to study complex systems by exploring how individual elements (agents) of a system behave as a function of individual characteristics and interactions with each other and the environment. Each agent interacts with other agents based on a set of rules and within an environment specified by the modeler, which leads to a set of specific outcomes, some of which may be unexpected. As ABMs
can be used to explore the potential impact of policies and interventions in dynamic social and physical environments, ABMs may be a useful tool to aid in decision making by policy makers. ABMs have been used to examine other public health interventions and policies, such as for infectious diseases (Epstein et al., 2007; Lee et al., 2010) and obesity (Auchincloss et al., 2011; Orr et al., 2014; Zhang et al., 2014), but they have not been fully explored and considered in the tobacco regulatory space.
Complex Tobacco Environment
Tobacco consumption continues to be the leading cause of preventable disease and death in the United States (HHS, 2014). More than 42 million Americans, representing 18 percent of the population, currently smoke cigarettes (Agaku et al., 2014; Jamal et al., 2014). Each day more than 3,200 children under age 18 smoke their first cigarette, and more than 700 children become daily cigarette smokers (SAMHSA, 2013). Many of these youth will become addicted and suffer adverse health consequences. At the current smoking rate, 5.6 million children alive today will die prematurely from smoking-related illness (HHS, 2014).
On June 22, 2009, the Family Smoking Prevention and Tobacco Control Act (Tobacco Control Act) gave FDA the authority to regulate the manufacture, distribution, and marketing of tobacco products—specifically cigarettes, cigarette tobacco, roll-your-own tobacco, and smokeless tobacco—to protect public health and reduce tobacco use in the United States. To oversee the implementation of the law, FDA established CTP, which works to prevent tobacco product use initiation, encourage current users to quit, and reduce the overall harm caused by tobacco use. New policies and regulations must be based on available medical, scientific, and other technological evidence as appropriate for the protection of the public’s health. Consequently, CTP is interested in forecasting the public health effects of potential changes in tobacco standards and other policies.
As described in Chapter 2, understanding the complicated environment in which tobacco products are used and sold is essential when attempting to model potential tobacco policies. This includes an understanding of the various tobacco products available and their addictive nature as well as tobacco use behaviors, including tobacco use initiation, progression, and cessation.
Center for Tobacco Products and Agent-Based Modeling
Through an interagency agreement between CTP and the U.S. Department of Energy, CTP commissioned Sandia National Laboratories to
develop an ABM that could help CTP understand the potential impacts of a variety of tobacco control policies on population health.1 The ABM developed by Sandia National Laboratories for CTP is titled Social Network Analysis for Policy on Directed Graph Networks (SnapDragon). The main purpose of SnapDragon is to explore the effects of various tobacco policies and interventions, such as public education campaigns, on opinion and tobacco use within social networks.
REPORT FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS
Why Use Agent-Based Models to Explore Tobacco Use?
Existing models in tobacco control have focused mostly on determining the long-term dynamics of population-level tobacco rates. These analyses have employed almost exclusively aggregate compartmental or system dynamics models, which assume a large degree of homogeneity among the population and generally do not consider interactions among members of the population. Given that smoking is largely related to social- and individual-level behaviors, it is becoming evident that these processes need to be modeled to understand the effects that a policy may have. Although analysis of survey data can help researchers identify the nature and strength of these social determinants of smoking behavior at the individual level, ABMs are needed to estimate the total population effects of those individual interactions. ABMs can account for individuals’ differences and the many ways in which such individuals can influence each other to estimate the combined effect of the multiple processes that constitute tobacco use behavior. These models can also account for important feedback mechanisms that have been, for the most part, ignored by existing aggregate models.
Given the strong social component inherent to tobacco use onset, cessation, and relapse, and given the heterogeneity of those social interactions, ABMs have the potential to be an essential tool in assessing the effects of policies to control tobacco. Many of the questions FDA faces require an understanding of the underlying behavioral mechanisms involved (e.g., initiation and cessation) and would require a model of those processes before those specific questions could be explored. Within the modeling community, it is often said that models need to be motivated by a specific question to be effective. However, the processes or mechanisms underlying these policy
_____________________
1The modeling efforts by Sandia National Laboratories under this agreement have included population health models that aim to help forecast potential long-term impacts on prevalence, morbidity, and mortality for the population in the United States as well as other types of models, including an ABM.
questions often need to be the focal point of the model before the specific question is addressed.
Finding 2-1: The committee finds that for many tobacco control policy questions, several key underlying processes—initiation, cessation, and relapse, among others—drive overall rates of tobacco use and have a strong social interaction component. An agent-based model could be a useful tool to represent these processes.
In the case of tobacco, a useful path will be to develop models of these processes first and to then apply them to the specific policy question. This does not imply that all efforts should be put into a single model of social processes which would then be applied to many different questions. Rather, accurately representing the underlying process of initiation, cessation, and relapse is, in some cases, essential to the development of a model of tobacco use behavior.2
Mechanisms That Generate Feedback Between Behavior and Social Environments
Policies can backfire when they fail to account for how people change their behaviors in response to an intervention, as individuals’ behaviors often depend on the behaviors of other people and on features of the social environment. A central challenge for policy makers, therefore, is to anticipate how organizations, corporations, and individuals will react to changes in incentive structures and features of the environment. However, anticipating this response can be difficult for several reasons, including limited knowledge of human behavior and the complex interactions that occur between individuals and the social environment.
Structural models,3 to which policy makers have long looked to guide policy decision making, typically attempt to uncover behavioral relation-
_____________________
2It is important to note that there are other features of tobacco control policy that are not directly related to initiation and cessation (e.g., tobacco companies responding to FDA regulatory changes in an attempt to undermine those changes), so the modeling decision to focus on a specific policy question versus initiation or cessation needs to be discussed early in model conceptualization.
3Structural models use a set of equations or rules, expressed analytically or in programming code, that describe different possible worlds. The specification of the model is dictated by theory, prior knowledge, and other inputs that determine what features of a given process to highlight and what to leave out. These assumptions, combined with data, produce a set of inferences about what will happen under a given set of conditions. This modeling approach includes (but is not limited to) macro-level simulation models, such as system dynamics, and individual-level models, such as ABMs. The appropriateness of a given modeling strategy depends on the theory brought to bear and on available empirical evidence.
ships or parameters that are invariant to specific circumstances or take those circumstances as inputs that condition behavior. Evidence across a number of policy domains suggests that if the incentives or risks associated with a given behavior are changed, people will likely behave differently. Thus, a non-superficial understanding of the incentives that drive behavior is required. To be useful for informing regulatory policy, modeling efforts must capture meaningful aspects of the social process under investigation. It is not enough to hypothesize different mechanisms and use a model to determine whether they lead to different outcomes. The model may be misspecified to the point where a “sensitivity analysis”4 provides no information at all on the true sensitivity of model outputs to inputs (Sanstad, 2015). At higher levels of aggregation, the behavior of organizations and other coalitions are also contingent on behavioral incentives. Failure to account for those incentives may lead to unexpected and undesirable results. The goal is to identify how people’s or organizations’ behaviors might change under a different incentive structure.
Conclusion 3-1: The committee concludes that a deep understanding of human behavior, decision making, and incentive structures is important for agent-based models and other models that are used to understand how interdependent behaviors shape the outcomes of a given policy. Regardless of the model type, if the behavior is not plausible, the model is not likely to be informative.
Recommendation 3-1: When developing an agent-based model (or similar modeling approach), the Center for Tobacco Products should consult with subject-matter experts to identify the plausible behaviors and focal processes at work from the beginning of the model development process.
Microsimulation and Agent-Based Models
Within the domain of individual models, some scholars semantically distinguish between two types: microsimulation and agent-based models. However, both involve the same basic procedure—assigning artificial agents a behavior and using simulation to assess the aggregate implications of that behavior—and both approaches are operationalized through computer code. This is an important commonality because if microsimulation and ABMs are viewed as two distinct approaches, their two research communities will be less likely to benefit from each other’s work. The committee
_____________________
4Sensitivity analysis is “an exploration, often by numerical (rather than analytical) means, of how model outputs (particularly QOIs [quantities of interest]) are affected by changes in the inputs (parameter values, assumptions, etc.)” (NRC, 2012, p. 117).
found that from a purely technical standpoint, microsimulations and ABMs are the same modeling enterprise and that they are differentiated mainly by differences in how they tend to be deployed, including in how agent interactions are specified and how agents’ environments are abstracted. Although there are no fundamental differences between ABMs and microsimulations, there are historical differences in how these models have been specified and used by their research communities.
Conclusion 3-2: Researchers who use the terms agent-based modeling and microsimulation have different approaches to model specification. However, the committee concludes that from a technical standpoint these are the same enterprise (an individual-level model implemented via computer code). The committee believes that modelers would greatly benefit from best practices and lessons learned from applications that have been performed by the two research communities to address policy questions.
High-Dimensional Models and Low-Dimensional Models
The appropriate level of model detail and empirical realism is a choice that modelers need to make. The appropriate level of model detail depends on the research question, the intended use of the model, and the data available to empirically ground the model. It is important to note, however, that at whatever level, models provide only an imperfect representation of the real world, as computational models in general are not reality mirrors, nor are they intended for this purpose. ABMs can represent anything from low-dimensional, abstract worlds where agents are defined by just one or two attributes and interact in a highly stylized environment based on simple rules to high-dimensional, highly detailed worlds where agents have many attributes, the environment contains a great deal of information, and agents engage in multiple behaviors. It may be tempting to create ABMs that pull in as much empirical data and knowledge as possible in an attempt to create a highly realistic “laboratory” to explore policy questions. However, this approach is not usually the most productive, because available data and knowledge of human behavior are almost never adequate to achieve this. ABM allows the developers to explore the importance of various mechanisms in the face of no data and to assess the potential value of collecting these data; however, this introduces an added layer of uncertainty and raises the possibility of model misspecification. Also, the model can become cumbersome and hard to manage when additional layers of detail are added, and it can be difficult to get clear analytic results. The success of a model is not determined by the level of granularity at which it represents a process; rather its success is based on how successfully it facilitates the understanding of the problem or question under study.
Conclusion 3-3: The committee concludes that low-dimensional and high-dimensional models have complementary virtues and weaknesses. A more complicated model may have greater verisimilitude, but added detail per se does not ensure realism. A low-dimensional model, while abstracting from some features of the real world, may generate forecasts that are easier to understand and interpret.
Recommendation 3-2: The Center for Tobacco Products should develop and employ both low- and high-dimensional models, using both as appropriate to shed light on policy impacts.
Making Decisions with Partial Knowledge
Models cannot predict the future with certainty. They provide only a partial representation of reality and have some level of abstraction. Models can mislead policy making if modelers present findings with greater certitude than is warranted. These challenges are pertinent to any type of model that seeks to inform policy, not just ABMs. A good model will quantify how uncertainty in model inputs translates into uncertainty in the likely outcomes of various policies and will generate a range of predictions that reflect that uncertainty (Manski, 2013; Wagner et al., 2010). The key challenge is separating what is known from what is unknown. Note that this is a very different enterprise from conducting a “parameter sweep” type sensitivity analysis, which merely provides more insight into the workings of the model itself and not into the relationship between the model and the actual world. Once analysts have generated a set of credible model outputs, they must use that information to draw a conclusion about the best course of action. The challenge for the policy maker is to evaluate candidate policy outcomes and weigh the risks and benefits. Thus, to use these models effectively to guide policy decisions, the model user needs a rule for translating these uncertain predictions into a policy decision.
Conclusion 3-4: The committee concludes that the common exercise of sensitivity analysis does not suffice to measure the uncertainty in model-based forecasts. Sensitivity analysis may provide some insight into the workings of the model itself, but it does not per se assess the potential relationship between model findings and the real world.
Recommendation 3-3: When the U.S. Food and Drug Administration uses the findings of any model, the agency should take into account the uncertainty of findings in order to evaluate policy outcomes and weigh the risks and benefits appropriately.
An Evaluation Framework for Policy-Relevant Agent-Based Models
Policy-relevant ABMs are complex, resource-intensive technical activities that are developed by large groups of people with varying areas of expertise and whose results need to be translated and communicated to various stakeholders in order to affect policy and improve health. Policy-relevant ABMs need to be built carefully using appropriate data and social science theories, rigorously tested, and clearly communicated. These requirements for ABMs are the same as for other types of computational models and simulations used to inform policy decisions. Given the amount of time, effort, and money required to build an effective policy-relevant model, it is critical to evaluate the process, its outcomes, and its overall value. Chapter 4 presents an evaluation framework for policy-relevant ABMs developed by the committee. Such an evaluation framework can help model developers improve their modeling efforts, help funders understand better how to use model results and how to guide future funding of modeling work, help policy makers understand how to translate model results into more effective policies and increase their trust in the analysis, and help modelers and scientists by suggesting new avenues for research, modeling, and data collection.
Fundamental Evaluation Categories
The evaluation framework developed by the committee has five fundamental evaluation categories that the committee believes need to be included in most ABM evaluations:
- Resources: The modeling team needs access to adequate financial, infrastructure, human, and knowledge resources to successfully design, build, and test the model.
- Technical Best Practices: Model implementation, testing, and validation phases need to be reviewed throughout model development.
- Model Suitability: Models need to be developed in a manner that makes them suitable for their intended purpose and allows for exploration or testing of specific policy options or conditions. Some models could be developed for very narrow questions related to tobacco use, others as a broad tool to look at a larger range of tobacco policies.
- Communication and Translation: Communication and translation strategies are essential during every stage of model development for enhancing the model-building process and ensuring that the model is focused on the key issues that will affect policy outcomes. Modeling requirements, descriptions, and results need to be com-
municated effectively to a variety of audiences, including agency staff, regulators, politicians, and the general public.
- Policy Outcomes: Ultimately, policy-relevant models will be used to inform policy and regulatory action or to advance scientific progress.
Recommendation 4-1: The Center for Tobacco Products should adopt an evaluation framework for its modeling work, either the one presented in this report or one similar to it. Key dimensions of the evaluation framework should include considerations of resources, technical best practices, model suitability, communication and translation, and policy outcomes. The evaluation plan should be designed early in the model development process and should be carried out throughout model development.
This evaluation framework would apply to all efforts funded by CTP (internal model development, interagency agreements, contracts, and grants). The evaluation—as well as periodic peer review5—should come from external experts in addition to internal reviewers. If CTP chooses to adopt the framework outlined by the committee, it should be used as a guideline and not as a mechanical exercise or checklist, as different ABMs will require differing evaluation strategies based on their intended use, modeling approach, and other aspects of model development.
Review of the Social Network Analysis for Policy on Directed Graph Networks Model (SnapDragon)
As a major component of its statement of task, the committee was asked by FDA to review the FDA-commissioned ABM developed through an interagency agreement with Sandia National Laboratories, entitled Social Network Analysis for Policy on Directed Graph Networks (SnapDragon). Chapter 5 describes and analyzes SnapDragon, applying the evaluation criteria from Chapter 4 where appropriate, and discusses the model’s usefulness for informing tobacco control policy. The SnapDragon model has not been published in a peer-reviewed journal, but two manuscripts on the model were undergoing peer review in two different journals during the course of this study and have since been accepted for publication (Moore
_____________________
5See the National Research Council (2007) report on modeling for guidance on peer review of models.
et al., in press a,b).6 The committee reviewed the model as it was specified as of July 31, 2014.7
To date, the SnapDragon model development team has focused primarily on the effects of multiple competing high- and low-opinion messages in social networks, illustrated through the study of education campaigns, in a single- or multiple-tobacco-product environment. The model distinguishes individuals by any number of characteristics and, in particular, according to their use of a variety of tobacco products, which allows for the investigation of the simultaneous use of different products. Currently, the model classifies individuals as either “users” or “nonusers” for each tobacco product under consideration. The user status is determined by an underlying construct termed “opinion.”8 Each individual carries an opinion about each tobacco product under consideration, which drives tobacco use behavior. SnapDragon explicitly models the time trajectory of individuals’ opinions as a result of their interactions with other individuals, and the modeling choice is based on theory stemming from the field of opinion dynamics. In the model, individuals are connected to others through predefined social networks. Connected individuals can affect each other’s opinions if such opinions do not differ by more than a specified tolerance range. As time goes on, individuals continuously adjust their opinions toward a weighted average of the opinions of the individuals who can influence them. The weights (or plasticity values) represent the importance given to the opinion of particular individuals. These plasticity values are not necessarily reciprocal, meaning that any two connected individuals can assign a different weight to the opinion of the other. As opinions adjust, they drive tobacco use behavior (i.e., become a user or a nonuser of a specific tobacco product). That is, nonusers whose opinions about the product increase beyond a certain level (termed the initiation threshold) become users of such tobacco product,
_____________________
6The draft manuscripts reviewed by the committee and other supporting documents are available upon request from the project public access file: http://www8.nationalacademies.org/cp/ManageRequest.aspx?key=49612.
7In addition to the draft manuscripts (dated May and November 2013), the committee received more information on SnapDragon from in-person presentations by the model developers during two open information-gathering sessions as well as from written question-and-answer documents exchanged between the committee and the SnapDragon development team. In July 2014 and January 2015, draft descriptions of the committee’s technical understanding of the model were sent to the model developers for technical review. In their January 2015 response, the developers noted several changes to the model that occurred after July 31, 2014, and identified additional changes they planned to make in the future. However, the review by the committee is based on the model as it existed on July 31, 2014.
8The modeling team defines “opinion” as an aggregate concept that captures the overall positive or negative attitude of a person toward a tobacco product. It is represented as a continuous variable, taking values between 0 and 1, with 0 standing for the most negative attitude of a person toward a tobacco product and 1 the most positive.
and users of a particular product whose opinions about such product fall below a certain level (termed the cessation threshold) become nonusers of such product. Interventions that can potentially influence individuals’ behaviors, such as tobacco control efforts, are modeled as modifying either the opinions of individuals about a certain tobacco product or the opinion thresholds that delimit possible user status.9
Review of SnapDragon
In Chapter 5 the committee provides a detailed review of the SnapDragon model; the key findings and conclusions are described here. While SnapDragon has been designed to evaluate a wide range of tobacco products, the committee focused on how the structure of the model can accommodate known facts about smoking behavior.
SnapDragon presents a novel framework to study the impact of policy measures on smoking initiation, cessation, and prevalence by attempting to model explicitly the processes of initiation and cessation as driven by social interactions. Instead of relying on externally supplied inputs for initiation and cessation rates that were determined outside the model, the model tries to derive these figures endogenously, by proposing a hypothesis about how these processes are generated. Specifically, SnapDragon attempts to explain the dynamics of tobacco use (i.e., how the system changes over time) as a result of a convergence of opinions about specific tobacco products through the interaction among individuals in the population, guided by a formulation from the field of opinion dynamics (see Chapter 5).
However, the committee found that several elements in SnapDragon’s formulation either do not conform to existing knowledge about tobacco use or defy face validity. First, the model does not consider a feedback mechanism from behavior to opinion. It is almost certain that the experience of using a particular tobacco product would influence the user’s opinion about such product. As SnapDragon only considers that behaviors are modified through opinions, this suggests that the model is missing an important feedback mechanism from behavior to opinion. Second, it is very unlikely that opinions about tobacco products are transmitted independently of individuals’ behavior toward such products, as SnapDragon stipulates. This formulation could lead to highly unrealistic scenarios, as explained in Chapter 5.
It is conceivable that imitation of smoking behavior could play a role in tobacco adoption. However, in SnapDragon the imitation component happens indirectly—by individuals sharing their opinions about a product—rather than directly through behavior. As opinions are not
_____________________
9A detailed description of SnapDragon is available in Chapter 5.
influenced by behaviors in the model, the observed number of tobacco users in the population will not affect the rates at which new adopters are generated. Similarly, observed quitting behavior cannot be imitated directly in the model.
Third, the rationale behind the modeling choice of making interacting opinions converge to a weighted average is not clear. This modeling choice, when applied to smoking behavior, can lead to inconsistencies with observed facts. It is likely that other mechanisms, not reflected in SnapDragon, play an important role in modifying smoking behavior throughout an individual’s lifespan. Furthermore, SnapDragon does not identify former smokers. As such, the model in its current form can track product prevalence but cannot accurately determine health risks,10 as a significant proportion of tobacco-related morbidity and mortality falls on former users of combustible products (HHS, 2014). These and other limitations of the model—including other aspects of tobacco use behavior, equilibrium patterns, and the use of data in the model—are outlined in Chapter 5.
The developers of SnapDragon have suggested that it could be used for a number of tobacco control policy applications, but the underlying assumptions of the model (as discussed in Chapter 5) suggest that this is unlikely. The committee statement of task calls for recommendations for improvement of SnapDragon, if needed, and although some changes could be made to address some of the weaknesses identified in this report, doing so would lead to the creation of a new model. SnapDragon does not encompass essential facts from the tobacco research literature, and many of its assumptions lack face validity. In addition, the data required to inform the parameters in SnapDragon have not yet been identified, and the model has not yet reached the stage of model validation for broad application to tobacco control policy. While SnapDragon is a very flexible model that can be manipulated in various ways to reproduce certain observed facts about tobacco use behavior, it currently lacks sufficient modeling structure to be informative for policy. Therefore, the committee has not included recommendations for improvement. Key findings and conclusions regarding SnapDragon are below:
Conclusion 5-1: As SnapDragon presumes that opinions may modify behavior but behavior does not modify opinion, the committee concludes that the model is missing an important feedback mechanism from behavior to opinion.
_______________________
10Although determining health risks was not listed as one of the purposes of SnapDragon, if CTP plans to use SnapDragon as a stand-alone model, this is a limitation. If CTP plans to use SnapDragon only to inform population models, this is not a limitation of the model.
Finding 5-1: The committee finds that the representation of behavior in SnapDragon does not align with what is currently known about tobacco use and dependence.
Conclusion 5-2: The committee concludes that the modeling decision of making interacting opinions about tobacco converge to a weighted average is not supported by evidence and is unlikely to be an accurate representation of tobacco use behavior.
Finding 5-2: Whereas some other models based on opinion dynamics have been able to replicate the equilibrium patterns of socially driven processes, the committee has not found applications in which the specific time path to equilibrium has been empirically validated.
Finding 5-3: The committee finds that there has been no assessment of SnapDragon’s ability to accurately predict initiation, prevalence, or cessation.
Conclusion 5-3: The committee concludes that a realistic parameterization of SnapDragon would be hard to achieve, so it is unlikely that the model will be able to generate credible assessments of policies.
Recommendation 5-1: SnapDragon should not be pursued by the Center for Tobacco Products as an aid for regulatory decision making.
Data Collection and Model Development at the Center for Tobacco Products
Chapter 6 provides a high-level overview of existing tobacco use data sources, identifies data gaps, and makes recommendations for the future implementation of ABM at CTP. Various types of existing data sources related to tobacco use can be used to inform and strengthen ABMs, but these sources do not contain all relevant agent attributes, behaviors, and social and spatial interactions related to tobacco use. One approach to access such data would be to try to maximize the use of available administrative data from states and regions, but except in unusual circumstances, this information is unlikely to contain many of the behaviors and interactions wanted. Another approach would be to combine data from various sources, such as large-area administrative information and small-area detailed surveys. However, using such combinations would require considerable care. A longer-term approach would be to try to anticipate critical data needs and fund or otherwise encourage the collection of data that best suit ABMs or other modeling approaches. Similarly, encouraging the standardization of
data collection items and methods might improve model quality. Even for administrative data that are “routinely” collected, such as tobacco marketing or sales information or population smoking prevalence estimates, it could be possible to evaluate those data periodically for validity and consistency. It may also be possible to substitute existing or newly developed biomarkers of certain smoking behaviors for other forms of data collection, and, in selected instances, information from other countries with similar populations may be of value.
The committee also discusses the importance of collecting data that inform agent interactions, either with other agents or with the agent’s environment, which are a key element in ABMs. Such interactions are difficult or impossible to capture empirically, but alternative data collection methodologies, including qualitative methods and experiential or situational sampling, could help overcome this challenge. Because ABMs and other individual-level modeling techniques are promising tools to further the understanding of tobacco use behavior, it is worthwhile to collect such data. As a major funder and user of tobacco data (including for the modeling of tobacco use), CTP can help shape the tobacco data environment in the future.
Conclusion 6-1: The committee concludes that agent-based models designed to inform policy decisions require data on the underlying mechanisms governing behavior and on agent-to-agent and agent-to-environment interactions. Currently, these data are not commonly collected.
Recommendation 6-1: The Center for Tobacco Products should identify and help develop data sources relevant to the questions it is trying to address using agent-based models and other modeling approaches.
Data already being collected (either by CTP or other sources) could be incorporated into the modeling process. CTP could consider coordinating with other activities, such as the Tobacco Centers of Regulatory Science, to gather this data.
To ensure that the processes of collecting the necessary data and identifying agent attributes based on those data are done successfully, it is crucial to address implementation issues. Funders for policy-relevant models require access to expertise if they are to issue effective funding opportunity announcements or contracts; to determine which modeling approaches are appropriate for the question at hand; to work with sponsored modeling teams throughout model development; to evaluate model inputs, processes, and outputs; and to appropriately interpret model results and translate them for decision makers.
FDA is regularly confronted with uncertainty within the complex tobacco environment. Because of this, the agency will continue to need models that represent potential tobacco policies in order to organize data, elucidate uncertainties, and forecast future scenarios. Because the use of models at CTP has the potential to affect regulatory decision making, it is essential that the development of these models be overseen by individuals who have the expertise and experience needed to maximize the benefit and reliability of the models.
Recommendation 6-2: The Center for Tobacco Products (CTP) should ensure that it has staff with, or access to, the necessary expertise to inform CTP’s research, contracting, and evaluation efforts and to translate model results for various stakeholders.
Although individual models are a useful tool for informing policy decisions, having a range of modeling techniques will offer a fuller picture of the policy questions confronted by CTP—for example, by creating various models to approach the same question or process (for example, multiple ABMs or ABMs and aggregate models). The documentation of model inputs, activities, and outputs by the model developers (as discussed in Chapter 4) and a comparison of results with a rigorous discussion by the developers on why the results differ—or do not differ—will create a richer understanding of the models and the model results and will help to address model uncertainty. Doing so will help to increase policy makers’ confidence in the model results or identify where assumptions need to be modified or where further data is needed.
Recommendation 6-3: The U.S. Food and Drug Administration should develop a range of models using various approaches. This would include agent-based models as well as other modeling approaches.
It is important to note that the range of models FDA could use includes not only those that FDA commissions or develops but also those that others have already developed or will develop to help guide tobacco control policy.
CONCLUSION
Although simulation modeling has been used for many years in tobacco control, CTP is still early in its efforts to use ABM to explore tobacco control policy and regulation. This report illustrates many of the challenging and technical aspects surrounding ABMs. However, the committee believes that ABMs are a useful tool and that they could add to the understanding of tobacco use initiation, cessation, and relapse processes. While the model de-
veloped for FDA (see Chapter 5) does not accurately represent many of the important characteristics of tobacco use, there is much that can be learned from its development that could be applied to future models of tobacco use. There are some barriers to overcome, such as the collection of data to inform the development of ABMs and understanding the empirical and theoretical challenges of specifying model inputs and appropriately interpreting model outputs (see Chapter 3). A strong evaluation framework (see Chapter 4) is needed to track model development. As discussed in Chapters 3 and 4, it will be important to consult an interdisciplinary modeling team, and subject-matter experts will need to be consulted at the earliest stage of model conceptualization and throughout the model development process to ensure that the model is grounded in the current state of tobacco science (that is, evidence-based research related to tobacco in the fields of epidemiology, social and behavioral sciences, biology, chemistry, and others), while carefully considering individual behavior. If the principles discussed in this report are followed, the role of ABMs for informing tobacco regulation will be greatly strengthened.
REFERENCES
Agaku, I. T., B. A. King, C. G. Husten, R. Bunnell, B. K. Ambrose, S. S. Hu, E. Holder-Hayes, H. R. Day, and Centers for Disease Control and Prevention. 2014. Tobacco product use among adults—United States, 2012–2013. Morbidity and Mortality Weekly Report 63(25):542–547.
Auchincloss, A. H., R. L. Riolo, D. G. Brown, J. Cook, and A. V. Diez Roux. 2011. An agent-based model of income inequalities in diet in the context of residential segregation. American Journal of Preventive Medicine 40(3):303–311.
Epstein, J. M., D. M. Goedecke, F. Yu, R. J. Morris, D. K. Wagener, and G. V. Bobashev. 2007. Controlling pandemic flu: The value of international air travel restrictions. PLoS ONE 2(5):e401.
Fultz, S. L. 2014. Committee on assessment of agent-based models to inform tobacco product regulation. Paper presented at first Committee Meeting of the Committee on Assessment of Agent-Based Models to Inform Tobacco Product Regulation, Washington, DC.
HHS (U.S. Department of Health and Human Services). 2014. The health consequences of smoking—50 years of progress: A report of the Surgeon General. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health.
Jamal, A., I. T. Agaku, E. O’Connor, B. A. King, J. B. Kenemer, and L. Neff. 2014. Current cigarette smoking among adults—United States, 2005–2013. Morbidity and Mortality Weekly Report 63(47):1108–1112.
Lee, B. Y., S. T. Brown, P. C. Cooley, R. K. Zimmerman, W. D. Wheaton, S. M. Zimmer, J. J. Grefenstette, T. M. Assi, T. J. Furphy, D. K. Wagener, and D. S. Burke. 2010. A computer simulation of employee vaccination to mitigate an influenza epidemic. American Journal of Preventive Medicine 38(3):247–257.
Manski, C. F. 2013. Public policy in an uncertain world: Analysis and decisions. Cambridge, MA: Harvard University Press.
Moore, T. W., P. D. Finley, N. S. Brodsky, T. J. Brown, B. Apelberg, B. Ambrose, R. J. Glass. In press a. Modeling education and advertising with opinion dynamics. The Journal of Artificial Societies and Social Simulation.
Moore, T. W., P. D. Finley, B. J. Apelberg, B. Ambrose, N. S. Brodsky, T. J. Brown, C. Husten, R.J. Glass. In press b. An opinion-driven behavioral dynamics model for addictive behaviors. European Physical Journal B.
NRC (National Research Council). 2007. Models in environmental regulatory decision making. Washington, DC: The National Academies Press.
———. 2012. Assessing the reliability of complex models: Mathematical and statistical foundations of verification, validation, and uncertainty quantification. Washington, DC: The National Academies Press.
Orr, M. G., S. Galea, M. Riddle, and G. A. Kaplan. 2014. Reducing racial disparities in obesity: Simulating the effects of improved education and social network influence on diet behavior. Annals of Epidemiology 24(8):563–569.
SAMHSA (Substance Abuse and Mental Health Services Administration). 2013. Results from the 2012 National Survey on Drug Use and Health: Summary of national findings, NSDUH series H-48, HHS publication No. (SMA) 14-4863. Rockville, MD: U.S. Department of Health and Human Services.
Sanstad, A. H. 2015. Assessing agent-based models for regulatory applications: Lessons from energy analysis. Paper commissioned by the Committee on the Assessment of Agent-Based Models to Inform Tobacco Product Regulation (see Appendix C).
Wagner, W., E. Fisher, and P. Pascual. 2010. Misunderstanding models in environmental and public health regulation. NYU Law Environmental Law Journal 18:293–356.
Zhang, D., P. J. Giabbanelli, O. A. Arah, and F. J. Zimmerman. 2014. Impact of different policies on unhealthy dietary behaviors in an urban adult population: An agent-based simulation model. American Journal of Public Health 104(7):1217–1222.
