Assessing the Use of Agent-Based Models for Tobacco Regulation (2015) / Chapter Skim
Currently Skimming:
4 An Evaluation Framework for Policy-Relevant Agent-Based Models
4 An Evaluation Framework for Policy-Relevant Agent-Based Models
Pages 87-118
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 87... ...
These types of questions can be answered by evaluating the model building process, the model outcomes, and the return on investment. The goal of this chapter is to present an evaluation framework for assessing ABMs for tobacco control policy and regulation.
|
From page 88... ...
The remainder of this chapter provides a grammar for describing policy-relevant ABMs, presents an evaluation framework, identifies highpriority categories for evaluation questions, and illustrates some of the evaluation concepts through two case-study descriptions of existing policyrelevant ABMs. MODEL DESCRIPTION Before establishing a framework for evaluating how ABMs inform public health or tobacco control policy and regulation, a consistent way to describe and talk about them is needed.
|
From page 89... ...
Data -- validation Are empirical data used to validate model results? Theories What are the primary social science and behavioral theories used in the model design?
|
From page 90... ...
for examples of how the descriptive grammar can be applied to three different policy-relevant ABMs. EVALUATION FRAMEWORK Fundamentally, systematic evaluations of policy-relevant ABMs are important because they can lead to better and more effective models in the future.
|
From page 91... ...
The development of the evaluation framework was based on a review of relevant literature and on committee members' experience in building and assessing ABMs and in developing public health policy evaluations. The evaluation framework is designed to cover the important aspects of designing, implementing, testing, and disseminating policy-relevant ABMs, especially for tobacco control regulatory and policy efforts.
|
From page 92... ...
• Development of full model or model modules • Empirical databases Intermediate • Diffusion of new tobacco control Human Resources knowledge among funders, regulators, • Availability of core modeling staff with Model Testing & Validation Model Results policymakers, scientists required expertise • Examine basic model behavior • New regulatory and funding • Systematic storage of model output data • Availability of relevant subject-matter • Identify problems and interesting model behavior announcements • Interpretation of important model results experts • Perform sensitivity analysis • Public health scientists collect new data • Model limitations and uncertainties • Availability of funding, policymaking, • Validate model outputs against theory and to inform future model development and regulatory staff (funders and endusers) existing data policy research Policy Results Internal Model Development Activities Knowledge Resources Policy Testing • Description of policy effects • Examine effects of introduction of new policies • Identification of promising policies, • Accurate information about current and policy leverage points, implementation Long-term or regulation proposed public health policies and strategies • Implementation of evidence-based • Conduct policy experiments that compare regulations tobacco control policies & regulation multiple policies or regulations • Access to relevant empirical data • Improve population health via: (epidemiologic, behavioral, public health • Reducing product harms and system, tobacco industry)
|
From page 93... ...
Another important group of people consists of the model funders and other policy makers who may use the results of the model to inform their policy and regulatory work. Finally, a modeling team will need access to a wide variety of knowledge resources, including relevant data, empirical findings, and current and proposed policies and regulatory options that may be addressed by the ABM.
|
From page 94... ...
, the process of model evaluation is more than just a strict validation or verification procedure; it is a process that builds confidence in model applications and increases the understanding of model strengths and limitations. "Regulatory model evaluation must consider how accurately a particular model application represents the system of interest while being reproducible, transparent, and useful for the regulatory decision at hand" (NRC, 2007, p.
|
From page 95... ...
Funders and sponsors need to be kept apprised of progress, and content experts need to talk frequently with the model building staff to avoid making major errors related to essential model implementation decisions (Kuntz et al., 2013) , such as programming agent behaviors, and to identify data gaps throughout model development.
|
From page 96... ...
However, this highlights the need for the policy maker to be involved with model development from its conception and the importance of translating the model results properly. Although this involvement will not guarantee that the model is used, it will increase the likelihood that the model addresses the current questions the policy makers are faced with and that they have a deep understanding of the value that the model offers (Wagner et al., 2010)
|
From page 97... ...
. Identifying High-Priority Evaluation Questions The evaluation framework for policy-relevant ABMs provides a guide for designing an evaluation of a specific ABM project or a broader modeling initiative.
|
From page 98... ...
In this chapter's annex, the committee provides a list of example questions based on the evaluation framework that could be appropriate for evaluating an ABM project, especially in the context of tobacco control policy and regulatory science. Fundamental Evaluation Categories As the committee developed its evaluation framework, five fundamental evaluation categories for policy-relevant ABMs emerged.
|
From page 99... ...
Recommendation 4-1: The Center for Tobacco Products should adopt an evaluation framework for its modeling work, either the one pre sented in this chapter or one similar to it. Key dimensions of the evalu ation framework should include considerations of resources, technical best practices, model suitability, communication and translation, and
|
From page 100... ...
This evaluation framework would apply to all efforts funded by the Center for Tobacco Products (CTP) -- internal model development, interagency agreements, contracts, and grants.
|
From page 101... ...
(2011) examined the effects of the following: gas prices; the ability of agents to consider fuel costs, PHEV purchase price, and rebates; PHEV all-electric battery range; consumer values regarding financial versus nonfinancial concerns in vehicle purchase; agent comfort threshold with the PHEV technology; social and media influence on PHEV market penetration; and fuel efficiency of the resulting fleet after 25 years.
|
From page 102... ...
Consequently, the updated model generated results that differed slightly from those of the original model.8 7 Some of the findings included if there are sufficient potential early adopters, readily acces sible estimates of lifetime vehicle fuel costs could be important for promoting PHEV market penetration; increasing gas costs could help people choose PHEV over traditional vehicles; temporal incentive programs like tax credits are not likely to have lasting effects on long-term fuel efficiency unless manufacturers are able to lower sticker prices after the rebates are discontinued; and increasing PHEV battery range may be an important leverage point. 8 The results of the modified model indicated, among other things, that consumer uneasiness with the new PHEV technology was the biggest barrier to potential PHEV market penetration; that manufacturers and policy makers may need to take more action to help consumers feel
|
From page 103... ...
Although the initial model design did not properly represent the agents' behaviors, the authors made needed adjustments to improve the model for its intended purpose (exemplifying the necessary feedbacks in the evaluation framework presented in this chapter)
|
From page 104... ...
. SimAmph provides a good example of several of the criteria laid out in the evaluation framework.
|
From page 105... ...
Because of the policy focus of DPMP, the researchers work with policy makers to ensure that the model is suitable for their purposes, and they regularly assign a "knowledge broker" to translate model findings into policy language and communicate the limitations of the modeled scenarios as well as the predictive ability of the model to the policy makers (MacDonald, 2012) .13 Because DPMP aims to ensure that modelers understand the needs of the model they are developing and to make certain that the models are used properly by policy makers, communication and translation strategies are considered throughout model development.
|
From page 106... ...
. TABLE 4-2 Application of Descriptive Grammar to Three PolicyRelevant ABMs Models PHEV Market Adoptiona SimAmphb SnapDragonc Basic Description Purpose To inform To test policies that To study the effects of policies affecting could influence drug tobacco control policies plug-in hybrid use and experience in a single- or multiple vehicle market among young tobacco product penetration.
|
From page 107... ...
Outcomes Primary outcome Fleet fuel Individual drug use and User or nonuser of efficiency population prevalence tobacco products resulting from of drug-related harm agent vehicle and of regular drug use purchase choices Proximal/distal Distal Proximal/Distal Proximal/Distal outcome Policy Policy definition Effects of Effects of mass media Introduction of non purchase rebatese drug prevention specific communications campaigns; effects of campaign; introduction using drug-sniffing of new products dogs Policy realism Realistic Realistic Abstract Policy tests Yes (although not Yes No (at the time of a primary goal of committee review) study)
|
From page 108... ...
is available at: http://cormas.cirad.fr/ A en/applica/simAmph.htm. EVALUATION QUESTIONS DERIVED FROM THE EVALUATION FRAMEWORK FOR POLICY RELEVANT AGENT-BASED MODELS Based on the evaluation framework presented on page 92, this document contains sample questions for each of the categories outlined.
|
From page 109... ...
What hardware resources did the model developers use?
|
From page 110... ...
ii. How did the particular theoretical framework enhance or weaken the validity of the model results?
|
From page 111... ...
iv. How do the results compare to the results of other models addressing similar policy questions or having similar purpose?
|
From page 112... ...
iv. Did the authors clearly discuss the model's strengths and weaknesses and implications for tobacco control policy?
|
From page 113... ...
ii. Based on model results, did policies and regulatory options change?
|
From page 114... ...
Have the results informed tobacco control knowledge and influenced decisions among funders, regu lators, policy makers, scientists?
|
From page 115... ...
2011. An agent-based model to study market penetration of plug-in hybrid electric vehicles.
|
From page 116... ...
2011. Comparative logic modeling for policy analysis: The case of HIV testing policy change at the Department of Veterans Affairs.
|
From page 117... ...
Paper presented at 2014 Tobacco Products Scientific Advisory Committee Meeting, Rockville, MD. Šalamon, T
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.