Decision Making and Oversight of MRTP Studies: Findings and Recommendations
In the previous three chapters, the committee describes the evidence domains for potential modified risk tobacco products (MRTPs), including studies of health effects, addictive potential, and risk perception and communication. The committee discussed the governance of those studies in Chapter 2. The Food and Drug Administration (FDA) will have to integrate the evidence from those diverse domains when making regulatory decisions about MRTPs. Many of the same issues and concerns related to the governance and design of studies arise regardless of the evidence domain. The committee’s findings and recommendations, therefore, cut across the evidence domains and focus on the types of evidence and studies that the FDA should use in making regulatory decisions about MRTPs, the design of studies of MRTPs for the FDA’s decision making, and the governance of those studies. This chapter focuses on those cross-cutting issues, ways the FDA can integrate information from those studies as part of its regulatory decisions, and the committee’s overarching findings and recommendations.
This chapter begins with a brief summary of the evidence domains discussed in more detail in the previous chapters, a discussion of the scientific and ethical issues in studying MRTPs, and the governance issues that accompany such studies. The committee then discusses (1) the integration of all the evidence for the FDA’s regulatory decisions about these products, including mathematical modeling and simulation techniques, and (2) the comparative nature of the harm reduction claims that the FDA will be evaluating. The committee then presents its findings and recommendations.
Evidence Domains
The evidence to support the marketing authorization of an MRTP will come from studies of health effects, addictive potential, and risk perception. In this section the committee provides a brief summary of the studies within those evidence domains.
Health Effects
Laboratory analysis of the performance and of the constituents of tobacco products will be the first step in the evaluation of any new product. These analyses involve standard methods of extraction, sample cleanup, analytic identification, and quantitation. There are important limitations to laboratory analysis of product performance and composition. Laboratory analysis of constituents may not reflect constituent uptake under conditions of use. Smoking machines do not replicate human smoking conditions, and there is no proven way to replicate the many ways humans use tobacco. It is crucial, therefore, to describe the smoking regimen or other extraction methods employed.
The second step in the evaluation of MRTPs will be preclinical studies of toxicity. These assays are essential in identifying particularly risky or toxic products that should not be tested in humans, as well as products that have reasonable potential to reduce risk and, therefore, should proceed to clinical evaluation. Evaluation of products in vitro should precede in vivo assays. Although it is not possible to make laboratory animals use tobacco products the way humans do, and there are inherent interspecies differences that prevents meaningful extrapolation of human effects, it is still informative to observe the effect of tobacco products in live animal models. The number of animal studies required to characterize an MRTP preclinically could potentially be reduced by setting composition standards or limits or establishing standards for certain categories of MRTPs. Assays of toxicity in humans will also be essential, in particular assays of urinary mutagenicity and sister chromatid exchange in peripheral lymphocytes.
Biomarkers of exposure measure human exposure to constituents of tobacco and could include the constituents themselves, their metabolites, or protein (or DNA) binding products of the constituents or their metabolites. These biomarkers have the potential to bypass many of the uncertainties in product composition analysis and provide a realistic and direct assessment of carcinogen and toxicant dose in an individual. Biomarkers of exposure should be validated before their use.
When the FDA evaluates studies, it is important that it ensures the constituents of a product are accurately and precisely measured, that
exposure methods are appropriate, and that any biomarkers of exposure have been validated.
Experimental designs, in particular randomized controlled trials (RCTs), provide data that can support the strong inferences about the effect of an MRTP on human health relative to conventional tobacco products. The use of appropriately designed clinical trials will be important to establish whether the use of the MRTP reduces exposure to toxicants or induces positive changes in surrogate markers. An RCT is an effective means of examining acceptability and use of the MRTP, the ability of the MRTP to increase cessation in users of conventional tobacco products, and the likelihood that MRTP availability will lead to dual use. RCT methods can also produce evidence on whether and how much individuals use an MRTP after they have used it to help them quit conventional products, changes in perception of the MRTP with its continued use, and the MRTP’s ability to suppress tobacco withdrawal symptoms. It is important to recognize that no single RCT can address all of these areas, and each study should have a focused objective with a primary endpoint.
Postmarket studies of marketed MRTPs will be critical to evaluating the effect of the MRTP on both individuals and on the public’s health. In particular, the prospective cohort design will be an essential tool to validating anticipated or claimed effects of marketed MRTPs. These studies can assess baseline tobacco and MRTP exposures; summarize product use as the study is ongoing, including any changes in product use habits; document and verify outcomes as they occur; and evaluate a wide variety of outcomes, including both intermediate and clinical outcomes.
In addition, other study designs will be necessary to provide evidence on the public health effects of MRTPs, including retrospective cohort studies, case-control studies, crossover or case-crossover designs, and comparative effectiveness research methods. Different study designs will be necessary depending on the circumstances and the research question.
Addictive Potential
Evaluation of the likelihood of initiation, maintenance, and persistence of use in both conventional tobacco users and nonusers is critical to estimating the public health effect of marketing MRTPs. Specifically, evaluation of the MRTP’s ability to promote initiation and continuation of its regular use, switching to its use and cessation of the consumption of more harmful products, dual use, and to promote relapse back to more harmful tobacco use are all essential. All of these outcomes are related to the reinforcing value of the MRTP (that is, how rewarding it is).
There is a continuum of reinforcement value. In theory, the MRTP should be somewhat more reinforcing than nicotine replacement therapies
but perhaps less reinforcing than conventional cigarettes. Ideally, an MRTP would be sufficiently reinforcing so as to attract smokers away from conventional cigarettes but not encourage the widespread dependent use of the product by individuals who were previously nonusers or who would have quit smoking. Data from all relevant measures should be integrated, taking into account not only the effects of the MRTP on important health outcomes, but also the prevalence of use and projected public health impact.
Evaluation of the abuse and addiction potential of a product can be accomplished with a variety of experimental designs and in a variety of contexts. Those include subjective evaluations in laboratory contexts, acute self-administration studies in laboratory contexts, use in extended residence facilities, and natural environment contexts where long-term use can be studied in real-world circumstances via RCTs, cross-sectional survey studies, and longitudinal cohort studies.
Evaluation of reinforcement value in a laboratory setting is particularly important because the results of these studies reliably correspond to an agent’s addictive potential in real-world use. One standard with regard to human abuse liability drug testing are acute dose-effect comparison studies, because of the correspondence between subjective ratings of drug effects and real-world abuse potential. Behavioral economic self-administration studies will also be important to evaluating the reinforcement potency of a product. The usefulness of all studies in forecasting the risk for initiation and abuse of a product depends on study design factors. Important design considerations include the size of the sample, the nature of the sample (whether the sample includes heavy smokers or light smokers, smokers who want to quit, and nonsmokers), the characterization of the sample (age, sex, gender, ethnicity, educational attainment, socioeconomic status, etc.), and the nature of the comparison product.
Risk Perception and Communication
Judgments about risk, otherwise known as risk perceptions, are a fundamental element to most theoretical models of health behavior and behavioral decision making. In general, those models argue that individuals’ perceptions about the value and likelihood of behavior-related positive and negative consequences and their vulnerability to those consequences play a key role in behavioral choices. As such, understanding individuals’ perceptions of tobacco-related products, including MRTPs, whether such perceptions change over time, and whether such perceptions play a role in tobacco use behavior, is critical. It will be important to identify consumers’ perceptions of disease risk, likelihood of addiction, likelihood of reducing or increasing others’ exposure to potentially hazardous
compounds, and perceptions of risk compared to other products already on the market. It is also important to assess intentions of using the product. It is essential that industry carefully crafts messages about risks and benefits of any MRTP and demonstrates through rigorous testing that people correctly understand and interpret the risks.
Studies evaluating risk perceptions and risk communication must be performed both before the marketing of an MRTP and after the MRTP has been marketed. Premarket research will play an essential role in developing the messages that the tobacco industry can use to communicate information about MRTPs to consumers. This research will determine consumers’ ability to accurately understand messages that communicate information about the risks, benefits, and conditions of use pertaining to the MRTP itself and compared to existing tobacco products. Studies should also test how these messages influence consumers’ perceptions of the risks, benefits, and likelihood of addiction related to the MRTP. The first stage of premarket research will involve formative work using focus groups. The second stage should include discussions with groups of similar individuals to assess how the messages that were developed in the first stage are received by consumers. Finally, the effects of these messages on consumer perceptions should be tested. It will be important to evaluate consumer understanding and to compare consumer perceptions of the MRTP to conventional products. After the product is released on the market, it is vital to continue monitoring consumer perceptions and behavior related to that product. Conducting nationally representative cohort-sequential longitudinal surveys will be essential.
Cross-Cutting Issues with Studies of MRTPs
When evaluating the studies of MRTPs, there are a number of considerations that are relevant regardless of whether a study is looking at health effects, addictive potential, or risk perception. Those cross-cutting issues, which include issues related to study design and governance, are discussed in this section.
Study Design
Issues that could affect the FDA’s evaluation of a study on MRTPs include the generalizability of the study and how well the study is conducted.
Studies should be designed appropriately to create an evidence base that can support a finding of public health benefit. The ultimate goal of studying the effect of MRTPs on human health and behavior is to be able to accurately predict the public health effects of allowing an MRTP to be
marketed. In other words, the ultimate goal of scientific studies it to produce generalizable data. The “generalizability” of data, or the reliability of predictions that can be made about the real world based on scientific observations, will depend on the design of the studies.
The FDA should carefully evaluate the size and nature of the sample to assess the generalizability of the study data. Sample sizes should be carefully determined and tailored to the study design and the effects studied. Statistically underpowered studies cannot support inferences or projections about the effects of a product. The nature of the study sample is critical to the usefulness of study results. Results from studies conducted in one population may not be applicable to other populations, because the characteristics that define the study population either are related to or cause the responses to the product. As such, it is important to study a wide range of populations. It is particularly important to include populations that have a high risk of using tobacco and populations that will be affected by the marketing of the product.
Study designs must also carefully consider the degree of control imposed on experimental designs. Internal and external validity should be balanced not only within studies but also across studies of the same product. Highly controlled experimental designs can eliminate many variables and confounders and support strong inferences, but simultaneously lose relevance to the FDA’s decisions because the conditions of product use do not reflect real-world circumstances and behaviors. Experimental designs that are less controlled can emulate circumstances that reflect real-world conditions and behaviors, and therefore may be more relevant in predicting real-world effects, but uncontrolled variables may confound meaningful associations or inferences.
Regardless of the type of study design, the planning and conduct of the study should meet good research practice standards. As discussed earlier in this report, there are minimum standards for studies in other settings, such as studies of pharmaceuticals. Consensus statements such as Consolidated Standards of Reporting Trials (CONSORT) for clinical trials, Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) for observational studies, publication criteria from the International Council of Medical Journal Editors, and reporting criteria of the International Conference on Harmonization have been implemented to ensure that the design, conduct, and reporting of studies are consistent with the state-of-the-art scientific standards. Studies of MRTPs should meet those or similar criteria to help ensure the overall quality and integrity of the studies. Certain types of studies will have specific criteria to ensure quality research. Some of those specific criteria were discussed in the preceding chapters.
Governance
Research on MRTPs will require some oversight or governance to ensure the research is free from bias and conflicts of interest and that appropriate controls are in place for human subject research. Such governance will also ensure the disclosure of data to ensure transparency and instill confidence in the research findings.
The tobacco industry has a history of hiding and misrepresenting information about the risks of tobacco products (Cummings, 2003; Cummings et al., 2002, 2007).1 This history has lead to profound public distrust in both the tobacco industry and in the research it sponsors, and the absence of governance in the tobacco industry has created an isolated industry that lacks not only the expertise to produce the necessary range of credible and reliable data, but also the trustworthiness to acquire external expertise and avenues to disseminate acquired data (Ashley and Cohen, 2003; Harris Interactive, 2003, 2010; NCI, 2008). The production of reliable and credible data depends upon building rigor, oversight, and independence into the entire research process. Data problems often cannot be detected after study completion, and therefore integrity and accountability need to be built into the research throughout the study’s execution. There is neither an established set of regulatory practices for the review of MRTPs nor an established set of federal research standards for the design, conduct, analysis, monitoring, and completion of studies in support of MRTPs.
There are also a number of ethical issues associated with conducting human subjects research involving MRTPs. The first issue is the risk of conducting clinical trials of MRTPs or other tobacco products in populations with a high risk for tobacco initiation and addiction, including but not limited to adolescents, certain ethnic minorities, and individuals with mental health disorders. Randomization of participants to a product known to be potentially addictive and hazardous is ethically problematic. The second issue is the risk of improperly disclosing the substance abuse of a minor to the minor’s parents or guardians in the process of obtaining parental consent for research. While the assent of minors is always necessary, investigators should also be aware of circumstances where waiver of parental consent is warranted because obtaining parental consent will violate the confidence of the study participant. The third issue is the inclusion of individuals from high-risk groups with reduced decision-making capacity. Some populations at a high risk for tobacco use, such as
_______________
1 The history of research conducted, funded, or supported by the tobacco industry is not raised to be retributive or punitive, but simply to acknowledge that past actions reflect on the credibility of the industry’s current research, which may pose a problem for regulators, particularly in the contentious area of MRTPs
adolescents and populations with mental health issues, may have a higher prevalence of individuals with reduced decision-making capacity. When investigators are conducting research involving these high-risk groups, they should be particularly cautious about the inclusion of individuals who lack the capacity to provide meaningful consent.
The tobacco industry is currently limited in its ability to produce credible and comprehensive data. The challenges created by that lack of credibility are augmented by the fact that it is inevitable that product sponsors will need to collect extensive data on the effect of products in populations vulnerable to high use rates. Because of those challenges, at least part of the research base in support of an MRTP may need to be generated by researchers and organizations independent of the sponsors of the MRTP in question (Rees et al., 2009a, 2009b). The creation of a third party or third parties for the conduct and oversight of studies of MRTPs could help overcome some of the issues discussed above. The Health Effects Institute (HEI), a nonprofit corporation with approximately one-half of its funding coming from the automobile industry and the other half coming from the federal government and other government sources, has successfully managed the boundary between industry and government, and between the research community in health effects and the research community in air quality (Keating, 2001). HEI, however, does not fund projects in support of specific marketing applications, but rather it funds projects that contribute to general knowledge. The Reagan-Udall Foundation (RUF) advises the FDA on modernizing regulatory science, and it conducts and oversees studies on regulatory science, particularly in the emerging fields of pharmacogenomics and genomic-based prediction of drug response and adverse event risk. The RUF has in place controls for bias and conflict of interest that are noteworthy.
Public access to the totality of the data on MRTPs is critical to the credibility of MRTP research. Registering studies funded by the National Institutes of Health or used in drug approvals on the National Library of Medicine website Clinicaltrials.gov has greatly improved the transparency of those studies. The FDA may have to balance the need for public access to key information with the need to protect proprietary information and promote innovation. Public availability of data provided in an MRTP application is discussed in the Family Smoking Prevention and Tobacco Control Act of 2009 (FSPTCA).2
_______________
2 Family Smoking Prevention and Tobacco Control Act of 2009, Public Law 111-31, 123 Stat. 1776 (June 22, 2009)
INTEGRATION OF EVIDENCE AND DECISION MAKING
Table 6-1 presents the evidence domains and example considerations for using evidence from the different domains. A key challenge facing the FDA will be integrating the various domains and levels of evidence provided by sponsors in support of an MRTP application. A systematic, explicit approach that weights outcomes in terms of their public health importance, identifies the measures and data most relevant to those outcomes, and combines the available evidence in a manner that is psychometrically sound, objective, and reproducible would be helpful. This effort could be informed by decision theory concepts and techniques such as expected utility, Bayesian, and improper linear model approaches. Mathematical modeling and simulation analysis, such as discussed earlier in Chapter 3, can also be used to predict population-level effects.
It is clear that no single class of evidence (e.g., preclinical, RCTs, consumer perception, epidemiologic) in itself will be sufficient to support an MRTP application. Studies offered in support of an MRTP application should address all key research domains needed to prognosticate the product’s likely public health impact, including the following:
• Product content (including but not limited to harmful and potentially harmful constituents), performance, and quality assurance regarding product consistency
• Self-administration and subjective evaluation
• Exposure assessment by state-of-the-art methods and measurements that focus on human exposure to harmful and potentially harmful constituents
• Consumer and nonconsumer perceptions; and assessment of biomarkers of exposure, biomarkers of risk, and where feasible, disease outcomes
The research should use designs that are properly powered, balance internal and external validity, and comprise multiple populations appropriate to the experimental questions being addressed. Target populations of special relevance include (but are not limited to) users of tobacco products, both those interested and uninterested in quitting; nonsmokers; former smokers; beginning smokers; and adolescents. In addition, study samples must permit inferences to populations with significant smoking prevalence such as those low in socioeconomic status and educational attainment, and certain ethnic minorities.
Beyond merely amassing evidence in support of the modified risk claims, higher-level processing of the evidence is needed to inform decision making. It is probable that, depending on their construction, intended uses, and desired claims, different MRTP applications could require different
TABLE 6-1 Evidence Domains Relevant to an MRTP Application and Examples of Types of Findings
| Class of Evidence | Examples of Types of Finding That May Be Required |
| Preclinical | • Assurance of manufacturing quality control |
| • Significant and substantial reduction in toxicant and carcinogen content in product | |
| • Significant reduction in exposure to toxicants and carcinogens in limited human study | |
| • No significant evidence for offsetting increases in content of or exposure to other toxicants | |
| Clinical trial | • Significant reduction in exposure to toxicants and carcinogens in relation to continued use of traditional product, preferably approaching nonsmoker levels |
| • Significant rates of cessation of conventional tobacco product use, or significant decrease in the rates of conventional tobacco product use | |
| • Significant reduction in biomarkers or surrogates of disease | |
| Abuse potential | • No more liable for abuse than currently marketed products |
| • No significant evidence of attractiveness to nonusers of tobacco | |
| Epidemiology | • Clear and consistent evidence of reduction in disease risk (e.g., cancer, cardiovascular disease, chronic obstructive pulmonary disease) or intermediate endpoint thereof |
| • No significant evidence of offsetting increased risk for other diseases | |
| • No significant evidence of uptake among nonusers or relapse among former users (postmarketing) | |
| Consumer and | • Evidence for accurate understanding of product claim |
| nonconsumer perceptions | • No significant evidence that consumers equate reduced exposure with reduced risk |
| • No significant evidence of intention to use product among nonusers (especially adolescents) | |
| • No significant evidence of switching from MRTP to other tobacco product usage | |
| Populations at | • No significant evidence of risk of initiation among nonusers |
| high risk for | (especially adolescents) |
| tobacco use | • Consistency of findings across relevant subpopulations of interest (e.g., low socioeconomic status, racial/ethnic minorities) |
| Modeling and | • Population predictions show reduction in smoking-related |
| synthesis | morbidity and mortality following the introduction of an MRTP with no significant evidence of uptake by nonusers (especially |
NOTE: This table is not comprehensive and is not intended to be a guideline or framework for the evaluation of MRTP applications.
balances of the various classes of evidence. As such, prescribing a universally applicable portfolio of evidence is difficult to envision. However, it remains possible to frame four broad scenarios for an MRTP application, as shown in the Table 6-2.
A product sponsor may file an MRTP application for a product that is already on the market or for a new product (meaning one not available on the U.S. market). For each of these, a sponsor could request a Modified Risk claim or a modified exposure claim under the Special Rule for Certain Products.
Existing products for which sponsors wish to make a risk/exposure claim raise particular concerns because of conscious and unconscious associations built over time with branding and other marketing messages. This is particularly true for MRTPs that are cobranded with other tobacco products. In the case of an existing product wishing to make a claim of modified risk, it would not be unreasonable of the FDA to expect epidemiologic evidence (e.g., case-control or cohort studies of users and nonusers of the MRTP about which the claim is made, with clinical or validated surrogate endpoints) and to have this evidence weigh heavily in its decision making. Supporting evidence for such an application could include switching studies and RCTs that conclusively demonstrate substantially reduced biomarkers of toxicant exposure or biomarkers of risk. It would be anticipated that preclinical evidence from human, animal, in vitro, or in vivo studies would play relatively minor roles (e.g., providing mechanistic context) in justifying a claim for a product that is already on the market. However, all MRTP products would benefit from having supportive preclinical studies completed prior to human studies, even if they are already on the market. Lastly, significant emphasis would need to be placed on extensive consumer and nonconsumer testing of the proposed advertising and marketing materials, product packaging, and design to determine (1) whether the typical consumer understands the message and is unlikely to be misled and (2) if the product is minimally attractive to nonusers of tobacco.
In the case of a product not currently available to consumers, claims of reduced risk (which require epidemiologic evidence) would be difficult to support. However, if the identical product were available in other
TABLE 6-2 Four Broad Scenarios for an MRTP Application
| Modified Risk | Modified Exposure (Special Rule) | |
| Product already on market | A | B |
| Product not on market | C | D |
markets, then epidemiologic evidence from users in those markets could be informative as to demonstrating disease risk reduction. In such a case, cautions must be applied, because emerging evidence suggests that clinical trial results from non-U.S. populations can differ substantially from those obtained in U.S. populations, even when identical drugs are given and identical study designs are employed (Glickman et al., 2009). The FDA should articulate guidance for the acceptance of data produced in foreign studies.
In the case of a truly new product (where nothing similar is sold elsewhere) an Investigational New Drug application model could be the most appropriate approach. In such cases, then, the FDA could require that preclinical laboratory testing be completed before moving to animal or human studies (e.g., Phase I), which would have to be justified by significant findings in the laboratory work. If preclinical findings pointed to potential reduced exposure, then the FDA could authorize Phase II or III trials to explore the experience of reduced exposure in larger human populations under controlled conditions (e.g., RCTs). Concurrent or subsequent to these trials, consumer perception studies would be required to understand how potential users and nonusers would respond to product claims. To gather important population-level data on truly novel products, the FDA should consider authorizing a limited test market (perhaps requiring sponsors to track a cohort of MRTP purchasers for later follow-up) to gather data on real-world use.
Mathematical Modeling and Simulation Analysis
Mathematical modeling and simulation analyses can be useful to estimate exposures, health effects, and broad public health effects, providing information to the FDA for its decisions. The role of modeling in evaluating the effect of MRTPs and considerations for the conduct and reporting of model-based analyses are discussed below.
The Role of Modeling
Empirical studies employing a variety of designs, such as prospective, retrospective, randomized, or cohort designs, are expected to provide fundamental information about the societal impact of MRTPs. Considerations for the design and conduct of such studies are discussed in other sections of this report. However, the evidence empirical studies can provide has several limitations, especially in the context of assessing societal impact. First, empirical studies typically address streamlined questions. For example, they may compare only two interventions, have limited follow-up, or involve narrow subsets of potential users and nonusers of tobacco products.
As a result, generalizable evidence from empirical studies may be slow to accumulate and may also need updating with future studies. Second, empirical studies often require significant resources and time to launch and carry out. The magnitude of the required resources (sample size, length of follow-up) may be very large for studies intended to derive generalizable assessments of the short- and long-term societal effects of MRTPs.
Mathematical modeling and simulation analysis provide a complementary approach to the conduct of empirical studies that can be useful at each stage of the regulatory process for MRTPs. Modeling has already been used in the evaluation of smoking behaviors and related interventions (Mendez et al., 2008; TPSAC, 2011). Modeling is also widely used in several areas of health care research and public policy, such as the analysis of the economic impact of therapeutic and diagnostic interventions, the health implications of the introduction of disease screening programs, and the health impact of environmental conditions and exposures (Rutter et al., 2011; Weinstein et al., 2003).
According to the National Research Council (NRC) report Improving Information for Social Policy Decisions: The Uses of Microsimulation Modeling, a simulation model is “a replicable, objective sequence of computations used for generating estimates of quantities of concern” (NRC, 1991). In the context of this report the committee adopts the NRC definition with the following clarifications that adapt well-established criteria for models used in health care evaluation (Weinstein et al., 2003) to the regulation of MRTPs:
a. Models account for smoking behaviors and outcomes (health, behavioral, economic) over time and across various cohorts of individuals.
b. Models are based on information from empirical studies.
c. Models are used to estimate the effects of the introduction and use of MRTPs on outcomes of interest to the regulatory process.
The literature on the methodology and uses of modeling is by now extensive and addresses modeling both in general and for particular domains. Decision trees are some of the simpler and most widely used models in health care evaluation (Sonnenberg and Beck, 1993). Cohort Markov models describe the trajectory of groups of individuals through time and conditions of the process being modeled (Sonnenberg and Beck, 1993). Micro-simulation models (discrete or continuous time, agent based, etc.) describe the trajectories of individuals in a population, incorporating information on exposures and other events and outcomes (NRC, 1991; Rutter et al., 2011). A survey of recently used methods in tobacco modeling is provided in the proceedings of the 2008 Tobacco Modelers Conference (2008).
Modeling analyses have multiple potential uses in the assessment of the societal impact of MRTPs, as required by the regulatory process. Those aspects can help the FDA in its decision-making process. First, model-based analyses can synthesize the available information from empirical studies of MRTPs. In doing so, models can help clarify the logical and scientific relations between the structural assumptions and information sources used in the model (inputs) and the final conclusions and recommendations from the analysis. In addition to combining available information, models can help clarify and possibly reconcile seemingly contradictory evidence from previous studies. Second, models can enable researchers and decision makers to explore complex interactions and systems that may be impractical to evaluate in empirical studies. In particular, models can include components relating to quantities or processes that are not directly observable in empirical studies. Thus, model-based analyses can inform and augment current knowledge from clinical practice and empirical observation for the underlying mechanisms of MRTP utilization, smoking behavior, and outcomes. Third, model-based analyses allow researchers and decisions makers to explore “what if” questions relevant to decision making, which would not be practical to assess in empirical studies. For example, models can be used to assess the potential impact of modifications in MRTP structure, delivery, and scheduling of intervention. The vast multitude of such potential modifications makes it impractical to expect, for example, that comparative studies will be conducted for each possible combination of changes. Fourth, and of major importance to the regulatory process, models can be used to make projections about the short- and long- term effects of the introduction of MRTPs. These projections can be used not only for current decision making but also for planning future studies designed to fill particular gaps in scientific knowledge or to reduce uncertainty in the evidence about key determinants of the outcome.
Considerations for the Conduct and Reporting of Model-Based Analyses
The adoption and use of modeling in the regulation of MRTPs will represent a relatively new dimension in the regulatory process. The decades-long experience of modeling in health care evaluation and other areas of policy and decision making can provide a solid foundation for the introduction of model-based analyses in the regulatory process. Of particular value in the eventual formulation of a regulatory framework could be the results of completed (Weinstein et al., 2003) and ongoing work on the development of good research practices for modeling.3
_______________
3 ISPOR-SMDM Modeling Good Research Practices Task Force Working Group, Report Parts 1-7, unpublished data, 2011.
As with models in general, transparency in all aspects of the model, validation of the model, and proper assessment of the uncertainty regarding model parameters and results are major aspects of modeling for the societal impact of MRTPs.
Transparency Transparency refers to the availability of detailed information on all aspects of model structure, sources of evidence used, computational approach, and the construction of summaries and reporting of the results. This information is essential for a proper scientific understanding of the modeling and for enabling model critique and validation by researchers and other participants in the regulatory process. Transparency is also essential for the proper interpretation of the results of modeling analyses and their usefulness to decision making at the individual and policy levels. A practical goal of transparency in modeling used for MRTP regulation is to enable others to reproduce the results of the model-based analysis.
Validation Validation of the model examines its structure and performance from several perspectives, including internal, face, cross, and predictive validity.4 The internal validity of a model refers to whether the components of the model perform their intended tasks, the programming is correct, and the summaries of the results are accurate. Thus internal validation requires verification of the correct functioning of each component of the model.
The external validity of a model refers to the model’s ability to simulate events and outcomes that have actually occurred in settings that are close to those captured in the assumptions of the model. Thus external validation requires its developers to select appropriate databases, simulate the outcomes for individuals in these databases, and compare the results of the simulation to the actual outcomes. When models are to be used in settings that are not very close to those used for their development, careful calibration is needed to ensure applicability of models to these new settings (Vanni et al., 2011).
The face validity of a model is a more subjective attribute than the previous two, because it refers to the degree to which the structure, inputs, and methods for summarizing the results correspond to currently accepted scientific knowledge and practice, as judged by experts in tobacco studies. Standard components of the regulatory process, such as advisory panels, can take on a significant role in the assessment of face validity for models used on MRTP regulation.
_______________
4 David M. Eddy et al., ISPOR-SMDM Modeling Good Research Practices Task Force Working Group Part 4, unpublished data, 2011.
The cross-validity of a model refers to whether the model’s results agree with those of other models developed for the same purpose. For example, the Cancer Intervention and Surveillance Modeling Network (CISNET) initiative funded by the National Cancer Institute involves the development and comparison of the results of alternative models for assessing the impact of technologies for the early detection of cancer (CISNET Breast Cancer Collaborators, 2006). Cross-validation can be time and resource intensive, because it requires the development of several alternative models and the comparison of their results.
The predictive validity of a model refers to the model’s ability to accurately predict future outcomes. This aspect of model validity is particularly important for the intended use of modeling in regulating MRTPs. For example, a model developed on the basis of currently available information can be linked to future empirical studies of the impact of an MRTP and validated partly or fully using data from those studies.
Uncertainty Uncertainty accounting refers to the systematic examination, assessment, and reporting of the uncertainty in model inputs and assumptions, estimates of model parameters, and summary measures of the results (Bilcke et al., 2011). There are three important sources of uncertainty in model-based analyses. First, there can be uncertainty about the structural assumptions of the model, such as assumptions about variables to include and causation and prediction pathways. The uncertainty about such assumptions is often difficult to quantify, but their impact on the conclusions of the analysis may be substantial (Bojke et al., 2009). Second, there can be uncertainty about model parameter estimates derived from available studies. Formal statistical measures for this uncertainty can be derived using meta-analysis methods. However, the assessment of how this uncertainty propagates through the modeling computations to the final results continues to be a challenge. Third, there can be variability among individuals in the population, which can be explained on the basis of characteristics of these individuals (systematic or explained variation) or is considered to be random.
The impact of the various forms of uncertainty on the final results of the model is typically assessed by deterministic or probabilistic sensitivity analysis. However, because of the extent and variety of uncertainty in a modeling analysis, a realistic account of its impact on the results of the analysis continues to be challenging. For example, most sensitivity analyses in practice do not account for the multivariate structure of inputs and the correlations in this structure. Guidelines on how the results of sensitivity
analyses should be reported are currently in development5 (Bilcke et al., 2011; Stout et al., 2009).
Selection of Comparison Products
The amount of harm reduction claimed by an MRTP sponsor in an application is a critical issue in deciding whether to issue an order for the marketing of the MRTP. Harm reduction is inherently relative; a reduction claim is by definition relative to a comparison product. Selection of an appropriate comparison product is essential for informed and accurate decision making. The FSPTCA recognizes this, giving the Secretary of the Department of Health and Human Services authority to require product sponsors to compare their product to a commercially marketed representative product. The choice of appropriate comparison products will be driven by the type of MRTP being tested, the anticipated claim, and the study design. And, indeed, the comparison products may differ between different classes of evidence. However, two reference products come to the forefront in terms of integration and synthesis of evidence: leading brands and smoking cessation products.
Leading Brands
Those products that are most commonly used by consumers are likely to provide a good comparison for products that claim to demonstrate reduced health risk. “Leading brands” represent a set of products that accounts for a significant portion of the market and could capture subgroups of interest (e.g., low socioeconomic status, who tend to use discount brands, and racial/ethnic minorities, who tend toward menthols). Using leading brands increases the likelihood that the findings will have broader applicability to the population, which is crucial given the public health standard against which MRTPs are evaluated. Using leading brands as a comparator also avoids potential mischief in comparing an MRTP to a product that is little used but may inflate the apparent risk reduction of the MRTP. In some cases, when desiring a reduced exposure or risk claim, the comparison product will be a product within the same product class (e.g., cigarette-like MRTP versus leading brand cigarette), and thus the comparison is relatively straightforward. In other cases, an MRTP may make a reduced risk/exposure claim across product classes (e.g., smokeless MRTP versus leading brand cigarettes). In this case, the product should also be compared to leading products in its own class
_______________
5 ISPOR-SMDM Modeling Good Research Practices Task Force Working Group Part 3, unpublished data, 2011.
(e.g., smokeless MRTP versus moist snuff). Following this example, a smokeless MRTP that succeeds in both within- and cross-class comparisons against leading brands could reduce risk/exposure for both smokers and traditional smokeless users. However, it is also possible that a smokeless MRTP does pose less risk or exposure than cigarettes but is no different than other smokeless products not seeking a claim.
Smoking Cessation
On the opposite end of the spectrum of exposure and risk reduction is the “gold standard” of smoking cessation (or tobacco cessation in the case of smokeless tobacco users). This provides an aspirational goal for risk and exposure for MRTPs—in principle, the closer risks and exposures from the MRTP are to cessation products, the more confident a regulator can be in the chances for net public health benefit. Note that the use of this comparison product is not the same as studying whether the MRTP acts as an aid to smoking cessation. Rather, the goal is to compare how the risk or exposure reduction attained with use of the MRTP compares to smoking cessation of similar duration. It is also important to consider that for some health conditions, such as acute cardiovascular outcomes and lung function decline, the benefits of cessation accrue more quickly than for cancer.
In the committee’s view, the fundamental problem that confronts the FDA is a shortage of credible and reliable evidence about the effects of MRTPs on both individual and public health. The history of deceptive behavior by the tobacco industry undermined the trust of the public as well as the public’s confidence in the industry’s ability to rigorously conduct studies that will generate the data needed to evaluate these products. Therefore, the committee’s recommendations are designed to articulate the minimum standards for producing credible and reliable evidence to demonstrate that the marketing of an MRTP is consistent with the protection of public health. The committee articulates a strategy for the production of scientific evidence by making recommendations in three areas:
1. Types of evidence and studies
2. Design and integration of studies
3. Governance of studies
Types of Evidence and Studies
Finding 1: Types of Evidence. The public health standard articulated by the FSPTCA requires collection of scientific evidence from a wide range of disciplines and research domains. Although the committee respects the FDA’s independence and discretion in regulating MRTPs, the committee maintains there is a minimum range of research domains required to evaluate the effect of MRTPs on individuals and public health. Individual methods may change as the technology or state of the science may evolve, but the minimum standards for the domains of evidence will be relevant regardless of the state of the science in the future.
Recommendation 1: The FDA should require that studies submitted in support of an MRTP application address all key research domains needed to forecast and monitor the product’s public health impact, including:
• product composition and performance;
• addiction potential and likelihood for initiation or persistence of use;
• human exposure to harmful and potentially harmful constituents;
• perceptions about the product’s effects and likelihood of addiction; and effects of the product on human health and surrogates of human health.
Finding 2: Phased Approach to New MRTPs. Many novel MRTPs are likely to be developed for marketing in the near future. There are inherent uncertainties and risks with new products that should be addressed. Risks should be minimized before new products are tested in humans. To address the risk of new products, a phased approach, similar to the New Drug Application framework for the regulation and control of new drugs, is appropriate for the evaluation of new MRTPs. A phased approach will help the FDA ensure that only products that are unlikely to be unsafe and have a reasonable expectation of reducing harm relative to conventional tobacco products will be used in human studies.
Recommendation 2: The FDA should establish guidance that conveys an expected sequencing of studies, such that preclinical work is completed and submitted to the FDA before clinical (human subjects) work commences, and that there is a reasonable expectation based on preclinical work that a reduction or lack of harm will be seen in humans.
Finding 3: Clinical Trial Studies. Although the use of randomized controlled trial methods will be constrained for a number of reasons (including the practical limitations of study cost, size, and follow-up, and ethical constraints on randomizing study participants to harmful exposures), they will continue to play an essential role in creating an evidence base on the public health effects of MRTPs. Randomized controlled trial methods can provide highly reliable data on the likelihood of addiction and initiation or cessation of product use. Also, these methods can provide reliable evidence on human exposure.
Recommendation 3: The FDA should require randomized controlled trials in the following domains:
• Exposure reduction
• Self-administration of the MRTP
• Effects on use of conventional tobacco products
These randomized controlled trials should include multiple comparison products (such as nicotine replacement products, conventional cigarettes or smokeless tobacco, placebo preparations, and alternative nicotine delivery systems). These trials should also assess the effect of the MRTP on human exposure and on human health and surrogates of human health.
Finding 4: Requirement for Postmarket, Prospective Epidemiologic Studies. Postmarket studies of MRTPs will be critical to evaluating the effect of MRTPs on both individuals and the public’s health. In particular, the prospective cohort design will be an essential tool to validating anticipated or claimed effects of marketed MRTPs. These studies have several important strengths: (1) biochemical tobacco and MRTP exposure can be assessed at baseline, offering “unbiased” exposure assessment before health outcomes occur; (2) there is less of a problem with retrospective recall of product use, beause this information can be summarized at the start of the study and followed prospectively; (3) changing product use habits can be monitored as the study progresses; (4) outcomes can be documented as they occur, and verification is more efficient; and (5) a wide variety of outcomes can be evaluated in the same study, particularly outcomes that are more common. Furthermore, cohort studies allow assessment of overall health status and outcomes.
Recommendation 4: The FDA should require prospective epidemiologic studies to commence upon issuance of a marketing order to confirm reduced exposure and reduced risk claims, and to examine
effects of MRTP availability on the population as a whole, including the likelihood of initiation and cessation. The FDA should issue guidance on the design, conduct, and analysis of such studies.
Finding 5: Modeling of Public Health Outcomes. Mathematical modeling and simulation analysis provides a complementary approach to the conduct of empirical studies that can be useful at each stage of the regulatory process for MRTPs. Model-based analyses can (1) synthesize the available information from empirical studies of MRTPs; (2) enable researchers and decision makers to explore complex interactions and systems that may be impractical to evaluate in empirical studies; (3) allow researchers and decisions makers to explore “what if” questions relevant to decision making, which would not be practical to assess in empirical studies; and (4) be used to make projections about the short- and long- term effects of the introduction of MRTPs.
Recommendation 5: The FDA should issue guidance on the development and use of simulation and modeling approaches to predict public health impact through the systematic integration of information about relevant assumptions and influences. Such approaches should be tested for robustness with regard to results and assumptions, they should be public and transparent, and they should be validated against postmarketing epidemiologic research.
Design and Integration of Studies
Finding 6: Standards for Sampling in MRTP Studies. To have regulatory usefulness, studies of MRTPs must be generalizable to the overall population of interest and to specific populations, including populations at high risk for tobacco use. Failure to include relevant populations in studies will result in incomplete evidence on the effect of an MRTP on the public’s health and, therefore, will be inadequate to support regulatory decisions about the marketing of MRTPs.
Recommendation 6: The FDA should require studies to include populations of special relevance, including (but not limited to)
• users of tobacco products, including users who are and are not interested in quitting;
• in certain circumstances, nonusers of tobacco products;
• former smokers;
• beginning smokers;
• adolescents; and
• populations at a high risk for tobacco use, including, but not limited to, those low in socioeconomic status and educational attainment, and certain ethnic minorities.
Finding 7: Quality of Studies. The usefulness of a study to inform a regulatory decision hinges on the quality and appropriateness of the design. In many cases, complementary studies might be needed to provide a breadth of evidence for an informed regulatory decision with appropriate control of confounders and internal and external validity.
Recommendation 7: For all studies of the effects of MRTPs on human health and behavior, the FDA should require a range of designs that are properly powered, balance internal and external validity, and comprise multiple populations appropriate to the experimental questions being addressed.
Finding 8: Standards for Good Research Practice. A significant amount of guidance on minimum standards for scientific studies directly relevant to the evaluation of MRTPs has already been developed. Guidelines for formatting, design, conduct, and reporting of science are articulated in consensus statements, such as the Consolidated Standards of Reporting Trials (CONSORT) reporting criteria for clinical trials, the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for observational studies, the publication criteria of the International Council of Medical Journal Editors, and the reporting criteria of the International Conference on Harmonization. These existing guidelines represent robust standards for the conduct of science across many of the research domains relevant to the evaluation of MRTPs.
Recommendation 8: The FDA should issue guidance to the industry regarding the format, design, conduct, and reporting of studies in support of MRTP applications that is based upon current generally accepted principles for scientific investigation.
Finding 9: Standards for Integration of Evidence. Regulatory decisions regarding MRTPs will be based on a wide range and variety of scientific evidence, and the integration of scientific evidence will play a pivotal role in that decision making. The assessment of MRTPs will typically require the evaluation and integration of evidence on risks and benefits across multiple diverse outcomes, such as measures of toxicity, biomarkers, addictiveness, and disease endpoints. Modeling and simulation approaches are relevant to estimating public health effects of tobacco and, therefore, the FDA will likely engage in various methods of data
integration, synthesis, and analysis, including, but not limited to, simulation and modeling. It is critical that these approaches are transparent and reproducible.
Recommendation 9: The FDA should develop and use an approach to data integration that is explicit and transparent with regard to the importance of the different outcomes, that uses optimal available evidence, and that employs objective and reproducible methods for data integration.
Governance of Studies
Finding 10: Independent Oversight and Conduct of Studies. It has been established in public records and as a matter of law that the tobacco industry has engaged in illegal and improper practices, including the destruction and manipulation of scientific data. As a result, the tobacco industry is profoundly isolated from the mainstream scientific community. Many major universities have policies against acceptance of tobacco funding, and many high-impact scientific and medical journals will not accept tobacco industry-supported manuscripts. The consequence of this isolation is a lack of the expertise and the resources necessary to produce high-quality science across the range of disciplines to support an application to market an MRTP. Use of a trusted third party, particularly for products developed by the tobacco industry, could provide an avenue for the production of credible evidence needed by the FDA to evaluate tobacco products. Ultimately, such a research structure could encourage and support the production and dissemination of credible and reliable evidence about the effects of tobacco products on the public’s health.
Recommendation 10: MRTP sponsors should consider use of independent third parties to undertake one or more key functions, including the design and conduct of research, the oversight of specific studies, and the distribution of sponsor funds for research. Such independent third parties should be approved by the FDA in advance of the research.
Finding 11: Public Disclosure of Research. Public availability of data not only builds credibility and public trust, but also benefits the public because it allows for independent analysis of study methods and data. The model of Clinicaltrials.gov is particularly compelling and relevant, and a similar model of public accounting and open disclosure should be expected of the tobacco industry.
Recommendation 11: The FDA should require all MRTP sponsors to place all data generated in the development and marketing of the MRTP in a public repository selected by the FDA.
Finding 12: Proper Conduct of Research. Standards for the conduct of science and the protection of human research participants have been established for biomedical research enterprises not only in academics but also in commercial research. The FDA has the tools to ensure studies adhere to established standards in the drug development framework, which can be applied to the development of MRTPs. Those standards not only protect human participants, but also build credibility into any data that are provided to the FDA, particularly by the tobacco industry. Institutional credibility and trustworthiness is particularly relevant in this context, given the history of unethical and illegal practices of the tobacco industry.
Recommendation 12: The FDA should require studies offered in support of an MRTP application to adhere to established standards and principles of good research governance, including appropriately qualified investigators, transparency, independent institutional review board or ethical review, and adherence to the Common Rule (21 CFR parts 50 and 56).
Ashley, M. J., and J. E. Cohen. 2003. What the public thinks about the tobacco industry and its products. Tobacco Control 12(4):396-400.
Bilcke, J., P. Beutels, M. Brisson, and M. Jit. 2011. Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: A practical guide. Medical Decision Making 31(4):675-692.
Bojke, L., K. Claxton, M. Sculpher, and S. Palmer. 2009. Characterizing structural uncertainty in decision analytic models: A review and application of methods. Value Health 12(5):739-749.
CISNET Breast Cancer Collaborators. 2006. The impact of mammography and adjuvant therapy on U.S. breast cancer mortality (1975-2000): Collective results from the cancer intervention and surveillance modeling network. JNCI Monographs 2006(36):1-126.
Cummings, K. M. 2003. A promise is a promise. Tobacco Control 12(2):117-118.
Cummings, K. M., C. Morley, and A. Hyland. 2002. Failed promises of the cigarette industry and its effect on consumer misperceptions about the health risks of smoking. Tobacco Control 11(Suppl. 1):i110-i117.
Cummings, K. M., A. Brown, and R. O’Connor. 2007. The cigarette controversy. Cancer Epidemiology, Biomarkers & Prevention 16(6):1070-1076.
Glickman, S. W., J. G. McHutchison, E. D. Peterson, C. B. Cairns, R. A. Harrington, R. M. Califf, and K. A. Schulman. 2009. Ethical and scientific implications of the globalization of clinical research. NEJM 360(8):816-823.
Harris Interactive. 2003. Attitudes to government regulation vary greatly for different industries; The Harris Poll #19. http://www.harrisinteractive.com/vault/Harris-Interactive-Poll-Research-Attitudes-to-Government-Regulation-Vary-Greatly-for-Different-Industries-2003-04.pdf (accessed December 5, 2011).
Harris Interactive. 2010. Oil, pharmaceutical, health insurance, and tobacco top the list of industries that people think should be more regulated; The Harris Poll #149. http://www.harrisinteractive.com/vault/HI-Harris-Poll-Industry-Regulation-2010-12-02.pdf (accessed December 5, 2010).
Keating, T. J. 2001. Lessons from the recent history of the Health Effects Institute. Science, Technology, & Human Values 26(4):409-430.
Mendez, D., S. P. Marcus, P. I. Clark, S. J. Leischow, and T. F. Pechacek. 2008. Tobacco modelers conference proceedings. In Tobacco modelers conference. Ann Arbor: University of Michigan.
NCI (National Cancer Institute). 2008. The role of the media in promoting and reducing tobacco use. Edited by R. M. Davis, NCI tobacco control monograph series. Bethesda, MD: U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute.
NRC (National Research Council). 1991. Improving information for social policy decisions: The uses of microsimulation modeling. Vol. 1. Washington, DC: National Academy Press.
Rees, V. W., J. M. Kreslake, K. M. Cummings, R. J. O’Connor, D. K. Hatsukami, M. Parascandola, P. G. Shields, and G. N. Connolly. 2009a. Assessing consumer responses to potential reduced-exposure tobacco products: A review of tobacco industry and independent research methods. Cancer Epidemiology, Biomarkers & Prevention 18(12):3225-3240.
Rees, V. W., J. M. Kreslake, R. J. O’Connor, K. M. Cummings, M. Parascandola, D. Hatsukami, P. G. Shields, and G. N. Connolly. 2009b. Methods used in internal industry clinical trials to assess tobacco risk reduction. Cancer Epidemiology, Biomarkers & Prevention 18(12):3196-3208.
Rutter, C. M., A. M. Zaslavsky, and E. J. Feuer. 2011. Dynamic microsimulation models for health outcomes. Medical Decision Making 31(1):10-18.
Sonnenberg, F. A., and J. R. Beck. 1993. Markov models in medical decision making: A practical guide. Medical Decision Making 13(4):322-338.
Stout, N. K., A. B. Knudsen, C. Y. J. Kong, P. M. McMahon, and G. S. Gazelle. 2009. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics 27(7):533-545.
TPSAC (Tobacco Products Scientific Advisory Committee). 2011. Menthol cigarettes and public health: Review of the scientific evidence and recommendations. Tobacco Products Scientific Advisory Committee.
Vanni, T., J. Karnon, J. Madan, R. G. White, W. J. Edmunds, A. M. Foss, and R. Legood. 2011. Calibrating models in economic evaluation: A seven-step approach. Pharmacoeconomics 29(1):35-49.
Weinstein, M. C., B. O’Brien, J. Hornberger, J. Jackson, M. Johannesson, C. McCabe, and B. R. Luce. 2003. Principles of good practice for decision analytic modeling in healthcare evaluation: Report of the ispor task force on good research practices—modeling studies. Value Health 6(1):9-17.
