The U.S. Environmental Protection Agency (EPA) is not the only agency or organization that must make decisions in the face of uncertainty. Other agencies do as well, and, as is the case with EPA, when making a decision those other agencies must consider the likelihood and magnitude of a risk, the number of people at risk, whether some people are more at risk than others, the likelihood that a given intervention will mitigate the risk, the cost of potential interventions, and the potential consequences of inaction.
A number of decisions about public health interventions that are now well understood were made at a point in time when there were more uncertainties. For example, it is now well accepted that the pasteurization of dairy products eliminates the risk of infections caused by Campylobacter jejuni, Salmonella species, and other pathogens (FDA, 2011a); that fortification of foods with vitamins and minerals decreases the health consequences of vitamin and mineral deficiencies, e.g., that the fortification of wheat products with folate decreases neural tube defects (Darnton-Hill and Nalubola, 2002); that vaccination against common childhood infections prevents serious morbidity and mortality (Bonanni, 1999); and that prenatal screening for HIV infection facilitates the immediate postdelivery administration of antiviral agents to prevent HIV infection in an infected baby (Anderson and Sansom, 2006). However, not all of those interventions were unanimously accepted when first proposed or implemented, primarily due to uncertainties surrounding the possible benefits, risks, costs, feasibility, and public values. Many of those uncertainties have been reduced through research, including research on the effects of the interventions or
treatments that were implemented. In contrast, other interventions that were once thought beneficial, such as bed rest after childbirth or a heart attack, were found not to be beneficial once uncertainties were reduced.
In this chapter the committee reviews the decision-making tools and techniques from a number of different areas of public health, focusing on how uncertainty is taken into account in decisions. In particular, these reviews are in response to two of the questions in the committee’s charge: “What are promising tools and techniques from other areas of decision making on public health policy? What are benefits and drawbacks to these approaches for decision makers at EPA and their partners?” The committee could not review all organizations that make public health decisions or all decision-making processes, so it focused on selected agencies and organizations that, as does EPA, assess benefits and risks to human health (and in some cases technological, economic, and other factors), identify uncertainties, and make regulatory or policy decisions on the basis of those analyses. The chapter begins with a general discussion of the decision-making processes at a number of government agencies and organizations. It then uses case studies to illustrate how different agencies and organizations have made difficult regulatory or policy decisions while accounting for uncertainties.
A number of U.S. agencies play important public health roles that involve weighing evidence and taking into account uncertainties in the making of a policy or regulatory decision that affects public health. Table 4-1 summarizes the processes and methods used by different public health agencies and organizations to evaluate the human health risks and benefits and other factors influencing the decisions, along with their inherent uncertainties. As can be seen in the table, many organizations have no formal guidance materials related to their decision-making processes, and many do not conduct formal uncertainty analyses.
Within the U.S. Food and Drug Administration (FDA), some divisions—such as the center responsible for overseeing drug approvals and postmarketing safety, the Center for Drug Evaluation and Review, and the center responsible for overseeing medical devices, the Center for Devices and Radiological Health—have published guidance material on risk assessments. Historically, however, neither center provides a thorough discussion of uncertainty analyses or of the communication of those uncertainties along with FDA decisions, although a recent report for FDA has highlighted the importance of communicating the uncertainties in the agency’s decisions and the data that underlie them (Fischhoff et al., 2011). The Occupational Safety and Health Administration (OSHA) and the Nuclear Regulatory
|Agency/Organization||Method of Assessing Risks||Uncertainty Analyses|
|EPAa||Conducts quantitative assessments of risks. The assessment method varies depending on the nature of the exposure (for example, inhalation exposure vs. ingestion) and the endpoint of concern (for example, cancer vs. non-cancer endpoints).
Has published extensive number of detailed guidance documents and other materials related to its assessment methods and assessments of risks, benefits, and other factors related to individual agents and regulatory decisions.
Participated in interagency working group on risk assessment guidelines.
|Conducts extensive quantitative uncertainty analyses of the risks of individual chemical or other agents. The uncertainty analyses of human health risk estimates often includes the uncertainties in the
• dose–response assessment,
• exposure assessment,
• toxicity assessment, and
• risk characterization.
Has conducted some assessments, including analysis of the uncertainty in estimates of benefits and costs.
|FDA–CFSAN||Conducts quantitative assessments of risks, including product-specific assessments, pathogen- and chemical-specific assessments, product-pathway assessments, and risk-ranking assessments (for example, Listeria monocytogenes in ready-to-eat foods, and methylmercury in seafood).
Published guidance for risk assessments for food terrorism (FDA, 2012a).
|Uncertainty analyses vary among the assessments, with some having qualitative and some having quantitative assessments. Some analyses have estimated the effects of different regulatory actions (for example, Listeria monocytogenes assessment in FDA, 2003). Food terrorism and vulnerability assessment guidance discusses the fact that uncertainty exists, but does not provide formal guidance for analysis of uncertainty.|
|Agency/Organization||Method of Assessing Risks||Uncertainty Analyses|
|FDA–CDER||Has published guidance for industry on premarketing risk assessment (for example, FDA, 2005).
Has published guidance on risk communication with the public in the context of drug safety (FDA, 2012b).
|Discusses the fact that uncertainty exists, but does not present any formal guidance for analysis of uncertainty.
The guidance does not contain a specific discussion of the
communication of uncertainty.
Has published guidance for industry for benefit–risk determinations (FDA, 2012c).
Guidance discusses the sources of uncertainty in the science supporting estimates of human health risks and benefits. There is no guidance related to how to analyze uncertainties.
|CDC–ACIP||Uses the GRADE system to review and classify evidence (Ahmed et al., 2011).||The GRADE system includes a discussion of the strengths and limitations of the evidence. Depending on the information available, detailed uncertainty information, including uncertainty in the analysis of costs and benefits, is considered in ACIP’s recommendations.|
|AHRQ–Evidence Based Practice Centers||Categorizes the strength of the evidence related to medical interventions using a process based on the GRADE system.||The categories used include a qualitative discussion of the uncertainties in evidence.|
Conducts quantitative assessments of risk estimates for different exposures (exposures might be, for example, individual chemical exposures, noise exposure, or job descriptions).
Some assessments include some quantitative analyses of uncertainties (such as the presentation of upper and lower bounds on estimates or the evaluation of the effect of using different models to generate estimates).
Conducts qualitative, semi-quantitative, and quantitative assessments of human health risks (for example, the Listeria monocytogenes risk assessment in FDA, 2003).
The uncertainty analysis varies among the assessments; some include qualitative analyses and some include quantitative analyses, sometimes including analyses of the effects of different regulatory actions (for example, the Listeria monocytogenes risk assessment in FDA, 2003).
|Agency/Organization||Method of Assessing Risks||Uncertainty Analyses|
|Nuclear Regulatory Commission||Conducts quantitative, probabilistic risk assessments to estimate the likelihood and consequences of different events to help develop “risk-informed, performance-based regulations” (NRC, 2012).||Conducts uncertainty and sensitivity analyses in its assessments.|
|WHO–IARC||Publishes IARC Monographs, which evaluate the increased risk of cancer associated with environmental factors (including chemicals, complex mixtures, occupational exposures, physical agents, biological agents, and lifestyle factors).
The monographs include a qualitative assessment to classify environmental factors into groups on the basis of the evidence of carcinogenicity.
|Monographs include a qualitative discussion of the uncertainties in the evidence and identify data gaps.|
|WHO–FAO||Has published detailed guidance and assessments for microbial risk characterization in food that include qualitative, semiquantitative, and quantitative human health risk assessments. Has published quantitative assessments of the health risks associated with various chemicals in food (WHO, 2013).||Detailed discussions of the uncertainties in estimates of health risks and analyses of those uncertainties. Some assessments discuss economic factors in decisions, including uncertainties in economic analyses. Some assessments include discussion of risk communication. Uncertainties are discussed, but no quantitative assessments of uncertainties.|
NOTES: ACIP, Advisory Committee on Immunization Priorities; AHRQ, Agency for Healthcare Research and Quality; CDC, Centers for Disease Control and Prevention; CDER, Center for Drug Evaluation and Review; CDRH, Center for Devices and Radiological Health; CFSAN, Center for Food Safety and Nutrition; EPA, Environmental Protection Agency; FAO, Food and Agriculture Organization of the United Nations; FDA, Food and Drug Administration; FSIS, Food Safety and Inspection Service; GRADE, Grading of Recommendations Assessment, Development and Evaluation; IARC, International Agency for Research on Cancer; OSHA, Occupational Safety and Health Administration; WHO, World Health Organization.
aEPA and FSIS, in conjunction with other public partners published microbial risk assessment guidelines (USDA/FSIS and EPA, 2012). The guidelines discuss uncertainty, uncertainty analysis, and how to communicate uncertainty for risk characterization.
Commission discuss and use uncertainty analyses when formulating regulations, as does the Agency for Toxic Substances and Disease Registry of the U.S. Centers for Disease Control and Prevention (CDC) when it conducts its health assessments. At the international level, the International Agency for Research on Cancer of the World Health Organization (WHO) evaluates the evidence for the carcinogenicity of different agents and classifies those agents into different categories according to their estimated carcinogenicity; uncertainties are presented qualitatively when discussing the gaps in evidence.
Both the Food Safety and Inspection Service (FSIS) and the Center for Food Safety and Nutrition (CFSAN) at FDA have published—individually and jointly—a number of assessments of the health risks associated with chemical or biological agents in different foods. Those assessments often contain quantitative analyses of uncertainties and sensitivity analyses. In addition, an interagency working group1 has published draft guidelines for microbial risk assessments for food and water. The guidelines discuss the analysis and communication of uncertainties in risk assessments. WHO and the Food and Agriculture Organization (FAO) of the United Nations have also published guidance for the characterization of the risks from microbial contamination of food. That guidance discusses qualitative and quantitative human health risk assessments and the analyses of uncertainties in those assessments. They also discuss economic analyses to support decision making and the concomitant uncertainties in those analyses. Other uncertainties are not discussed, nor are issues related to the communication of uncertainties in the assessments of health risks and economics.
The U.S. Preventive Services Task Force (USPSTF), an independent task force that is supported and administered by the Agency for Healthcare Research and Quality, uses an evidence-based approach to evaluate health care interventions and make recommendations for clinical practices, including medical screening tests. To do so, the USPSTF uses an adapted version of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system that qualitatively characterizes as high, moderate, or low the likelihood that a practice or treatment is beneficial. A working group designed the GRADE system as “a common, sensible and transparent approach to grading quality of evidence and strength of recommendations” (GRADE Working Group, 2012b). A number of different organizations, such as the USPSTF and CDC’s Advisory Committee on Immunization Practices, have used the GRADE system to characterize evidence and recommendations (GRADE Working Group, 2012a). The use of GRADE when making recommendations related to vaccines is briefly discussed later in this chapter.
1 The working group included representatives from FDA, FSIS, and EPA.
The organizations that use more sophisticated uncertainty analysis, such as the Nuclear Regulatory Commission, FSIS, CFSAN, OSHA, and FAO, do so using methods and approaches that are similar to those used by EPA. A few organizations discuss the presence of uncertainty in economic analyses, but even those organizations do not explicitly discuss how or whether that uncertainty affected their decisions. Furthermore, they rarely consider factors other than health risks, health benefits, and economic analyses in their decision-making process. Many of the organizations elicit input from stakeholders through public meetings and comments on proposed action, much as EPA does; they do not, however, set forth an explicit process for incorporating uncertainties, such as a heterogeneity of stakeholder perspectives, into decision making.
In reviewing the processes of these public health agencies and organizations, the committee identified a number of assessments or decisions that illustrate the techniques and approaches that have helped—or, in one instance, handicapped—decision makers in their efforts to make decisions in the face of uncertainty. These cases include the following, which are discussed below: (1) the assessment of the health effects associated with secondhand smoke; (2) FSIS and FDA’s assessment of regulations related to Listeria monocytogenes; (3) FSIS’s assessment of the human health risks associated with bovine spongiform encephalopathy; (4) FSIS’s and FDA’s decisions surrounding the contamination of the food supply with melamine; (5) FDA’s decisions related to the diabetes medication Avandia® (rosiglitazone); and (6) assessments related to vaccinations. The committee did not attempt to develop a thorough evaluation or critique of each case; rather, it focused on aspects of the different cases that demonstrate useful approaches to evaluating and considering uncertainty in regulatory or policy decisions.2
Smoking bans that limit exposures to secondhand smoke (SHS) have been enacted in many places despite some stakeholders pointing to uncertainties in economic and other data as well as to uncertainties in estimates of health risks as a reason not to enact bans. Those uncertainties are thought to have been generated or at least exaggerated by the tobacco industry (Muggli et al., 2003; Ong and Glantz, 2000; Tong and Glantz, 2007). This section discusses what evidence was available on the economic
2 The committee uses these cases to illustrate the types of analyses and processes conducted in public health settings that can facilitate decision making. The committee is not endorsing, commenting on, or drawing any conclusions about the appropriateness or correctness of the regulatory or policy decisions themselves.
impacts and public acceptance of smoking bans at the time of decisions and what lessons can be learned from the implementation of these bans.
Human Health Risks
Although many of the human health risks associated with cigarette smoking were well established by the 1960s (U.S. Surgeon General’s Advisory Committee on Smoking and Health, 1964), it was not until 1986 that a surgeon general’s report concluded that SHS increased the risk for many different adverse health outcomes (HHS, 1986). The evidence of the risks from SHS comes from environmental chemistry and toxicology, including animal models of disease, as well from as observational studies (most of which were case-control studies or meta-analyses of those case-control studies). Federal and state human health risk assessments have concluded that SHS is harmful to humans (Cal EPA, 2005; EPA, 1992; NTP, 2011). Most of the risk-assessment findings were based on quantitative, well-conducted studies, although the findings were not always consistent among the studies (see HHS et al.  and IOM [2010a] for reviews of the studies). Concerns about variations in findings for a specific condition were allayed by the large number of studies, their general consistency, and the results of a number of meta-analyses conducted.
Despite the scientific evidence indicating adverse effects of SHS—including EPA’s assessment of environmental tobacco smoke (EPA, 1992), some individuals and groups, many of whom had financial interests in not having smoking bans, called the evidence into question (Oreskes and Conway, 2010). Similarly, public comment periods on health risk assessments and proposed policies and regulations were often dominated by individuals or groups criticizing the studies who were often allied with the tobacco industry, and tobacco industry documents indicate that they had a strategy of maintaining the scientific debate around the health effects of secondhand smoke (Bryan-Jones and Bero, 2003). To set smoking policies, therefore, decision makers had to distinguish between true uncertainties in the evidence and unfounded criticisms of the evidence motivated by financial interests, and they had to not only consider the results of each study, but also carefully scrutinize the quality of each study under consideration.
One economic factor that was taken into account when considering smoking bans was the potential economic effects on the establishments that would be subject to the bans (for example, bars and restaurants). Before the advent of state and local regulation, few studies had evaluated the
economic effects that smoking restrictions and bans might have on those establishments. Because detailed studies of the economic consequences of likely regulations and policies were often unavailable, there was also no characterization of the uncertainties surrounding those economic factors. Legislatures were left to make decisions on environmental controls for SHS exposure in the face of large uncertainty and intense lobbying. As smoking bans were enacted and implemented, studies have looked at the economic consequences of the bans, for example, on restaurants and bars (Glantz and Charlesworth, 1999), which has decreased the uncertainty around the economic factors.
In accordance with national or local rules, decisions on smoking restrictions and bans generally included the opportunity for the public to comment on the proposed policies and on the science underlying them (Bero et al., 2001). As mentioned above, many of the people commenting spoke out against the policies, but some of that opposition was orchestrated by the tobacco industry and allied parties (Mangurian and Bero, 2000). In addition, several commentators on the process suggested that there was “burnout” by the public on SHS issues and a loss of advocacy that came with many years of direct cigarette regulation, particularly at the local level (WHO, 2006). Furthermore, the national environmental and public health organizations and agencies that could have supported local and state regulations often did not weigh in strongly, possibly because of a coordination (Bero et al., 2001). Those aspects increased the uncertainty about the percentage of people and which sectors of the public were for or against smoking bans and restrictions.
Further problems with the interactions with stakeholders may have been caused by communication issues, including a lack of communication about the uncertainties surrounding the issue. Most communication with stakeholders about uncertainty used standard statistical presentations of epidemiological studies and meta-analyses (Hackshaw et al., 1997; Law et al., 1997), and there appears to have been little attempt in the federal (EPA, 1992; NTP, 2011) and state risk analyses (see, for example, Cal EPA ) to present uncertainty in lay terms. The dose–response phenomenon was also not discussed extensively, with the exception of questions concerning the relevance of studies of home exposures to social exposures. Those discussions could have led the public to believe that the extent and implications of uncertainties in the data and analyses were greater than they actually were.
Decisions in the Face of Uncertainty: Lessons Learned
Given the lack of any known health benefits from exposure to SHS, health assessments centered on the risks associated with exposure to SHS. Despite a large amount of evidence related to those risks, discussions often focused on the uncertainties in the evidence rather than on its consistencies; individuals and groups with a financial stake in blocking smoking restrictions and bans often drove those discussions (Bryan-Jones and Bero, 2003). There was large uncertainty about the potential costs from lost revenues to establishments subject to bans and about the financial benefits from avoided medical costs. The discussions of smoking bans also raised social issues related to infringing on personal, voluntary behaviors and personal rights, with a large amount of heterogeneity in people’s opinions on those issues.
States and local jurisdictions where there was either the political will or higher public acceptance of bans were the first to implement smoking restrictions and bans. Researchers took advantage of some of those bans to investigate whether they were associated with any health effects, to study the public reaction to the bans, and to see whether the bans had any economic consequences on establishments covered by the bans. Epidemiology studies indicated that smoking bans or restrictions were associated with decreases in adverse cardiovascular events (Barone-Adesi et al., 2006; Bartecchi et al., 2006; Cesaroni et al., 2008; IOM, 2010a; Juster et al., 2007; Khuder et al., 2007; Lemstra et al., 2008; Pell et al., 2008; Sargent et al., 2004; Seo and Torabi, 2007; Vasselli et al., 2008). Research surveys showed that the public approval of various state and local laws and regulations was generally, although not uniformly, positive after implementation, both in the United States and other countries (Borland et al., 2006; Kelly et al., 2009; Miller et al., 2002; Pursell et al., 2007; Tang et al., 2003). For example, in 2000 73.2 percent of people surveyed in California who had visited a bar at least once in the previous year approved of California’s smoke-free laws, up from 59.8 percent in 1998, the year that a ban of smoking in all bars was implemented (odds ratio [OR] = 1.95; 95 percent confidence interval [CI] = 1.58, 2.40) (Tang et al., 2003). Studies of the economic effects of the bans on restaurants and other establishments decreased the economic uncertainties related to smoking restriction and bans (Glantz and Smith, 1994, 1997; Hyland et al., 1999; Sciacca and Ratliff, 1998; Scollo et al., 2003). With the decreased uncertainty provided by all these types of studies, other state and local regulators had stronger evidence on which to base their decisions. As of October 5, 2012, more than 3,581 municipalities had laws that restrict where smoking is allowed, and “36 states, along with American Samoa, the Northern Mariana Islands, Puerto Rico, the U.S. Virgin Islands, and the District of Columbia” had workplace laws that restrict smoking (ANRF, 2012). U.S. efforts to characterize the
risks of environmental tobacco smoke have also been used to support international smoking bans.
The passing and implementation of smoking bans provides many lessons for EPA and for other regulators. First, it emphasizes the importance of scientists and policy makers scrutinizing the quality of individual studies as part of appropriately determining the overall weight of the evidence and the uncertainty in it. Second, it demonstrates the need to consider the sources of scientific criticisms and uncertainties that are raised and to separate valid scientific criticisms from invalid ones. Third, it emphasizes that when considering economic factors and other factors, such as public acceptance, uncertainty based on anecdotal concerns about potential financial consequences might not reflect the actual effects of a regulation. Fourth, it illustrates the heterogeneity in public values and how acceptance of health-protective policies can shift over time, leading to new societal norms.
Listeria monocytogenes is a bacterium that causes listeriosis, a potentially fatal bacterial infection that can result from eating food contaminated with the bacterium (FDA and FSIS, 2003a). Listeriosis primarily affects pregnant women, older adults, and persons with weakened immune systems (FDA and FISIS, 2003a). Infections during pregnancy can lead to premature delivery, infection of the newborn, or stillbirth. Death occurs in 20 percent of cases of listeriosis (Swaminathan and Gerner-Smidt, 2007); CDC estimates that L. monocytogenes causes nearly 1,600 illnesses each year in the United States, including more than 1,400 hospitalizations and 255 deaths (Scallan et al., 2011). FDA and FSIS collaborated, in consultation with the CDC, to conduct a risk assessment of L. monocytogenes. In this section, the committee discusses that risk assessment and the uncertainty analyses in it and also discusses how FDA has used the results of that risk assessment to refine its policies around the control of L. monocytogenes in different food products within its regulatory purview.
L. monocytogenes can contaminate food contact surfaces and also non-food contact surfaces, such as floors and drains in food-processing facilities. The growth of L. monocytogenes is more difficult to control than the growth of most other bacteria. Temperatures at or below 40°F control the growth of most bacteria, but L. monocytogenes survives on cold surfaces and can multiply slowly at 32°F; temperatures of 0°F are required to completely stop L. monocytogenes from multiplying (FDA and FSIS, 2003b).
Both FDA and FSIS have regulations related to L. monocytogenes in ready-to-eat (RTE) foods. FSIS has a zero-tolerance policy for L. monocytogenes on RTE meat products within its regulatory purview, such as hot dogs and luncheon meats (FSIS, 2003). Until FDA published proposed draft guidelines in 2008 that established two categories of RTE foods, it had a zero-tolerance policy for L. monocytogenes in all RTE foods within its regulatory purview. Under those zero-tolerance policies, the presence of L. monocytogenes indicated the product was “adulterated” and unfit for commerce (FDA, 2008a).
The food industry argued that not all RTE foods support the growth of L. monocytogenes equally and, therefore, that not all RTE foods should be subject to the same regulations. The food industry identified low pH, low water activity, and the presence of an “inhibitory” substance as factors that inhibit the growth of L. monocytogenes. The industry also argued that the risk of listeriosis depends on the type and frequency of RTE consumption as well as on home refrigeration factors (i.e., temperature and duration). In essence, the industry was arguing that the variability in food susceptibilities and the uncertainty in home refrigeration had not been considered in the regulations.
In light of those arguments, FDA reviewed its zero-tolerance policy for L. monocytogenes. The agency was faced with determining whether or not to relax the zero-tolerance standard for some foods and, if they were relaxed, what guidance it should issue to industry for controlling L. monocytogenes (FDA, 2008a).
Human Health Risks
The 2003 FDA/FSIS risk assessment was designed to predict the relative risk of listeriosis from eating certain ready-to-eat foods among people in three age-based groups: perinatal (16 weeks after fertilization to 30 days after birth), elderly (60 years of age and older), and intermediate-age (general population, less than 60 years of age) (FDA and FSIS, 2003a,b). The assessment evaluated 23 categories of foods considered to be the principal potential sources of L. monocytogenes. In particular, it evaluated the heterogeneity among different foods and different age groups of people. Consistent with previous assessments, the 2003 assessment concluded that foodborne listeriosis is a moderately rare but severe disease, and it supported the findings from epidemiologic investigations of sporadic illness and outbreaks of listeriosis that certain foods (for example, pâté, fresh soft cheeses, smoked seafood, frankfurters, and foods typically purchased from deli counters) are potential vehicles of listeriosis for susceptible populations.
The assessment estimated the human health risks for different exposures. It also used various scenarios—different food consumption rates,
different growth rates, different contamination rates, and so on—to evaluate which points in the farm-to-table continuum were most susceptible to contamination or had the greatest potential for risk mitigation. Through those sensitivity analyses, the assessment identified five main factors that affect consumer exposure to L. monocytogenes at the time of food consumption: (1) the amount and frequency of consumption of a food, (2) the frequency and levels of L. monocytogenes in RTE food, (3) the potential to support growth of L. monocytogenes in food during refrigerated storage, (4) refrigerated storage temperature, and (5) the duration of refrigerated storage before consumption. Those factors point toward several control strategies to mitigate the risks of listeriosis, which ranged from “reformulation of products to reduce their ability to support the growth of L. monocytogenes” to “encouraging consumers to keep refrigerator temperatures at or below 40°F and reduce refrigerated storage times” (FDA and FSIS, 2003a, p. 27).
Rather than providing a single risk estimate, the health risk assessment provided a range of estimates using sensitivity analyses and probabilistic methods for the different food categories, the populations with different susceptibilities to listeriosis, and the strains of L. monocytogenes with varying virulence. The assessment “attempt[ed] to capture both the variability inherent in the incidence of foodborne listeriosis and … the uncertainty associated with the data analysis” (FDA and FSIS, 2003a, p. 15). Presenting the different sensitivity analyses allowed decision makers to target strategies to mitigate risks for different populations and food categories. For example, specific strategies could be developed to prevent exposures in pregnant women, the elderly, and susceptible individuals within the intermediate-age group. In addition to the uncertainty analyses discussed above, FDA/FSIS discussed other uncertainties that remained, including the need for evidence related to changes in food processing, distribution patterns, preparation, and consumption practices.
FDA used the results of the risk assessment and its analyses of variability to develop regulations that differentiated between foods that pose higher and lower risks for listeriosis. In February 2008, FDA issued a draft Compliance Policy Guide (FDA, 2008a) that proposed two risk-based limits for L. monocytogenes in RTE foods, differentiating between foods that support the growth of the pathogen and those that do not. That regulation takes into account the conclusions from the sensitivity analyses in the assessment that the risks from foods with a pH less than or equal to 4.4, foods with water activity less than or equal to 0.92, and frozen foods do not support the growth of L. monocytogenes and, therefore, pose very low risk of listeriosis.
The process for developing the FDA/FSIS risk assessment included a period for public comment on a January 2001 draft of the L. monocytogenes risk assessment (FDA and FSIS, 2003b). The final risk assessment summarized the changes made to the draft in response to the comments received. Many of those changes reflected decreased uncertainties as a result of the feedback received. They included changes to the food categories to better incorporate characteristics that contribute to the support of growth of L. monocytogenes (for example, moisture content and pH), updated data on contamination, growth rates of L. monocytogenes in different foods and for different storage durations, the frequency and prevalence of L. monocytogenes on different foods, consumer habits, and modifications to the model used.
Decisions in the Face of Uncertainty: Lessons Learned
The assessment of L. monocytogenes and FDA’s use of that assessment highlight how analyses of health risks that account for uncertainties—such as those for different food types, different storage conditions, and different susceptible populations—can provide decision makers with information to help design policies that target mitigation strategies to the greatest risks, either for specific foods with a higher likelihood of being associated with human illness, or for populations that are more susceptible to illness. The detailed and specific risk characterization allowed FDA and FSIS to develop specific guidance for different foods and to develop outreach strategies to protect the populations at highest risk from consumption of foods contaminated with L. monocytogenes.3 In other words, the assessment and how it was used in FDA’s decision highlights the importance of analyzing the heterogeneity in an assessment and demonstrates how—when uncertainty about that heterogeneity is reduced—an agency can better tailor risk-mitigation options.
In light of the complexity of the risk assessment, the agency also evaluated methods for grouping the results for communication purposes (FDA and FSIS, 2003b). The assessment concluded, “One approach that appears to be very useful for risk management/communication purposes is the evaluation of the relative risk ranking results using cluster analysis”
3 A recent outbreak of L. monoctyogenes in cantaloupes, however, shows that even lower-risk foods are not immune to contamination. See CDC (2011a) for a description of the outbreak. FDA investigated that outbreak, identified a number of factors that contributed to the outbreak (FDA, 2011c), and highlighted the need for all processors to employ good agricultural and management practices (FDA, 2011c), as are provided in FDA and USDA guidance to industry (FDA, 2008c).
(p. 228). That analysis allowed the development of a matrix to depict five overall risk designations: very high, high, moderate, low, and very low. For example, deli meats are considered very high risk because they were in the high cluster for both per-serving and per-annum consumption.
Bovine spongiform encephalopathy (BSE), commonly referred to as mad cow disease, is a chronic degenerative disease that affects the central nervous system of cattle. BSE is one of a number of transmissible spongiform encephalopathies that are caused by infectious agents associated with an abnormally folded protein known as a prion (IOM, 2004). In cattle, the infectious agent is transmitted through ingestion of contaminated feed. In humans, exposure to beef products that are infected with BSE can lead to Creutzfeldt-Jakob disease (CJD), a fatal neurodegenerative disease (IOM, 2004). Because there are no vaccines against or treatments for BSE or CJD and because it is extremely difficult to destroy the infectious agent, preventing the spread of BSE among cattle and preventing cattle infected with BSE from entering the human food supply are key control mechanisms. The potential human health consequences from CJD are severe, and the costs of an outbreak are high. For example, hundreds of thousands of infected animals had to be destroyed, and trade restrictions were instituted by other countries, following an outbreak of BSE in Britain in 1985 (IOM, 2004).
In 1998 the U.S. Department of Agriculture (USDA) requested that the Harvard Center for Risk Analysis (HCRA) evaluate measures to control the spread of BSE among animals and from animals to humans. In response, in 2003 the HCRA developed a model to simulate the consequences of introducing BSE into the United States by various means (Cohen et al., 2003a,b). USDA and the public provided comments on the assessment, and HCRA published an updated assessment in 2005 (Cohen and Gray, 2005), and the results of additional simulations were published in 2006 (Cohen, 2006). The assessment demonstrates how, when faced with uncertainty about the best regulatory option, the effects of different management options can be modeled to inform the decision-making process.
Assessment of BSE Infection Risks
To assess the risks to cattle and to humans from the introduction of BSE in the United States, HCRA designed a model that could predict “the number of newly infected animals that would result from introduction of BSE, the time course of the disease following its introduction, and the potential for human exposure to infectious tissues” (Cohen et al., 2003b, p. vii). The model also incorporated “key processes and procedures that
make the spread of disease more or less likely” (p. vii) to estimate the effectiveness of different control measures in stemming the spread of BSE to cattle and decreasing the likelihood of BSE-infected meat entering the human food supply. Table 4-2 lists the processes and procedures evaluated in the 2006 simulations (see notes section in Table 4-2 below). For example, the model was used to estimate the number of infected cows anticipated 20 years after 500 infected animals were introduced into the United States, using different beef consumption rates or with different detection rates for BSE in antemortem inspections. Using the model, HCRA was able to evaluate different scenarios and look at the effects on the human food supply up to 20 years after the introduction of a given number of BSE-infected cattle into the United States.
The model makes it possible to examine the degree to which different parameters affect the estimated risks. For example, the sensitivity analyses demonstrated that the model is very sensitive to the rate of either accidental or intentional misfeeding of cattle with feed containing ruminant protein but that there is a large uncertainty in the rate at which that misfeeding occurs. Changing the incubation time also affects the model outputs;
NOTES: Explanations of the sensitivity analyses:
Sensitivity 1 – Pessimistic MBM/feed production mislabeling and contamination assumptions
Sensitivity 2 – Pessimistic misfeeding assumptions
Sensitivity 3 – Pessimistic render reduction factor assumptions
Sensitivity 4 – Higher assumed beef on bone consumption rates
Sensitivity 5 – Pessimistic antemortem inspection BSE detection rates
Sensitivity 6 – Longer incubation period
Sensitivity 7 – Evaluate the importance of the proportion of cattle showing no clinical signs of diseases that are nonambulatory
Sensitivity 8 – Evaluate the importance of the proportion of clinical animals that are nonambulatory
Abbreviations: BSE = bovine spongiform encephalopathy; MBM = meat and bone meal.
SOURCE: BSE Risk Assessment (Cohen, 2006; FSIS and USDA, 2004).
the number of newly infected cattle decreased from 180 to 43 when the incubation period was lengthened by a factor of two. In contrast, other parameters—such as the rates of mislabeling and contamination, render reduction factors, the consumption rates of beef on bone, and the effectiveness of antemortem inspections—had less influence on model outputs (Cohen and Gray, 2005). The 2006 assessment analyzed other regulatory options, ranging from a ban on slaughter for human consumption of all nonambulatory disabled cattle to prohibiting human consumption of the brain, skull, eyes, trigeminal ganglia, spinal cord, vertebral column, and dorsal root ganglia of cattle 30 months of age or older, in order to evaluate the impacts of these options on risks. That assessment also presented the effects of different regulatory compliance rates on risks, and it presented, in addition to means, the 5th, 25th, 50th, 75th, and 95th percentile values for the outputs (see Table 4-2). That information gave decision makers an indication of how the uncertainty in the data used in the assessment might affect the output from the assessment. Overall, the 2006 simulations indicated that the preventive measures enacted by USDA decrease the potential for human exposures but have little effects on the spread of BSE in the U.S. cattle population.
Economic Factors and Public Sentiment
FSIS implemented a number of rules to protect the public’s health in the wake of finding a BSE-positive cow in the United States,4 and it also conducted a regulatory impact analysis of those rules. The analyses used probabilistic models to estimate the “costs and revenues changes (a partial budget analysis) associated with the final rule compared with the baseline; the net total monetary changes of decreased revenues and increased costs versus increased revenues and decreased costs; and the distribution of those net revenues and costs changes among producers” (FSIS and USDA, 2007, p. 24).
The inputs in the economic model included ranges of values for the variables related to the costs for industry to comply with the rule, such as the number of affected establishments, the number of different types of cattle slaughtered (for example, bulls, cows, or steers), the weight of different cattle, the number of days a week a slaughter facility operates, labor costs, the one-time capital costs for equipment, ongoing material costs of compliance, and disposal costs. The analyses presented minimum, maximum, and most likely values for the compliance costs as well as for associated biological parameters. Using that approach, FSIS estimated probability distributions for compliance costs and for the potential reductions in
4 FSIS described the actions as “emergency actions to protect public health” (FSIS/USDA, 2004, p. 1).
the risks to human health for different risk-mitigation options. The agency then calculated the cost-effectiveness ratio for each of the options and the incremental cost-effectiveness ratios for the options. Although the data tables in an appendix of the final FSIS/USDA report presented a range of values—specifically, the minimum, mean, and maximum costs—the tables in the main body of the report showed only the mean (or most likely) costs.
The report identified a number of data gaps in the analyses. For example, it was not known how much it would cost to redesign facilities to allow for the segregation of animals and animal carcasses, nor was it known how many facilities had already implemented such changes; those costs, therefore, were not included in the analysis. The regulatory impact analysis supported most of the rules FSIS had implemented on an emergency basis, although the analysis did lead to changes being made to some aspects of the rule (FSIS and USDA, 2007).
FSIS also held a number of public meetings to obtain input on the various potential impacts of regulatory actions and assessed how best to communicate the issues concerning BSE to the public, given the large uncertainties in both the data and the analyses.
Decisions in the Face of Uncertainty: Lessons Learned
The analyses that FSIS performed in response to the discovery of a BSE-positive cow in the United States provide an example of a decision-driven risk assessment. The probabilistic models were designed to evaluate specific regulatory options and to identify where within the beef-processing system the agency could have the greatest effect on risks. The benefit and cost-effectiveness analyses provided FSIS with information about the comparative costs of different actions to help decision makers focus resources. The assessment demonstrates how analyses can evolve and be updated in response to new information, stakeholder input, emergency situations, and evolving regulations. The agency’s actions demonstrate an adaptive management process used in the face of deep uncertainty about the health risks of BSE and the appropriate mitigation strategies necessary to protect against it. FSIS implemented emergency regulations and modified them, as appropriate, after a full regulatory impact assessment was conducted.
Meat, Chicken, Eggs, and Catfish
In 2007 FDA, in conjunction with FSIS, conducted an interim assessment of the risks in the human food supply of melamine, which people can get from eating pork, chicken, fish, and eggs. The concern arose because
some animals that are raised to be consumed by humans were inadvertently fed animal feed that may have been adulterated with melamine and its analogues. Those compounds had been associated with kidney failure in pets that had eaten contaminated pet food. Because of these concerns, animals on farms where contaminated feed was distributed were held (either voluntarily or by state quarantine) pending an assessment of the risks to human health (FDA and FSIS, 2007).
When the contamination occurred, there was a great deal of uncertainty surrounding the human health risks from melamine. No information was available on the relative potency of melamine and its analogues. It had been hypothesized that melamine might act synergistically with its analogues, but that hypothesis had not been tested. There had been no studies in humans, and high-dose studies in dogs, rats, and mice had shown hepatic toxicity but not liver failure. Furthermore, the mechanism of those effects was not understood, and it was not known if the adverse effects would occur in humans or at lower doses. In the face of those uncertainties, the agencies needed to estimate the risks to humans from the consumption of meat and fish possibly contaminated with melamine and to quickly decide what, if any, action was needed to protect the public’s health.
A 13-week study in rats that had been orally exposed to melamine indicated a no-observed-adverse-effect level (NOAEL; see Chapter 2 for an explanation of NOAELs) for bladder stones of 63 mg/kg body weight/day. Given the lack of data and the need for a quick decision, a quantitative analysis of the uncertainty was not possible, and the agencies used safety factors to evaluate the risks. That NOAEL was divided by a safety factor of 100 (two 10-fold safety factors, or uncertainty factors, to account for inter- and intraspecies sensitivity) in order to come up with a tolerable daily intake (TDI)5 of 0.63 mg/kg body weight/day. The agencies estimated human exposures to melamine for three different scenarios, including a worst-case scenario, and presented the mean and 90th percentile exposures in each scenario based on a consumption of catfish, chicken, eggs, pork, or a combination of all four (FDA and FSIS, 2007). Given the concentrations of melamine measured in samples of meat collected from animals exposed to the melamine-contaminated feed and estimated human exposures from consumption of meat, the agencies concluded that, even using the worst-case consumption scenario that assumed all solid food was contaminated with melamine, the estimated potential exposures were well below the
5 The TDI is defined as the estimated maximum amount of an agent to which individuals in a population may be exposed daily over their lifetimes without an appreciable health risk with respect to the endpoint from which the NOAEL is calculated (http://www.fda.gov/Food/FoodSafety/FoodContaminantsAdulteration/ChemicalContaminants/Melamine/ucm164658.htm (accessed January 3, 2013).
TDI. Given the estimated risks, FSIS decided that products from animals fed contaminated feed would not be considered adulterated and, therefore, could be made available for slaughter (FSIS, 2007).
In September 2008 FDA learned that some infant formula from a Chinese manufacturer might contain melamine. Consumption of melamine-contaminated infant formula in China had resulted in a reported 52,857 cases of nephrolithiasis (and, in some instances, renal failure), including about 13,000 hospitalizations and 3 confirmed deaths (FDA, 2008d). The exposure scenario and the sensitivity of the population in the case of infant formula were very different from those in the case of the contamination of meat, fish, and eggs. Some of the specific differences were
1. the contaminated product, infant formula, represented the total caloric intake for most of these infants;
2. the exposure was chronic over a number of months;
3. the population exposed to the products consisted of infants and toddlers whose renal systems had not yet been fully developed; and
4. the human exposure had not been mitigated by the melamine passing through the digestive system of an animal (FDA, 2008d).
Many of the uncertainties discussed above in the case of the contamination of meat and other food products were still in place, and once again FDA had to quickly estimate the risks to humans from melamine—in this case, the risks to infants—and to decide what actions, if any, were necessary to protect public health. Additional uncertainties stemmed from the possibility that premature infants with immature kidney function who had been fed formula as the sole source of nutrition were getting a significantly larger exposure to the melamine than was seen in other cases; they could be ingesting more of the chemical per unit body weight than adults who ate the meat and other products, and they could have been exposed for a longer time period than full-term term infants.
Some studies had been published since the 2007 risk assessment (FDA and FSIS, 2007), including studies related to melamine metabolism and studies on the pathology resulting from exposures in pets (cats and dogs). Data since the 2007 interim risk assessment raised further concerns about an increased toxicity from combined exposure to melamine and its analogue cyanuric acid. Because of those unknowns, FDA concluded in its October 2008 assessment that it could not “establish a level of melamine and its analogues in these products that does not raise public health concerns” (FDA, 2008d). Given the risks, infant formula from China was recalled,
and FDA initiated a sampling program to test formula and other products for melamine contamination (FDA, 2008b).
In November 2008, FDA updated its assessment after finding infant formula that had very low concentrations of either melamine or one of its analogues, but not both (FDA, 2008e). That contaminated formula, which was manufactured in the United States, contained concentrations ranging from 0.137 ppm of melamine in one product to 0.247 ppm of cyanuric acid in another. Those concentrations were “up to 10,000 times less than the levels of melamine reported in Chinese-manufactured infant formula” (FDA, 2008e). FDA had to estimate the risks to infants, taking into account the uncertainty. To do so, FDA began with the TDI previously calculated for adult exposures to melamine in meat, chicken, eggs, and catfish.
Given the potential for infants to be more sensitive than adults for the reasons discussed above, FDA applied a 10-fold safety factor to the TDI to get a TDI of 0.063 mg melamine/kg body weight/day to evaluate the additional risks. Assuming a worst-case scenario in which all of an infant’s total dietary intake (0.15 kg of powdered infant formula) was contaminated with melamine, 100 percent of the diet would have to be contaminated with 1.26 ppm of melamine for an infant to reach the TDI. FDA concluded, therefore, that the “levels of melamine or one of its analogues alone below 1.0 ppm in infant formula do not raise public health concerns” (FDA, 2008e). In light of that conclusion, FDA did not recall infant formula that contained concentrations of melamine below 1.0 ppm, and it continued its sampling program to test for melamine in the food products that it regulates.
Decisions in the Face of Uncertainty: Lessons Learned
As was the case in the contamination of the food supply with melamine, agencies sometimes have to make regulatory decisions in the face of deep uncertainties about the health risks and—because of the potential for imminent public health consequences—with little or no time to investigate those risks. The responses to contamination of the food supply demonstrate how, even in the absence of probabilistic modeling of uncertainty, a human health risk assessment can provide information for an important regulatory decision. The use of scenarios, including a worst-case scenario as above, can help with such decisions.
The regulation of some prescription pharmaceuticals provides an example of decision making in the face of many uncertainties. The case of the diabetes medication Avandia (rosiglitazone) illustrates the uncertainties and scientific disagreements surrounding drugs and how FDA makes
decisions under those uncertainties. Private industry, government agencies, health professionals, and the public are important stakeholders for those decisions, and each has different roles, responsibilities, levels of scientific literacy, and values.
Premarketing Benefit and Risk Assessments
Before a pharmaceutical can be marketed, it must undergo a lengthy approval process involving the review of clinical trial data on efficacy, toxicity data, and pharmacokinetic profiles. But the information available, especially related to the safety of the drug, is limited at the time of approval. Those limitations include the facts that the products are generally tested in individuals with single illnesses for relatively short periods of time (compared with the lifetime use of some products) and in many fewer people than will be taking the drug once it goes to market (approval can be obtained with as few as 4,000 participants, and possibly many fewer) (IOM, 2012a).
FDA has published general guidance about the approval process for prescription drugs, but that guidance does not include recommendations for formal, quantitative analysis of uncertainties. For example, FDA has guidelines for the design and conduct of premarketing clinical trials (FDA, 1998), but the manufacturers have broad latitude in designing their studies; the FDA scientists who review the material interpret those guidelines on a case-by-case basis for the specifics of each drug. Ultimately, FDA decision makers evaluate the evidence for the benefits and risks of a drug in the context of the public health need for the drug and decide whether to approve it. FDA sometimes convenes advisory committees of experts to make recommendations regarding the approval of drugs (and regarding other topics as well), and FDA advisory committees might interpret the data or the guidelines differently from the FDA reviewers. FDA often, but not always, follows the recommendations of its advisory committees. The vote of an advisory committee on whether a drug should be approved for marketing is often not unanimous, indicating uncertainty in one or more of the factors involved in drug approval (FDA, 2011b).
Regulatory and Study History of Avandia
FDA’s regulatory decisions related to Avandia (rosiglitazone) provide another example of decision making under uncertainty. In 1999 FDA approved Avandia for the treatment of diabetes, but it requested that the drug’s sponsor conduct further clinical trials because of concerns about the drug’s effects on lipids. In 2007 a meta-analysis of the results of the clinical trials raised concerns about an increased risk of cardiovascular events
associated with Avandia (Nissen and Wolski, 2007). Following the publication of that meta-analysis, an FDA advisory committee concluded that the use of Avandia “was associated with a greater risk of myocardial ischemic events than placebo,” but because of the limitations of the meta-analysis6 it recommended label warnings and education rather than a withdrawal of the drug. FDA required a boxed warning on the drug’s package and also required a long-term randomized controlled head-to-head clinical trial to evaluate the drug’s cardiovascular risks. Observational studies carried out over the next few years indicated an elevated risk of adverse cardiovascular events, and in 2010 FDA once again investigated the drug’s benefit–risk profile to decide if any changes in the drug’s marketing approval conditions were needed (FDA, 2010a). At that time the agency heard from a number of stakeholders about the importance of having the drug available to them, despite the risks of adverse cardiovascular events. After reviewing the scientific evidence and hearing the views of patients and other stakeholders, in September 2010 FDA stopped its required trial and placed a number of restrictions on access to Avandia, but it allowed the drug to remain on the market in the United States (FDA, 2010c).
In making that decision FDA had to weigh the benefits of a widely used drug that was effective at combating diabetes—a large public health problem—with the risks of adverse cardiovascular outcomes—another large public health problem—when great uncertainty existed. As is typically the case given the nature of the data for evaluating drugs, no formal quantitative uncertainty analysis was conducted. In describing the uncertainty, the director of FDA’s Center of Drug Evaluation and Review (CDER) stated that
there are multiple and conflicting signals of … risk. … The current cardiovascular safety database for [the drug] does not provide an assurance of safety at the level set out in FDA’s guidance for marketed … drugs. … Many highly experienced clinical trialists and methodologists, both within and external to the FDA, who have examined these data find it hard to arrive at definitive conclusions. … This uncertainty about the risk of [the drug] is overwhelmingly the most important reason for the differing opinions on what regulatory action should be taken … [and] various members of the Committee had quite disparate opinions on these matters. These differences of opinion stem from varied conclusions about the existing data. … Similarly, several [Agency] Offices have different recommendations. The Office of Surveillance and Epidemiology … recommends market withdrawal. … [The] Office of New Drugs recommends additional
6 Most of the committee members agreed that there was at least a strong signal for increased cardiac ischemic risk, although concerns were raised about the short duration of the trials, the quality of the data, the low number of cardiac events, the lack of cardiac event adjudication, and concerns about the heterogeneity of the study population (FDA, 2007).
warnings on the drug label, without restrictions on marketing. … The basis of these recommendations is the uncertainty about the existence of the cardiovascular ischemic safety risk.
Despite the lack of clarity in the data, I believe it is most prudent, given the current uncertainty about the safety risk, to restrict access to the product, and ensure that patients and prescribers are fully informed of the evidence of risk, until and unless more information is obtained. … The evidence pointing to a … risk with [the drug] is not robust or consistent. … Nevertheless, there are multiple signals of concern, from varied sources of data, without reliable evidence that refutes them. Additionally, evidence available to date … does not reveal a signal of … risk with the other … drug available on the US market. … Therefore, based on this safety information, it is necessary to restrict access … until more substantial evidence of its safety becomes available. (Woodcock, 2010)
FDA qualitatively discussed the scientific uncertainties in an unusually transparent manner not only by publishing on its website the CDER director’s final decision (in this case, regarding safety information generated after approval), but also by openly discussing the different opinions of agency scientists at a science advisory meeting and by posting nine scientific review documents on the agency website (FDA, 2010b). As detailed in a recent IOM report (IOM, 2012a) those memoranda highlighted scientific disagreements among agency scientists. For example, agency scientists disagreed about how mechanistic data should affect the consideration of different data from humans, whether missing data points were appropriately handled in studies of Avandia, what endpoints should have been used in studies, and how generalizable data from studies are. Those documents present a detailed analysis of the uncertainties in data and the differing opinions that agency scientists had about the policy options available to the agency decision makers. FDA decision makers, in the absence of quantitative analyses of those uncertainties beyond the presentation of different scientific opinions, made the regulatory decision.
Once FDA approves a drug, other stakeholders have one or more decisions to make. For example, payers decide whether they will pay or reimburse for a given drug; health care providers, in discussion with their patients, decide which drug should be prescribed; and the patient decides whether the side effects of a drug are worth the benefit. Each of these decisions includes uncertainties, some of which can be better understood or reduced through more research, but the decisions often need to be made before that research is available. For each of those choices, in a process that can be either formal or informal, organizations or individuals assess the benefits and the risks, identify uncertainties, and make their decisions.
Decisions in the Face of Uncertainty: Lessons Learned
This example illustrates the sorts of pervasive uncertainties that a decision maker can face and the different ways that government scientists can interpret the same sets of data. (See IOM, 2012a, for more details.) Ultimately, the decision maker must chose from a range of options. The Avandia decision illustrates how an agency can transparently represent those different interpretations and explain what mattered for the final decision. Despite the many uncertainties, a decision was made and the rationale explained. The decision demonstrates how stakeholders’ input, such as the need to keep a drug available on the market, can be considered in a regulatory decision. It also highlights the limitations of preclinical data, and the importance of reevaluating and revisiting decisions as new information emerges.
Decisions about vaccination of the public, such as whom to vaccinate and when, are often associated with uncertainty. Those decisions might have to be made for relatively low-probability, high-consequence events, such as vaccinations to avoid a pandemic, as well as for relatively high-probability events with potentially high consequences, such as vaccinations to prevent infection with the human papillomavirus (HPV),7 the causative agent for cervical cancer. This section discusses some of the considerations and lessons learned from decisions about influenza and HPV vaccination programs.
Pandemic Prevention and Vaccinations
Decisions related to low-probability, high-consequence events are particularly challenging and fraught with uncertainty. The preparations needed for pandemic influenza or bioterrorist events are examples of such a decision problem. Pandemic influenza occurs when the influenza virus, which circulates around the globe constantly, undergoes a major change in its antigenic presentation (antigenic shift), leaving many in the population with
7 The overall prevalence of HPV was estimated at 42.5 percent in 14- to 59-year-old females in the United States using data from the 2003 through 2006 National Health and Nutrition Examination Surveys (NHANES) (Hariri et al., 2011). The estimated prevalence of HPV types 6, 11, 16, and 18 was 8.8 percent (95 percent confidence interval [CI], 7.8–10.0 percent). Types 16 and 18 are estimated to be responsible for 70 percent of cervical cancers (Dunne et al., 2011).
no immunity.8 Influenza pandemics have occurred three times in the 20th century and in the first decade of the 21st century. Judging when a pandemic will occur and how to prepare for it is great challenge in the face of tremendous uncertainty. Because of the time period required to develop and administer a vaccine, decisions about how to respond to emerging information occur with great uncertainty about various issues, such as whether the strain identified is circulating widely, whether it is a strain that would incur serious morbidity and mortality, whether a vaccine could be developed in time to mitigate the virus, whether the vaccine would be safe, and whether citizens would get vaccinated.
The government’s swine flu program that began in March 1976 and ended in March 1977 illustrates some of the problems of making decisions when there is great uncertainty. The decision process that went into the pandemic preparedness and response for the 1976 swine influenza epidemic has widely been viewed as a failure. Neustadt and Fineberg (1978) analyzed that decision-making process with the goal of informing future decisions that might have to be made under similar levels of uncertainty. Some of those lessons are also applicable to EPA.
In 1976 a previously unknown swine flu virus was identified from four cases of influenza, including one fatal case, at a training center for Army recruits (Neustadt and Fineberg, 1978). Concerns were raised because of the possibility of human-to-human transmission; because of a lack of built-up antibodies from previous, similar infections in anyone under 50 years of age; and because of memories of the very virulent strain of swine flu that led to the pandemic of 1918 which killed 500,000 mostly young and able-bodied people in the United States alone. In light of those concerns, a decision was made to implement a mass vaccination plan for all Americans. The pandemic never materialized, vaccine production was delayed, the population did not rush to get vaccinated, and after a rare neurologic side effect—Guillain-Barré syndrome—was identified, the vaccination program was ended.
Uncertainties surrounded a number of issues: the infectivity of the influenza virus, the appropriate dose for children, the production schedules of the vaccine, and the ability to carry out—and the public’s willingness to participate in—mass vaccinations. Those uncertainties were considered in the decision-making process and were discussed by policy makers, vaccine
8 Antigenic drift refers to the small changes in antigenic presentation that the virus undergoes constantly. Exposed people have some immunity from previous years’ exposures. Antigenic shift is a much more serious change in the influenza virus and can lead to a pandemic.
manufacturers, and scientific, medical, and public health experts; they were not, however, thoroughly analyzed and adequately considered. The broad lessons that Neustadt and Fineberg (1978) gleaned from their evaluation of the immunization program were (1) the need to build in points at which a program will be reviewed, that is, to take an iterative approach to implementation if possible; (2) the need to consider when a plan is feasible; (3) the need to be prepared to deal with questions from the media and the public, but not to let the possibility of questions dictate the decision; (4) the importance of an agency’s credibility; and (5) the need to think twice about medical knowledge, including the uncertainties in that knowledge. That last point offers perhaps the most important lesson from the swine flu experience: the importance of framing the decision in terms of what data are available, what uncertainties there are in the data, and, importantly, what new facts or evidence, if at hand, would lead to a different decision and when such different decisions would be made (Neustadt and Fineberg, 1978).
Given the many uncertainties about vaccine availability, effectiveness, and utilization, in recent years much of the discussion about recent pandemic preparedness has involved the issue of public values regarding who should be vaccinated in the early days of the pandemic when vaccine availability is limited. Engaging the public to explore their values regarding vaccine distribution has revealed important differences between what scientists thought and what the general public thought about who should be vaccinated first. For example, in a pilot project designed to elicit input from “approximately 300 citizens and stakeholders in different parts of the United States” there was strong agreement among the participants that “‘Assuring the Functioning of Society’ should be the first goal and ‘Reducing Individual Deaths and Hospitalizations Due to Influenza’ should be the second priority goal. … There was little support for other goals to vaccinate young people first, or to use a lottery system or a first come first served approach as top priorities” (Bernier and Marcuse, 2005, p. 7).
The uncertainty about vaccine availability has also led to intensive efforts to model the effects of community containment (such as school closures, the discouragement of public gatherings, and such hygienic measures as the use of respiratory masks and advice about hand washing) on the spread of the pandemic. The evidence base for the use of those measures has been quite meager (IOM, 2006), and has, in part, depended on analysis of historical data from the 1918–1919 pandemic. Despite the many uncertainties, however, the federal government needed to provide advice to state governments and to the public, so a national plan was developed, and
when a new pandemic strain emerged in 2009, the plan was implemented. A recent IOM report (2012b) describes the initial version of a modeling tool to help prioritize the development of vaccines. The modeling tool could be used to identify the likely effects of uncertainty in different factors that play a part in determining the priority of developing a vaccine.
The Human Papillomavirus Vaccine
During the past 20 years researchers have identified HPV as the causative agent in cervical cancer and have characterized the virus and its components (see IOM, 2010b, for an overview), and pharmaceutical companies have developed, tested, and marketed a vaccination against the virus (Baer et al., 2002; Carter et al., 2000; Harper et al., 2004; Harro et al., 2001; Petter et al., 2000). In 2006 the FDA approved the marketing of Gardasil, the first vaccine against HPV, for use in 9- to 26-year-old females. The vaccine and vaccination programs are expensive, however, and CDC’s Advisory Committee on Immunization Practices (ACIP) was faced with taking costs into consideration and making a recommendation about who should be vaccinated and when they should be vaccinated.9
Researchers have evaluated the costs associated with different vaccination programs in preventing cervical cancer and have outlined the uncertainties in those costs. The annual cost from HPV in the United States was estimated to be $4 billion dollars or more, including the costs associated with the management of genital warts, costs associated with cervical cancer, and costs associated with routine cervical cancer screening and the follow-up of abnormal Pap smears (Markowitz et al., 2007). Models have been used to predict decreases in Pap test abnormalities, cervical cancer precursor lesions, and cervical cancer rates in order to estimate the benefits of vaccination. The ACIP considered four published studies on cost effectiveness (Elbasha et al., 2007; Goldie et al., 2004; Sanders and Taira, 2003; Taira et al., 2004). Markowitz et al. (2007), Sanders and Taira (2003), and Goldie et al. (2004) estimated cost per quality-adjusted life-year (QALY) using Markov models, and Taira et al. (2004) and Elbasha et al. (2007) “applied dynamic transmission models to incorporate the benefits of herd immunity in estimating the cost effectiveness of HPV vaccination” (Markowitz et al., 2007, p. 15). Uncertainties were considered and incorporated in those cost-effectiveness evaluations. Goldie et al. (2004) used a range of inputs for the rates of incidence and the clearance of HPV infection, the natural history of cervical intraepithelial neoplasia (CIN), the natural history of
9 In addition to the estimated health benefit and risk estimates and the economic factors related to the HPV vaccine, there were a number of social and political issues related to whether vaccinations should be mandatory. The committee does not discuss those issues here.
invasive cervical cancer, vaccine efficacy, age at vaccination and vaccine coverage, the specificity and sensitivity of cervical cancer screening tests, vaccination costs (including the costs of the patients’ time), the costs of cervical cancer screening and treatment (including the costs of the patients’ time), and costs associated with health-related quality of life. Sanders and Taira (2003) conducted sensitivity analyses for various factors, including the optimal vaccination age, universal vaccination of adolescent girls versus targeting high-risk girls, the probabilities of occurrence and progression of HPV, squamous cell lesions and cervical cancer, the probability of death, and the costs and quality of life with various health states. Elbasha et al. (2007) evaluated the cost effectiveness of different HPV vaccination strategies, including (1) routine HPV vaccination of females by 12 years of age, (2) routine vaccination of females and males by 12 years of age, (3) routine vaccination of females by 12 years of age in combination with catch-up vaccinations for females between 12 and 24 years of age who were not previously vaccinated, and (4) routine vaccination of females and males by 12 years of age in combination with catch-up vaccinations for females and males between 12 and 24 years of age who were not previously vaccinated. The authors used a dynamic model that included demographic and epidemiologic components. Groups were formed on the basis of age and the extent of sexual activity (low, medium, or high sexual activity). The epidemiologic component “simulates HPV transmission and the occurrence of CIN, cervical cancer, and external genital warts” (Elbasha et al., 2007, p. 29). Vaccine characteristics included the degree of protection and the duration of protection. Taira et al. (2004) considered ranges of HPV prevalence, infection rates, age groups, sexual activity, and sex in order to estimate the cost effectiveness of different vaccination programs, including programs that would and would not include males.
In 2007, after considering those analyses and other factors, the ACIP recommended a three-dose vaccination series for females 11 to 12 years of age and, as a catch-up, vaccination for females 13 to 26 years of age who had not previously been vaccinated (Markowitz et al., 2007).
Since that decision, the ACIP has made three additional or updated recommendations related to HPV vaccination. In 2010 the committee developed a recommendation for Cervarix, a second HPV vaccine (CDC, 2010a). At the same time, the ACIP issued guidance stating that males aged 9 to 36 may be given Gardasil (CDC, 2010b). In 2011 the ACIP replaced that 2009 recommendation for males, recommending instead the use of Gardasil in males aged 11 or 12 years and a catch-up vaccination in males aged 13 through 21 who had not been vaccinated (or who had not completed the three-dose series) (CDC, 2011b). It also stated that males aged 22 to 26 years may be vaccinated.
In October 2010 the ACIP adopted a new framework for developing its recommendations, the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach (Ahmed et al., 2011). The key factors in that approach include “the balance of benefits and harms, type of evidence, values and preferences of the people affected, and health economic analyses” (Ahmed et al., 2011, p. 9171). In the GRADE approach, ACIP includes tables that summarize the evidence on which it basis its decision. The tables include a description of the strength and limitations of the body of evidence. Those tables are intended to “enhance the ACIP’s decision-making process by making it more transparent, consistent and systematic” (Ahmed et al., 2011, p. 9171).
Decisions in the Face of Uncertainty: Lessons Learned
Vaccinations illustrate a number of challenges experienced by those making decisions in the face of uncertainty. Viruses are “moving targets” that evolve and change, often requiring major policy decisions to be made in the absence of a full knowledge of the characteristics of future viruses. Faced with a possible pandemic in 1976, the government embarked on a vaccination campaign that has often been criticized. The lessons learned from that decision include the need for deliberative problem formulation, the importance of having different options, and the value of considering the potential effectiveness of those options in advance (for example, shutting down schools to control the spread of influenza). Other lessons from this case study are the importance of implementing an iterative approach in the presence of uncertainty, considering the feasibility of a plan, the importance of preparing for media communications, the importance of credibility, and the value of questioning knowledge and considering uncertainties in knowledge (Neustadt and Fineberg, 1978).
CDC’s ACIP was faced with making recommendations about whom and when to vaccinate with a newly available, but expensive, vaccine against HPV that could prevent cervical cancer. A number of very detailed cost-effectiveness analyses were available that provided the ACIP with estimates of the benefits and costs associated with a number of different vaccination programs, assuming different levels of benefit. Those extensive analyses—which included analyses of a number of uncertainties—allowed ACIP to consider the effects of different scenarios and the range of benefits and costs under those scenarios using different estimates of vaccine effectiveness. ACIP’s decision would potentially affect the medical care of the entire U.S. adolescent population. Having the detailed analyses showing ranges of cost effectiveness provided the ACIP with the evidence it needed to make a decision.
At the level of individual patients, physicians often make decisions or recommendations about preventive or therapeutic interventions with very uncertain information. They do so based on experiences from their training, advice from their professional societies, and information from government agencies and relevant private industries and by keeping current with the published medical literature and knowing their patients. In contrast to the population-based decisions that EPA makes, the decisions that physicians make or the advice they give is for one patient at a time, and the evidentiary basis for that advice typically comes from studies in populations and expert systematic reviews of the published literature, which may have different applicability to different patients. There is often conflicting evidence about the safety and effectiveness of the interventions. For example, although the U.S. Preventive Services Task Force (USPSTF) does not conduct quantitative uncertainty analysis when making its recommendations, the language of its recommendations qualitatively describes the level of uncertainty in the evidence (USPSTF, 2008). The USPSTF conclusions have significant policy implications. The role of certainty is explicitly stated in the conclusions and in the communication of the scientific evidence, providing an example of how a qualitative assessment and the description of uncertainties in an evidence base can provide important information to a decision maker. For example, the most strongly worded conclusion—for preventive interventions receiving a grade of A—reads, “The USPSTF recommends the intervention. There is high certainty that the net benefit is substantial.” The Grade B conclusion reads, “The USPSTF recommends the service. There is high certainty that the net benefit is moderate or there is moderate certainty that the net benefit is moderate to substantial” (USPSTF, 2008). Unfortunately, for many interventions there is no clinical guidance given because the evidence is confusing or inadequate.
Decisions in the Face of Uncertainty: Lessons Learned
The USPSTF guidelines demonstrate how, for some types of decisions, simple, qualitative descriptions of the uncertainty can be helpful for the decisions.
• Some agencies, such as FSIS and FDA, have conducted quantitative uncertainty analyses on public health estimates. They use tools and techniques, such as probabilistic analysis and Monte Carlo simulations, that are similar to those that EPA has used.
• A phased or iterative approach to regulations, as occurred with the implementation of smoking bans, can allow for the collection of data, including data for economic analyses, that can decrease uncertainty.
• Regulations that are likely to be met with opposition, such as smoking bans and mandatory vaccinations, are well served to engage stakeholders in problem formulation and uncertainty analysis early to ensure that uncertainties are well understood and to ensure that stakeholders’ information needs are met.
• Uncertainty analyses, such as those conducted in the assessment of L. monocytogenes, can characterize heterogeneity and its consequences and provide decision makers with the information to decide upon regulations and risk-mitigation options that target the public health goal in the most effective manner.
• Well-planned, decision-driven modeling of uncertainty, such as was conducted in the BSE risk assessment, can provide information about the likelihood of different regulatory options decreasing the risks to the public.
• Decisions must sometimes be made quickly in the face of large or deep uncertainty, such as was the case with the threatened pandemic influenza in 1976 and with melamine in the food supply in 2007 and 2008. Under such circumstances, probabilistic models are not typically available to help with decision making. The analysis of scenarios can be useful under such circumstances, and iterative management approaches can avoid mistakes that are costly either in terms of resources or to the reputation of and trust in the agency.
• Detailed economic analyses that outline the ranges of likely cost-effectiveness of different scenarios, such as those conducted for HPV vaccinations, can provide the evidence needed to make a decision.
• As demonstrated by FDA’s activities around its decision on Avandia, making public the uncertainty from scientific disagreements—and even the disagreements among agency scientists—can increase the transparency and the public understanding of a decision. That decision also highlights how quantitative uncertainty analyses are not always needed to make an informed decision.
• All of these examples provide some characterization of uncertainty—some quantitative, some qualitative, some using safety factors—some in public health factors only and some in costs and economic impacts as well. These examples show the wide range of approaches that can be taken and provide some indication of the situations in which each approach may be appropriate. These examples also
show how uncertainty analysis is typically focused on public health impact estimates and demonstrate the few examples of assessing uncertainty in other factors. The tools and techniques used are typically those that EPA is already using.
Ahmed, F., J. L. Temte, D. Campos-Outcalt, and H. J. Schünemann. 2011. Methods for developing evidence-based recommendations by the Advisory Committee on Immunization Practices (ACIP) of the U.S. Centers for Disease Control and Prevention (CDC). Vaccine 29(49):9171–9176.
Anderson, J. E., and S. Sansom. 2006. HIV testing among US women during prenatal care: Findings from the 2002 National Survey of Family Growth. Maternal and Child Health Journal 10(5):413–417.
ANRF (American Nonsmokers Rights Foundation). 2012. Overview list—How many smoke-free laws? http://www.no-smoke.org/pdf/mediaordlist.pdf (accessed July 10, 2012).
Baer, A., N. B. Kiviat, S. Kulasingam, C. Mao, J. Kuypers, and L. A. Koutsky. 2002. Liquid-based papanicolaou smears without a transformation zone component: Should clinicians worry? Obstetrics and Gynecology 99(6):1053–1059.
Barone-Adesi, F., L. Vizzini, F. Merletti, and L. Richiardi. 2006. Short-term effects of Italian smoking regulation on rates of hospital admission for acute myocardial infarction. European Heart Journal 27(20):2468–2472.
Bartecchi, C., R. N. Alsever, C. Nevin-Woods, W. M. Thomas, R. O. Estacio, B. B. Bartelson, and M. J. Krantz. 2006. Reduction in the incidence of acute myocardial infarction associated with a citywide smoking ordinance. Circulation 114(14):1490–1496.
Bernier, R. H., and E. K. Marcuse. 2005. Citizen voices on pandemic flu choices: A report of the Public Engagement Pilot Project on Pandemic Influenza (PEPPI). http://ppc.unl.edu/publications/documents/PEPPPI_FINALREPORT_DEC_2005.pdf (accessed January 3, 2013).
Bero, L. A., T. Montini, K. Bryan-Jones, and C. Mangurian. 2001. Science in regulatory policy making: Case studies in the development of workplace smoking restrictions. Tobacco Control 10(4):329–336.
Bonanni, P. 1999. Demographic impact of vaccination: A review. Vaccine 17(Suppl 3):S120–S125.
Borland, R., H. H. Yong, M. Siahpush, A. Hyland, S. Campbell, G. Hastings, K. M. Cummings, and G. T. Fong. 2006. Support for and reported compliance with smoke-free restaurants and bars by smokers in four countries: Findings from the International Tobacco Control (ITC) Four Country Survey. Tobacco Control 15(Suppl 3):34–41.
Bryan-Jones, K., and L. A. Bero. 2003. Tobacco industry efforts to defeat the Occupational Safety and Health Administration indoor air quality rule. American Journal of Public Health 93(4):585.
Cal EPA (California Environmental Protection Agency). 2005. Proposed identification of environmental tobacco smoke as a toxic air contaminant, part b: Health effects. California Environmental Protection Agency.
Carter, J. J., L. A. Koutsky, J. P. Hughes, S. K. Lee, J. Kuypers, N. Kiviat, and D. A. Galloway. 2000. Comparison of human papillomavirus types 16, 18, and 6 capsid antibody responses following incident infection. Journal of Infectious Diseases 181(6):1911–1919.
CDC (U.S. Centers for Disease Control and Prevention). 2010a. FDA licensure of bivalent human papillomavirus vaccine (HPV2, cervarix) for use in females and updated HPV vaccination recommendations from the Advisory Committee on Immunization Practices (ACIP). Morbidity and Mortality Weekly Report 59(20):626–629.
———. 2010b. FDA licensure of quadrivalent human papillomavirus vaccine (HPV4, gardasil) for use in males and guidance from the Advisory Committee on Immunization Practices (ACIP). Morbidity and Mortality Weekly Report 59(20):630–632.
———. 2011a. Multistate outbreak of listeriosis linked to whole cantaloupes from Jensen Farms, Colorado. http://www.cdc.gov/listeria/outbreaks/cantaloupes-jensen-farms/120811/index.html (accessed May 21, 2012).
———. 2011b. Recommendations on the use of quadrivalent human papillomavirus vaccine in males—Advisory Committee on Immunization Practices (ACIP), 2011. Morbidity and Mortality Weekly Report 60(50):1697–1728.
Cesaroni, G., F. Forastiere, N. Agabiti, P. Valente, P. Zuccaro, and C. A. Perucci. 2008. Effect of the Italian smoking ban on population rates of acute coronary events. Circulation 117(9):1183–1188.
Cohen, J. T. 2006. Harvard risk assessment of bovine spongiform encephalopathy update. Phase IA: Supplemental simulation results. Harvard Center for Risk Analysis. http://www.fsis.usda.gov/PDF/BSE_Risk_Assess_Report_2006.pdf (accessed January 2, 2013). Cohen, J. T., and G. M. Gray. 2005. Harvard risk assessment of bovine spongiform encephalopathy update. Harvard Center for Risk Analysis. http://www.fsis.usda.gov/PDF/BSE_Risk_Assess_Report_2005.pdf (accessed January 2, 2013).
Cohen, J. T., K. Duggar, G. M. Gray, and S. Kreindel. 2003a. Evaluation of the potential for bovine spongiform encephalopathy in the United States: Report to the U.S. Department of Agriculture (revised October 2003). Harvard Center for Risk Analysis. http://www.agcenter.com/mad%20cow/Harvard%20Study%20on%20madcow.pdf (accessed January 2, 2013).
Cohen, J. T., K. Duggar, G. M. Gray, S. Kreindel, H. Gubara, T. HabteMariam, D. Oryang, and B. Tameru. 2003b. A simulation model for evaluating the potential for spread of bovine spongiform encephalopathy to animals or to people. In Prions and mad cow disease, edited by B. K. Nunnally and I. S. Krull. New York: Marcel Dekker. Pp. 53–111.
Darnton-Hill, I., and R. Nalubola. 2002. Fortification strategies to meet micronutrient needs: Successes and failures. Proceedings of the Nutrition Society 61(02):231–241.
Dunne, E. F., M. Sternberg, L. E. Markowitz, G. McQuillan, D. Swan, S. Patel, and E. R. Unger. 2011. Human papillomavirus (HPV) 6, 11, 16, and 18 prevalence among females in the United States—National Health and Nutrition Examination Survey, 2003–2006: Opportunity to measure HPV vaccine impact? Journal of Infectious Disease 204(4):562–565.
Elbasha, E. H., E. J. Dasbach, and R. P. Insinga. 2007. Model for assessing human papillomavirus vaccination strategies. Emerging Infectious Diseases 13(1):28. http://wwwnc.cdc.gov/eid/article/13/1/06-0438.htm (accessed January 2, 2013).
EPA (U.S. Environmental Protection Agency). 1992. Respiratory health effects of passive smoking: Lung cancer and other disorders. EPA/600/6-90/006F. Washington, DC: Environmental Protection Agency.
FDA (U.S. Food and Drug Administration). 1998. Guidance for industry: Providing clinical evidence of effectiveness for human drugs and biological products. Washington, DC: Food and Drug Administration. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm078749.pdf (accessed January 3, 2013).
———. 2003. Reducing the risk of Listeria monocytogenes. FDA/CDC 2003 update of the Listeria Action Plan. http://www.foodsafety.gov/~dms/lmr2plan.html#update (accessed March 3, 2012).
———. 2005. Guidance for industry: Premarketing risk assessment. http://www.fda.gov/downloads/RegulatoryInformation/Guidances/ucm126958.pdf%20 (accessed January 3, 2013).
———. 2007. Summary minutes of the Joint Meeting of the Endocrinologic and Metabolic Drugs Advisory Committee and the Drug Safety and Risk Management Advisory Committee held on July 30, 2007. http://www.fda.gov/ohrms/dockets/ac/07/minutes/2007-4308m1-final.pdf (accessed January 2, 2013).
———. 2008a. Compliance policy guide—guidance for FDA staff, Sec. 555.320: Listeria monocytogenes draft guidance. Washington, DC: U.S. Food and Drug Administration. http://www.fda.gov/OHRMS/DOCKETS/98fr/FDA-2008-D-0058-GDL.pdf (acessed January 3, 2013).
———. 2008b. “Dear colleague” letter to the United States food manufacturing industry, regarding melamine. http://www.fda.gov/Food/FoodSafety/FoodContaminantsAdulteration/ChemicalContaminants/Melamine/ucm164514.htm (accessed January 3, 2013).
———. 2008c. Guidance for industry: Guide to minimize microbial food safety hazards of fresh-cut fruits and vegetables. http://www.fda.gov/food/guidancecomplianceregulatoryinformation/guidancedocuments/produceandplanproducts/ucm064458.htm (accessed January 2, 2013).
———. 2008d. Interim safety and risk assessment of melamine and its analogues in food for humans [a]. http://www.fda.gov/Food/FoodSafety/FoodContaminantsAdulteration/ChemicalContaminants/Melamine/ucm164522.htm (accessed January 3, 2013).
———. 2008e. Update: Interim safety and risk assessment of melamine and its analogues in food for humans. http://www.fda.gov/Food/FoodSafety/FoodContaminantsAdulteration/ChemicalContaminants/Melamine/ucm164520.htm (accessed January 3, 2013).
———. 2010a. FDA drug safety communication: Ongoing review of Avandia (rosiglitazone) and cardiovascular safety. http://www.fda.gov/Drugs/DrugSafety/PostmarketDrugSafetyInformationforPatientsandProviders/ucm201418.htm (accessed January 2, 2013).
———. 2010b. FDA significantly restricts access to the diabetes drug Avandia. http://www.fda.gov/Drugs/DrugSafety/PostmarketDrugSafetyInformationforPatientsandProviders/ucm226956.htm (accessed January 2, 2013).
———. 2010c. FDA significantly restricts access to the diabetes drug Avandia: Makes regulatory decisions on record and tide trials. http://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm226975.htm (accessed January 3, 2013).
———. 2011a. Raw Milk Misconceptions and the Danger of Raw Milk Consumption. http://www.fda.gov/Food/FoodSafety/Product-SpecificInformation/MilkSafety/ConsumerInformationAboutMilkSafety/ucm247991.htm (accessed January 17, 2013).
———. 2011b. Advisory committees: Critical to the FDA’s product review process. http://www.fda.gov/Drugs/ResourcesForYou/Consumers/ucm143538.htm (accessed January 2, 2013).
———. 2011c. Environmental assessment: Factors potentially contributing to the contamination of fresh whole cantaloupe implicated in a multi-state outbreak of listeriosis. http://www.fda.gov/Food/FoodSafety/FoodborneIllness/ucm276247.htm (accessed January 2, 2013).
———. 2012a. Alert: The basics. http://www.fda.gov/Food/FoodDefense/ToolsResources/ucm296009.htm (accessed January 3, 2013).
———. 2012b. Guidance drug safety information—FDA’s communication to the public. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM295217.pdf (accessed January 3, 2013).
———. 2012c. Guidance for industry and Food and Drug Administration staff. Factors to consider when making benefit-risk determinations in medical device premarketing approval and de novo classifications. http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM296379.pdf (accessed January 3, 2013).
FDA and FSIS (Food Safety and Inspection Service). 2003a. Interpretive summary: Quantitative assessment of relative risk to public health from foodborne Listeria monocytogenes among selected categories of ready-to-eat foods. http://www.fda.gov/downloads/Food/ScienceResearch/ResearchAreas/RiskAssessmentSafetyAssessment/UCM197329.pdf (accessed January 3, 2013).
———. 2003b. Quantitative assessment of relative risk to public health from foodborne Listeria monocytogenes among selected categories of ready-to-eat foods. http://www.fda.gov/downloads/Food/ScienceResearch/ResearchAreas/RiskAssessmentSafetyAssessment/UCM197330.pdf (accessed January 3, 2013).
———. 2007. Interim Melamine and Analogues Safety/Risk Assessment. http://www.fda.gov/Food/FoodSafety/FoodContaminantsAdulteration/ChemicalContaminants/Melamine/ucm164658.htm (accessed January 3, 2013).
Fischhoff, B., N. T. Brewer, and J. S. Downs (Eds.). 2011. Communicating risks and benefits: An evidence-based user’s guide. Washington, DC: Food and Drug Administration.
FSIS (Food Safety and Inspection Service). 2003. Control of Listeria monocytogenes in ready-to-eat meat and poultry products: Final rule. Federal Register, June 6, 2003 (Volume 68, Number 109. http://www.fsis.usda.gov/oppde/rdad/frpubs/97-013f.htm (accessed January 3, 2013).
———. 2007. Disposition of hogs and chickens from farms identified as having received pet food scraps contaminated with melamine and melamine-related compounds and offered for slaughter. FSIS, Docket No. 2007-0018.
FSIS and USDA (U.S. Department of Agriculture). 2004. Preliminary analysis of interim final rules and an interpretive rule to prevent the BSE agent from entering the U.S. food supply. http://www.fsis.usda.gov/oppde/rdad/frpubs/03-025N/BSE_Analysis.pdf (accessed January 2, 2013).
———. 2007. Economic analysis: Final regulatory impact analysis. Final rule. FSIS Docket No. 03-025F.
Glantz, S. A., and A. Charlesworth. 1999. Tourism and hotel revenues before and after passage of smoke-free restaurant ordinances. Journal of the American Medical Association 281(20):1911–1918.
Glantz, S. A., and L. R. Smith. 1994. The effect of ordinances requiring smoke-free restaurants on restaurant sales. American Journal of Public Health 84(7):1081–1085.
———. 1997. The effect of ordinances requiring smoke-free restaurants and bars on revenues: A follow-up. American Journal of Public Health 87(10):1687–1693.
Goldie, S. J., M. Kohli, D. Grima, M. C. Weinstein, T. C. Wright, F. X. Bosch, and E. Franco. 2004. Projected clinical benefits and cost-effectiveness of a human papillomavirus 16/18 vaccine. Journal of the National Cancer Institute 96(8):604–615.
GRADE Working Group. 2012a. Organizations that have endorsed or that are using GRADE. http://www.gradeworkinggroup.org/society/index.htm (accessed June 13, 2012).
———. 2012b. Welcome. http://www.gradeworkinggroup.org/index.htm (accessed June 23, 2012).
Hackshaw, A. K., M. R. Law, and N. J. Wald. 1997. The accumulated evidence on lung cancer and environmental tobacco smoke. British Medical Journal 315(7114):980–988.
Hariri, S., E. R. Unger, M. Sternberg, E. F. Dunne, D. Swan, S. Patel, and L. E. Markowitz. 2011. Prevalence of genital human papillomavirus among females in the United States, the National Health and Nutrition Examination Survey, 2003–2006. Journal of Infectious Diseases 204(4):566–573.
Harper, D. M., E. L. Franco, C. Wheeler, D. G. Ferris, D. Jenkins, A. Schuind, T. Zahaf, B. Innis, P. Naud, N. S. De Carvalho, C. M. Roteli-Martins, J. Teixeira, M. M. Blatter, A. P. Korn, W. Quint, and G. Dubin. 2004. Efficacy of a bivalent L1 virus-like particle vaccine in prevention of infection with human papillomavirus types 16 and 18 in young women: A randomised controlled trial. The Lancet 364(9447):1757–1765.
Harro, C. D., Y.-Y. S. Pang, R. B. S. Roden, A. Hildesheim, Z. Wang, M. J. Reynolds, T. C. Mast, R. Robinson, B. R. Murphy, R. A. Karron, J. Dillner, J. T. Schiller, and D. R. Lowy. 2001. Safety and immunogenicity trial in adult volunteers of a human papillomavirus 16 L1 virus-like particle vaccine. Journal of the National Cancer Institute 93(4):284–292.
HHS (U.S. Department of Health and Human Services). 1986. The health consequences of involuntary smoking: A report of the Surgeon General. Washington, DC: U.S. Department of Health and Human Services.
HHS, CDC, Coordinating Center for Health Promotion, National Center for Chronic Disease Prevention and Health Promotion, and Office on Smoking and Health. 2006. The health consequences of involuntary exposure to tobacco smoke: A report of the surgeon general. Washington, DC: U.S. Department of Health and Human Services.
Hyland, A., K. M. Cummings, and E. Nauenberg. 1999. Analysis of taxable sales receipts: Was New York City’s Smoke-Free Air Act bad for restaurant business? Journal of Public Health Management and Practice 5(1):14–21.
IOM (Institute of Medicine). 2004. Advancing prion science: Guidance for the National Prion Research Program. Edited by R. Erdtmann and L. B. Sivitz. Washington, DC: The National Academies Press.
———. 2006. Modeling community containment for pandemic influenza: A letter report. Washington, DC: The National Academies Press.
———. 2010a. Secondhand smoke exposure and cardiovascular effects: Making sense of the evidence. Washington, DC: The National Academies Press.
———. 2010b. Women’s health research: Progress, pitfalls, and promise. Washington, DC: The National Academies Press.
———. 2012a. Ethical and scientific issues in studying the safety of approved drugs. Washington, DC: The National Academies Press.
———. 2012b. Ranking vaccines: A prioritization framework: Phase I: Demonstration of concept and a software blueprint. Edited by G. Madhavan, K. Sangha, C. Phelps, D. Fryback, T. Lieu, R. M. Martinez, and L. King. Washington, DC: The National Academies Press.
Juster, H. R., B. R. Loomis, T. M. Hinman, M. C. Farrelly, A. Hyland, U. E. Bauer, and G. S. Birkhead. 2007. Declines in hospital admissions for acute myocardial infarction in New York State after implementation of a comprehensive smoking ban. American Journal of Public Health 97(11):2035–2039.
Kelly, B. C., J. D. Weiser, and J. T. Parsons. 2009. Smoking and attitudes on smoke-free air laws among club-going young adults. Social Work and Public Health 24(5):446–453.
Khuder, S. A., S. Milz, T. Jordan, J. Price, K. Silvestri, and P. Butler. 2007. The impact of a smoking ban on hospital admissions for coronary heart disease. Preventive Medicine 45(1):3–8.
Law, M. R., J. Morris, and N. J. Wald. 1997. Environmental tobacco smoke exposure and ischaemic heart disease: An evaluation of the evidence. Bristish Medical Journal 315(7114):973–980.
Lemstra, M., C. Neudorf, and J. Opondo. 2008. Implications of a public smoking ban. Canadian Journal of Public Health 99(1):62–65.
Mangurian, C. V., and L. A. Bero. 2000. Lessons learned from the tobacco industry’s efforts to prevent the passage of a workplace smoking regulation. American Journal of Public Health 90(12):1926–1930.
Markowitz, L. E., E. Dunne, M. Saraiya, H. Lawson, H. Chesson, and E. Unger. 2007. Quadrivalent human papillomavirus vaccine. Morbidity and Mortality Weekly Report 56(RR02):1–24.
Miller, C., M. Wakefield, S. Kriven, and A. Hyland. 2002. Evaluation of smoke-free dining in South Australia: Support and compliance among the community and restauranteurs. Australia and New Zealand Journal of Public Health 26(1):38–44.
Muggli, M. E., R. D. Hurt, and D. D. Blanke. 2003. Science for hire: A tobacco industry strategy to influence public opinion on secondhand smoke. Nicotine and Tobacco Research 5(3):303–314.
Neustadt, R. E., and H. V. Fineberg. 1978. The swine flu affair: Decision-making on a slippery disease. Washington, DC: National Academy Press.
Nissen, S. E., and K. Wolski. 2007. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. New England Journal of Medicine 356(24):2457–2471.
NRC (Nuclear Regulatory Commission). 2012. Risk assessment in regulation. http://www.nrc.gov/about-nrc/regulatory/risk-informed.html (accessed January 3, 2013).
NTP (National Toxicology Program). 2011. Report on carcinogens, twelfth edition. Washington, DC: U.S. Department of Health and Human Services, Public Health Service, National Toxicology Program. http://ntp.niehs.nih.gov/ntp/roc/twelfth/roc12.pdf (accessed January 2, 2013).
Ong, E. K., and S. A. Glantz. 2000. Tobacco industry efforts subverting International Agency for Research on Cancer’s second-hand smoke study. Lancet 355(9211):1253–1259.
Oreskes, N., and E. M. Conway. 2010. Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. New York: Bloomberg Press.
Pell, J. P., S. Haw, S. Cobbe, D. E. Newby, A. C. H. Pell, C. Fischbacher, A. McConnachie, S. Pringle, D. Murdoch, and F. Dunn. 2008. Smoke-free legislation and hospitalizations for acute coronary syndrome. New England Journal of Medicine 359(5):482–491.
Petter, A., K. Heim, M. Guger, A. Ciresa–König, N. Christensen, M. Sarcletti, U. Wieland, H. Pfister, R. Zangerle, and R. Höpfl. 2000. Specific serum IGG, IGM and IGA antibodies to human papillomavirus types 6, 11, 16, 18 and 31 virus-like particles in human immunodeficiency virus-seropositive women. Journal of General Virology 81(3):701–708.
Pursell, L., S. Allwright, D. O’Donovan, G. Paul, A. Kelly, B. J. Mullally, and M. D’Eath. 2007. Before and after study of bar workers’ perceptions of the impact of smoke-free workplace legislation in the republic of Ireland. BMC Public Health 7:131.
Sanders, G. D., and A. V. Taira. 2003. Cost effectiveness of a potential vaccine for human papillomavirus. Emerging Infectious Diseases 9(1):37–48.
Sargent, R. P., R. M. Shepard, and S. A. Glantz. 2004. Reduced incidence of admissions for myocardial infarction associated with public smoking ban: Before and after study. British Medical Journal 328(7446):977–980.
Scallan, E., R. M. Hoekstra, F. J. Angulo, R. V. Tauxe, M. A. Widdowson, S. L. Roy, J. L. Jones, and P. M. Griffin. 2011. Foodborne illness acquired in the United States—Major pathogens. Emerging and Infectious Diseases 17(1):7–15.
Sciacca, J. P., and M. I. Ratliff. 1998. Prohibiting smoking in restaurants: Effects on restaurant sales. American Journal of Health Promotion 12(3):176–184.
Scollo, M., A. Lal, A. Hyland, and S. Glantz. 2003. Review of the quality of studies on the economic effects of smoke-free policies on the hospitality industry. Tobacco Control 12(1):13–20.
Seo, D. C., and M. R. Torabi. 2007. Reduced admissions for acute myocardial infarction associated with a public smoking ban: Matched controlled study. Journal of Drug Education 37(3):217–226.
Swaminathan, B., and P. Gerner–Smidt. 2007. The epidemiology of human listeriosis. Microbes and Infection 9(10):1236–1243.
Taira, A. V., C. P. Neukermans, and G. D. Sanders. 2004. Evaluating human papillomavirus vaccination programs. Emerging Infectious Diseases 10(11):1915–1923.
Tang, H., D. W. Cowling, J. C. Lloyd, T. Rogers, K. L. Koumjian, C. M. Stevens, and D. G. Bal. 2003. Changes of attitudes and patronage behaviors in response to a smoke-free bar law. American Journal of Public Health 93(4):611–617.
Tong, E. K., and S. A. Glantz. 2007. Tobacco industry efforts undermining evidence linking secondhand smoke with cardiovascular disease. Circulation 116(16):1845–1854.
U.S. Surgeon General’s Advisory Committee on Smoking and Health. 1964. Smoking and health. Report of the Advisory Committee to the Surgeon General of the Public Health Service. Public Health Service Publication No. 1103 United States. Public Health Service. Office of the Surgeon General.
USDA (U.S. Department of Agriculture)/FSIS (Food Safety and Inspection Service) and EPA (U.S. Environmental Protection Agency). 2012. Microbial risk assessment guideline: Pathogenic organisms with focus on food and water. FSIS Publication No. USDA/ FSIS/2012-001; EPA Publication No. EPA/100/J12/001.
USPSTF (U.S. Preventive Services Task Force). 2008. Grade definitions. http://www.uspreventiveservicestaskforce.org/uspstf/grades.htm#post (accessed January 2, 2013). http://www.fsis.usda.gov/PDF/Microbial_Risk_Assessment_Guideline_2012-001.pdf (accessed January 3, 2013).
Vasselli, S., P. Papini, D. Gaelone, L. Spizzichino, E. De Campora, R. Gnavi, C. Saitto, N. Binkin, and G. Laurendi. 2008. Reduction incidence of myocardial infarction associated with a national legislative ban on smoking. Minerva Cardioangiologica 56(2):197–203.
WHO (World Health Organization). 2006. Building blocks for tobacco control: A handbook. Geneva: World Health Organization.
———. 2013. Microbiological risks publications. http://www.who.int/foodsafety/publications/micro/en (accessed January 3, 2013).
Woodcock, J. 2010. Memorandum from Janet Woodcock to NDA 021071 (dated September 22, 2010). Re: Decision on continued marketing of rosiglitazone (Avandia, Avandamet, Avandaryl). http://www.fda.gov/downloads/Drugs/DrugSafety/PostmarketDrugSafetyInformationforPatientsandProviders/UCM226959.pdf (accessed January 3, 2013).