National Academies Press: OpenBook

A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration (2011)

Chapter: Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks

« Previous: 7 Conclusions and Future Directions
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Appendix A
Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks

Committee on Ranking FDA Product Categories Based on Health Consequences


Board on Environmental Studies and Toxicology


Division on Earth and Life Studies


National Research Council


Institute of Medicine


NATIONAL RESEARCH COUNCIL AND
INSTITUTE OF MEDICINE
OF THE NATIONAL ACADEMIES


THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

National Research Council

Division on Earth and Life Studies

Board on Environmental Studies and Toxicology

500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 2347 Fax: 202 334 2752

February 17, 2009

Dr. Kathleen M. Koehler

Science Policy Analyst

Office of Science and Data Policy

U.S. Department of Health and Human Services

Hubert Humphrey Building, Room 434E 200 Independence Avenue, SW Washington, DC 20201

Dear Dr. Koehler:

At your request, the National Academies convened the Committee on Ranking FDA Product Categories Based on Health Consequences. The committee members were selected on the basis of their expertise in food safety, health economics, medical devices, vaccine safety, pharmacoepidemiology, biostatistics, comparative risk analysis, and decision analysis.

The committee was tasked with developing and applying a conceptual model to rank product categories in FDA program areas on the basis of health risks, both positive and negative aspects (that is, the committee was to consider beneficial aspects of the product categories in the context of possible adverse health consequences). The study was divided into two phases: selection of the model (phase I) and development, refinement, and application of the model to conduct a risk ranking of FDA product categories (phase II). The committee’s task is described in greater detail below. This letter report fulfills the task specified for phase I of this project.

The committee held two meetings. The first included a public session during which FDA staff and other invited experts made presentations. During that session, some indicated that a model that incorporates evaluations of interventions would be particularly valuable. The committee agrees but notes two complicating factors: evaluating baseline risks among product categories is a task of great magnitude and complexity, and it is the nature of interventions to be at the individual-product level and not the product-category or program level. Therefore, the model dictated by the committee’s task is not directly applicable to

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

“intervention” analysis and cannot be used to evaluate strategies to reduce risk. However, the committee acknowledges that the existence of intervention capabilities is an important measure in determining risk, and model parameters will need to capture that aspect. Given the size and complexity of the task, the committee will attempt to keep the model as simple as possible, recognizing that rough estimates of risk may be all that is possible at the product-category level.

This letter report first provides background information on comparative risk analysis. Next, it outlines the conceptual model. Considerations regarding the product categories and their attributes are provided. The report concludes with a discussion of the steps needed to refine the model and conduct a risk-ranking exercise. (There are also several attachments: a verbatim statement of the committee’s task, a committee roster and biographies, a bibliography, and acknowledgment of reviewers.) The report reflects the consensus of the committee and has been reviewed in accordance with standard National Research Council review procedures.

Sincerely,

Robert Lawrence,

Chair

Committee on Ranking FDA Product Categories Based on Health Consequences

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

LETTER REPORT ON THE DEVELOPMENT OF A MODEL FOR RANKING FDA PRODUCT CATEGORIES ON THE BASIS OF HEALTH RISKS

BACKGROUND

In 1986, U.S. Environmental Protection Agency (EPA) Administrator Lee Thomas asked 75 scientists and managers to develop a report on the “relative importance” of various environmental threats that were mainly in EPA’s jurisdiction. In 1987, the group issued Unfinished Business (EPA 1987), which categorized environmental threats in 31 problem areas, defined largely along existing programmatic lines. The group identified and divided the risks according to four important attributes with respect to the characterization of the 31 environmental problem areas: cancer risk, noncancer risk, ecologic risk, and “welfare” effects. The resulting report provided separate assessments (ranked low, medium, or high) for each of the four attributes. A key insight of Unfinished Business was that EPA's resource allocations appeared to be more in line with what the public perceived as the most important risks than with the priorities identified by the agency's experts. EPA asked its Science Advisory Board (SAB) to review the report, and the SAB released a follow-up report, Reducing Risk (EPA SAB 1990), which endorsed the broad comparative risk analysis (CRA) approach and produced findings similar to those in Unfinished Business.

In addition to spawning many applications of CRA at the office, region, state, and local levels (Minard 1996; Jones 1997), the early CRA efforts led to questions about how best to facilitate comparisons and identify useful attributes for characterizing risks or risk-reduction opportunities. The EPA SAB noted that ranking risks or ranking the alternative actions that might be available for reducing risks would probably yield different rankings (EPA SAB 1990). In particular, although some risks might rank high, they might also be associated with very expensive or uncertain risk-reduction actions and therefore be unamenable to intervention according to cost-benefit criteria. In addition, if risks associated with some low-priority areas can be addressed effectively with certainty at low or no cost, their low-priority status should not prevent these “bargains” from being recognized.

Progress on CRA method development continues, although its use remains relatively limited. Finkel and Golding (1995) noted that the “comparison of risks involves values in at least five areas: defining what we mean by ‘risk’; selecting the endpoints to consider; categorizing the risks for comparison; selecting a time frame for evaluating the adverse effects; and gauging the seriousness of the consequences.” In February 1994, a workshop organized by Resources for the Future for the President’s Office of Science and Technology Policy brought together researchers in CRA with the goal of developing a systematic process for comparing risks among different federal agencies (Davies 1996). As part of that work, researchers from Carnegie Mellon University developed a framework for

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

ranking risks that included both quantitative and qualitative measures of relevant programmatic attributes (Fischhoff 1995; Morgan et al. 1996). They included health-impact measures (such as morbidity and mortality) and psychometric measures that research shows play an important role in the evaluation of risks (such as fairness, scientific understanding, and uncertainty). That work spawned a series of research projects and papers that refined and applied the framework (e.g., Morgan et al. 1999, 2001; Long and Fischhoff 2000; Morgan et al. 2000; DeKay et al. 2001; Florig et al. 2001; Willis et al. 2004, 2005; Fischhoff 2006; Gutiérrez et al. 2006; Bronfman et al. 2007, 2008a,b), including a discussion directly related to food safety (DeKay et al. 2005). Recently, those risk-ranking methods have been adopted by a variety of national and international entities. For example, the U.S. Army Corps of Engineers is using the methods to rank hurricane mitigation opportunities on the Louisiana Gulf Coast (USACE 2008), researchers at the University of North Carolina School of Public Health and RAND Corporation are using them to develop an environmental-health strategy and action plan for the United Arab Emirates (UNC 2008), and the British government is using them to communicate with and gather information from the general public on health-related priority-setting strategies (HM Treasury 2004, 2005a,b; OGC 2008). Regardless of the application, such projects share the goals of collecting and presenting risk information in a systematic manner to guide and assess informed judgments. After assessment, those judgments may serve as a valuable input into a decision-making process focused on evaluating difficult policy choices.

STATEMENT OF TASK AND COMMITTEE’S APPROACH

In light of the increased use of CRA by federal agencies, the Department of Health and Human Services (DHHS) and the Food and Drug Administration (FDA) asked the National Research Council to convene an expert committee to develop and apply an evidence-based conceptual model and methods for ranking categories of products addressed by FDA programs. The conceptual model and methods were to focus on ranking product categories according to the ranges of magnitude of various potential health consequences to U.S. users of the products at the individual level and the population level, taking both adverse and beneficial effects into consideration. To accomplish its task, the committee was to include the following activities: consult with the sponsor to select FDA product categories to be ranked; consider products currently in use and near-term future products expected to come under FDA purview; review selected scientific literature bearing on adverse and beneficial health consequences; consider the scientific literature broadly to include social-science and economics literature, gray literature, and regulatory-policy literature; seek opportunities to assess health consequences in a way that allows results to be compared among broad product categories; identify information needed to address key uncertainties; assess the performance of the evidence-based model for ranking the selected product cate-

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

gories and identify next steps for model refinement; and where applicable and feasible, consider the potential effects on population health if risk-reduction strategies curtail the beneficial use of products.

The committee was asked first to produce a brief letter report that describes the scientific conceptual framework to be used to rank product categories (phase I) and then to perform ranking exercises by using the proposed conceptual framework (phase II). In neither phase was the committee to recommend regulatory strategies; those choices entail policy judgments that transcend scientific and technologic considerations. This letter report fulfills phase I of the project.

To accomplish its task, the committee held a public session at its first meeting, during which it heard presentations from FDA staff in the various program areas and from experts in the fields of decision analysis and CRA. The committee reviewed numerous scientific publications on CRA and literature provided by FDA. On the basis of its review and the statement of task, the committee selected a model that has the capacity to evaluate multiple product categories and compare them; to evaluate the magnitude and variation in distribution of both favorable and unfavorable effects; to improve FDA’s discharge of its responsibilities as they affect public health; to evaluate new product categories, risks, benefits, and other considerations; and to include multiple non-health-related outcomes of interest, such as equity and the quality of scientific understanding. The committee recognized that the model should be able to function to the greatest extent possible with sparse information. Although the primary focus of the committee was human health, the model considers animal health and welfare to be consistent with the full scope of activities conducted by FDA.

The CRA exercise requested in the statement of task is a valuable tool in determining relative risks among product categories, but such exercises are not sufficient to guide many policy decisions unless they incorporate additional concerns. For example, the absolute risk in a category may not be a good indicator of the potential to reduce risk in that category or of the potential to reduce risk by any specific action. Measures of the potential benefits of specific actions are critical for resource-allocation decisions. Likewise, the presence of a health risk may or may not be associated with the economic costs or benefits of addressing the risk, the equity concerns (who pays and who benefits), the likelihood and timeframe of achieving the stated risk reduction, or the public’s perceptions of the risk-mitigation options. To the extent that any of those concerns are or should be important in making policy decisions, the proposed CRA alone would not be sufficient for making decisions. On the basis of the guidance that the committee received from the statement of task and from clarification offered by the sponsor during its first public meeting, it concluded that discussion of the merits of other theoretical frameworks that might be valuable in assessing risk-mitigation alternatives was outside its task. Baseline ranking of risks is a necessary but not sufficient step in the more general decision-making process.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

RANKING MODEL

Conceptual Framework

On the basis of recent literature, the committee concluded that the best approach to ranking FDA product categories on the basis of health risk is to use a conceptual framework similar to that described by Florig et al. (2001). Unlike other comparative methods (such as the World Health Organization’s Global Burden of Disease Study [Murray and Lopez 1996; WHO 2008], which ranks solely on the basis of utility loss from illness), the approach described by Florig and colleagues allows for disparate items, such as cosmetics and vaccines, to be ranked, as will be necessary for FDA. Furthermore, although this is not explicitly required by FDA, the selected approach is designed to accommodate qualitative and quantitative variables in the formal ranking process. That will facilitate inclusion of important variables and may greatly improve the utility of the proposed approach for FDA.

Figure 1 summarizes the two phases and the multistep process envisioned by the committee. As illustrated in Figure 1, steps A and B involve defining the FDA product categories to rank and identifying the risk attributes to describe the categories, respectively. In phase I of this project, which is summarized in this letter report, the committee has proposed a preliminary list of categories and attributes. The final determination of the categories and attributes will require further input from knowledgeable FDA staff, iteration, and refinement and will be completed in phase II of this project. The committee’s final report will discuss the final categorization and identification of attributes and the process used to make those determinations. Step C requires describing the categories in terms of the attributes, step D involves performing the risk-ranking exercises, and step E involves summarizing and evaluating the results of the risk-ranking process. Steps C-E will be accomplished in phase II of this project, the results of which will be described in the committee’s final report. Each step in the multistep process is described in greater detail in the sections that follow.

FIGURE 1 Framework of health-risk ranking model. Source: Adapted from Florig et al. 2001. Reprinted with permission; copyright 2001, Risk Analysis.

FIGURE 1 Framework of health-risk ranking model. Source: Adapted from Florig et al. 2001. Reprinted with permission; copyright 2001, Risk Analysis.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Uncertainties in the risk-ranking model will be captured in two ways. For some attributes, such as those measuring mortality risks, a quantitative approach can be used by providing a mean (or median) with population upper and lower bounds. Other attributes can be selected to represent uncertainties with a qualitative description. For example, an attribute that captures the quality of scientific understanding will indicate knowledge gaps, which will lead to uncertainty in the ranking exercise. The committee recognizes that successful ranking will require both iteration and further engagement with FDA.

Step A:
Defining the Product Categories

Any ranking process must begin with development of a list of the risk items to rank. Many approaches exist to categorize risk, and as Morgan et al. (1996) note, “no single categorization scheme is likely to serve all … needs.” Morgan et al. (2000) describe criteria for defining categories and state that categories should be “exhaustive so that no relevant risks are overlooked,” “mutually exclusive so that risks are not double-counted,” and “homogenous so that all risk categories can be evaluated on the same set of attributes.” Furthermore, the categories should be relevant to the organizational structure, legislative mandates, and risk-management activities of the organization. Among the other criteria listed by Morgan et al. (2000) is the goal of keeping the number of categories to a number that makes the risk-ranking exercise feasible. Depending on the techniques used during the ranking, a feasible number would generally be in the range of 15-30. Given the number of products that FDA regulates, the committee recognizes that the number of categories could be expanded too much and make risk-ranking impossible. That means that the task of ranking products for FDA as a whole must of necessity focus on highly aggregated product categories.

FDA provided the committee with an initial list of product categories, which is shown in Table 1. The list includes 28 categories, and it primarily mirrors FDA’s organizational structure and statutory and regulatory authorities (that is, it is broken down according to FDA’s existing five product-focused centers). Although the committee noted that the categorization could alternatively focus on type or magnitude of hazard, it concluded that the final selection of the product categories for ranking will require further input from FDA staff who have specific expertise in the FDA products. Valuable input from FDA will include data on the size of each potential category (for example, with respect to numbers of regulated individual products or firms and relative market sizes in dollars). The committee expects that some of the categories will expand, others will contract, and some will be substantially revised. For example, the committee questioned the product categories suggested for medical devices. Currently, medical devices are defined according to risk and classified as class I, II, or III devices. Accordingly, this scheme may be more appropriate for categorization of medical devices. Furthermore, the committee notes that the present list includes catego-

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

ries at different levels of specificity, and this could obscure the value of the ranking exercise. The committee will determine whether additional categories will be needed to address near-term future products, such as nanomaterials, or whether those products can be integrated into the existing categories. Using the criteria of Morgan et al. (2000) discussed above and input from FDA, the committee will be able to determine the most appropriate product categories for evaluation in phase II of this project.

Step B:
Identifying the Attributes

Ranking risks requires identifying the important attributes of the risks. Morgan et al. (1996) described criteria for selecting attributes and noted that attributes should be comprehensive, non-overlapping, stand-alone, measurable, and minimal to reduce the complexity of the risk-ranking exercise. As a preliminary scoping exercise, the committee selected five attribute groups related to exposure, severity of effect, ability to anticipate and prevent adverse events, ability to mitigate adverse events, and benefits of products or product categories. Each group contains multiple specific attributes, which are shown in Table 2. The committee emphasizes that Table 2 is only a preliminary list. Overlapping attributes must be explicitly noted to avoid double-counting in the risk-ranking exercise, and most important, attributes must be selected that are applicable between and within the broad FDA product categories. That exercise will be challenging and will require input from FDA staff who have specific expertise in FDA product categories. The committee will use the criteria of Morgan et al. (1996) discussed above and further input from FDA to finalize the list of risk attributes in phase II of this project. At the conclusion of this process, the attributes will be clearly defined and well understood by FDA staff and the risk rankers.

The committee defined exposure as the condition of being subject to some effect or influence and considered the five risk attributes shown in Table 2 to be appropriate for quantifying or describing it. The exposed population is the percentage of the U.S. population potentially exposed. Cumulative incidence is the number of new cases of illness, injury, or other health-related events attributable to an exposure during a specified period in a specified population and is expressed as a rate. Prevalence is the number of cases of a health-related state or event that exist in a specified population at a particular time, regardless of when they began or how long they have existed, and can be expressed as a rate. Vulnerable groups refer to people who have increased susceptibility to adverse outcomes because of genetics, age, socioeconomic status, occupational or environmental exposure, or physiologic state; this attribute could be described in terms of the number and size of vulnerable populations. Cluster refers to a group of people who are at excess risk for adverse events that are related temporally, by proximity, or by source; this attribute could be described in terms of group size.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

TABLE 1 Suggested Initial List of Product Categories for Ranking Provided by FDA

Program Area

Product Categories

Food, cosmetics, and dietary supplements

Food

 

Produce

 

Eggs and dairy

 

Processed food

 

Seafood

 

Cosmetics

 

Dietary supplements

 

Food and color additives

Drugs and biologics regulated as drugs

Over-the-counter drugs

 

Diagnostic prescription drugs

 

Preventive prescription drugs

 

Prescription drugs that are life-sustaining

 

Prescription drugs for treatment for symptoms or improvement in quality of life

 

Prescription drugs that are used cosmetically

Biologic products other than those regulated as drugs

Vaccines

 

Blood and tissue products

 

In vitro diagnostics related to donor testing

 

Devices regulated as biologics

 

Allergenics

 

Cell and gene therapy

Veterinary products

Approved animal drugs

 

Unapproved animal drugs

 

Animal feeds

 

Pet food

Medical devices and radiation-emitting products

Critical devices for professional use

 

Noncritical devices for professional use

 

Noncritical devices for lay use

 

Nonmedical radiation-emitting devices

 

Patient cables and lead wires

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

TABLE 2 Risk Attributes for Model for Ranking FDA Product Categories on the Basis of Health Risk

Attribute Groups

Risk Attributes

Exposure

Exposed population

 

Cumulative incidence

 

Prevalence

 

Vulnerable groups

 

Cluster

Severity of effect

Mortality

 

Morbidity

 

Vulnerable groups

 

Catastrophic event

 

Diffusion effects beyond intended usea

 

Animal health

Ability to anticipate and prevent adverse events

Quality of scientific understanding

 

Availability of substitutes

 

History of problems and corrective actions

 

Availability of quality standards, guidelines, or standard operating procedures (quality assurance and quality control, good manufacturing practices)

 

Variability in product composition or performance

 

Vulnerability of supply chain

Ability to mitigate adverse events

Availability of substitutes

 

Availability of corrective actions

 

Traceability

 

Latency

 

Ability to recall

 

Reversibility

Product benefits

Mortality reduction

 

Morbidity reduction

 

Efficacy and effectiveness

 

Animal welfare

aDiffusion effects is an attempt to capture effects on people who do not use the product.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Severity of the effect is described by six attributes as shown in Table 2. Mortality could be measured as expected number (or range of numbers) of deaths per year in the United States attributable to the product category. Morbidity could be quantified by one or more of the following: number of doctor visits per year, number of hospitalizations (or hospital days) per year, number of lost work days per year, total cost of treatment, and number of chronic cases per year. Metrics need to account for acute and chronic illnesses. The committee discussed using quality-adjusted life years (QALYs) to capture the differences but concluded that the method might be difficult to apply to something as heterogeneous or complex as the product categories given the expected paucity of data. However, if data are available, QALYs should be included in the risk-ranking model. Vulnerable groups could be represented by the percentage of all deaths that occur in vulnerable groups or the percentage of deaths in a vulnerable group that are attributable to the product category. Catastrophic event refers to a low-probability event with the potential for a severe outcome and could be characterized by the number of deaths in a worst-case scenario. The attribute diffusion effect is an attempt to capture effects on people who do not use the product (for example, transgenerational effects resulting from product use); this attribute could be characterized by a minor-moderate-major significance descriptor. Animal health could be measured as the number of animals that die from use of animal products; food animals and companion animals could be evaluated separately.

The committee considered six specific attributes to characterize the ability to anticipate and prevent adverse events. Quality of scientific understanding is related to product knowledge, that is, background information available on a product, the number of scientific studies, and the quality of the studies. The availability of substitutes attribute is related to how critical a product is. The attribute history of problems and corrective actions reflects the availability of information on such events as adverse reactions and manufacturing defects and whether actions have been needed to address problems. The attribute availability of quality standards, guidelines, or standard operating procedures attempts to capture the idea that manufacturing products in accordance with standards, guidelines, or standard operating procedures improves FDA’s ability to control product quality and minimize adverse events (for example, the availability of practices to produce uniform, high-quality products and the limits of the practices). Variability in product composition or performance refers to inherent differences in products that result even when good manufacturing practices are adhered to or that result from intended or unintended deviations from good manufacturing practices, including inadvertent contamination, equipment malfunction, or deliberate adulteration. Variability in performance is also related to person-person differences in response to a product, such as the side effects experienced after taking a drug; such variability may be related to individual characteristics or to how the product is used. Finally, vulnerability of the supply chain refers to the potential disruption in manufacture or distribution of a product if there are any problems with obtaining inputs to any step of the process. All

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

the risk attributes in this group could be defined by a binary yes-no descriptor or a high-moderate-low descriptor.

Six attributes are used to describe the ability to mitigate adverse events. Availability of substitutes and corrective actions would provide methods to mitigate an adverse event by replacing a product with an equivalent or similar product, by modifying manufacturing or distribution processes, or by communicating with consumers about potentially harmful products. Traceability is the ability to identify the sources of all product components, affected products, or components in a supply chain, or people potentially exposed to products. Latency is defined as the interval between the first exposure and the observation of an adverse event; long latency would make it difficult to anticipate and prevent adverse events. Ability to recall is the ability to remove from the supply chain a product that is identified as carrying unacceptable risks. Reversibility is the amelioration of adverse events. Each of the attributes could be defined with a yes-no descriptor or a high-moderate-low descriptor.

The final attribute group attempts to incorporate the beneficial aspects of products to the general and target populations. Some benefits are linked to protecting the population from adverse consequences; some are linked to diagnosing, treating, or preventing disease; and others are linked to promoting health, such as those related to nutrition. The benefits may be best quantified by expected reduction in mortality (that is, the number of lives saved per year in the United States from intended use of the product category) and by expected reduction in morbidity (for example, the number of disabilities or of hospitalizations avoided per year in the United States). The efficacy and effectiveness attribute captures how well the current program is working (for example, Is it cost-effective?). Animal welfare tries to capture the idea that food production animals are important food sources and contribute substantial benefits to the U.S. economy and that companion animals enhance quality of life and may provide some health benefits. Products that enhance the quality of life of the animals would provide a benefit to society. A high-moderate-low descriptor could be used to characterize that attribute. The committee emphasizes that capturing beneficial aspects of product categories will be a challenging task. Some benefits are obvious and often discussed, and others are less well known, especially if products have long-term consequences. Many benefits will be difficult to quantify or rank, such as those related generally to improving quality of life or social values by providing people with a variety of choices or with novel products.

Because it is highly desirable to have an attribute table that allows easy side-by-side comparisons, there should be fewer than 20, and ideally fewer than 15, attributes. Winnowing the attribute list down will require merging some of the attributes shown in Table 2. For example, the “Ability to anticipate and prevent adverse events” set, which currently has six specific attributes, will probably contain only one or two in the final version. If those six attributes were all described by using binary variables, 64 combinations would be possible. Given that only 15-30 product categories will be ranked, that number of attributes overspecifies the ranking task. Winnowing of the attribute list will occur natu-

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

rally as overlapping attributes are highlighted and preferred attributes are selected. In some cases, entire attributes groups could be eliminated (for example, is the “exposure” group necessary if the “severity of effect” group contains mortality and morbidity rates and counts?).

The committee notes that for each product category the risk rankers will receive a summary “pamphlet” that will include the attribute table (which will display all the attribute values in a consistent format compatible with easy side-by-side comparisons) and supporting information that helps to put the attribute values in context. Because the attributes are selected to be universal and applicable between product categories, there are relevant details that cannot be captured in a simple listing of quantified attributes. For example, the amplifying text could include descriptions of the vulnerable groups relevant for the product category, the extent to which life-cycle calculations were completed or limited, or animal impacts that were not explicitly shown in the attribute table. The supplemental information will be concise and organized in a standard manner so that comparisons between product categories are possible.

Step C:
Describing the Product Categories by Using the Attributes

In this initial phase of its work, the committee has not reached formal recommendations for the follow-on steps in the framework, but it recognizes the difficulties ahead, particularly in completing step C, in which the values of each attribute (step B) must be described for each product category (step A). In some ranking exercises, the items for ranking are unique and narrowly defined. In this case, however, the broad product categories, such as vaccines, contain many individual products, each with their own attribute values. Thus, aggregation will present important challenges with respect to attribute characterization. For some attributes, such as those measuring mortality risks, a summation across all items in the product category may capture the relevant values, with upper and lower bounds on the quantified values providing an indication of the uncertainties in the calculations. However, because of the heterogeneity of some of the product categories, difficulty will arise in assigning unique values for many attributes. It may be desirable to split large product categories to improve homogeneity of ranking (for example, vaccines might be divided into killed vaccines and live vaccines). For some attributes, aggregation may have the effect of making them nonvariant among the set of items being ranked. That problem could be mitigated by focusing the attributes in question on the mechanisms that lead to the greatest concern for morbidity and mortality. Thus defined, the attribute would distinguish product categories with well-established cause-effect linkages from categories that are less well understood.

The necessary refinement of the attribute definitions will not be possible until the committee can re-engage the necessary FDA personnel through follow-on workshops in phase II of this project. Discussions could lead to a regrouping of the product categories and attribute description. The process by which the

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

first three steps are completed will generate valuable information. In fact, the committee emphasizes that the process that it undertakes with FDA is likely to be a more important tool and provide more insights than the list of ranked product categories alone. Creating robust attributes that are systematically and accurately measured across the wide array of FDA regulatory responsibilities will focus agency thinking and help to communicate the considerations underlying agency decision-making to a broad range of stakeholders.

Steps D and E:
Conducting the Ranking and Analyzing the Results

Once the attributes are fully defined for each product category, the ranking will begin, probably with at least two approaches: a holistic ranking based on the rankers’ overall preferences and a ranking based on application of a formal mutiattribute model. The committee anticipates selecting the ranking procedures after further discussion with FDA to learn how it plans to use the rankings.

In a holistic ranking, the preferences for, importance of, and trade-offs between the attributes are not made explicit by the rankers. The rankers carefully review the summary material and, using their own judgment, rank the product categories. Guidance as to how to structure the ranking procedure can make the task more manageable. For example, guidance provided could be to complete a rough sorting into three preliminary categories—high, moderate, and low risk—before completing the series of pairwise comparisons. In contrast, the formal multiattribute model approach requires the rankers to consider each attribute and their range of values and to state explicitly the relative trade-offs between attributes through the elicitation of attribute weights. A model is built for each ranker by using those weights, and a ranking of the product categories is calculated. Sensitivity analyses are performed on the ranks to determine the influence of the assessed weights and the value functions.

Although the two basic approaches to complete the rankings differ substantially, there are advantages in using both approaches in a single exercise (Palmgren et al. 2000). With a holistic approach, overlapping attributes can be considered (that is, the combination of attribute values may provide insights into relevant details); however, with a formal multiattribute model, double counting is not allowed, and attribute weight assessment must reflect this requirement. The results of using one approach can improve the understanding of the other, and ultimately a revised ranking informed by the results of both is likely to be preferred by the rankers. Analyses conducted after the ranking can determine the relative importance of the two approaches in generating a ranker’s final ordering. Developing a satisfactory multiattribute model for the ranking task has many potential benefits, including the ability to add new or revised product categories and to see quickly how they fit among the categories already ranked. Once again, understanding how FDA will use the ranking results will determine the preferred approaches. The rankings could be used internally by senior administrators as input into their own strategic planning or as a way to capture

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

public perceptions and communicate policy to the general public. Both applications offer merit but would certainly dictate different approaches to documentation and communication of the process. Regardless of the specific uses, determining the level of agreement among the rankers could provide valuable insights that are not now available to FDA.

NEXT STEPS

The committee has proposed a conceptual model for ranking FDA product categories and has suggested preliminary categories and risk attributes. The committee emphasizes that participation of FDA staff in each program area is essential for the development of a successful and useful model. Development of the model will require a series of three workshops. The first workshop will involve discussion of the model’s product categories and attributes that will lead to refinement, revision, and adoption of both for further use in steps C-E. This iterative cycling between product categories and attribute definitions will ensure that the follow-on steps can be completed. In addition, the engagement of FDA personnel will serve as an educational opportunity so that the model, its application, and its limitations can be understood and appreciated. The second workshop will involve populating the model matrix with data to allow the ranking exercises to be performed and will require much effort on the part of FDA personnel who will be responsible for gathering data and providing input values. Populating the model matrix will almost certainly require the determination of values that have not been estimated previously and rely on the use of subjective expert judgment. As indicated above, the difficulty in providing values will probably vary considerably among product categories. The third workshop will involve conduct of the actual ranking exercises. The committee emphasizes that the development of the model is an iterative process as reflected in Figure 1. Findings from any of the three workshops may necessitate adjustments and refinements of earlier steps in the model. The committee’s final report will summarize and evaluate the outcome of the workshops and provide recommendations for using the risk-ranking model as an input in a decision-making process.

The committee notes that the risks and benefits vary substantially between and within product categories, and that will pose a challenge in developing and implementing the ranking process. However, that challenge makes the ranking exercise and resulting ranking valuable for FDA because unaided comparisons also face the challenge of comparing apparently incomparable product categories but without the common metrics that the committee will be recommending. Therefore, the ranking exercise is a logical first step for FDA. The committee provides an outside perspective on the challenge of comparing risk among disparate categories, and its recommendations, with input from FDA, will help to identify a framework for making advances in FDA management processes and decisions.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Attachment A
STATEMENT OF TASK

An expert committee will develop and apply an evidence-based conceptual model and methods to rank product categories within the broad types of products addressed by programs of the U.S. Food and Drug Administration (FDA). The conceptual model and methods will focus on ranking product categories according to the potential ranges of magnitude of various health consequences to U.S. users of the products at individual and population levels, taking both adverse and beneficial effects into consideration. The committee will begin by selecting, in consultation with DHHS and FDA, categories of products within FDA mandates for human and veterinary drugs, biologics, medical devices, foods, cosmetics, and products that emit radiation. The committee will then review selected scientific literature bearing on adverse and beneficial health consequences related to these product categories. It will develop a scientific conceptual framework for potential use in guiding product category rankings based on expert judgments and related analysis of the types and potential ranges of magnitude of health consequences to U.S. users of the products (phase I). Using this framework, the committee will perform ranking exercises through expert elicitation and analysis or other appropriate methods (phase II).


In carrying out its task, the committee will include the following activities:

  • In selecting product categories for ranking, consider products currently in use and near-term future products expected to come under FDA purview.

  • Seek opportunities to assess health consequences in a way that allows results to be compared across broad product categories.

  • Where data or assessment methods are deficient for evaluating a product category, identify information needs for addressing key uncertainties and present evaluations.

  • Assess the performance of the evidence-based model for ranking the selected product categories and identify next steps for further refinement of the model.

  • In assessing health consequences, consider both the risks and the beneficial aspects of product use, and where applicable and feasible, consider the potential impact on population health if beneficial product use is curtailed through risk reduction strategies.

  • In reviewing selected scientific literature, the committee shall consider the scientific literature broadly, to include, as appropriate, social science and economic literature, grey literature, and regulatory policy literature.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

The committee will not recommend regulatory strategies, because those choices will entail policy judgments that transcend scientific and technologic considerations.

Seven months after initiation of the study, the committee will prepare a brief letter report describing the conceptual model and methods it will use to rank product categories in its final report.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Attachment B
COMMITTEE MEMBERSHIP

ROBERT LAWRENCE, Chair, Johns Hopkins University, Baltimore, MD

JAMES ANDERSON, Case Western Reserve University, Cleveland, OH

FRANCISCO DIEZ-GONZALEZ, University of Minnesota, St. Paul

KATHRYN EDWARDS, Vanderbilt University, School of Medicine, Nashville, TN

SUSAN ELLENBERG, University of Pennsylvania, Philadelphia

PAUL FISCHBECK, Carnegie Mellon University, Pittsburgh, PA

HELEN JENSEN, Iowa State University, Ames

ROBIN KELLER, University of California, Irvine

DAVID MELTZER, University of Chicago, Chicago, IL

SANFORD MILLER, University of Maryland, College Park

RICHARD PLATT, Harvard Medical School, Boston, MA

KIMBERLY THOMPSON, Harvard School of Public Health, Boston, MA

STAFF

ELLEN MANTUS, Project Director

DAVID A. BUTLER, Senior Program Officer

NORMAN GROSSBLATT, Senior Editor

HEIDI MURRAY-SMITH, Research Associate

PANOLA GOLSON, Senior Program Assistant

BIOGRAPHIES

Robert S. Lawrence (IOM), Chair, is the Center for a Livable Future (CLF) professor and director of the CLF in the Department of Environmental Health Sciences, professor of health policy and international health at the Johns Hopkins Bloomberg School of Public Health and professor of medicine at the Johns Hopkins University School of Medicine. His expertise and research interests include community and social medicine, human rights, health promotion and disease prevention, evidence-based decision rules for prevention policy, and food security. Dr. Lawrence is a master of the American College of Physicians and a fellow of the American College of Preventive Medicine. He is a member of the Institute of Medicine and has served on numerous National Academies committees, most recently the Committee on Adolescent Health Care Services and Models of Care for Treatment, Prevention, and Health Development and the Committee to Evaluate Measures of Health Benefits for Environmental, Health,

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

and Safety Regulation. Dr. Lawrence received his MD from Harvard Medical School and trained in internal medicine at the Massachusetts General Hospital.


James M. Anderson (IOM) is professor of pathology, macromolecular science, and biomedical engineering at Case Western Reserve University. His research interests range from his activity as a pathologist in clinical implant retrieval and evaluation to fundamental mechanistic studies focused on tissue, cell, and blood interactions with biomaterials. Dr. Anderson is the recipient of the Elsevier Biomaterials Gold Medal for the most significant contributions to biomaterials science from 1980 to 2005 and the Society of Investigative Pathology Chugai Mentoring Award. He has been involved in the International Standards Organization Task Force to Develop Standards for Medical Device Safety for the last 18 years. He is editor-in-chief of the Journal of Biomedical Materials Research. Dr. Anderson is a member of the Institute of Medicine and has served as a member of its Committee on Postmarket Surveillance of Pediatric Medical Devices and Committee on Capturing the Full Power of Biomaterials for Military Medical Needs. He received his MD from the Case School of Medicine and his PhD in chemistry from Oregon State University.


Francisco Diez-Gonzalez is an associate professor in the Department of Food Science and Nutrition at the University of Minnesota. His research expertise is in food-safety microbiology, foodborne pathogens, safety of fresh fruits and vegetables, preharvest control of pathogenic E. coli, bioterrorism agents, and safety of organic food. Dr. Diez-Gonzalez teaches courses in food safety and food microbiology. He has served on the University of Minnesota Institutional Biosafety Committee, and he has advised both undergraduate and graduate students. He is also the recipient of the New Career Excellence Award for the College of Human Ecology at the University of Minnesota. He is member of the Editorial Board of the Journal of Food Protection and the Journal of Food Analytical Methods. Dr. Diez-Gonzalez received his PhD in food science from Cornell University.


Kathryn M. Edwards (IOM) is Sarah H. Sell Chair in Pediatrics and the director of the Vanderbilt Vaccine Research Program at Vanderbilt University Medical Center. Her research focuses on the evaluation of vaccines for the prevention of infectious diseases in adults and children. She is a fellow of the Infectious Diseases Society of America and of the American Academy of Pediatrics. Dr. Edwards has served as a member of the Advisory Committee on Immunization Practices of the Centers for Disease Control and Prevention and the Vaccines and Related Biological Products Advisory Committee of the Food and Drug Administration. She has also served as a member of the National Academies Committee to Assess the Safety and Efficacy of the Anthrax Vaccine. Dr. Edwards received her MD from the University of Iowa College of Medicine.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Susan S. Ellenberg is professor of biostatistics and associate dean for clinical research at the University of Pennsylvania School of Medicine. Her research focuses on the design and analysis of clinical trials and the assessment of medical-product safety. Dr. Ellenberg is associate editor of Clinical Trials and of the Journal of the National Cancer Institute. She is a fellow of the American Statistical Association, the Society for Clinical Trials, and the American Association for the Advancement of Science. She has served as a member of the National Academies Planning Committee for the IOM Drug Safety Report: Resource Implications, Committee on the Assessment of the U.S. Drug Safety System, and Committee on Applied and Theoretical Statistics. Dr. Ellenberg received her PhD in mathematical statistics from George Washington University.


Paul S. Fischbeck is professor of social and decision sciences, professor of engineering and public policy, and director of the Center for the Study and Improvement of Regulation at Carnegie Mellon University. His research focuses on the quantification and communication of uncertainty, including theoretical improvements in decision analysis and numerous applied real-world problems. Dr. Fischbeck has written extensively on various applications of decision and risk-analysis methods and has won several awards from the Institute for Operations Research and the Management Sciences. He is a member of the National Research Council Marine Board and has served on several committees, including the Committee on Marine Salvage Response Capability: A Workshop and the Committee on Risk Assessment and Management of Marine Systems. Dr. Fischbeck received a PhD in industrial engineering and engineering management from Stanford University.


Helen H. Jensen is a professor of economics and head of the Food and Nutrition Policy Division of the Center for Agricultural and Rural Development at Iowa State University. Her research fields are food and nutrition policy, analysis of food-consumption behavior, economics of food safety, and health risk assessment. Dr. Jensen is on the Board of Directors of the American Agricultural Economics Association and of the Council on Food, Agricultural and Resource Economics and has recently been on the Editorial Boards of Food Economics, Agricultural Economics, and Agribusiness: An International Journal. She has served on U.S. Department of Agriculture expert review panels, including the Panel on Measuring Food Security in the United States and the Panel on the Health Eating Index. She has served on several National Academies committees and is currently involved with the Committee on Nutrition Standards for National School Lunch and Breakfast Programs and the Committee on Economic Development and Current Status of the Sheep Industry in the United States. Dr. Jensen received her PhD in agricultural economics from the University of Wisconsin-Madison.


L. Robin Keller is a professor of operations and decision technologies at the University of California, Irvine. Her research is in decision analysis, risk analy-

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

sis, creative problem-structuring, and behavioral decision theory. She is the editor-in-chief of Decision Analysis. Dr. Keller has served as program director for the Decision, Risk, and Management Science Program of the U.S. National Science Foundation, and she has conducted studies funded by the U.S. Environmental Protection Agency and the Department of Energy. She has served as a member of the National Research Council Committee to Assess the Distribution and Administration of Potassium Iodide in the Event of a Nuclear Incident, and she is currently a member of the U.S. National Committee for the International Institute for Applied Systems Analysis. Dr. Keller received her PhD from the University of California, Los Angeles.


David O. Meltzer is an associate professor in the Department of Medicine, chief of the Section of Hospital Medicine, and an associate faculty member of the Harris School and the Department of Economics at the University of Chicago. He is also director of the Center for Health and the Social Sciences and co-director of the Program on Outcomes Research Training. Dr. Meltzer’s research explores problems in health economics and public policy, with a focus on theoretical foundations of medical cost-effectiveness analysis and the effects of managed care and medical specialization on the cost and quality of care. He is the recipient of numerous awards, including the National Institutes of Health Medical Scientist Training Program Fellowship, the National Science Foundation Graduate Fellowship in Economics, and the Lee Lusted Prize of the Society for Medical Decision Making, of which he is the immediate past president. Dr. Meltzer has served on several National Academies committees, most recently the Committee on the Assessment of the U.S. Drug Safety System and the Committee on Establishing a National Cord Blood Stem Cell Bank Program. He received his MD and his PhD in economics from the University of Chicago.


Sanford A. Miller is a senior fellow at the Joint Institute for Food Safety and Applied Nutrition at the University of Maryland. He was named professor and dean emeritus of the Graduate School of Biomedical Sciences at the University of Texas Health Science Center in December 2000 after serving as dean from 1987 to 2000. He is a former director of the Center for Food Safety and Applied Nutrition in the Food and Drug Administration. Dr. Miller has served on many national and international government and professional-society advisory committees, including the National Advisory Environmental Health Sciences Council of the National Institutes of Health and the Joint World Health Organization-United Nations Food and Agricultural Organization Expert Advisory Panel on Food Safety. He is a member of the National Academies Food and Nutrition Board and the Committee on Use of Dietary Supplements by Military Personnel. Dr. Miller received his PhD in physiology and biochemistry from Rutgers, The State University of New Jersey, New Brunswick.


Richard Platt is professor and chair of the Department of Ambulatory Care and Prevention and a professor of medicine at Harvard Medical School. His research

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

focuses on the safety and effectiveness of marketed drugs and vaccines and on infectious diseases in the community and hospital settings. Dr. Platt is a former chair of the Food and Drug Administration Drug Safety and Risk Management Advisory Committee. He is a member of the Advisory Panel for Research of the Association of American Medical Colleges and has chaired the Executive Committee of the HMO Research Network, the Epidemiology and Disease Control Study Section of the National Institutes of Health, and the Steering Committee of the Centers for Disease Control and Prevention Office of Health Care Partnerships. He has served on several National Academies committees and is a member of the Roundtable on Evidence-Based Medicine. Dr. Platt received his MD from Harvard Medical School.


Kimberly M. Thompson is associate professor of risk analysis and decision science at the Harvard School of Public Health. Her research is related to developing and applying quantitative methods for risk assessment and risk management and the public-policy implications of including uncertainty and variability in risk characterization. She has served on several National Academies committees, including the Committee for the Study of a Motor Vehicle Rollover Rating System and the Subcommittee to Update the 1999 Arsenic Report. She is a member of the Board on Environmental Studies and Toxicology. Dr. Thompson received her ScD in environmental health from the Harvard School of Public Health.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Attachment C
REFERENCES

Bronfman, N.C., L.A. Cifuentes, M.L. DeKay, and H.H. Willis. 2007. Accounting for variation in the explanatory power of the psychometric paradigm: The effects of aggregation and focus. J. Risk Res. 10(4):527-554.

Bronfman, N.C., L.A. Cifuentes, and V.V. Gutierrez. 2008a. Participant-focused analysis: Explanatory power of the classic psychometric paradigm in risk perception. J. Risk Res. 11(6):735-753.

Bronfman, N.C., E.L. Vazquez, V.V. Gutierrez, and LA. Cifuentes. 2008b. Trust, acceptance and knowledge of technological and environmental hazards in Chile. J. Risk Res. 11(6):755-773.

Davies, J.C. 1996. Comparing Environmental Risks: Tools for Setting Government Priorities. Washington, DC: Resources for the Future.

DeKay, M.L., H.K. Florig, P.S. Fischbeck, M.G. Morgan, K.M. Morgan, B. Fischhoff, and K.E. Jenni. 2001. The use of public risk ranking in regulatory development. Pp. 208-230 in Improving Regulation: Cases in Environment, Health, and Safety, P.S. Fischbeck, and R.S. Farrow, eds. Washington, DC: Resources for the Future.

DeKay, M.L., P.S. Fischbeck, H.K. Florig, M.G. Morgan, K.M. Morgan, B. Fischhoff, and K.E. Jenni. 2005. Judgment-based risk ranking for food safety. Pp. 198-226 in Toward Safer Food: Perspectives on Risk and Priority Setting, S.A. Hoffmann, and M.R. Taylor, eds. Washington, DC: Resources for the Future.

EPA (U.S. Environmental Protection Agency). 1987. Unfinished Business: A Comparative Assessment of Environmental Problems. EPA/230/2-87/025. Office of Policy, Planning, and Evaluation, U.S. Environmental Protection Agency, Washington, DC.

EPA SAB (U.S. Environmental Protection Agency Science Advisory Board). 1990. Reducing Risk: Setting Priorities and Strategies for Environmental Protection. SAB-EC-90-021. U.S. Environmental Protection Agency Science Advisory Board, Washington, DC [online]. Available: http://yosemite.epa.gov/sab/sabproduct.nsf/28704D9C420FCBC1852573360053C692/$File/REDUCING+RISK++++++++++EC-90-021_90021_5-11-1995_204.pdf [accessed Dec. 2, 2008].

Finkel, A.M., and D. Golding. 1995. Worst Things First? The Debate over Risk-Based National Environmental Priorities. Washington, DC: Resources for the Future. Fischhoff, B. 1995. Ranking risks. Risk Health Saf. Environ. 6:189-200.

Fischhoff, B. 2006. Cognitive processes in stated preference methods. Pp. 937-968 in Handbook of Environmental Economics, Vol. 2. Valuing Environmental Changes, K.G. Mäler, and J.R. Vincent, eds. Amsterdam: Elsevier.

Florig, H.K., M.G. Morgan, K.M. Morgan, K.E. Jenni, B. Fischhoff, P.S. Fischbeck, and M.L. DeKay. 2001. A deliberative method for ranking risks (I): Overview and test-bed development. Risk Anal. 21(5): 913-921.

Gutiérrez, V.V., L.A. Cifuentes, and N.C. Bronfman. 2006. The influence of information delivery on risk ranking by lay people. J. Risk Res. 9(6):641-655.

HM Treasury. 2004. The Orange Book: Management of Risk-Principles and Concepts. London, UK: Her Majesty’s Stationery Office. October 2004 [online]. Available: http://www.who.int/management/general/risk/managementofrisk.pdf [accessed Dec. 2, 2008].

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

HM Treasury. 2005a. Managing Risks to the Public: Appraisal Guidance. London, UK: Her Majesty’s Stationery Office. June 2005 [online]. Available: http://www.hm-treasury.gov.uk/d/Managing_risks_to_the_public.pdf [accessed Dec. 2, 2008].

HM Treasury. 2005b. Principles of Managing Risks to the Public. HM Treasury Cabinet Office, London, UK [online]. Available: http://www.hm-treasury.gov.uk/d/risk_principles_180903.pdf [accessed Dec. 2, 2008].

Jones, K. 1997. A Retrospective on Ten Years of Comparative Risk. Prepared for American Industrial Health Council, Washington, DC, by Green Mountain Institute for Environmental Democracy, Montpelier, VT. January 24, 1997 [online]. Available: http://heartland.temp.siteexecutive.com/pdf/23157i.pdf [accessed Dec. 2, 2008].

Long, J., and B. Fischhoff. 2000. Setting risk priorities: A formal model. Risk Anal. 20(3):339-352.

Minard, R.A. 1996. Comparative risk assessment and the States: History, politics and results. Pp. 23-62 in Comparing Environmental Risks: Tools for Setting Government Priorities, J.C. Davies, ed. Washington, DC: Resources for the Future.

Morgan, K.M., M.L. DeKay, and P.S. Fischbeck. 1999. A multi-attribute approach to risk prioritization. Risk Policy Report 6(6):38-40.

Morgan, M.G., H.K. Florig, M.L. DeKay, and P. Fischbeck. 2000. Categorizing risks for risk ranking. Risk Anal. 20(1):49-58.

Morgan, K.M., M.L. DeKay, P.S. Fischbeck, M.G. Morgan, B. Fischhoff, and H.K. Florig. 2001. A deliberative method for ranking risks (II): Evaluation of validity and agreement among managers. Risk Anal. 21(5):923-937.

Morgan, M.G., B. Fischhoff, L. Lave, and P. Fischbeck. 1996. A proposal for ranking risks within federal agencies. Pp. 111-148 in Comparing Environmental Risks: Tools for Setting Government Priorities, J.C. Davies, ed. Washington, DC: Resources for the Future.

Morgan, M.G., H.K. Florig, M.L. DeKay, and P. Fischbeck. 2000. Categorizing risks for risk ranking. Risk Anal. 20(1):49-58.

Murray, C.J.L., and A.D. Lopez, eds. 1996. The Global Burden of Disease: A Comprehensive Assessment of Mortality and Disability from Diseases, Injuries and Risk Factors in 1990 and Projected to 2020. Cambridge, MA: Harvard University Press.

OGC (Office of Government Commerce). 2008. Management of Risk (M_o_R). Office of Government Commerce, HM Treasury, London, UK [online]. Available: http://www.ogc.gov.uk/guidance_management_of_risk_4441.asp [accessed Dec. 2, 2008].

Palmgren, C.R., M.L. DeKay, P.S. Fischbeck, B. Fischhoff, and M.G. Morgan. 2000. Evaluating a Risk-Ranking Methodology. Society of Risk Analysis Annual Meeting Applications of Risk Analysis in Industry and Government, December 3-6, 2000, Washington, DC.

UNC. 2008. UNC School of Public Health to help UAE assess environmental health risks. UNC Gillings School of Global Public Health. School of Public Health News: June 9, 2008 [online]. Available: http://www.sph.unc.edu/school_of_public_health_news/unc_school_of_public_health_to_help_uae_assess_environmental_health_risks_7546_1957.html [accessed Dec. 2, 2008].

USACE (US Army Corps of Engineers). 2008. Risk-Informed Decision Framework Appendix, Draft. Louisiana Coastal Protection and Restoration Technical Report. U.S. Army Corps of Engineers, New Orleans District, Mississippi Valley Division. February 2008 [online]. Available: http://lacpr.usace.army.mil/\Report\DraftAppendices\RiskInformedDecisionFrameworkAppendix.pdf [accessed Dec. 2, 2008].

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

WHO (World Health Organization). 2008. The Global Burden of Disease: 2004 Update. Geneva: World Health Organization [online]. Available: http://www.who.int/healthinfo/global_burden_disease/GBD_report_2004update_full.pdf [accessed Jan. 9, 2009].

Willis, H.H., M.L. DeKay, M.G. Morgan, H.K. Florig, and P.S. Fischbeck. 2004. Ecological risk ranking: Development and evaluation of a method for improving public participation in environmental decision making. Risk Anal. 24(2):363-378.

Willis, H.H., M.L. DeKay, B. Fischhoff, and M.G. Morgan. 2005. Aggregate, disaggregate, and hybrid analyses of ecological risk perceptions. Risk Anal. 25(2):405-428.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

Attachment D
ACKNOWLEDGMENT OF REVIEWERS

This report has been reviewed in draft form by persons chosen for their diverse perspectives and technical expertise in accordance with procedures approved by the Report Review Committee of the National Research Council. The purposes of the independent review are to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following for their review of this report: Ann Bostrom, University of Washington; Paul Citron, Medtronic, Inc.; Alan M. Garber, Stanford University; Sandra A. Hoffman, Resources for the Future; Robert E. Johnston, University of North Carolina, Chapel Hill; Ralph L. Keeney, Duke University; Harley W. Moon, Iowa State University; Joseph V. Rodricks, ENVIRON; and Brian L. Strom, University of Pennsylvania School of Medicine.

Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of the report was overseen by the review coordinator, Lauren Zeise, California Environmental Protection Agency, and the review monitor, John C. Bailar, III, University of Chicago. Appointed by the National Research Council, they were responsible for making certain that an independent examination of the report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of the report rests entirely with the committee and the institution.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×

This page intentionally left blank.

Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 149
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 150
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 151
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 152
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 153
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 154
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 155
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 156
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 157
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 158
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 159
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 160
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 161
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 162
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 163
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 164
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 165
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 166
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 167
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 168
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 169
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 170
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 171
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 172
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 173
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 174
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 175
Suggested Citation:"Appendix A: Letter Report on the Development of a Model for Ranking FDA Product Categories on the Basis of Health Risks." National Research Council and Institute of Medicine. 2011. A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration. Washington, DC: The National Academies Press. doi: 10.17226/13156.
×
Page 176
Next: Appendix B: Statement of Task for the Committee on Ranking FDA Product Categories Based on Health Consequences, Phase II »
A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration Get This Book
×
Buy Paperback | $46.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

With the responsibility to ensure the safety of food, drugs, and other products, the U.S. Food and Drug Administration (FDA) faces decisions that may have public-health consequences every day. Often the decisions must be made quickly and on the basis of incomplete information. FDA recognized that collecting and evaluating information on the risks posed by the regulated products in a systematic manner would aid in its decision-making process. Consequently, FDA and the Department of Health and Human Services (DHHS) asked the National Research Council (NRC) to develop a conceptual model that could evaluate products or product categories that FDA regulates and provide information on the potential health consequences associated with them.

A Risk-Characterization Framework for Decision-Making at the Food and Drug Administration describes the proposed risk-characterization framework that can be used to evaluate, compare, and communicate the public-health consequences of decisions concerning a wide variety of products. The framework presented in this report is intended to complement other risk-based approaches that are in use and under development at FDA, not replace them. It provides a common language for describing potential public-health consequences of decisions, is designed to have wide applicability among all FDA centers, and draws extensively on the well-vetted risk literature to define the relevant health dimensions for decision-making at the FDA. The report illustrates the use of that framework with several case studies, and provides conclusions and recommendations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!