THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

National Research Council

Division on Earth and Life Studies

Board on Environmental Studies and Toxicology

500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 2347 Fax: 202 334 2752

February 17, 2009

Dr. Kathleen M. Koehler

Science Policy Analyst

Office of Science and Data Policy

U.S. Department of Health and Human Services

Hubert Humphrey Building, Room 434E 200 Independence Avenue, SW Washington, DC 20201

Dear Dr. Koehler:

At your request, the National Academies convened the Committee on Ranking FDA Product Categories Based on Health Consequences. The committee members were selected on the basis of their expertise in food safety, health economics, medical devices, vaccine safety, pharmacoepidemiology, biostatistics, comparative risk analysis, and decision analysis.

The committee was tasked with developing and applying a conceptual model to rank product categories in FDA program areas on the basis of health risks, both positive and negative aspects (that is, the committee was to consider beneficial aspects of the product categories in the context of possible adverse health consequences). The study was divided into two phases: selection of the model (phase I) and development, refinement, and application of the model to conduct a risk ranking of FDA product categories (phase II). The committee’s task is described in greater detail below. This letter report fulfills the task specified for phase I of this project.

The committee held two meetings. The first included a public session during which FDA staff and other invited experts made presentations. During that session, some indicated that a model that incorporates evaluations of interventions would be particularly valuable. The committee agrees but notes two complicating factors: evaluating baseline risks among product categories is a task of great magnitude and complexity, and it is the nature of interventions to be at the individual-product level and not the product-category or program level. Therefore, the model dictated by the committee’s task is not directly applicable to “intervention” analysis and cannot be used to evaluate strategies to reduce risk. However, the committee acknowledges that the existence of intervention capabilities is an important measure in determining risk, and model parameters will need to capture that aspect. Given the size and complexity of the task, the committee will attempt to keep the model as simple as possible, recognizing that rough estimates of risk may be all that is possible at the product-category level.

This letter report first provides background information on comparative risk analysis. Next, it outlines the conceptual model. Considerations regarding the product categories and their attributes are provided. The report concludes with a discussion of the steps needed to refine the model and conduct a



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
National Research Council 500 Fifth Street, NW Division on Earth and Life Studies Washington, DC 20001 Board on Environmental Studies and Toxicology Phone: 202 334 2347 Fax: 202 334 2752 February 17, 2009 Dr. Kathleen M. Koehler Science Policy Analyst Office of Science and Data Policy U.S. Department of Health and Human Services Hubert Humphrey Building, Room 434E 200 Independence Avenue, SW Washington, DC 20201 Dear Dr. Koehler: At your request, the National Academies convened the Committee on Ranking FDA Product Categories Based on Health Consequences. The committee members were selected on the basis of their expertise in food safety, health economics, medical devices, vaccine safety, pharmacoepidemiology, biostatistics, comparative risk analysis, and decision analysis. The committee was tasked with developing and applying a conceptual model to rank product categories in FDA program areas on the basis of health risks, both positive and negative aspects (that is, the committee was to consider beneficial aspects of the product categories in the context of possible adverse health consequences). The study was divided into two phases: selection of the model (phase I) and development, refinement, and application of the model to conduct a risk ranking of FDA product categories (phase II). The committee’s task is described in greater detail below. This letter report fulfills the task specified for phase I of this project. The committee held two meetings. The first included a public session during which FDA staff and other invited experts made presentations. During that session, some indicated that a model that incorporates evaluations of interventions would be particularly valuable. The committee agrees but notes two complicating factors: evaluating baseline risks among product categories is a task of great magnitude and complexity, and it is the nature of interventions to be at the individual-product level and not the product-category or program level. Therefore, the model dictated by the committee’s task is not directly applicable to “intervention” analysis and cannot be used to evaluate strategies to reduce risk. However, the committee acknowledges that the existence of intervention capabilities is an important measure in determining risk, and model parameters will need to capture that aspect. Given the size and complexity of the task, the committee will attempt to keep the model as simple as possible, recognizing that rough estimates of risk may be all that is possible at the product-category level. This letter report first provides background information on comparative risk analysis. Next, it outlines the conceptual model. Considerations regarding the product categories and their attributes are provided. The report concludes with a discussion of the steps needed to refine the model and conduct a 1

OCR for page 1
risk-ranking exercise. (There are also several attachments: a verbatim statement of the committee’s task, a committee roster and biographies, a bibliography, and acknowledgment of reviewers.) The report reflects the consensus of the committee and has been reviewed in accordance with standard National Research Council review procedures. Sincerely, Robert Lawrence, Chair Committee on Ranking FDA Product Categories Based on Health Consequences 2

OCR for page 1
LETTER REPORT ON THE DEVELOPMENT OF A MODEL FOR RANKING FDA PRODUCT CATEGORIES ON THE BASIS OF HEALTH RISKS BACKGROUND In 1986, U.S. Environmental Protection Agency (EPA) Administrator Lee Thomas asked 75 scientists and managers to develop a report on the “relative importance” of various environmental threats that were mainly in EPA’s jurisdiction. In 1987, the group issued Unfinished Business (EPA 1987), which categorized environmental threats in 31 problem areas, defined largely along existing programmatic lines. The group identified and divided the risks according to four important attributes with respect to the characterization of the 31 environmental problem areas: cancer risk, noncancer risk, ecologic risk, and “welfare” effects. The resulting report provided separate assessments (ranked low, medium, or high) for each of the four attributes. A key insight of Unfinished Business was that EPA's resource allocations appeared to be more in line with what the public perceived as the most important risks than with the priorities identified by the agency's experts. EPA asked its Science Advisory Board (SAB) to review the report, and the SAB released a follow-up report, Reducing Risk (EPA SAB 1990), which endorsed the broad comparative risk analysis (CRA) approach and produced findings similar to those in Unfinished Business. In addition to spawning many applications of CRA at the office, region, state, and local levels (Minard 1996; Jones 1997), the early CRA efforts led to questions about how best to facilitate comparisons and identify useful attributes for characterizing risks or risk-reduction opportunities. The EPA SAB noted that ranking risks or ranking the alternative actions that might be available for reducing risks would probably yield different rankings (EPA SAB 1990). In particular, although some risks might rank high, they might also be associated with very expensive or uncertain risk-reduction actions and therefore be unamenable to intervention according to cost-benefit criteria. In addition, if risks associated with some low-priority areas can be addressed effectively with certainty at low or no cost, their low- priority status should not prevent these “bargains” from being recognized. Progress on CRA method development continues, although its use remains relatively limited. Finkel and Golding (1995) noted that the “comparison of risks involves values in at least five areas: defining what we mean by ‘risk’; selecting the endpoints to consider; categorizing the risks for comparison; selecting a time frame for evaluating the adverse effects; and gauging the seriousness of the consequences.” In February 1994, a workshop organized by Resources for the Future for the President’s Office of Science and Technology Policy brought together researchers in CRA with the goal of developing a systematic process for comparing risks among different federal agencies (Davies 1996). As part of that work, researchers from Carnegie Mellon University developed a framework for ranking risks that included both quantitative and qualitative measures of relevant programmatic attributes (Fischhoff 1995; Morgan et al. 1996). They included health-impact measures (such as morbidity and mortality) and psychometric measures that research shows play an important role in the evaluation of risks (such as fairness, scientific understanding, and uncertainty). That work spawned a series of research projects and papers that refined and applied the framework (e.g., Morgan et al. 1999, 2001; Long and Fischhoff 2000; Morgan et al. 2000; DeKay et al. 2001; Florig et al. 2001; Willis et al. 2004, 2005; Fischhoff 2006; Gutiérrez et al. 2006; Bronfman et al. 2007, 2008a,b), including a discussion directly related to food safety (DeKay et al. 2005). Recently, those risk-ranking methods have been adopted by a variety of national and international entities. For example, the U.S. Army Corps of Engineers is using the methods to rank hurricane mitigation opportunities on the Louisiana Gulf Coast (USACE 2008), researchers at the University of North Carolina School of Public Health and RAND Corporation are using them to develop an environmental-health strategy and action plan for the United Arab Emirates (UNC 2008), and the British government is using them to communicate with and gather information from the general public on health-related priority-setting strategies (HM Treasury 2004, 2005a,b; OGC 2008). Regardless of the application, such projects share the goals of collecting and presenting risk information in a systematic 3

OCR for page 1
manner to guide and assess informed judgments. After assessment, those judgments may serve as a valuable input into a decision-making process focused on evaluating difficult policy choices. STATEMENT OF TASK AND COMMITTEE’S APPROACH In light of the increased use of CRA by federal agencies, the Department of Health and Human Services (DHHS) and the Food and Drug Administration (FDA) asked the National Research Council to convene an expert committee to develop and apply an evidence-based conceptual model and methods for ranking categories of products addressed by FDA programs. The conceptual model and methods were to focus on ranking product categories according to the ranges of magnitude of various potential health consequences to U.S. users of the products at the individual level and the population level, taking both adverse and beneficial effects into consideration. To accomplish its task, the committee was to include the following activities: consult with the sponsor to select FDA product categories to be ranked; consider products currently in use and near-term future products expected to come under FDA purview; review selected scientific literature bearing on adverse and beneficial health consequences; consider the scientific literature broadly to include social-science and economics literature, gray literature, and regulatory-policy literature; seek opportunities to assess health consequences in a way that allows results to be compared among broad product categories; identify information needed to address key uncertainties; assess the performance of the evidence-based model for ranking the selected product categories and identify next steps for model refinement; and where applicable and feasible, consider the potential effects on population health if risk-reduction strategies curtail the beneficial use of products. The committee was asked first to produce a brief letter report that describes the scientific conceptual framework to be used to rank product categories (phase I) and then to perform ranking exercises by using the proposed conceptual framework (phase II). In neither phase was the committee to recommend regulatory strategies; those choices entail policy judgments that transcend scientific and technologic considerations. This letter report fulfills phase I of the project. To accomplish its task, the committee held a public session at its first meeting, during which it heard presentations from FDA staff in the various program areas and from experts in the fields of decision analysis and CRA. The committee reviewed numerous scientific publications on CRA and literature provided by FDA. On the basis of its review and the statement of task, the committee selected a model that has the capacity to evaluate multiple product categories and compare them; to evaluate the magnitude and variation in distribution of both favorable and unfavorable effects; to improve FDA’s discharge of its responsibilities as they affect public health; to evaluate new product categories, risks, benefits, and other considerations; and to include multiple non-health-related outcomes of interest, such as equity and the quality of scientific understanding. The committee recognized that the model should be able to function to the greatest extent possible with sparse information. Although the primary focus of the committee was human health, the model considers animal health and welfare to be consistent with the full scope of activities conducted by FDA. The CRA exercise requested in the statement of task is a valuable tool in determining relative risks among product categories, but such exercises are not sufficient to guide many policy decisions unless they incorporate additional concerns. For example, the absolute risk in a category may not be a good indicator of the potential to reduce risk in that category or of the potential to reduce risk by any specific action. Measures of the potential benefits of specific actions are critical for resource-allocation decisions. Likewise, the presence of a health risk may or may not be associated with the economic costs or benefits of addressing the risk, the equity concerns (who pays and who benefits), the likelihood and timeframe of achieving the stated risk reduction, or the public’s perceptions of the risk-mitigation options. To the extent that any of those concerns are or should be important in making policy decisions, the proposed CRA alone would not be sufficient for making decisions. On the basis of the guidance that the committee received from the statement of task and from clarification offered by the sponsor during its first public meeting, it concluded that discussion of the merits of other theoretical frameworks that might 4

OCR for page 1
be valuable in assessing risk-mitigation alternatives was outside its task. Baseline ranking of risks is a necessary but not sufficient step in the more general decision-making process. RANKING MODEL Conceptual Framework On the basis of recent literature, the committee concluded that the best approach to ranking FDA product categories on the basis of health risk is to use a conceptual framework similar to that described by Florig et al. (2001). Unlike other comparative methods (such as the World Health Organization’s Global Burden of Disease Study [Murray and Lopez 1996; WHO 2008], which ranks solely on the basis of utility loss from illness), the approach described by Florig and colleagues allows for disparate items, such as cosmetics and vaccines, to be ranked, as will be necessary for FDA. Furthermore, although this is not explicitly required by FDA, the selected approach is designed to accommodate qualitative and quantitative variables in the formal ranking process. That will facilitate inclusion of important variables and may greatly improve the utility of the proposed approach for FDA. Figure 1 summarizes the two phases and the multistep process envisioned by the committee. As illustrated in Figure 1, steps A and B involve defining the FDA product categories to rank and identifying the risk attributes to describe the categories, respectively. In phase I of this project, which is summarized in this letter report, the committee has proposed a preliminary list of categories and attributes. The final determination of the categories and attributes will require further input from knowledgeable FDA staff, iteration, and refinement and will be completed in phase II of this project. The committee’s final report will discuss the final categorization and identification of attributes and the process used to make those determinations. Step C requires describing the categories in terms of the attributes, step D involves performing the risk-ranking exercises, and step E involves summarizing and evaluating the results of the risk-ranking process. Steps C-E will be accomplished in phase II of this project, the results of which will be described in the committee’s final report. Each step in the multistep process is described in greater detail in the sections that follow. Phase I Phase II PRELIMINARY FINAL Step A: Define the Step A: Define the FDA product FDA product categories to be ranked categories to be ranked Step E: Describe Step C: Describe Step D: Select the issues the categories in participants and identified and terms of perform the risk the resulting attributes rankings rankings PRELIMINARY FINAL Step B: Identify the risk Step B: Identify the attributes that should risk attributes to be be considered for each considered for each of of the product the product categories categories FIGURE 1 Framework of health-risk ranking model. Source: Adapted from Florig et al. 2001. Reprinted with permission; copyright 2001, Risk Analysis. 5

OCR for page 1
Uncertainties in the risk-ranking model will be captured in two ways. For some attributes, such as those measuring mortality risks, a quantitative approach can be used by providing a mean (or median) with population upper and lower bounds. Other attributes can be selected to represent uncertainties with a qualitative description. For example, an attribute that captures the quality of scientific understanding will indicate knowledge gaps, which will lead to uncertainty in the ranking exercise. The committee recognizes that successful ranking will require both iteration and further engagement with FDA. Step A: Defining the Product Categories Any ranking process must begin with development of a list of the risk items to rank. Many approaches exist to categorize risk, and as Morgan et al. (1996) note, “no single categorization scheme is likely to serve all … needs.” Morgan et al. (2000) describe criteria for defining categories and state that categories should be “exhaustive so that no relevant risks are overlooked,” “mutually exclusive so that risks are not double-counted,” and “homogenous so that all risk categories can be evaluated on the same set of attributes.” Furthermore, the categories should be relevant to the organizational structure, legislative mandates, and risk-management activities of the organization. Among the other criteria listed by Morgan et al. (2000) is the goal of keeping the number of categories to a number that makes the risk- ranking exercise feasible. Depending on the techniques used during the ranking, a feasible number would generally be in the range of 15-30. Given the number of products that FDA regulates, the committee recognizes that the number of categories could be expanded too much and make risk-ranking impossible. That means that the task of ranking products for FDA as a whole must of necessity focus on highly aggregated product categories. FDA provided the committee with an initial list of product categories, which is shown in Table 1. The list includes 28 categories, and it primarily mirrors FDA’s organizational structure and statutory and regulatory authorities (that is, it is broken down according to FDA’s existing five product-focused centers). Although the committee noted that the categorization could alternatively focus on type or magnitude of hazard, it concluded that the final selection of the product categories for ranking will require further input from FDA staff who have specific expertise in the FDA products. Valuable input from FDA will include data on the size of each potential category (for example, with respect to numbers of regulated individual products or firms and relative market sizes in dollars). The committee expects that some of the categories will expand, others will contract, and some will be substantially revised. For example, the committee questioned the product categories suggested for medical devices. Currently, medical devices are defined according to risk and classified as class I, II, or III devices. Accordingly, this scheme may be more appropriate for categorization of medical devices. Furthermore, the committee notes that the present list includes categories at different levels of specificity, and this could obscure the value of the ranking exercise. The committee will determine whether additional categories will be needed to address near-term future products, such as nanomaterials, or whether those products can be integrated into the existing categories. Using the criteria of Morgan et al. (2000) discussed above and input from FDA, the committee will be able to determine the most appropriate product categories for evaluation in phase II of this project. Step B: Identifying the Attributes Ranking risks requires identifying the important attributes of the risks. Morgan et al. (1996) described criteria for selecting attributes and noted that attributes should be comprehensive, non- overlapping, stand-alone, measurable, and minimal to reduce the complexity of the risk-ranking exercise. As a preliminary scoping exercise, the committee selected five attribute groups related to exposure, severity of effect, ability to anticipate and prevent adverse events, ability to mitigate adverse events, and benefits of products or product categories. Each group contains multiple specific attributes, which are shown in Table 2. The committee emphasizes that Table 2 is only a preliminary list. Overlapping 6

OCR for page 1
attributes must be explicitly noted to avoid double-counting in the risk-ranking exercise, and most important, attributes must be selected that are applicable between and within the broad FDA product categories. That exercise will be challenging and will require input from FDA staff who have specific expertise in FDA product categories. The committee will use the criteria of Morgan et al. (1996) discussed above and further input from FDA to finalize the list of risk attributes in phase II of this project. At the conclusion of this process, the attributes will be clearly defined and well understood by FDA staff and the risk rankers. The committee defined exposure as the condition of being subject to some effect or influence and considered the five risk attributes shown in Table 2 to be appropriate for quantifying or describing it. The exposed population is the percentage of the U.S. population potentially exposed. Cumulative incidence is the number of new cases of illness, injury, or other health-related events attributable to an exposure during a specified period in a specified population and is expressed as a rate. Prevalence is the number of cases of a health-related state or event that exist in a specified population at a particular time, regardless of when they began or how long they have existed, and can be expressed as a rate. Vulnerable groups refer to people who have increased susceptibility to adverse outcomes because of genetics, age, socioeconomic status, occupational or environmental exposure, or physiologic state; this attribute could be described in terms of the number and size of vulnerable populations. Cluster refers to a group of people who are at excess risk for adverse events that are related temporally, by proximity, or by source; this attribute could be described in terms of group size. TABLE 1 Suggested Initial List of Product Categories for Ranking Provided by FDA Program Area Product Categories Food, cosmetics, and dietary supplements Food Produce Eggs and dairy Processed food Seafood Cosmetics Dietary supplements Food and color additives Drugs and biologics regulated as drugs Over-the-counter drugs Diagnostic prescription drugs Preventive prescription drugs Prescription drugs that are life-sustaining Prescription drugs for treatment for symptoms or improvement in quality of life Prescription drugs that are used cosmetically Biologic products other than those regulated as drugs Vaccines Blood and tissue products In vitro diagnostics related to donor testing Devices regulated as biologics Allergenics Cell and gene therapy Veterinary products Approved animal drugs Unapproved animal drugs Animal feeds Pet food Medical devices and radiation-emitting products Critical devices for professional use Noncritical devices for professional use Noncritical devices for lay use Nonmedical radiation-emitting devices Patient cables and lead wires 7

OCR for page 1
TABLE 2 Risk Attributes for Model for Ranking FDA Product Categories on the Basis of Health Risk Attribute Groups Risk Attributes Exposure Exposed population Cumulative incidence Prevalence Vulnerable groups Cluster Severity of effect Mortality Morbidity Vulnerable groups Catastrophic event Diffusion effects beyond intended usea Animal health Ability to anticipate and prevent adverse events Quality of scientific understanding Availability of substitutes History of problems and corrective actions Availability of quality standards, guidelines, or standard operating procedures (quality assurance and quality control, good manufacturing practices) Variability in product composition or performance Vulnerability of supply chain Ability to mitigate adverse events Availability of substitutes Availability of corrective actions Traceability Latency Ability to recall Reversibility Product benefits Mortality reduction Morbidity reduction Efficacy and effectiveness Animal welfare a Diffusion effects is an attempt to capture effects on people who do not use the product. Severity of the effect is described by six attributes as shown in Table 2. Mortality could be measured as expected number (or range of numbers) of deaths per year in the United States attributable to the product category. Morbidity could be quantified by one or more of the following: number of doctor visits per year, number of hospitalizations (or hospital days) per year, number of lost work days per year, total cost of treatment, and number of chronic cases per year. Metrics need to account for acute and chronic illnesses. The committee discussed using quality-adjusted life years (QALYs) to capture the differences but concluded that the method might be difficult to apply to something as heterogeneous or complex as the product categories given the expected paucity of data. However, if data are available, QALYs should be included in the risk-ranking model. Vulnerable groups could be represented by the percentage of all deaths that occur in vulnerable groups or the percentage of deaths in a vulnerable group that are attributable to the product category. Catastrophic event refers to a low-probability event with the potential for a severe outcome and could be characterized by the number of deaths in a worst-case scenario. The attribute diffusion effect is an attempt to capture effects on people who do not use the product (for example, transgenerational effects resulting from product use); this attribute could be characterized by a minor-moderate-major significance descriptor. Animal health could be measured as the number of animals that die from use of animal products; food animals and companion animals could be evaluated separately. The committee considered six specific attributes to characterize the ability to anticipate and prevent adverse events. Quality of scientific understanding is related to product knowledge, that is, 8

OCR for page 1
background information available on a product, the number of scientific studies, and the quality of the studies. The availability of substitutes attribute is related to how critical a product is. The attribute history of problems and corrective actions reflects the availability of information on such events as adverse reactions and manufacturing defects and whether actions have been needed to address problems. The attribute availability of quality standards, guidelines, or standard operating procedures attempts to capture the idea that manufacturing products in accordance with standards, guidelines, or standard operating procedures improves FDA’s ability to control product quality and minimize adverse events (for example, the availability of practices to produce uniform, high-quality products and the limits of the practices). Variability in product composition or performance refers to inherent differences in products that result even when good manufacturing practices are adhered to or that result from intended or unintended deviations from good manufacturing practices, including inadvertent contamination, equipment malfunction, or deliberate adulteration. Variability in performance is also related to person- person differences in response to a product, such as the side effects experienced after taking a drug; such variability may be related to individual characteristics or to how the product is used. Finally, vulnerability of the supply chain refers to the potential disruption in manufacture or distribution of a product if there are any problems with obtaining inputs to any step of the process. All the risk attributes in this group could be defined by a binary yes-no descriptor or a high-moderate-low descriptor. Six attributes are used to describe the ability to mitigate adverse events. Availability of substitutes and corrective actions would provide methods to mitigate an adverse event by replacing a product with an equivalent or similar product, by modifying manufacturing or distribution processes, or by communicating with consumers about potentially harmful products. Traceability is the ability to identify the sources of all product components, affected products, or components in a supply chain, or people potentially exposed to products. Latency is defined as the interval between the first exposure and the observation of an adverse event; long latency would make it difficult to anticipate and prevent adverse events. Ability to recall is the ability to remove from the supply chain a product that is identified as carrying unacceptable risks. Reversibility is the amelioration of adverse events. Each of the attributes could be defined with a yes-no descriptor or a high-moderate-low descriptor. The final attribute group attempts to incorporate the beneficial aspects of products to the general and target populations. Some benefits are linked to protecting the population from adverse consequences; some are linked to diagnosing, treating, or preventing disease; and others are linked to promoting health, such as those related to nutrition. The benefits may be best quantified by expected reduction in mortality (that is, the number of lives saved per year in the United States from intended use of the product category) and by expected reduction in morbidity (for example, the number of disabilities or of hospitalizations avoided per year in the United States). The efficacy and effectiveness attribute captures how well the current program is working (for example, Is it cost-effective?). Animal welfare tries to capture the idea that food production animals are important food sources and contribute substantial benefits to the U.S. economy and that companion animals enhance quality of life and may provide some health benefits. Products that enhance the quality of life of the animals would provide a benefit to society. A high- moderate-low descriptor could be used to characterize that attribute. The committee emphasizes that capturing beneficial aspects of product categories will be a challenging task. Some benefits are obvious and often discussed, and others are less well known, especially if products have long-term consequences. Many benefits will be difficult to quantify or rank, such as those related generally to improving quality of life or social values by providing people with a variety of choices or with novel products. Because it is highly desirable to have an attribute table that allows easy side-by-side comparisons, there should be fewer than 20, and ideally fewer than 15, attributes. Winnowing the attribute list down will require merging some of the attributes shown in Table 2. For example, the “Ability to anticipate and prevent adverse events” set, which currently has six specific attributes, will probably contain only one or two in the final version. If those six attributes were all described by using binary variables, 64 combinations would be possible. Given that only 15-30 product categories will be ranked, that number of attributes overspecifies the ranking task. Winnowing of the attribute list will occur naturally as overlapping attributes are highlighted and preferred attributes are selected. In some cases, entire 9

OCR for page 1
attributes groups could be eliminated (for example, is the “exposure” group necessary if the “severity of effect” group contains mortality and morbidity rates and counts?). The committee notes that for each product category the risk rankers will receive a summary “pamphlet” that will include the attribute table (which will display all the attribute values in a consistent format compatible with easy side-by-side comparisons) and supporting information that helps to put the attribute values in context. Because the attributes are selected to be universal and applicable between product categories, there are relevant details that cannot be captured in a simple listing of quantified attributes. For example, the amplifying text could include descriptions of the vulnerable groups relevant for the product category, the extent to which life-cycle calculations were completed or limited, or animal impacts that were not explicitly shown in the attribute table. The supplemental information will be concise and organized in a standard manner so that comparisons between product categories are possible. Step C: Describing the Product Categories by Using the Attributes In this initial phase of its work, the committee has not reached formal recommendations for the follow-on steps in the framework, but it recognizes the difficulties ahead, particularly in completing step C, in which the values of each attribute (step B) must be described for each product category (step A). In some ranking exercises, the items for ranking are unique and narrowly defined. In this case, however, the broad product categories, such as vaccines, contain many individual products, each with their own attribute values. Thus, aggregation will present important challenges with respect to attribute characterization. For some attributes, such as those measuring mortality risks, a summation across all items in the product category may capture the relevant values, with upper and lower bounds on the quantified values providing an indication of the uncertainties in the calculations. However, because of the heterogeneity of some of the product categories, difficulty will arise in assigning unique values for many attributes. It may be desirable to split large product categories to improve homogeneity of ranking (for example, vaccines might be divided into killed vaccines and live vaccines). For some attributes, aggregation may have the effect of making them nonvariant among the set of items being ranked. That problem could be mitigated by focusing the attributes in question on the mechanisms that lead to the greatest concern for morbidity and mortality. Thus defined, the attribute would distinguish product categories with well-established cause-effect linkages from categories that are less well understood. The necessary refinement of the attribute definitions will not be possible until the committee can re-engage the necessary FDA personnel through follow-on workshops in phase II of this project. Discussions could lead to a regrouping of the product categories and attribute description. The process by which the first three steps are completed will generate valuable information. In fact, the committee emphasizes that the process that it undertakes with FDA is likely to be a more important tool and provide more insights than the list of ranked product categories alone. Creating robust attributes that are systematically and accurately measured across the wide array of FDA regulatory responsibilities will focus agency thinking and help to communicate the considerations underlying agency decision-making to a broad range of stakeholders. Steps D and E: Conducting the Ranking and Analyzing the Results Once the attributes are fully defined for each product category, the ranking will begin, probably with at least two approaches: a holistic ranking based on the rankers’ overall preferences and a ranking based on application of a formal mutiattribute model. The committee anticipates selecting the ranking procedures after further discussion with FDA to learn how it plans to use the rankings. In a holistic ranking, the preferences for, importance of, and trade-offs between the attributes are not made explicit by the rankers. The rankers carefully review the summary material and, using their own judgment, rank the product categories. Guidance as to how to structure the ranking procedure can make the task more manageable. For example, guidance provided could be to complete a rough sorting into 10

OCR for page 1
three preliminary categories—high, moderate, and low risk—before completing the series of pairwise comparisons. In contrast, the formal multiattribute model approach requires the rankers to consider each attribute and their range of values and to state explicitly the relative trade-offs between attributes through the elicitation of attribute weights. A model is built for each ranker by using those weights, and a ranking of the product categories is calculated. Sensitivity analyses are performed on the ranks to determine the influence of the assessed weights and the value functions. Although the two basic approaches to complete the rankings differ substantially, there are advantages in using both approaches in a single exercise (Palmgren et al. 2000). With a holistic approach, overlapping attributes can be considered (that is, the combination of attribute values may provide insights into relevant details); however, with a formal multiattribute model, double counting is not allowed, and attribute weight assessment must reflect this requirement. The results of using one approach can improve the understanding of the other, and ultimately a revised ranking informed by the results of both is likely to be preferred by the rankers. Analyses conducted after the ranking can determine the relative importance of the two approaches in generating a ranker’s final ordering. Developing a satisfactory multiattribute model for the ranking task has many potential benefits, including the ability to add new or revised product categories and to see quickly how they fit among the categories already ranked. Once again, understanding how FDA will use the ranking results will determine the preferred approaches. The rankings could be used internally by senior administrators as input into their own strategic planning or as a way to capture public perceptions and communicate policy to the general public. Both applications offer merit but would certainly dictate different approaches to documentation and communication of the process. Regardless of the specific uses, determining the level of agreement among the rankers could provide valuable insights that are not now available to FDA. NEXT STEPS The committee has proposed a conceptual model for ranking FDA product categories and has suggested preliminary categories and risk attributes. The committee emphasizes that participation of FDA staff in each program area is essential for the development of a successful and useful model. Development of the model will require a series of three workshops. The first workshop will involve discussion of the model’s product categories and attributes that will lead to refinement, revision, and adoption of both for further use in steps C-E. This iterative cycling between product categories and attribute definitions will ensure that the follow-on steps can be completed. In addition, the engagement of FDA personnel will serve as an educational opportunity so that the model, its application, and its limitations can be understood and appreciated. The second workshop will involve populating the model matrix with data to allow the ranking exercises to be performed and will require much effort on the part of FDA personnel who will be responsible for gathering data and providing input values. Populating the model matrix will almost certainly require the determination of values that have not been estimated previously and rely on the use of subjective expert judgment. As indicated above, the difficulty in providing values will probably vary considerably among product categories. The third workshop will involve conduct of the actual ranking exercises. The committee emphasizes that the development of the model is an iterative process as reflected in Figure 1. Findings from any of the three workshops may necessitate adjustments and refinements of earlier steps in the model. The committee’s final report will summarize and evaluate the outcome of the workshops and provide recommendations for using the risk- ranking model as an input in a decision-making process. The committee notes that the risks and benefits vary substantially between and within product categories, and that will pose a challenge in developing and implementing the ranking process. However, that challenge makes the ranking exercise and resulting ranking valuable for FDA because unaided comparisons also face the challenge of comparing apparently incomparable product categories but without the common metrics that the committee will be recommending. Therefore, the ranking exercise is a logical first step for FDA. The committee provides an outside perspective on the challenge of 11

OCR for page 1
comparing risk among disparate categories, and its recommendations, with input from FDA, will help to identify a framework for making advances in FDA management processes and decisions. Attachments: A Statement of Task B Committee Membership C Bibliography D Acknowledgment of Reviewers 12