Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 139
7 Conclusions and Future Directions The committee applied its risk-characterization framework to four case studies, each of which was based on decision scenarios provided to the commit- tee by the Food and Drug Administration (FDA). For each case study, the com- mittee illustrated how its framework could be applied; it defined the specific decision options to be compared and developed a risk-attribute table to charac- terize the public-health consequences of the alternative decisions; and it pro- vided a discussion of how the risk characterization could be used to support the specific decision options being compared. In doing so, the committee relied on members’ expert judgments and data gleaned primarily from publicly available literature and databases. This chapter presents the committee’s perspective on the challenges and the lessons learned from its experience in applying the framework to the case studies. General conclusions and suggestions for future directions are provided at the end of the chapter. LESSONS LEARNED FROM THE CASE STUDIES Value of Discussion and Multiple Points of View The development and analysis of each case study in this report benefited greatly by involving both subject-matter expertise and decision-analysis exper- tise. The decision focus of the framework, specifying and comparing the out- comes of specific decisions, did not come naturally to committee members who had more detailed scientific expertise related to FDA products and product cate- gories. They were initially inclined to look more broadly at the effects of the product being considered, and some iteration and detailed discussion were nec- essary to narrow the focus of discussion to the comparison of specific options. For example, in discussions about the evaluation and comparison of the risks associated with various food products (Chapter 4), the committee was initially tempted to include a discussion of the health benefits of each food product as 139

OCR for page 139
140 A Risk-Characterization Framework for Decision-Making at FDA well as the risks from each. After focusing on the decisions that the comparison might be used to support (that is, allocation of food-inspection resources), the committee determined that the focus should be on the risks and, more specifi- cally, on the risks that could be averted by improved or more rigorous inspec- tions. The committee notes, however, that the targeting-decision case study did not explicitly compare the health consequences of the current inspection proc- esses with those of changes in the inspections; if that decision were to be evalu- ated, additional steps would be necessary. The decision analysts on the committee were able to focus the subject- matter experts on a relatively constrained decision context, to identify the se- quence of information needs, and to assist the subject-matter experts in making judgments about the array of possible effects on the basis of sparse data. The decision analysts, of course, could not provide the specialized and detailed knowledge necessary to identify and recognize the most relevant data for a spe- cific decision context. The committee was hampered in one case study (the ef- fects of potential melamine contamination of infant formula) by the lack of de- tailed subject-matter expertise among the committee members; as a result, the committee had much less confidence in the estimates of the risk attributes of the case study than in the estimates of the other three case studies. In all case studies, the discussions and interactions between committee members with different backgrounds and expertise were critical for the use of the risk-characterization framework. On the basis of its experience, the commit- tee concludes that FDA will benefit from including multiple stakeholders in its decision-making process, from defining decisions to gathering information and ultimately formulating conclusions. Just as shared decision-making (Charles et al. 1997) is beneficial for medical treatment decision-making (including infor- mation-sharing and consensus-building), it will be beneficial for FDA strategic decision-making. Defining the Decision Context The committee found that it was critical in each case to define the decision options to be evaluated and compared clearly, so that appropriate risk informa- tion for the decision-making process could be obtained. In all cases, decision- analytic structuring was used to organize thinking about the decision context. Analytic reasoning and basic structuring tools, such as influence diagrams (see, for example, Figure 3-2), were used to identify the various factors that needed to be considered to develop estimates of the public-health consequences of the al- ternative decision options. For mitigation-selection decisions, as illustrated with the vaccine- withdrawal case study in Chapter 3, defining clear and distinct decision options to be compared was straightforward. Although the example was deliberately chosen to be a simple comparison of a yes-no variety, it would have been easy to

OCR for page 139
Conclusions and Future Directions 141 expand the set of options being considered to include more nuanced options and to make it a more complex example. For targeting decisions, as illustrated with the evaluation of three food categories in Chapter 4 and the evaluation of melamine testing in Chapter 6, defining the decision context and the options to be compared was more com- plex. In fact, the food case study focused on comparing the health consequences associated with consumption of the different food categories but stopped short of evaluating different resource allocations. The evaluation and comparison pre- sented in Chapter 4 could be used to support a risk ranking or could be used as one input into a targeting decision. For example, if FDA were deciding where to target additional food-safety inspection resources, understanding the public- health consequences as characterized in Chapter 4 would be an important input. As described in Chapter 2, for targeting decisions, the options or alternatives theoretically available to FDA are vast; virtually any amount of a resource could be allocated to the identified products or product categories and is constrained only by the total resources available. However, before substantial time and effort are invested, the many options possible need to be narrowed judiciously, and such narrowing will necessarily involve input from FDA management in addi- tion to the technical staff. Finally, strategic-investment decisions, as illustrated by the evaluation of enhanced surveillance of medical implants in Chapter 5, proved the most diffi- cult to formulate and evaluate with the framework. In theory, defining the op- tions for this case study—current surveillance compared with enhanced surveil- lance—was simple. In practice, however, the committee members had to speculate about the details of what an enhanced surveillance program would entail to enable them to estimate its effects. That proved to be a difficult task and one that clearly had substantial effect on the estimates derived. As described by FDA (Bertoni 2010), strategic-investment decisions are typically long-term ca- pacity-building investments. Characterizing the Public-Health Consequences of Each Option For each case study, various tools were used to develop the estimates nec- essary to characterize the public-health consequences and populate the attribute tables shown in each case study. In some cases, the quantities of interest could be estimated directly from available data; for others, several steps—some with considerable uncertainty—were required to generate estimates. In simple cases, exploratory descriptive statistics and bounding analysis were used. For example, estimates of the number of people exposed to the risk of foodborne illness caused by pathogens in leafy greens required an estimate of the number of peo- ple who consume leafy greens in a year, which could be based on readily avail- able information on food consumption. In more complex cases, a series of esti- mates and relatively complex calculations were used to derive estimates for the attribute table. For example, estimating the number of deaths that might occur

OCR for page 139
142 A Risk-Characterization Framework for Decision-Making at FDA from melamine contamination of infant formula required estimates of the frac- tion of infant formula potentially contaminated, the concentration of melamine in that contaminated formula, the amount of contaminated formula consumed by an infant (which varied with age), the estimated dose received by infants con- suming contaminated formula, and an estimate of the dose-response relation- ship. To develop the final estimate, the committee had to identify and structure the various factors and their relationships, estimate each of the critical factors (by using a combination of descriptive histories, bounding analyses, and judg- ment), and calculate the resulting numbers (in this case, using Monte Carlo simulation methods). The risk-characterization framework provides guidance on the estimates that are necessary to compare decision options but not on how those estimates are to be developed. In working through the case studies, the committee encoun- tered several challenges that FDA will also face in applying this framework. Some of the challenges are discussed below. Challenges in Finding and Interpreting Data The success of the proposed risk-characterization framework depends on the ability to populate the attribute table. Common challenges among all case studies were finding and interpreting data to support the required estimates. In the vaccine-withdrawal case study described in Chapter 3, for example, deter- mining the excess risk of intussusception attributable to the vaccine was diffi- cult; there were few data on the background rate of intussusceptions and little information on whether the rotavirus infection might cause intussusceptions in some cases. There was also speculation at the time that the cases of intussuscep- tions occurring after vaccination would have happened anyway: that is, the causal relationship between the RRV-TV vaccine and intussusception was speculative. For the food case study described in Chapter 4, there were several data challenges. The industry segment is so large and diverse that information on volumes, producers, and distribution is not readily available. The committee chose a simple measure of the size of the exposed population (the number who consume any of the product over the course of a year) partly because more de- tailed data about annual consumption and consumption quantities were not read- ily available. An additional complication for many food categories is the lack of morbidity and mortality data. Although information exists on the estimated number of illnesses, deaths, and hospitalizations because of foodborne patho- gens generally, no direct data exist on the attribution of those illnesses to spe- cific commodities. The committee used other sources of data and made a num- ber of assumptions to support estimates of the attribution of the illnesses to the specific food categories. In the melamine case study described in Chapter 6, the committee was se- verely hampered by lack of data. The committee notes, however, that the lack of

OCR for page 139
Conclusions and Future Directions 143 data reflects the reality of the situation being evaluated; at the time of the case study, virtually no information was available on the concentrations of melamine in various products in the United States. Regardless, the committee was able to estimate attribute values by using the available data, assumptions, and judg- ments and to produce a table that would have been helpful for decision-making. The case study on strategic-investment decisions described in Chapter 5 highlighted some additional challenges. Although lack of data is clearly a prob- lem, inaccuracies in the available data are barriers to accurate evaluations and make it difficult to identify newly emerging risks. Furthermore, when data are difficult to obtain because they are in multiple locations and in inconsistent for- mats, developing the required estimates is again hampered. Thus, having data in a format that will support decision-making is clearly advantageous. For exam- ple, in the case study on strategic-investment decisions, the committee observed that a simple count of adverse event reports in the databases (MAUDE and MDR databases) does not yield a suitable estimate for determining the probabil- ity of an adverse event. There is potential for both over-reporting and under- reporting in the information contained in those databases, and there is no infor- mation on the total number of devices implanted. Furthermore, it is unclear in the reported data whether an adverse health effect suffered by a patient who has a medical device is a result of the device or is a result of some cause unrelated to the device. Use of Expert Judgment Expert judgment and data were inextricably intertwined in the commit- tee’s approach to each case study in this report. In some case studies, the com- mittee did not have much direct information; in others, a large variety of data were available. In all cases, assumptions were necessary about how to interpret the data to complete the risk-attribute table. Among the case studies, evaluating the potential strategic-investment decision of enhanced postmarket surveillance of implanted medical devices proved challenging with respect to data and the need to rely more heavily on “pure” expert judgments. Those challenges arose partly from the fact that the specifics of the enhanced surveillance system had to be hypothesized, and it was not clear precisely what new information would be attained or how it might be used. As discussed in that case study, for example, it is clear that better information would reduce uncertainty in the estimated num- ber of adverse health effects (which would reduce the range between the 5th and 95th percentiles), but it is not possible to estimate what the new range would be before collecting the information. Some type of expert judgment is always required in evaluating and com- paring the potential outcomes of different decisions. Within the risk- characterization framework, decision options are to be evaluated and compared on the basis of whatever type, quantity, and quality of data are available when the decision must be made. In some cases, detailed peer-reviewed risk analyses

OCR for page 139
144 A Risk-Characterization Framework for Decision-Making at FDA might be available; in other cases, one may need to rely primarily on expert judgments. That flexibility allows risk information to be considered by decision- makers for any risk-relevant decisions even if detailed quantitative risk analyses are not available. The framework provides a structured way to document the data and the associated expert judgments clearly; as the framework is used more extensively, some of the analyses and data sources used for earlier studies can be leveraged to make related studies less burdensome, although some new data and new expert judgments will probably be required. Using the Risk Characterization to Support Decision-Making The risk-attribute table provides a succinct comparison of the decision op- tions that were evaluated and should be useful to decision-makers interested in understanding the key differences in the public-health consequences of those options. The comparisons alone, however, are not likely to provide all the deci- sion-relevant information that decision-makers and policy-makers need to con- sider, nor are they intended to do so. The focus of the framework is to enable a comparison of the potential public-health outcomes of different decisions and to provide a common language for discussing those consequences within FDA. The committee concludes that such risk information is relevant to many FDA decisions and that clear characterization of the consequences will lead to more consistent consideration of those issues. As discussed in Chapter 2, however, the committee clearly recognizes that many other factors must be considered by FDA in its risk-management decisions. The case studies illustrate that careful examination of the attribute table may lead to clear conclusions about the relative public-health consequences of different options, as in the mitigation-selection case study in which one option dominated the other. That will not always be the case; the summary table may simply highlight that one option is better on some attributes but worse on other attributes than the alternative, as in the comparison of food categories. In the latter case, the FDA may ultimately want to consider more formal approaches for weighing the tradeoffs among the different risk attributes to determine which option, on balance, would be preferred in terms of public-health consequences alone, as discussed in Chapter 2. Extending the Framework to Estimate the Value of Information When scientists conclude that more or better information is necessary and time and resources are available to obtain that information, the risk- characterization framework can be used to highlight what type of additional in- formation on public-health consequences would be most useful by using the decision-analytic concept of the value of information. As discussed in Chapter 2, new information is valuable only if it has the potential to change decisions and thus potentially improve outcomes (Clemen 1996). In a well-defined decision

OCR for page 139
Conclusions and Future Directions 145 context, value of information can be quantified, and the cost of data collection could be compared with its value. Two of the case studies provide some insight into the potential value of in- formation: the case study on a strategic-investment decision and the case study on a targeting decision spanning FDA centers. The former focused specifically on evaluating and comparing the public-health consequences of two levels of information collection (the current system vs an enhanced system). The latter included discussion of an extension of the case study to one in which alternative decisions would be explicitly included in the evaluation. In the context of the decision-relevant value of an enhanced postmarket-surveillance system, many changes in decisions may result from the gathered enhanced information, includ- ing possible device recalls, revised guidelines for patient selection or patient monitoring, and different device designs. If decisions to take different actions lead to different health outcomes (or lead to other decision-relevant aspects, such as operational efficiencies, public perception and trust of FDA, or political support for FDA activities), the enhanced system will have delivered informa- tion of value. Comparing the value of that information with the costs of collect- ing it is outside the scope of the present committee’s charge, but it could be done by FDA. CONCLUSIONS In developing the risk-characterization framework and conducting the case studies, the committee came to the following general conclusions: The committee found that framing the evaluation in a decision context was more straightforward and created an evaluation that would be more relevant for decision-making than simply conducting a risk ranking of products or prod- uct categories. The committee found that a risk-characterization framework could be developed with a relatively small number of attributes that are applicable within and among FDA programs. Those attributes can provide FDA with a common vocabulary for discussing risk-related decisions across centers and can be used as the basis of a consistent approach for including risk components in decision- making. There is a learning process for developing and refining the attributes, and comfort with the risk-attribute vocabulary grows over time. On the basis of its experience in developing the case studies, the com- mittee found that it is possible to characterize decision options by using the risk attributes and that they could be estimated by using existing data and expert judgment. The judgments that were required were not always easy, and commit- tee members were not always comfortable in making them, but in the end the committee concluded that the case studies would provide useful, relevant, and sufficiently accurate information to be of use to a decision-maker. The commit- tee recognizes FDA’s strong preference for “data” over “expert judgment” for

OCR for page 139
146 A Risk-Characterization Framework for Decision-Making at FDA obtaining estimates or making decisions. However, it is important to recognize that when a decision must be made immediately, the committee’s suggested approach can provide useful information about the public-health consequences of various options in a clear and consistent way on the basis of the best informa- tion available at the time the decision must be made. FUTURE DIRECTIONS As a result of its efforts to develop the risk-characterization framework and the case studies, the committee offers the following suggestions concerning implementation of the framework: FDA should consider using the concepts defined by the risk- characterization framework and particularly the risk attributes defined in the present report as a common language for discussing risk-related aspects of vari- ous decisions. In risk-related decisions, considering the outcomes of alternative decisions in terms of the attributes identified in the present report will begin to establish consistency in risk vocabulary throughout the agency and will build a base of understanding that will enable more detailed use of the approach for evaluating and comparing decision options in the future. As FDA begins to use the risk attributes and risk comparisons, such as those illustrated in the case studies for comparing decision options, it may find that some aspects of the method need to be modified. The committee believes that such modifications are entirely appropriate and that this approach should evolve to meet the agency’s needs as its staff gain experience in implementation of the approach. In its interactions with FDA, the committee came to recognize that in many cases the agency has a substantial amount of data but that the data are not collected, organized, or accessible in a format that is useful for supporting risk- based decision-making. More focus on developing and implementing structured decision processes that are based on clearly defined risk attributes and metrics will allow the agency to improve its approaches and mechanisms for collecting information. The committee emphasizes that simply collecting more data is not necessarily the best use of resources; collecting more relevant data and organiz- ing them so that they are useful for decision-making is the key. The committee acknowledges that new data-collection approaches and efforts will require in- formation management and technology support. The committee recognizes that precise predictions of the outcomes of different decisions based on the risk attributes may be difficult to develop. Data may be lacking, and scientists may be uncomfortable in making or even unwill- ing to make the necessary judgments to estimate the risk attributes. However, the committee emphasizes that decisions in which risk information could be valuable are made regularly and recommends that FDA use internal or external experts who are trained in and comfortable with decision analysis, risk assess-

OCR for page 139
Conclusions and Future Directions 147 ment, risk management, and specifically the assessment of uncertainties to fa- cilitate the use of the committee’s framework in its initial implementation. The committee recognizes that FDA will need specific expertise, train- ing, and staffing to implement the proposed risk-characterization framework consistently. As a first step, the agency could convene a workshop to educate staff in the use of the framework and use the case studies in the committee’s report as models. The agency could also provide resources to staff in various programs who have innovative ideas for implementing the framework. In addi- tion, an intra-agency group could be formed and meet regularly to share ideas and discuss the challenges of implementing the risk-based approach. REFERENCES Bertoni, M.J. 2010. Opening Remarks. Presentation at the 5th Meeting on Ranking FDA Product Categories Based on Health Consequences, Phase II, February 3, 2010, Washington, DC. Charles, C., A. Gafni, and T. Whelan. 1997. Shared decision-making in the medical en- counter: What does it mean? (or it takes at least two to tango). Soc. Sci. Med. 44(5):681-692. Clemen, R.T. 1996. Making Hard Decisions: An Introduction to Decision Analysis. Bel- mont, CA: Duxbury Press. 664 pp.

OCR for page 139