7
Modeling Approach and Implementation

This chapter of the report differs from previous chapters in that it does not directly review a section of the draft E. coli O157:H7 risk assessment. Instead, it reviews the basis of the approach and implementation of the model and offers the committee’s observations and recommendations regarding it. The discussion thus touches on and overlaps some of the observations offered earlier, as well as providing an overall assessment of the modeling work done to date.

At the outset, it should be said that the effort underlying this risk assessment is impressive. The authors have undertaken an extraordinary task of collection, analysis, and integration of information. It will be an important assessment and will undoubtedly serve as an exemplar for future assessments. The analysts are to be commended for undertaking this work. Nevertheless, several issues remain to be resolved as development continues. The committee notes that the draft’s authors have already implemented some of the suggestions discussed below.

The US Department of Agriculture (USDA) Food Safety and Inspection Service (FSIS) effort faced a number of substantial methodologic hurdles whose solutions have not been described in textbooks or in literature peculiar to microbial risk assessment. In addition to methodologic hurdles, the FSIS team has been forced to cope with the inadequacy of the knowledge base. As a result, it is appropriate that they interrupt their risk-assessment effort to allow for peer review and to reassess their solutions to some challenging issues.

The committee commends them both for the magnitude of the effort and for the principles behind their efforts. The committee believes that



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment 7 Modeling Approach and Implementation This chapter of the report differs from previous chapters in that it does not directly review a section of the draft E. coli O157:H7 risk assessment. Instead, it reviews the basis of the approach and implementation of the model and offers the committee’s observations and recommendations regarding it. The discussion thus touches on and overlaps some of the observations offered earlier, as well as providing an overall assessment of the modeling work done to date. At the outset, it should be said that the effort underlying this risk assessment is impressive. The authors have undertaken an extraordinary task of collection, analysis, and integration of information. It will be an important assessment and will undoubtedly serve as an exemplar for future assessments. The analysts are to be commended for undertaking this work. Nevertheless, several issues remain to be resolved as development continues. The committee notes that the draft’s authors have already implemented some of the suggestions discussed below. The US Department of Agriculture (USDA) Food Safety and Inspection Service (FSIS) effort faced a number of substantial methodologic hurdles whose solutions have not been described in textbooks or in literature peculiar to microbial risk assessment. In addition to methodologic hurdles, the FSIS team has been forced to cope with the inadequacy of the knowledge base. As a result, it is appropriate that they interrupt their risk-assessment effort to allow for peer review and to reassess their solutions to some challenging issues. The committee commends them both for the magnitude of the effort and for the principles behind their efforts. The committee believes that

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment many of its criticisms and suggestions regarding this model would apply to most previous and current microbial risk-assessment models if they were subject to the same intensity of review. DESCRIPTION OF THE OVERALL MODELING APPROACH The approach taken in this modeling effort is to create a highly complex probabilistic simulation model that extends from estimation of the pattern of prevalence of enterohemorrhagic E. coli (EHEC) among various types of cattle through propagation of the exposure predictions related to slaughter, processing, and preparation of meals to the estimation of the distribution of dose-response relationships. The dose-response relationships are derived by fitting predicted distributions of exposure to estimates of the population health risk attributable to ground beef as estimated from epidemiologic data. In its final step, however, the model departs from the standard approach to risk assessment in a way that merits careful attention. Specifically, the risk characterization is carried out in part within the hazard-characterization stage by estimating (on the basis of epidemiologic data and investigations) the annual number of cases of EHEC illness associated with ground beef. Because the dose-response relationship is inferred from an algorithm that was designed to recreate samples from the distribution of the annual number of cases of EHEC illness, the risk estimates provided by the draft model cannot be considered to be independent of the epidemiological data. Risk Modeling, But Not Risk Assessment as Commonly Understood A key observation regarding the draft model is that it does not provide a risk assessment in the form that many readers would expect. To label the product a risk assessment implies that the effort is directed toward providing an estimate of risk by collecting evidence and applying mathematical tools; the estimate of risk would be a dependent output of the model. In particular, the use of the terms farm-to-fork and process risk model will imply to most readers that the many factors involved in the model are aggregated mathematically and propagated forward to generate an estimate of population health risk. The standard approach to risk assessment is that the information input and the predictive output of the exposure assessment and the dose-response assessment are derived from independent scientific sources and that the dependent output is estimates of risk that are derived from the combination of the two subassessments. With an estimate of risk as the

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment dependent output, the label risk assessment is appropriate to describe the analysis in the standard case. The present modeling effort alters that arrangement by deriving the exposure assessment and the population risk estimates from separate sources and then inferring a dose-response relationship that is mathematically compatible with the calculated exposure assessment and the distribution of population risk estimates. Such a risk assessment might be considered inverted because the nominal risk equation risk = function of (exposure x dose-response relation) has been reorganized to dose-response relationship = function of (exposure x risk). An analogy for the present approach may be useful. Consider an analyst estimating the area of a rectangle. The analyst, on the basis of various predictive models, has simulated a distribution of possible lengths of the rectangle. The analyst also has a separate source of information regarding the total area and provides a range of estimates for the area. Given great uncertainty in the width of the rectangle, the analyst decides to generate estimates of the width of the rectangle by dividing samples from the distribution of area by samples from the distribution of length. That generates a set of candidate widths that are compatible with the other two kinds of information. In an analogous way, the area assessment is inverted to become a width assessment. Moving forward, the analyst uses the estimates of length and the derived set of widths to generate estimates of area. The questions faced in this situation are whether the calculation in the model should be described as providing an estimate of area, whether and how the model can be validated, and what can be inferred (statistically) from the set of width estimates generated in the process and how they can be used. If we modify the length or width of the rectangle in some way by using a management strategy, is it reasonable to estimate the resulting area simply by multiplying the new length by the inferred widths? For the FSIS assessment, the question is whether and how the inferred dose-response relationship can be used in future assessments and management planning to predict the benefits (in terms of risk reduction) of altering ground-beef production, delivery, or preparation. Assessment of the Rationale for the Inverted Assessment Approach The nonstandard treatment of dose-response assessment appears to be based on the judgment that this component of the risk estimation carries the most—and least likely to be resolved—uncertainty. Furthermore, the uncertainty in the dose-response relationship is judged to be less than the uncertainty in the population risk estimate that typically would be considered the goal of the risk-assessment effort. That judgment seems

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment reasonable in light of practical and ethical constraints in performing human dose-response experiments and the continued lack of high-quality evidence suitable for dose-response characterization from, for example, outbreak investigations. The departure from the standard approach can be justified, in principle, by the assertion that the primary goal of risk assessment is better understanding of the mechanisms of the generation, transmission, and attenuation of risk through the system. To be considered appropriate, that goal would have to be considered more important than providing an estimate of population health risk that is derived solely from simulation of the components of the risk-generating system. Once it has established and communicated an appropriately limited set of goals (and acknowledged that not all the candidate goals can be equally served by the same effort), the decisions made by the modeling group to use various techniques can be understood and judged relative to the stated goals rather than to a presumed or standard goal. In light of the concerns raised in the earlier sections of this review, particularly those addressing the Production and Slaughter Modules, the authors may wish to reconsider whether the dose-response assessment is truly the most uncertain modular component of the model. From a modeling perspective, the current state of knowledge available to predict the transmission and ultimate fate of EHEC from the farm to the ground-beef patty may be at least as uncertain as the dose-response relationship. Given the complexity of the steps involved, combined with the legal and regulatory data-collection and -reporting environment, there may be little hope of gaining insight into this process without a fundamental change in the situation. It may be possible to place more faith in the use of Shigella dysenteriae 1 as a surrogate for the best estimate of a dose-response function for EHEC, as is proposed in the Hazard Characterization chapter, than in the implemented exposure model as a surrogate for the reality of the ecology and transmission of EHEC within and between farms, feedlots, slaughter-houses, combo bins, and grinders and through the multitude of potential cooking practices and consumer behaviors. An Example of the Standard Approach After the FSIS draft report was released for public comment, a risk assessment by Nauta and colleagues (2001) became available. It addresses the risk of EHEC illness from steak tartare consumption in the Netherlands and was produced for the Rijksinstituut voor Volksgezondheid en Milieu (RIVM). The risk assessment had not been peer-reviewed when it became available to the committee, but it is described in sufficient detail to compare the modeling efforts at the level of the overall approach. Nauta

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment et al. carry out the risk assessment in the more standard format, providing independent estimates of the distribution of exposure through predictive modeling and of the dose-response relationship on the basis of data from a 1997 outbreak in Japan (Shinagawa, 1997). Those are combined to form a prediction of the population health risk attributable to consumption of steak tartare contaminated with EHEC. The results of the analysis can be compared on a truly independent basis with population risk estimates from the Netherlands. In this case, the baseline predicted number of cases of EHEC illness due to steak tartare is higher than the total number of cases associated with EHEC from all foods. That independent assessment demonstrates that some components of the model are overstating the risk. The point of this comparison is to suggest that there may be some merits of “face-value” validation in which the model generates a risk estimate independent of illness surveillance data. The Nauta et al. model may not be satisfactory in its performance, but it has the considerable potential benefit of independent and transparent validation. The FSIS risk model cannot provide that degree of output validation until the dose-response relationship is generated from a source independent of the validation data. It would also be possible to validate the model if one or more surrogate dose-response relationships (for a foodborne pathogen other than O157:H7) were adopted for use. Alternatively, it is possible to validate the inferred dose-response function as was attempted in the Risk Characterization chapter with an outbreak investigation. Regardless of any validation efforts, transparency would be greatly improved by a simple demonstration of the range of risk estimates that are generated by using a variety of dose-response relationships (such as the upper and lower bounds of the dose-response envelope) to show the performance and the sensitivity of the model in various possible dose-response scenarios. The committee notes that the RIVM model is not included here as an example of a better model. Its authors acknowledge many limitations, and the analysis is of a much smaller scope and less detail with respect to the evidence base. However, it is included as an example of a model that provides a risk estimate as a dependent outcome and that would more closely match the standard definition of risk assessment. Having maintained separation between the risk assessment and the national surveillance data, the RIVM model has the benefit that its output can be compared with independent epidemiologic data for purposes of validation. Additional Comments Regarding the Inverted Approach The alternative approach used by FSIS carries some disadvantages. As noted, the primary drawback is the loss of the face-value validation of

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment the output through comparison with independent epidemiologic data, in that the data have essentially become part of the assessment. The impact of the inverted assessment is that—from the point of view of comparison with population health risk data—the model can never be wrong (because it constitutes a circular argument). An overestimate in the exposure distribution would be accommodated by underestimating the probability of illness, and underestimation in the exposure distribution could be accommodated by overestimating the probability of illness. The committee assumes that the motivation behind the algorithm is to practice a form of model updating, that is, using independent observation to improve the accuracy of the model. The updating algorithms are described later, including some suggestions for making the updating more compatible with these goals. Another drawback in the approach is the lack of a scientific evidence base for the dose-response relationship. The dose-response relationship is derived from model assumptions that are not related to the pathogenicity of EHEC. Any change in the parameters of the exposure assessment (for example, an improved estimate of the prevalence in a population of cattle) or in the assumptions leading to the baseline population health risk estimate (for example, a change in the etiologic fraction estimates) changes the basis of the inferred dose-response relationship. It is not clear where an appropriate end to this cycle of revision would be. The approach is much harder to understand than a straightforward assessment. The resulting uncertainty in the simulated dose-response parameters is a product of the uncertainty in the exposure assessment and the uncertainty in the population health risk estimate. The only information provided in the algorithm that is directly related to the issue of the pathogenicity of EHEC is the envelope that limits the search space in inferring the dose-response function and the assumption that the functional form will be beta-Poisson. The complexity of the relationship between the many sources of uncertainty and the final distribution of dose-response parameters may be a threat to real transparency for the great majority of external reviewers. This is a different matter from the strict transparency of the approach demonstrated through provision of the report, appendixes, and underlying model code. Some judgment is required as to whether a dose-response relationship derived in this way is preferable to a simpler model with a more transparent depiction of the underlying uncertainty. Despite the potential problems, it must be recognized that the mere departure from a standard approach does not in itself constitute an error. A decision to depart from the standard approach could be considered entirely appropriate to the situation. But, it is important to communicate the nature of the departure and its impact on the overall utility of the

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment model and to ensure that readers do not misunderstand the output of the model as being a risk assessment as it is commonly understood. Regardless of the presence of any technical errors, there is the risk of errors of omission and commission. The potential error of omission lies in failing to ensure the full communication of these issues. The potential error of commission lies in describing the effort as “generating” or “predicting” population risk estimates when in fact population risk estimates are provided as input to the model on the basis of epidemiologic data. A serious miscommunication could result if readers form the impression that the model propagates evidence forward to generate population health risks and then judge the model to be appropriate on the basis of the quality of the match to what are thought to be independent epidemiologic data. Having departed from the standard approach, the authors have the burden of ensuring that readers do not construct an inappropriate mental model of the approach and thereby form a judgment of its validity. There may also be some concern about the utility of a model generated in this way if risk-management decision-making requires the provision of a risk-assessment model that can be validated to some extent by national-level epidemiologic data. Such a requirement is not stated for this situation. Therefore the committee recommends that the authors communicate more clearly the nature of, the rationale for, and the impact of the departure from the standard risk-assessment approach and should consider relabeling the product as a system risk model to avoid implying that the model generates an estimate of risk independent of that derived from epidemiologic data. The authors should reconsider the approach taken to infer the dose-response relationship in light of the loss of the potential for model-output validation, a desire to improve transparency, and concerns regarding whether the uncertainty is actually greatest in the dose-response characterization. Chapter 5 of this report offers comments regarding the choice of surrogate pathogens. DESCRIPTION OF MODEL-UPDATING (ANCHORING) ALGORITHMS At several points in the development of the model, algorithms are invoked to adjust the simulation outputs of the model to make them more compatible with observed data. This approach, called “anchoring” in the draft, is applied at the end of the simulation of grinder loads to adjust the simulation results to be compatible with FSIS sampling data. A variation of model updating is also applied in the hazard characterization stage to

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment match simulated exposure distributions with predictions of the numbers of cases of illness. The application of model updating is well founded in health risk assessment and related fields of environmental modeling (see Brand and Small, 1995; Small and Fischbeck, 1999). Updating is of particular value when models are created under conditions of high levels of uncertainty. In such cases, the variance in model estimates grows as the evidence is propagated through each linked submodel. In the end, the distribution representing uncertainty in the predicted risk can be too broad to provide discriminating evidence in support of decisions. Thus, it is generally desirable to include any source of information that can reduce uncertainty in a model’s output. The algorithms for model updating used in the draft report are described below. Updating Grinder-Load Concentrations The exposure-assessment modules yield distributions for the number of grinder loads predicted to contain levels of EHEC ranging from 1 colony-forming unit (CFU) to 1012 CFU. FSIS carries out microbiologic testing for E. coli O157:H7 in raw ground beef, so there is an opportunity to provide additional information to the model by using the results of the sampling. The approach taken is as follows: Infer the distribution of the proportion of positive samples taken from grinders that would be statistically consistent with the FSIS sampling evidence. Calculate (by simulation) the proportion of positive samples by simulating the exposure model up to the point of grinder-load concentration and simulating the sampling and detection process. For each simulation, compare the calculated proportion of positive samples with the distribution of the proportion of positive samples inferred from FSIS results. Each simulation is treated in one of three ways: If the simulated prevalence falls between the 5th and 95th percentile of the inferred prevalence distribution, it is accepted as a plausible simulation. If the simulated prevalence exceeds the 95th percentile of the inferred prevalence distribution, the simulation is rejected from future calculations as implausible. If the simulated prevalence is below the 5th percentile of the in

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment ferred prevalence distribution, the simulation is amended by shifting the histogram of grinder-load concentrations to the right (that is, increasing it in 0.5 log increments) until the calculated proportion of positive samples approximates the mean of the inferred proportion of positive samples. That approach raises a number of concerns in that it contains arbitrary measures and unsupported suppositions: The choice of the 5th and 95th percentiles of the distribution as critical limits is arbitrary and effectively censors or distorts data that fall outside these bounds. The uncertainty in the range of proportions inferred from the sampling evidence may be underestimated. Specifically, the inference uses a point estimate of the sensitivity of the detection set at exactly 4 times the point estimate of the sensitivity of another test that is itself uncertain. The effect is to overestimate the inferential value of the FSIS sampling process and therefore to limit artificially the acceptable range of exposure simulations. The distinction made to accept and adjust simulations below 5% and to reject unconditionally those above 95% appears arbitrary. One could just as easily find a mechanistic justification to adjust the concentration histogram downward as to adjust it upward. The fabrication process is modeled as contributing to grinder loads only in situations in which other uncertain factors may be underestimating the pathogen load in grinders, as opposed to having an independent contribution in each simulation, as might be expected; in this way, the impact of fabrication depends on unrelated factors and not on any explicit assumptions regarding the process of fabrication. The shift of all grinder-load profiles that fall below the 5th percentile toward a distribution leading to the mean proportion is an arbitrary distortion of the grinder concentration distribution. The overall effect of the algorithm is to limit the simulations to those which are compatible with the central portion of the sampling evidence and to distort other simulations to reinforce the mean estimate from the sampling evidence. This is particularly problematic in that it eliminates lower-probability high-risk situations, which are normally of great interest in risk assessment. Generally speaking, the parameters of the dose-response algorithm are not well specified, making it difficult to understand and evaluate the derivation. The draft does not provide summary statistics associated with the proportion of simulations that are accepted, rejected, or adjusted. It is

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment therefore not possible to judge how large an impact the algorithm has on the overall simulation process. In addition, there does not appear to be an analysis of the factors underlying the rejection of simulations—that is, an assessment of the patterns of inputs that are associated with the set of rejected or adjusted simulations. The draft’s Appendix A refers to a paper on a Bayesian synthesis method by Green et al. (2000) as a source for the procedure, but the inferential approach described in the paper does not appear to have been used. Estimating the Dose-Response Relationship The process for estimating the dose-response relationship is based on iterative matching of two pieces of evidence. In each iteration, the first source of evidence is a sample from a set of simulated exposure distributions. Each simulation generates a histogram of the frequency of servings at discrete levels of number of CFU per serving. The second source of evidence is 19 percentile estimates (from 5% to 95%, in 5% increments) from an uncertainty distribution of the population risk estimate of the annual number of illnesses, on the basis of epidemiologic analysis. The assumed dose-response curve is the beta-Poisson function with parameters α and ID50. A fitting algorithm then finds a value for ID50 that will translate each exposure distribution into each of the 19 discrete estimates of population risk. This process is repeated for seven potential values of the α parameter. The result is a total of 19 × 7 × N dose-response relationships (combinations of α and ID50) where N is the number of simulated exposure distributions for which the fitting is done (N appears to be set at 100). The percentiles of the ID50 parameter are then calculated from the entire pool of results. This is not specified in the report, but the median dose-response curve appears to be based on the 50th percentile from the pool of ID50 values. The value of α that is assumed to apply for the “50th percentile” dose-response curve is not clear after a review of both the model implementation and the draft. The committee has the following concerns with this approach: There is no description of the mathematical or statistical basis of the approach, nor is there any reference to a similar approach applied elsewhere in the literature. The basis of including the 19 percentiles for developing a pool of ID50 values is not clear; it appears to be arbitrary. The meaning of an ID50 value that is the 50th percentile of such a pool of fitting results is not clear and is not explained beyond the state

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment ment that it is the median value of a pool of data whose elements do not appear to have a formal basis. The approach does not appear to conform to any established process of inference. Some reference is made in the spreadsheet to prior and posterior estimates, implying a Bayesian updating process, but no evidence of a likelihood function or other expected components of such an inferential approach is given. Although it may be reasonable on scientific and qualitative grounds to state that a pair of surrogate pathogens form a plausible envelope, this is not equivalent to stating that the values of α and ID50 must be limited to those achieved by the fitting algorithm for the two pathogens. The uncertainty in the values of both α and ID50 that result from fitting to the feeding-trial data for Shigella and enteropathogenic E. coli would be expected to be broad. In the dose-response estimation method, the range of uncertainty in the α parameter is limited to 0.16–0.22 in steps of 0.01, thereby providing seven alternative values of α. That is particularly relevant, given the committee’s finding that Shigella dysenteriae 1 may constitute a reasonable surrogate for a “best estimate,” rather than its current role as an estimate of the upper bound of the envelope. Alternative Model-Updating Strategies Both the model-updating processes described above appear to lack a formal statistical basis. Given the use of Bayesian updating processes at various points in the model and the overall reliance on Monte Carlo simulation, it seems appropriate to consider using a form of Bayesian Monte Carlo simulation (or some of its more advanced resampling relatives) to incorporate properly the information provided by the observational data (see, for example, Brand and Small, 1995; Dilks et al., 1992; Gelman et al., 1995; Small and Fischbeck, 1999). There are a number of key differences in the application of an algorithm based on the Bayesian Monte Carlo methods: It does not place the burden of model adjustment on any one part of the model (that is, both the exposure and dose-response modules would be updated). The updating process works both “upstream” and “downstream” of the observation point. It does not allow for arbitrary adjustments. The quality of the process generating the observational data must be carefully scrutinized and quantified in the development of likelihood functions.

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment bined with other variables. The inconsistencies mean that analysts must arbitrarily pick a scale on which to express their uncertainty and resist comparing it across different scales. Expert Elicitation Formal expert elicitation does not appear to have been used explicitly in the draft model, but the committee suggests that it could be judiciously applied in some circumstances (notably in the Preparation Module) where single or small numbers of observations are extrapolated to the entire population. There are various approaches to eliciting information about input variables from experts or other knowledgeable persons. They range from simply asking them in informal and uncontrolled settings to using elaborate formal schemes (Cooke, 1991; Meyer and Booker, 1991; Morgan and Henrion, 1990; Warren-Hicks and Moore, 1998). Formal elicitation schemes can often be expensive. It might be reasonable to let experts define the shapes of input distributions subjectively, but this is not always a workable strategy and, when experts disagree, it can lead to even more controversy about the inputs. In the final analysis, the committee believes that there are circumstances in which it would be appropriate to solicit expert opinion regarding point estimates and distributions and that, if found useful, such information should be documented in the text and used in the model until data become available. Dependencies It appears that most of the variables in the FSIS draft risk assessment are assumed to be mutually independent. Overall, the draft is practically silent on the potential for input variables to be dependent. Although such assumptions make the computation for a model substantially easier, their justification, whether theoretical or empirical, is lacking. For instance, average carcass weight may be correlated with within-feedlot prevalence (Dargatz et al., 1997). Other candidates for dependence include any variables describing consumer behavior and preferences that may share a risk factor, such as age, ethnicity, sex, or health status. In the interest of discovering particularly high-risk scenarios, the modelers should review the list of model inputs for pairs or triplets of inputs that are intuitively likely to be dependent. The review could be based on data (perhaps rarely), reasoning regarding a common cause, or where the variables may be expected to be affected by a common general risk factor (knowledge of preparer regarding appropriate food handling or preparation practices) that is otherwise important in the assessment (such as serv

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment ing size, frequency of consumption of raw ground beef, or the age of the consumer). The committee recognizes that finding concrete evidence for individual important variables is difficult; evidence of the dependence structure of two or more variables will certainly be more rare. An explicit model of such dependence may not be feasible, but the effect of plausible dependence scenarios should be considered on a case-by-case basis and presumably prioritized through causal reasoning of the plausibility of the dependence relationships. This will assist in better characterizing the potential for high-risk scenarios and may help to explain higher proportions of attributable risk to particular exposure pathways. In addition, explicit reference to the plausibility of key variable dependencies (even if difficult to quantify) is useful information for risk-management and data-collection priority-setting. Therefore the committee recommends that the final risk assessment should address the potential for input-variable dependence in the model, based on causal reasoning and other evidence of such relationships. For potentially important dependencies, a sensitivity analysis should be performed to evaluate the nature and magnitude of the potential dependence structure. Seasonality The draft pays considerable attention to assumptions regarding the seasonal pattern of on-farm prevalence. As mentioned in Chapter 2 and Appendix D of this report, the data cited as justification for modeling seasonality in prevalence do not actually provide evidence of such seasonality. In face-to-face meetings, analysts of the development team suggest that the best evidence comes from studies conducted outside the United States; they were omitted out of relevance concerns, but the omission leaves the assertion about seasonality essentially unsupported. The relevance concerns about foreign studies should be resolved and documented so that justification for seasonality assumptions is clarified. Therefore the committee recommends that the authors should reconsider the evidence of and the approach for inferring seasonality in on-farm prevalence, including the potential for using data from outside the United States. Evidence of seasonality might also be sought in the upper tails of the internal or external pathogen load among E. coli O157:H7 -positive animals. That may have a stronger effect (by several orders of magnitude) than simple variations in prevalence on the number of contaminated ground-beef patties. The committee recognizes that, given the substantial uncertainty associated with other important quantities in the model that affect prevalence downstream, further refinement of the exact pattern of on-farm seasonality may not have high priority.

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment OVERALL MODEL UNCERTAINTY AND RELIABILITY It appears that several of the methods used in developing the draft risk assessment may tend to understate uncertainty. That would typically be considered disadvantageous, if not dangerous, in risk analyses. There are several ways this happens: incomplete reconstruction of statistical regressions, overreliance on empirical distributions, use of means other than raw data, and modeling of sampling variation without representing the underlying uncertainty arising from measurement error. Each of those is discussed below. Uncertainty Encoded in Regressions Not Reconstructed The draft makes use of several regression analyses, but in doing so it seems to have not fully reconstructed the uncertainty in the relationship among the random variables in the original data sets. For instance, in Equation 3.28 (described on p. 80 of the draft and p. 196 of Appendix C), constants are used to transform temperature linearly into E. coli generation time. The coefficients of the linear scaling appear to be regression parameters (Marks et al., 1998), but the regression model has not been used to re-express the uncertainty in the predicted variable. Instead, the linear scaling is the prediction of the mean value of generation time expected, given a particular temperature. That approach does not reconstruct even the scatter of the original data, much less account for sampling uncertainty. Similar applications of slope and intercept values are made in the equations for calculating the lag period for E. coli O157:H7 for a specific step of handling or storage (Equation 3.27) and the maximum population density of the pathogen in ground beef (Equation 3.29). Overreliance on Empirical Distributions The fact that empirical distributions typically underestimate the probabilities of values in distribution tails is a straightforward result of the reality that, given any limited random sample, the chance of ever observing the rare events in the tails is low. The authors seem to place too much credence in distributions that are based on remarkably few data points. In the case of Equation 3.11.1, described on p. 179 of Appendix C, the distribution of the number of bacteria per square centimeter of carcass surface area is specified from only four points. No accounting is made of the sampling error or measurement error implied by that small number. The problem is essentially the same as one encounters when using only a single study in a scientific review. Because of the importance of distribution tails,

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment the problem is also similar to using the observed maximum of a sample as the theoretical maximum of a random variable. Modelers need to account for sampling error to see beyond the limited data available. In principle, Kolmogorov-Smirnov confidence intervals could be computed that characterize the sampling uncertainty about the distribution as a whole, assuming that the samples were independent and identically distributed. In any case, the uncertainty about distribution shape should be explicitly and quantitatively characterized. Doing so will allow analysts to revisit the choice of distribution shape in later sensitivity and robustness studies. Use of Averages At many points in the E. coli assessment, the modelers use average values of variables rather than a distribution of various quantities. An average does not capture variability; it erases it. Consequently, averages are of little use in a risk assessment, where one of the primary concerns is to account for variability in the system. In the case of some variables, a distribution is used, but it is a distribution of values that turn out to be averages over one or more dimensions. In such cases, the distributions represent only part of the true variability of the underlying variable. Untenably Precise Submodels The overall assessment will underestimate uncertainty if it is composed of submodels that underestimate the uncertainties in the variables they are used to model. One example in the draft is the model of cooking loss. Figure 3-25 on p. 88 in the draft risk assessment depicts the modeled frequency distribution of log reductions in E. coli abundance caused from cooking. It also depicts the uncertainty about this distribution by displaying 20 realizations of the distribution. It seems implausible that the true distribution of reductions, whatever it is, has the multimodality that this figure shows. The problem is not the “bumpiness” of the distribution itself, but the fact that the same bumpiness persists in all the realizations of the distribution—that is, the bumpiness is stronger than the overall uncertainty about the distribution. As explained in Appendix D of the FSIS draft (“Modeling Issues”), the origin of the multimodality can be traced to the cooking-temperature data on which the model of log reductions was based. Each distribution of the ensemble displayed in the figure is based on integrating (apparently, actually a stochastic mixture of) nine distributions representing the effect of cooking with different pretreatments. A mixture model would be appropriate if the analysts knew the relative frequencies of the various pretreat

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment ments in homes and institutions. However, it seems that the mixture in this case is used to represent model uncertainty rather than the variability among kitchens. For each pretreatment, a regression is computed from 18 points (six replicates at each of three cooking temperatures), and the mean reduction is taken from the regression line. That seems to be the real reason that the uncertainty in the distribution of log reductions is so small. The mean reduction does not seem entirely relevant. It is the outliers of the distribution (small reductions) that are likely to induce illness. It seems that the modelers strive to make their assessment consistent with the observed data but sacrificed the reliability of their results to do so. As noted in Appendix D of the FSIS draft, the bumpiness of the distribution can be traced finally to the preferences for round numbers in the values reported for final cooking temperature on the Fahrenheit scale (for example, 150°F rather than 153°F). It seems clear that the modality is completely artifactual. That is not at all a criticism of the original data—it is how the survey turned out. However, that the bumpiness persisted through the many transformations made on the data suggests that the true scope of their inherent uncertainty was never fully recognized. In summary, the original data, with their multimodality, seem fine; what is insufficient is the breadth of uncertainty about the final distribution of log reductions. The committee notes that if the modelers had simply smoothed the distribution before computing and displaying the uncertainty about it, the problem might never have been noticed. Their openness in portraying the details of this particular variable and the data it was based on demonstrates the utility of transparency in presentation. Inappropriate Use of Bayesian Formulations The committee suspects that the use of beta distributions in Equation 3.10.1, described on p. 177 of Appendix C of the FSIS draft, exemplifies another way in which the uncertainty present in the system is underestimated in the draft model. That equation defines the random variable TR (transformation ratio), which is the multiplier of the E. coli prevalence in cattle that predicts the prevalence in carcasses. Data collected from four slaughter plants during July and August (Elder et al., 2000) suggest that TR is about 160%. (The ratio is larger than 1 presumably as a result of cross contamination during dehiding.) The counts actually reported were 91 of 307 cattle and 148 of 312 carcasses being contaminated. The authors explain that the beta distributions are used to model the uncertainty about TR. The beta distributions arise in this context from a simple Bayesian updating argument about frequency estimates. However, the specified

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment beta distributions have extremely small variances; both are less than 0.001. Given the ordinary fluctuations one might anticipate from day to day and across slaughter plants, it seems entirely unreasonable to expect that such distributions could be realistic models of the contaminated fractions of cattle or carcasses. Because the numerator and denominator in Equation 3.10.1 are combined under an (unsupported) assumption of independence, the quotient has a very small variance, which suggests that the ultimate TR has little uncertainty. Potentially Dominant Model Uncertainties Model uncertainty is a class of uncertainty that pertains to the adequacy of a model’s representation of reality. Strictly on the basis of qualitative judgment, the committee suggests that the following general model uncertainties may dominate: The ability to define an appropriate output or set of outputs from the onfarm module that is adequately correlated with the level of risk to the ground-beef supply. As described in the review of the Production Module (Chapter 3 of this report), prevalence estimates can vary considerably with detection method, type, and definition (for example, cattle with 1 or more EHEC on the entire surface or in the entire gut). It is intuitively reasonable to suggest that the number and relative proportion of animals with very high bacterial loads will dominate the overall contamination level in the combo bin, given the fact that the contamination (through pooling in the combo bins and grinders) is proportional to the total across many individual contributions. Given the expectation of logarithmic variability in pathogen loads, the highest-shedding animals will contribute more bacteria (by a factor of several powers of 10) whenever there is a transfer to the carcass surface. Conversely, the animal with an average burden may contribute comparably little. The ability to represent the transfer of organisms from the hide or gut of the animal to the carcass surface and the relationship between measurements of cell density on the carcass surface and the amount on trim that becomes ground beef. The ability to represent the potential for reservoirs of contamination in a slaughter plant and for cross contamination of carcasses. The potential pathways and reservoirs of contamination in a slaughter plant are numerous, complex, and likely to be dynamically changing, even if measured. The use of indicator organisms should be reconsidered for their ability to track the sources and extent of fecal contamination in the plant. Without a drastic change in the approach to data gathering and modeling, this part of the model will remain a “black box.” Attempts to model its internal mecha

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment nisms may be misguided until appropriate data are made available to the modelers. The prevalence of exposure to uncooked ground beef. This variable will be inherently difficult to estimate because of the uncommon nature of the exposure and the response bias associated with its estimation from surveys or other methods. However, it could be important because there may be a relatively high probability of illness associated with the pathway compared with a cooked product. The extent of the exposure that occurs through cross contamination. The committee recognizes that there are no examples of risk-assessment modules that could adequately describe the potential for risk associated with cross contamination during food preparation. Even an emerging experimental database that focuses on cross contamination is unlikely to support such a model in sufficient detail to form predictions of illness through this complex pathway in the near future. This remains a major methodologic hurdle in microbiologic risk assessment. The dose-response relationship for EHEC. Despite the committee’s suggestion that a Shigella species may constitute an adequate surrogate for the best estimate of the dose-response function, there remains broad uncertainty associated with this representation and even with the characterization of the dose-response function for Shigella itself. While it may ultimately be possible to narrow the uncertainty envelope used in the draft risk assessment, large uncertainties will remain regarding the dose that corresponds to a given probability of illness. Use of Undefined Variables One of the features of an Excel spreadsheet is the automatic initialization (to zero) of variables that are not otherwise explicitly defined. Automatic initialization is regarded as a feature of convenience when beginning a model, but can become a serious hindrance as the model develops. Appendix D, which contains comments on the model presented to the committee by outside reviewer Edmund Crouch, cites specific examples where references were found to undefined cells in the draft model spreadsheet. Some of these cells had been given a “hatched” format, presumably to indicate that the values were not available. Because of automatic initialization, Excel regarded the undefined cells as zeros. Although this may turn out to be the correct value to use for these particular variables, the committee suggests that the final risk assessment explicitly define all variables and constants to be used in the model, simply as a matter of good modeling practice.

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment Unit Conformance Experience with complex assessments has shown that profound errors can arise from simple or careless mistakes in units (Isbell et al., 1999). It is thus important that quantities to be added, subtracted, or compared for magnitude have conforming units; and quantities to be used as exponents or powers or the arguments of logarithms be dimensionless (Hart, 1995). The FSIS draft report’s Appendix C—a partial list of the model equations and code—contains some but not all of the information needed to check unit conformance. The committee was thus unable to conduct a rigorous review of the dimensions and units of the equations and variables used in the draft model. It suggests that the final model’s mathematical expressions be checked for dimensional soundness and the input quantities be checked for unit conformity with the variables in the expression. Overall Assessment of Model Reliability The committee believes that a forthright and comprehensive characterization of the uncertainty in the assessment would show the overall model uncertainty to be very large (much larger than suggested by the Risk Characterization chapter of the draft), even after anchoring. It may be that the uncertainty is so large as to appear to overwhelm any quantitative predictions based on the assessment. But even if that is so, the authors should not shy away from being as forthright and comprehensive as possible. Large uncertainty itself does not preclude useful applications of an assessment. But underestimated uncertainty can threaten the credibility of an assessment and lead to unwarranted confidence on the part of decision-makers regarding the response of the system being modeled to simulated mitigations. Therefore the committee recommends that the final report clearly describe the magnitude of model uncertainty related to key modules in the risk assessment and include strategies for reducing the uncertainty, if they exist. REFERENCES Brand KP, Small MJ. 1995. Updating uncertainty in an integrated risk assessment: Conceptual framework and methods. Risk Analysis 15(6):719–731. Bukowski J, Korn L, Wartenberg D. 1995. Correlated inputs in quantitative risk assessment: The effects of distributional shape. Risk Analysis 15:215–219. Burmaster DE, Anderson PD. 1994. Principles of good practice for the use of Monte Carlo techniques in human health and ecological risk assessments. Risk Analysis 14:477–481.

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment Cassin MH, Lammerding AM, Todd EC, Ross W, McColl RS. 1998. Quantitative risk assessment of Escherichia coli O157:H7 in ground beef hamburgers. International Journal of Food Microbiology 41:21–44. Cooke RM. 1991. Experts in Uncertainty. Cambridge: Oxford University Press. Cullen AC, Frey HC. 1999. Probabilistic Techniques in Exposure Assessment: A Handbook for Dealing with Variability and Uncertainty in Models and Inputs. New York: Plenum Press. Dargatz DA, Wells SJ, Thomas LA, Hancock DD, Garber LP. 1997. Factors associated with the presence of Escherichia coli O157 in feces in feedlot cattle. Journal of Food Protection 60:466–470. DeWaal CS, Barlow K, Alderton L, Jacobson MF. 2001. Outbreak Alert! Closing the Gaps in Our Federal Food Safety Net. Washington, DC: Center for Science in the Public Health Interest. October. Dilks DW, Canale RP, Meier PG. 1992. Development of a Bayesian Monte Carlo method for determining water quality model uncertainty. Ecological Modeling 62:149–162. Elder RO, Keen JE, Siragusa GR, Barkocy-Gallagher GA, Koomaraie M, Laegreid WW. 2000. Correlation of enterohemorrhagic E. coli O157 prevalence in feces, hides, and carcasses of beef cattle during processing. Proceedings of the National Academy of Sciences 97(7):2999–3003. Finley B, Proctor D, Scott P, Harrington N, Pasutenbach D, Price P. 1994. Recommended distributions for exposure factors frequently used in health risk assessment. Risk Analysis 14:533–553. Gelman A, Carlin JB, Stern HS, Rubin DB. 1995. Bayesian Data Analysis. London: Chapman and Hall. Grandy WT Jr, Schick LH. 1991. Maximum Entropy and Bayesian Methods. Dordrecht: Kluwer Academic Publishers. Green, EJ, MacFarlane DW, Valentine HT. 2000. Bayesian synthesis for quantifying uncertainty in predictions from process models. Tree Physiology 20:415–419. Haimes YY, Barry T, Lambert JH. 1994. When and how can you specify a probability distribution when you don’t know much? Risk Analysis 14:661–706. Hart GW. 1995. Multidimensional Analysis: Algebras and Systems for Science and Engineering. New York: Springer Verlag. Isbell D, Hardin M, Underwood J. 1999. Mars Climate Orbiter team finds likely cause of loss. Mars Climate Orbiter/Mars Polar Lander News & Status Release 99-113 (dated September 30, 1999) http://mars.jpl.nasa.gov/msp98/news/mco990930.html. Jaynes ET. 1957. Information theory and statistical mechanics. Physical Review 106:620–630. Kassenborg H, Hedberg C, Hoekstra M, Evans MC, Chin AE, Marcus R, Vugia D, Smith K, Desai S, Slutsker L, Griffin P, and the FoodNet Working Group. 2001. Farm visits and undercooked hamburgers as major risk factors for sporadic Escherichia coli O157:H7 infections—Data from a case-control study in five FoodNet sites. Manuscript in preparation. Lee RC, Wright WE. 1994. Development of human exposure-factor distributions using maximum-entropy inference. Journal of Exposure Analysis and Environmental Epidemiology 4:329–341. Levine RD, Tribus M. 1976. The Maximum Entropy Formalism. Cambridge: MIT Press. Marks HM, Coleman ME, Lin CTJ, Roberts T. 1998. Topics in risk assessment: Dynamic flow tree process. Risk Analysis 18:309–328. Meyer MA, Booker JM. 1991. Eliciting and Analyzing Expert Judgment: A Practical Guide. New York: Academic Press. Morgan MG, Henrion M. 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge: Cambridge University Press.

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment Nauta MJ, Evers EG, Takumi K, Havelaar AH. 2001. Risk assessment of Shiga-toxin producing Escherichia coli O157 in steak tartare in the Netherlands. Rijksinstituut voor Volksgezondheid en Milieu (RIVM) . Report 257851 003. Shinagawa K. 1997. Correspondence and problem for enterohemorrhagic E. coli O157 outbreak in Morioka city, Iwate. Koshu Eisei Kenkyu 46:104–112. Small MJ, Fischbeck PS. 1999. False precision in Bayesian updating with incomplete models. Journal of Human and Ecological Risk Assessment 5(2):291–304. Tilwari JL, Hobbie JE. 1976. Random differential equations as models of ecosystems. II. Initial condition and parameter specification in terms of maximum entropy distributions. Mathematical Biosciences 31:37–53. Warren-Hicks WJ, Moore DRJ. 1998. Uncertainty Analysis in Ecological Risk Assessment. Pensacola, FL.: SETAC Press, Chapter 3.

OCR for page 97
Escherichia Coli O157:H7 In Ground Beef: Review of a Draft Risk Assessment This page in the original is blank.