Appendix D
E. Coli Assessment
Some comments^{1} by Edmund Crouch, PhD, Cambridge Environmental Inc.
to the
Second Meeting of the
Committee on the Review of the Draft USDA E. coli O157:H7
FarmtoTable Process Risk Assessment
Monday, December 17, 2001
Author’s note to the reader:
These remarks were originally prepared after the IOM committee requested that I comment on the implementation of the E. coli risk assessment model and they have not been peerreviewed. Some minor editorial changes have been made from the version of the paper submitted to the committee. The remarks address the draft model contained in the spreadsheet “ECRA 010801a.xls,” dated August 8, 2001. They were specifically written for an audience who had ready access to the draft USDA risk assessment and spreadsheet, and it will be difficult for anyone who does not have these documents to understand them.
GENERAL
I was requested to examine in particular the implementation of the model described in this Risk Assessment, so the following concentrates on the spreadsheet, although unavoidably I have to comment on other matters as well.
There is some evidence that the spreadsheet provided to the committee is not the spreadsheet that was actually used to provide results in the document, but one in the stages of being modified to something different (perhaps to incorporate other functions, and/or other studies). The evidence lies in:

Compilation errors. The VBA code would not compile as supplied (see below, under “VBA code”). Some of the compilation errors looked as though they were occurring because what was provided was a work in progress (e.g., possible modifications in progress).

Crossreferencing errors in the spreadsheet. See the sections on the “BreedingHerds” and “Feedlots” sheets under “Herd prevalence”: studies used do not match those documented. These crossreferencing errors could have been introduced during an update that was adding/modifying studies.
SOME IMPLEMENTATION ISSUES
Notation
In what follows, an unqualified page number refers to the E. coli risk assessment, PDF version. I occasionally refer to a page number in Appendix C, but I am then explicit (and the page number might be slightly different from other copies, because Appendix C is in Word, so page numbers depend slightly on printer type).
References within spreadsheets are generally given in A1 notation. The sheet name is also given (e.g., SlaughterData!F39) if the sheet reference is not the sheet that is currently under discussion (i.e., is not the current section heading).
Documentation in Appendix C
“It was decided to use specific cell references rather than named ranges throughout the workbook. Although this makes changes to the model more difficult to accomplish, it should make it easier to follow and audit the flow in the procedures.”
This is contrary to my experience, in particular with this program, but also in general, and contrary to discussions of programming methods of which I am aware. This implementation is poorly documented and difficult to follow. The statement quoted is especially unusual since the default method of cell labeling (A1 style, letter column followed by numbered row) that is used in the documentation does not correspond to the method used within Visual Basic for Applications (VBA). In VBA, the preferred method of cell reference (and that used in much of, perhaps all, the programming here) is an entry like “cell(14,3)”, which means the cell at row 14 and column 3. This corresponds to the alternative labeling method in Excel, called R1C1 style, where a cell is referenced by row number and column number (i.e., the reverse order to the default, and using numbers for both row and column). Thus one has to get used to and translate be
tween a labeling style that is not the default, and is in reverse order to standard Cartesian axis labeling.
As an example of the problems, the following are said to be uncertainty distributions. Without some translation, I find this list uninformative. Trying to compare with the VBA code SimUncertainty (where the cell references are given in the cell(row,col) form) presents considerable difficulty. Here are some translations.
Feedlots!C31 
Oct.–May herd prevalence 
Feedlots!C32 
JuneSept. herd prevalence 
Feedlots!F33 
Herd prevalence (not seasonspecific) 
BreedingHerds!C31 
Oct.–May herd prevalence 
BreedingHerds!C32 
JuneSept. herd prevalence 
BreedingHerds!F33 
Herd prevalence (not seasonspecific) 
SlaughterData!E8 
Carcass prevalence/fecal prevalence (summer) 
SlaughterData!G8 
Carcass prevalence/fecal prevalence (winter) 
SlaughterData!C78 

SlaughterData!C81 

SlaughterData!C83 

SlaughterData!C88 

SlaughterData!C92 

SlaughterData!E12 
Trim/Vacuum/Wash, most likely decontamination 
SlaughterData!F12 
Trim/Vacuum/Wash, maximum decontamination 
SlaughterData!D15 
Evisceration, probability of rupture 
SlaughterData!E20 
Steam pasteurization, most likely decontamination 
SlaughterData!F20 
Steam pasteurization, maximum decontamination 
SlaughterData!D22 

SlaughterData!L29 

SlaughterData!L30 

SlaughterData!L31 

SlaughterData!F38 

SlaughterData!F39 

SlaughterData!F40 

SlaughterData!G25 

SlaughterData!G26 

GrowthData!C6:E6 
GrowthData!F13:H22 
GrowthData!M8:M21 
GrowthData!J22 
GrowthData!P22 
GrowthData!B26 
Cooking!O7:O15 
Cooking!J20 
Cooking!K20 
Cooking!G59 
DR!M2 
VBA Code
Compilation
Attempting to compile the VBA code resulted in the following errors.
Module RunSegments; Subroutine GrowthOnly
sGrinder not defined
sCoreModelBook not defined
sResultsBook not defined
Module Functions; Function SortString
When @Risk is not loaded, the builtin VBA function Mid is not recognized unless all references to @Risk are removed (for example, on the Tools:References list in the VBA editor). Similarly, other string functions (e.g., Left) are not recognized in similar circumstances. This behavior is odd, and may indicate that @Risk is doing something odd. However, it may simply be a bug in the behavior of the VBA compiler when it has unresolved external references (it is a bug because it is falsely signaling an error).
Module Functions; Function TrapezoidInv
dMidArea not defined
mean not defined
pMid2 incomplete statement
TriangInv incorrectly referenced (multiple times)
pMid not defined
plus many more.
It was necessary to “comment out” this entire routine to get compilation.
Module FullRun, Subroutine RunGrowthOnly
sUncertFilt not defined
sUncertainty not defined
Module X NotUsed, Subroutine SimulateAll
Call to subroutine MultiSimSlaughter not defined
Questions On The Code
Module Uncertainty. Subroutine SimUncertainty.
Why does this recalculate the Uncertainty sheet every 20 iterations of the i loop? This appears superfluous, and simply slows the calculation.
BREEDING HERDS SHEET
1. Herd prevalence: studies used do not match those documented.
At page 38 (just above Equation 3.4) it is stated that “True breeding herd prevalence (Figure 32) was estimated by combining the results from Equation 3.2 across all seven studies using Equation 3.4.” The seven studies referred to are in Table 31 on page 36. The spreadsheet model lists 10 studies (columns C:L), of which the first 7 are used in the estimate (columns P:V reference columns C:I). These are not the 7 studies in Table 31. The effect is to omit the studies labeled as Hancock et al., 1998; Lagreid et al., 1999; and Hancock et al., 1997a in Table 31 (taking account of the different names assigned in the spreadsheet). The extra studies included have no effect—they are included with zero entries that lead to likelihoods of unity.
Some columns in the spreadsheet match these experiments (with minor changes in attributed dates), but the entries that are used for computation do not, as illustrated in the following tables:
Table 31 
Spreadsheet 
Column 
Hancock et al., 1997a 
Hancock, 1997a 
I 
Hancock et al., 1997b 
Hancock, 1997b 
L 
Hancock et al., 1998a 
Hancock, 1998 
J 
Garber et al., 1999 
Garber, 1998 
C 
Lagreid et al., 1999 
Lagreid, 1998 
K 
Sargeant et al., 2000 
Sargeant, 2000 
G 
Hancock, 2001 
Hancock/FDA, 2001 
H 
As used in spreadsheet
Column Name 
Column 
Column Referenced 
Study Referenced 
Garber, 1998 
P 
C 
Garber, 1999 
Hancock, 1997a 
Q 
D 
Besser, 1997 (null) 
Hancock, 1998 
R 
E 
Rice, 1997 (null) 
Lagreid, 1998 
S 
F 
Hancock, 1994 (null) 
Hancock, 1997b 
T 
G 
Sargeant, 2000 
Sargeant, 2000 
U 
H 
Hancock, 2001 
Hancock/FDA, 2001 (7) 
V 
I 
Hancock, 1997a 
Entries listed as (null) have no entries in the relevant columns, and contribute no variation to the likelihood. Thus the spreadsheet headings do not match internally, nor with Table 31 (page 36).
This confusion presumably originally arose because of the additional information included in the spreadsheet at I3, that some studies did not include the sampling of adult cattle.
2. The method of application of Bayes theorem is crude. There are much better options available.
The aim was to find a distribution of probabilities (between 0 and 1). The method adopted was to compute the likelihood for increments of 0.01 in probability (between 0.01 and 1 inclusive; 0 necessarily has zero likelihood), then return a random variate with discrete values (integer multiples of 0.01) with probability proportional to the likelihood. This replaces a continuous variate with a discrete one, and may possibly produce odd effects in some simulations—e.g., we have no idea about the random number generator used; it is possible that one could get an interaction with the structure of the random number generator.
A better approach is to design a generator that produces random variates with a distribution function proportional to the likelihood—in the notation of Equation 3.2, page 37, all that is required is a generator with distribution function proportional to
where i labels the different sets of observations. This is relatively straightforward. For example, this distribution should be amenable to processing using a “universal” method of random number generation (see e.g., UNURAN at http://statistik.wuwien.ac.at/unuran/).
3. The method of calculation of the combined likelihood is unnecessarily complicated, and unnecessarily slows the computations.
The implementation proceeds in stages, appending each experiment one after the other. At each stage, a binomial term (see Equation 3.2, page 37) is computed (by a call to a builtin Excel function BINOMDIST, columns P:V) for each of the 100 values of probability (0.01 step 0.01 to 1.00, column O) . Then the likelihood is updated by multiplying by the binomial term, and the result is renormalized. All the renormalization steps are unnecessary. All the binomial calculations unnecessarily include the computation of a “combination” term (^{N}C_{S}) that is relatively expensive (timeconsuming) to compute (this is internal to the BINOMDIST function). All that is required is the product given by the equation above. The final normalization is not necessary, because the RiskDiscrete function subsequently used (at F33) performs such a normalization internally (this feature is apparently undocumented, but is essential for such a function).
4. The values of Hsens used in the computations do not match the values given in the documentation.
The values for Hsens for Breeding Herds are listed on page 38. The values used in the spreadsheets differ. The values are (rounded to 2 significant figures):
Experiment 
Documentation 
Spreadsheet (P2:V2) 
Garber et al., 1999 
0.75 
0.75 
Lagreid et al., 1999 
0.86 
not used 
Sargeant et al., 2000 
0.86 
0.99 
Hancock et al., 1998a 
0.89 
not used 
Hancock, 2001 
0.89 
0.87 
Hancock et al., 1997b 
0.96 
not used 
Hancock et al., 1997a 
0.99 
0.89 
This is undoubtedly because of the mixup already described above.
5. The uncertainty of Herd Sensitivity (page 37, symbol Hsens) is ignored.
The uncertainty in the estimate of Hsens is considerable. None of that uncertainty is propagated into the calculations. Hsens is fixed (it is listed in P2:V2)—and the VBA code setting up uncertainty estimates does not alter this fixed value.
6. The calculation of Hsens appears to be incorrect.
The calculation method for Hsens is simply described as “using Monte Carlo methods” (page 38, first paragraph), so we cannot be sure what was done. However, the spreadsheet contains a calculation of a quantity called “Herd sensitivity” (at C12:L12). Equation 3.3 on page 37 defines Hsens as
Hsens = 1− ∫ (1 − p)^{n}f(p)dp
where p is the “apparent withinherd prevalence,” f(p) is the “frequency” of p, and n is the number of animals tested in a herd. No limits were specified for the integral. Apparent withinherd prevalence is then said to be exponentially distributed between herds (pages 39–41), and proportional to the true prevalence (page 43, Equation 3.5). Let η be the test sensitivity and q the true withinherd prevalence, with distribution across herds g(q). Then
If f is exponential with parameter β (page 39, last paragraph), it must necessarily be a truncated exponential (so that its mean and standard deviation are not quite β, contrary to page 39). Examination of the spreadsheet entries at C12:L12 (in the row labeled “Herd sensitivity”) shows a formula equivalent to
Hsens(?) = 1− (1 − p)^{n}
where the “?” has been added because it is not clear what is being calculated here, even though it is labeled “Herd sensitivity.” This formula is clearly different from the “Herd sensitivity” listed in the documentation. However, this value does not appear to be used anywhere in the spreadsheets, although presumably it may have been used to evaluate the values used for Hsens in a subsidiary calculation. Otherwise, these entries are unnecessary, misleading, and simply serve to slow down the spreadsheet calculation.
7. The method of application of the Test Sensitivity is incorrect.
Test sensitivity is discussed at page 43, and used as in Equation 3.5. Obviously, Equation 3.5 can be correct only so long as “apparent prevalence” is lower than the test sensitivity. However, in the spreadsheet, the
impossible situation of “apparent prevalence” higher than “test sensitivity” is handled incorrectly (see, for example, F24).
It appears that the methodology attempted in the spreadsheet is based on likelihood methods (this is explicit for “true herd prevalence” Φ, and implicit for other parameters where uncertainty distributions for binomial observations are taken to be beta distributions). In fact, it appears that an attempt is being made to draw samples from conditional likelihoods—hence the conditioning of the distribution for Φ on the herd sensitivity, for example.
At F24, the equation used is MIN(RiskBeta(2+1,84–2+1)/$F$15,1). The RiskBeta function samples from a beta distribution corresponding to a binomial observation, and the result is divided by a test sensitivity that is itself a sample drawn from a beta distribution, again based on a binomial distribution. The impossible situation of the “apparent prevalence” higher than the “test sensitivity” is handled by substituting unity for the ratio, but the correct approach to obtain a sample distribution proportional to the conditional likelihood is to censor this sample combination, not to arbitrarily replace it. The effect of the replacement performed is to drastically distort the uncertainty distribution. The information on test sensitivity provided by the field study is being rejected, and instead replaced with some other (undefined) assumption.
Similar problems arise for all the estimates of seasonal prevalence. Note that the MIN function has only been applied in the spreadsheet to the Hancock (1994) study. Presumably that is the only one where the field study provides any real extra information about the test sensitivity. (The apparent prevalence is too rarely higher than the test sensitivity in the other cases to ever have arisen in any of the samples, or such cases were not noticed.)
8. Table 34 and associated text and calculations are undocumented and possibly incorrect.
The origin of the frequency distribution in Table 34 is not documented. The symbol x used in the text is clearly supposed to be the concentration (not the log_{10} of the concentration) of CFU/gram, so that the usual meaning of f(x) would be the distribution function for the concentration of CFU/gram, not the distribution function for log_{10}(CFU/gram). In that case the calculations performed in Table 34 are incorrect, because they ignore the nonlinear (indeed, logarithmic) scale of CFU/gram on the left (as does the normalization of f(x)). More likely, f(x) is supposed to represent the distribution function for log_{10}(CFU/gram). Even then, the sum calculated probably does not adequately represent the required cal
culation, because of the high nonlinearity in the exponential term being averaged.
9. At least one test sensitivity used does not agree with the documented value.
The study of Hancock (1994) is documented (page 45) as using 0.1gram samples and TSBvSMAC. This is estimated on page 45 to have a test sensitivity of 2%. The value used in the spreadsheet has an expected value of 7.7%, almost 4 times higher. The calculation (cell C38) leading to this higher estimate is not documented.
10. There is no evidence for “seasonal variability” presented in the documentation.
“Examining the monthly prevalence evidence, there appears to be a high prevalence season (June to September) and a low prevalence season (October to May).” (Page 45). The only evidence presented is: “For example, Garber et al. (1999) sampled cattle from February through July. These data show that 7 of 193 cattle sampled in infected herds were fecal positive during the period from February to May. In contrast, 44 of 1,075 cattle sampled in infected herds during June and July were fecal positive.” This is not a contrast, however—the rates presented could hardly be more similar (p = 0.47, onesided Fisher exact test). The only other study with results known separately for sampling in these two periods is Hancock/FDA (2001), showing 15/2,831 in December through March, versus 23/2,878 in June through September (documented only in a spreadsheet comment. This shows a larger contrast in rates, but again there is no significant difference (p = 0.11, onesided Fisher exact test).
The only evidence for seasonal difference in prevalence appears to come from comparisons between different studies, although this evidence was not discussed. The difference is most apparent between Hancock (1994) (performed only in summer) and the other studies. However, the prevalence estimate in Hancock (1994), is highly inflated by the estimated test sensitivity (and maybe should be inflated more—see above). Moreover, this study contains very few animals compared with the others. An adequate test for difference in seasonal variability needs to be carried out.
11. The method for averaging withinherd prevalence over “seasons” necessarily mixes any true seasonal difference with betweenstudy differences.
The method adopted for obtaining seasonal averages of withinherd prevalence appears to be ad hoc. Were completeyear data available for
every study, the imputations implied in the method adopted necessarily would reduce any seasonal differences that exist. Each study is imputed to have tested equal numbers of animals in every month of the study, and, where explicit timeresolved data are not available, the withinherd prevalence for every month for which aggregate data are available is assumed to be equal. For example, for Besser (1997), the aggregate results over 12 months were known to be 53 positives in 2,074. This rate was assumed to apply for each month (e.g., see D19:D30). The first imputation (equal numbers of animals) is perhaps plausible; but the second (equal rates in all seasons) is contrary to the assumption of the analysis. If there is any contrast between seasons, the second imputation forces the analysis to underestimate it.
On the other hand, since some studies provide information only within particular months, and there are possibly substantial differences between studies in the same months, the method adopted necessarily confounds seasonal differences with study differences.
It would be straightforward to perform a likelihood analysis that makes an assumption of different rates in different months. However, in view of the lack of any evidence of contrast between seasons, this appears unnecessary. Almost the entire “seasonal” difference indicated in the documentation is due to the inclusion of the high values from Hancock (1994), even though that has low weight. But the values obtained in Hancock et al. (1994) are highly skewed by a highly uncertain test sensitivity value (see previous comments).
FEEDLOTS
Most or all of the problems discussed above for the BreedingHerds sheet also occur in the Feedlots sheet. I only mention a few of them here, for example, to give the spreadsheet references (they usually differ from the BreedingHerds references).
12. Herd prevalence: studies used do not match those documented.
As for BreedingHerds, there is a mismatch between documentation and spreadsheet. The documentation says that Dargatz et al., 1997; Hancock et al., 1998b; Smith, 1999; and Elder et al., 2000, were used (page 48, and Table 36). These correspond to columns C, E, F, and G. There is an additional “Hancock 1999” in column D. However, the columns concatenated are C, D, E, and F. The effect is to omit Elder et al., 2000 (the erroneously included “Hancock 1999” has zero entries that contribute a constant to the likelihood).
13. The values of Hsens used in the computations do not match the values given in the documentation.
The values for Hsens for feedlots are listed on page 48. The values used in the spreadsheets differ. The values are (rounded to 2 significant figures):

Page 48 
Spreadsheet 
Dargatz et al., 1997 
0.77 
0.75 
Hancock et al., 1998b 
0.86 
0.99 (labeled as Smith, 1999, in column N) 
Smith, 1999 
0.99 
0.81 (labeled as Elder, 2000, in column O) 
Elder et al., 2000 
0.81 
This reference not used in the spreadsheet. 
This is undoubtedly because of the mixup already described above.
14. The uncertainty of Hsens is ignored.
Hsens is fixed (it is listed in L2:O2).
SLAUGHTERDATA SHEET
15. The distributions for several quantities are not justified.
At pages 65–66, we have “the reduction from decontamination (D1) was modeled using a triangular distribution with a minimum value of 0 logs, an uncertain most likely value ranging from 0.3 to 0.7 logs, and an uncertain maximum value ranging from 0.8 logs to 1.2 logs.” In fact, although not specified here, the two uncertainty ranges given were used to define uncorrelated uniform distributions. See E12, F12, SlaughterXXModel!I16 and SlaughterXXModel!K16 where XX is CB or SH. No basis whatever is given for selecting uniform distributions for the uncertainty, nor for selecting a triangular distribution for the variability distribution.
Other similar distributions occur at various places, all without justification. For example, see page 67, steam pasteurization in the second decontamination (step 5). In the spreadsheet, this is at E20, F20, and SlaughterXXModel!K16 where XX is CB or SH. Similarly for the chiller (step 6), at page 67, where we have a “normal distribution with an uncertain mean ranging from –0.5 to 0.5 logs and a standard deviation of 1 log.”
The spreadsheet uses a uniform for the mean (see D22 and SlaughterXXModel!F10 where XX is CB or SH).
16. Small plant, second decontamination (wash). Ambiguous documentation, and implementation that differs from any recognized interpretation.
Page 67: “It was assumed that small plants typically use a hot water rinse, sometimes supplemented with organic acids. The effectiveness of hot water rinsing is assumed equivalent to that described for decontamination Step 1 (D1).”
The meaning of this statement is ambiguous. What does “equivalent” mean—is the effectiveness of hot water rinsing within a given plant assumed to be identical to D1 for that plant, or to be equal in probability to D1 for that plant, or something else? What was implemented was equality in probability. The spreadsheet takes the same values for the “most likely” and “maximum” log reductions from their uncertainty distributions, but implements the variability distributions through separate instances of triangular distributions.
A FEW OTHER SPECIFIC COMMENTS
Page 18, para –2
In Washington state... The numbers have switched somewhere. 13/ 445 is 2.9%, and 5/445 is 1.1%, but these are cited as the other way round.
Page 43, last para
10^{7} in what units?
Page 43, last para
“A minimum shedding concentration of 10^{1} colonyforming units (CFU) per gram of feces can be assumed, based on a 10gram sample.” What is the evidence for this statement? That may be the detection limit, but it does not necessarily correspond to the lowest possible concentration.
Page 44
“Therefore, the observed difference in sensitivity between these methods approximates the effect of different sample quantities.” Is this proposed as a justification for f(x) in Table 34? It isn’t.
Page 44 and page 45
“Yet a 1.0gram sample from infected cattle is only 85% likely to contain E. coli O157:H7.” “A 0.1gram sample from infected cattle is only 73%
likely to contain E. coli O157:H7.” These statements apparently rely on Table 34, for which no evidence is presented.
Page 45
Hancock et al. (1994) sampling was estimated to have only 2% sensitivity, yet they detected 4.4% positive. And the difference is statistically significant even if the herd is 100% infected (p = 0.001). There is something wrong with this estimate. Also, equation 3.5 then predicts more than 100% prevalence, an impossibility. See the discussion of the BreedingHerds sheet.
Page 55, Equation 3.6
This represents a documentation failure, and a failure (at least in the documentation) to realize obvious speedups. The random variate B generated by this complicated sum is simply Binomial(40,H*w) as it is written. What the writers intended to say was that w is selected from an exponential variability distribution on each variability iteration, or something of that nature. The spreadsheet removes the unnecessary summation.
Page 56, Equation 3.7
Same problem. Again, the spreadsheet removes the unnecessary summation.
Page 61
“individual carcass contributes (n) ranged from 2 to 6. In cow/bull plants, this range was 2 to 4. Uncertainty about the most likely number of combo bins per carcass was modeled as a uniform(2,5) and uniform(2,3) for steer/heifer and cow/bull plants, respectively. The ranges and most likely values were modeled using triangular (min, most likely, max) distributions.” These three sentences all contradict one another, the first two as to numbers, the second two as to distributions assumed. And none gives any evidentiary basis for the selections.
Page 62, Equation 3.9
It is not clear whether the number of truckloads should be an integer, or this is estimating the average number of truckloads.
Page 63, high and low prevalence seasons for TR

Why is the approach taken for estimating TR different for the high and low seasons?

Such measurements as are discussed suggest that there is a nonlinear effect here—at low prevalence TR is less than unity; at high prevalence, it is higher than unity. However, the modeling completely ignores

this nonlinearity. But it might be very important to include it. It is stated that “more uncertainty is modeled about TR during this season”—but that is not what the discussed measurements indicate. The Bacon et al. (2000) measurements showing lower TR appear to have been entirely ignored— there is no discussion of their uncertainty.

It is not stated here what is meant by a “a mixture of the beta distributions based on the Elder et al. (2000) data and a uniform distribution with a minimum approaching 0 and a maximum of the summer TR.” What is done in the spreadsheet is to use a 50% probability mixture between the ratio of betas and a uniform ranging from zero to that ratio. No evidence is presented that such a mixture provides a good representation of the uncertainty involved.