6

Evaluation of Epidemic Modeling

The use of computational models for epidemic forecasting is challenging. Epidemic models are constructed by narrowing down broad scientific understandings to specific parameter estimates and assumptions. Gaps in scientific knowledge, limitations on data-collection resources, and the complexity of the transmission processes themselves all make it impossible to precisely predict the consequences of an infectious disease outbreak. The very process of model construction requires simplifying assumptions that introduce more uncertainty. The use of models to inform disease control policies in the face of animal disease epidemics has been the subject of considerable debate (Anonymous, 2001; Kitching et al., 2005, 2006; Dickey et al., 2008; Mansley et al., 2011; Smith, 2011). Kitching et al. (2005, 2006) and Mansley et al. (2011) comment that the misapplication of foot-and-mouth disease (FMD) epidemic forecasting can be misleading and can promote a false sense of security. Forecasts in most fields of natural sciences are best viewed skeptically. Despite the limitations, epidemic modeling can be a useful conceptual resource because it forces a systematic review of all components of an infectious disease outbreak, including critical assessment of knowledge and uncertainty about each component.

OVERVIEW OF METHODS AND ANALYSIS

Section 6 of the updated site-specific risk assessment (uSSRA) estimates the consequences of a potential release of FMD virus (FMDv) from the proposed National Bio- and Agro-Defense Facility (NBAF) in Manhattan, Kansas. As in the 2010 site-specific risk assessment (SSRA), the uSSRA uses



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 53
6 Evaluation of Epidemic Modeling The use of computational models for epidemic forecasting is challeng- ing. Epidemic models are constructed by narrowing down broad scientific understandings to specific parameter estimates and assumptions. Gaps in scientific knowledge, limitations on data-collection resources, and the com- plexity of the transmission processes themselves all make it impossible to precisely predict the consequences of an infectious disease outbreak. The very process of model construction requires simplifying assumptions that introduce more uncertainty. The use of models to inform disease control policies in the face of animal disease epidemics has been the subject of con- siderable debate (Anonymous, 2001; Kitching et al., 2005, 2006; Dickey et al., 2008; Mansley et al., 2011; Smith, 2011). Kitching et al. (2005, 2006) and Mansley et al. (2011) comment that the misapplication of foot- and-mouth disease (FMD) epidemic forecasting can be misleading and can promote a false sense of security. Forecasts in most fields of natural sciences are best viewed skeptically. Despite the limitations, epidemic modeling can be a useful conceptual resource because it forces a systematic review of all components of an infectious disease outbreak, including critical assessment of knowledge and uncertainty about each component. OVERVIEW OF METHODS AND ANALYSIS Section 6 of the updated site-specific risk assessment (uSSRA) estimates the consequences of a potential release of FMD virus (FMDv) from the proposed National Bio- and Agro-Defense Facility (NBAF) in Manhattan, Kansas. As in the 2010 site-specific risk assessment (SSRA), the uSSRA uses 53

OCR for page 53
54 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT the North American Animal Disease Spread Model (NAADSM) in conjunc- tion with data, statistical methods, and references from scientific literature. Simulation outputs from NAADSM were used to evaluate the impact of FMD spread through Kansas and into six adjoining states in different release events. The analysis estimated the consequences of large epidemics and the potential effects of some mitigation measures on an epidemic. De- pending on the risk scenarios, the outputs suggested that an epidemic in the seven states could last 18 months or more and result in the loss of tens of millions of animals. The 2010 SSRA results were criticized by the previous National Research Council committee for a lack of transparency, structural limitations in NAADSM, and some specific modeling choices (NRC, 2010). The uSSRA makes a variety of changes and attempts to address all the pre- viously identified shortcomings of the 2010 SSRA. The revised model in the uSSRA now estimates an FMD epidemic in these seven states to last about twice as long as and affect several times more animals than the 2010 SSRA. SUMMARY ASSESSMENT The overall methodology and presentation of epidemic modeling in the uSSRA are substantially improved compared to those in the 2010 SSRA. Part of the reason is the uSSRA’s better description of model limitations and uncertainty. Issues of reliability, uncertainty, and sensitivity are acknowl- edged at the beginning of Section 6 of the uSSRA and addressed again throughout. The breadth of epidemiological material collected in the uSSRA could make it a useful reference for future FMD research and planning. However, the epidemic modeling in the uSSRA still provides only a limited picture of the likely possibilities involved in an FMD epidemic originating in Manhattan, Kansas. Some of the limitations result from inadequacy of available tools, including NAADSM, some from lack of data and incomplete scientific understandings, and some from incomplete characterization of the resources and capacity for mitigation responses. Practical considerations have imposed a number of those limitations, as the uSSRA acknowledges. The committee finds that the modeling results underestimate the absolute size and duration of epidemics, in part because of a number of specific assumptions used in the uSSRA. Overly optimistic assumptions were made about response resources and capacities anticipated to be available by 2020, and these in turn would lead to an underestima- tion of the magnitude, duration, and economic impact of an FMDv escape from the NBAF in Manhattan, Kansas. The uSSRA underestimated contact risks and used overly optimistic parameter values for diagnostic capabili- ties, surveillance, contact rates, vaccination, and response. Consequently, the uSSRA spread model results incorrectly indicate foreshortened spread and low impact estimates. The incomplete data on interstate direct contacts,

OCR for page 53
55 EVALUATION OF EPIDEMIC MODELING including illicit livestock movements and interstate indirect contacts (fomi- tes), would inhibit simulated movement, including secondary and tertiary spread of virus and infected animals from Kansas to the six other states. Considering the aggregate of design, methods, data, and assumptions, the committee finds that the methodology as a whole lacks the overall valid- ity necessary to predict with reasonable confidence the outcome of an FMD epidemic emanating from an FMDv release from the NBAF in Manhattan, Kansas. Much of the lack of validity was unavoidable, due in large part to many ill-defined or unknown factors. These factors lead to considerable uncertainty stemming from an absence of quality data and the vagaries of proposed mitigation policies on the outcome of an FMD outbreak. It is also important to note that these limitations may well lead to underestimates or misestimates of the consequences of an epidemic, which are carried over into the economic analysis. However, the committee strongly agrees with the uSSRA’s broad conclusion that negative consequences of an FMD epidemic originating in Manhattan, Kansas, will probably be severe. The committee therefore agrees that great emphasis needs to be placed on pre- venting release of FMDv and detecting and containing FMDv if it escapes. METHODOLOGICAL LIMITATIONS Limitations of the Scope of Model The committee noted two major shortcomings related to the geo- graphical and outcome scopes. First, the spread model incorporated only seven states. According to the uSSRA, no suitable model for nationwide FMD prediction is yet available. Thus, absolute impacts reported in the uSSRA are acknowledged to be underestimates. The committee concurs that extension of the assessment to include spread through the contiguous United States, Mexico, and Canada would require several-fold greater ef- fort (GAO, 2002). Second, there was no scenario involving FMD becoming endemic. Endemic FMD would require different long-term control strate- gies, such as a vaccination-to-live strategy, extensive laboratory testing for surveillance, and an expensive long-term eradication program. Limitations of NAADSM Like all models, NAADSM provides an imperfect representation of FMD spread and control and is based on a variety of simplified assump- tions. As pointed out repeatedly in the uSSRA, use of only NAADSM, without application of support models, carries a number of structural limitations that force many ad hoc approximations to transmission and mitigation processes, resulting in a significant decrease in the reliability

OCR for page 53
56 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT of simulation results under at least a subset of important conditions. The limitations include the following: • NAADSM can describe only regional transmission, in this case, within a single state; it cannot account for bidirectional transmission across state borders. • NAADSM cannot address infection in wildlife, including feral swine populations. • NAADSM is not designed to include facilities that house multiple animal species. • The spread submodels between facilities make artificial assump- tions about movement mechanisms and do not allow for accurate repre- sentation of livestock movement patterns. • There are no means of representing zoned movement controls in response to an outbreak. • The current implementation does not allow realistic modeling of the livestock culling process, inasmuch as NAADSM cannot adequately ac- count for handling times and logistic limitations (p. 451 of the uSSRA). Nor does NAADSM allow options other than culling for the final disposition of herds that are immune after infection (p. 478 of the uSSRA). • The current implementation does not allow realistic modeling of the distribution and use of vaccines during an outbreak; it does not allow for simultaneous administration of vaccines directly by producers, and it assumes an unlimited vaccine supply (p. 456 of the uSSRA). • NAADSM allows users enormous latitude in defining the quali- tative and quantitative components of transmission. This is one of the strengths of NAADSM and also its major weakness, as it relies on expert opinion to define components. Model outcomes are very sensitive to param- eter assumptions, and even when expert opinions are used they can vary and lead to wide probability distributions (Bates et al., 2003). The uSSRA discusses those limitations and the ad hoc approximations that they necessitated. Whereas these approximations likely prevent devel- opment of accurate and nuanced understandings of the consequences of variation in the logistics of mitigation, they serve as reasonable placeholders for the broad-brush results obtained in the uSSRA. Resolving these limita- tions will eventually require redesign of NAADSM or a switch to a more flexible simulation platform.

OCR for page 53
57 EVALUATION OF EPIDEMIC MODELING Limitations of Available Data The uSSRA also points out many limitations in the data available for use in NAADSM modeling, which add to the uncertainty of the results presented. The limitations in the data include the following: • The relationship between dose of FMDv and probability of an infection of an individual animal in large (e.g., thousands of animals) and in small (e.g., less than 100 animals) herds was not clarified. The relation- ship is expected to vary, depending on FMDv serotype and strain, animal species, and route of exposure, as well as on the size of the herd. • Potential exposure of Kansas State University faculty, staff, and students; NBAF employees; and Foreign Animal Disease Diagnostic School participants to livestock. • Distributions and movements of feral swine and of susceptible wildlife, such as elk and deer that may have potential for transmitting FMDv to livestock (Rhyan et al., 2008; Moniwa et al., 2012). • Animal movement (direct contact) and fomite movement (indirect contact) within and among states in the region modeled and long-range movement of susceptible animals from the region to other states. • Data on producers who are noncompliant with state and federal regulations regarding veterinary inspection, animal identification, and per- mitting and documentation of animal movement for those who buy and sell through informal arrangements and who contribute to disease spread through comingling of livestock at non-regulated events (such as swap meets) or illegal animal movements. • Some data sources used in setting model parameters are not pub- licly available, which obstructs transparency and hinders independent rep- lication of the uSSRA’s results. • Although the uSSRA’s livestock database created for the Manhat- tan area is a strong data contribution for a snapshot in time, such data can become quickly outdated with changing numbers of animals, species, and livestock movements. The uSSRA did not reference any state or federal documents that would describe a mechanism for accurately identifying and updating active premises. In the face of an FMD outbreak, it will be critical to already have in place well-validated state animal health databases, active surveillance, and premises identification. Dose–Response Modeling and Minimum Infectious Dose The uSSRA uses probit analysis to estimate the population probability of infection associated with low doses of FMDv; the risk depends on the probability of exposure to at least one viable virion when index cases are

OCR for page 53
58 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT simulated. Probit analyses can provide appropriate low-dose risk estimation for some pathogens, but the committee has concerns about the development and use of probit analyses for FMDv in the uSSRA. First, use of the probit model instead of other dose–response models merits more examination and justification than was included in the uSSRA. The [log-]probit model can in some cases underestimate dose–response (Gale, 2001) compared with the estimates produced with the exponential or beta-Poisson models. The committee is aware that uSSRA Appendix Section A6.1.2.1 states (p. A6-7) that “the exponential and beta-poisson [sic] model were also considered; however, the potential of these models to characterize the dose-response relationship of FMDv in cattle and sheep was previously studied by French et al. and was found to be unsatisfactory, particularly at low doses.” The cited dose–response modeling from French et al. (2002), however, is at odds with the text in the uSSRA, and the uSSRA does not adequately consider the French et al. analyses or earlier work by others (Sutmoller and Vose, 1997; Cannon and Garner, 1999) that were cited by French et al. (2002). The uSSRA should have provided a more accurate and transparent analysis of the cited literature and provided further details to compare results of an exponential analysis with those of a probit analysis. Second, it appears that relevant data from experimental studies were excluded, and their omission may limit the range of data used in the pro- bit estimates. Specifically, the excluded data were related to animals that seroconverted but did not show evidence of shedding in the once-daily sampling schemes. Those animals could be the very ones that should be included in the probit analysis. The animals had become infected by virtue of the seroconversion, perhaps by a low dose that resulted in short-duration shedding that was not detected in 12-hour or 24-hour sampling intervals. Inclusion may have improved the probit-derived probability estimates of low-dose infectivity. Third, the committee is concerned about continued use of the “mini- mum infectious dose” (MID) concept. The uSSRA states on p. 408 that Many researchers have proposed that there is no risk of infection for doses of FMDv lower than a certain amount, called the ‘minimum infectious dose’.... These values might represent a phenomenon in which a minimum number of pathogen particles are required to overcome host defenses and establish an infection, or they could be an artifact of the use of a small number of animals in infection experiments (i.e., if five animals were used, identifying doses that cause less than a 20% probability of infection is difficult). The latter argument is legitimate in that these experiments have sample sizes that are statistically inadequate to estimate the risk of infection at low doses (Haas et al., 1999; NRC, 2005). However, the “minimum infectious

OCR for page 53
59 EVALUATION OF EPIDEMIC MODELING dose” concept is not credible. The comment about researchers proposing no risk below a particular threshold is related mainly to the older mi- crobial risk-related literature. Recent work (e.g., Haas et al. 1999; Gale, 2001) typically applies dose–response modeling with the best available data (possibly including meta-analyses) and extrapolates low-dose with probit, beta-Poisson, exponential, or other dose–response models. For pathogens on which reliable data for dose–response analyses are available, there is no population threshold dose (NRC, 2005). Fourth, the uSSRA does not provide an adequate discussion of the un- certainties in the FMDv dose–response modeling using the probit model or its alternatives nor does it provide an adequate discussion of their applica- tion to predicting herd response. Failure to do so may leave the impression that the dose–response predictions used in the probit model are highly certain, and this is not the case. The statistical reliability of dose–response modeling is briefly discussed, but its impact on the results is not adequately analyzed. Results could be sensitive to uncertainties such as FMDv strain differences, experimental dosing regimen (often bolus) compared to the potential herd exposures resulting from a leak, and differences between the experimental animals’ status and that of the target animal herds (e.g., species or breed, immune competency, concurrent infections, environmental stresses). The direction and magnitude of these effects may be unknown for FMDv, but they nevertheless remain as uncertainties in the extrapolation to herd response that were not adequately addressed in the uSSRA. Assumptions About Available Response Resources and Capacities The uSSRA makes various assumptions about foreign animal and zoo- notic disease response capabilities presumed to be in place at the time of the anticipated NBAF opening in 2020. It will be important to have these tested capabilities in place from day 1 to mitigate the effects of an acciden- tal release of an infectious agent. The committee notes that many of these assumptions are unrealistic today and that making them realistic would require major investments and considerable political will before the NBAF opens. Whereas the uSSRA does not discuss future investment require- ments, it does acknowledge that capabilities will be changing over the next 8 years. Concerns about the assumptions related to capabilities include the following: • Vaccination would begin very early (on day 7) in an FMD epi- demic. Also, once vaccination is initiated, single-dose, high-potency, 100% efficacious emergency vaccine would be available in unlimited quantities. It is further assumed that 100% of vaccinated animals would be protected from infection. These assumptions would apply for all 7 serotypes of

OCR for page 53
60 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT FMDv and for all the strains within each serotype that could escape from the NBAF. These assumptions are inconsistent with the current state of knowledge. • It would take 3–11 days to vaccinate all herds in Kansas (an aver- age of 90 herds per day). • 100% of cattle and swine producers will report a suspicious case in less than 2 weeks following infection; this is unrealistic. • Laboratory testing capacity for the presence of FMD virus (RRT- PCR [Callahan et al., 2002]) and virus isolation and for the presence of antibody is assumed to be unlimited, with a range in the turnaround time for testing of only 0–2 days. Laboratories in the National Animal Health Laboratory Network currently do not have the capability to conduct sero- logical tests for FMD. • Diagnostic laboratory tests are assumed to be of exceptionally high accuracy and reliability, and perfect accuracy is assumed in detecting FMD on 100% of infected premises. Furthermore, the lack of real-time FMD surveillance, as acknowledged in the uSSRA, diminishes the likelihood of early detection and control. The uSSRA states that “economic estimates based on the outputs of the economic model for the Updated SSRA will, again, underestimate the absolute impact of the outbreak of FMD originating from the NBAF be- cause the outbreak is artificially limited to the region modeled instead of the whole of North America” (p. 405). Many of the limitations listed above are also likely to result in underestimation of the extent and cost of a potential release of FMDv from the NBAF in Manhattan, Kansas. Other Sensitivity Analysis The epidemic modeling section provides the only sensitivity analysis that has any degree of rigor in the three volumes of the uSSRA. This section of the uSSRA provides a correct and important caveat about the useful- ness of the estimates (p. 534): This analysis informs how much confidence can be placed in the results as absolute reflections of the impact of an FMD outbreak given that some of the modeling parameters are based on scanty evidence. As discussed, epidemiological models are best used to understand relative risk and rela- tive benefit of risk mitigation measures because inaccuracies in a model are reflected in the baseline and experimental cases, largely cancelling each other out. The uSSRA further discusses that variation in the contact rate of less than an order of magnitude (a factor of 0.5–2) changes the duration of an

OCR for page 53
61 EVALUATION OF EPIDEMIC MODELING epidemic by over an order of magnitude (see pp. 537–538). However, many parameter values have greater uncertainty—with ranges that span several orders of magnitude. Distributions based on these wider ranges should have been provided in the sensitivity analyses because that would provide better information on the most important components of uncertainty in the results. REFERENCES Anonymous. 2001. Lessons from an epidemic. Nature 411(6841):977. Bates, T.W., M.C. Thurmond, and T.E. Carpenter. 2003. Description of an epidemic simulation model or use in evaluating strategies to control an outbreak of foot-and-mouth disease. Am J Vet Res 64(2):195-204. Callahan, J.D., F. Brown, F.A. Osorio, J.H. Sur, E. Kramer, G.W. Long, J. Lubroth, S.J. Ellis, K.S. Shoulars, K.L Gaffney, D.L. Rock, and W.M. Nelson. 2002. Use of a portable real- time reverse transcriptase-polymerase chain reaction assay for rapid detection of foot- and-mouth disease virus. J Am Vet Med Assoc 220(11):1636-1642. Cannon, R.M., and M.G. Garner. 1999. Assessing the risk of wind-borne spread of foot-and- mouth disease in Australia. Environ Int 25(6-7):713-723. Dickey, B.F., T.E. Carpenter, and S.M. Bartell. 2008. Use of heterogeneous operation-specific contact parameters changes predictions for foot-and-mouth disease outbreaks in complex simulation models. Prev Vet Med 87(3-4):272-287. French, N.P., L. Kelly, R. Jones, and D. Clancy. 2002. Dose-response relationships for foot and mouth disease in cattle and sheep. Epidemiol Infect 128(2):325-332. Gale, P. 2001. Developments in microbiological risk assessment for drinking water. J Appl Microbiol 91(2):191-205. GAO (U.S. Government Accountability Office). 2002. Foot and Mouth Disease: To Protect U.S. livestock, USDA Must Remain Vigilant and Resolve Outstanding Issues. Available online at http://www.gao.gov/new.items/d02808.pdf (accessed March 30, 2012). Haas, C.N., J.B. Rose, and C.P. Gerba. 1999. Quantitative Microbial Risk Assessment. New York: John Wiley. Kitching, R.P., A.M. Hutber, and M.V. Thrusfield. 2005. A review of foot-and-mouth disease with special consideration for the clinical and epidemiological factors relevant to predic- tive modelling of the disease. Vet J 169(2):197-209. Kitching, R.P., M. Thrusfield, and N.M. Taylor. 2006. Use and abuse of mathematical models: An illustration from the 2001 foot and mouth disease epidemic in the United Kingdom. Rev Sci Tech 25(1):293-311. Mansley, L.M., A.I. Donaldson, M.V. Thrusfield, and N. Honhold. 2011. Destructive tension: Mathematics versus experience—the progress and control of the 2001 foot and mouth disease epidemic in Great Britain. Rev Sci Tech 30(2):483-498. Moniwa, M., C. Embury-Hyatt, Z. Zhang, K. Hole, A. Clavijo, J. Copps, and S. Alexandersen. 2012. Experimental foot-and-mouth disease virus infection in white tailed deer. J Comp Pathol Apr 18. NRC (National Research Council). 2005. Reopening Public Facilities After a Biological Attack: A Decision-Making Framework. Washington, DC: The National Academies Press. NRC. 2010. Evaluation of a Site-Specific Risk Assessment for the Department of Homeland Security’s Planned National Bio- and Agro-Defense Facility in Manhattan, Kansas. Wash- ington, DC: The National Academies Press.

OCR for page 53
62 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT Rhyan, J., M. Deng, H. Wang, G. Ward, T. Gidlewski, M. McCollum, S. Metwally, T. McKenna, S. Wainwright, A. Ramirez, C. Mebus, and M. Salman. 2008. Foot-and- mouth disease in North American bison (Bison bison) and elk (Cervus elaphus nelsoni): Susceptibility, intra- and interspecies transmission, clinical signs, and lesions. J Wildl Dis 42(2):269-279. Smith, G. 2011. Models of macroparasitic infections in domestic ruminants: A conceptual review and critique. Rev Sci Tech 30(2):447-456. Sutmoller, P., and D.J. Vose. 1997. Contamination of animal products: The minimum pathogen dose required to initiate infection. Rev Sci Tech 16(1):30-32.