Evaluation of a Site-Specific Risk Assessment for the Department of Homeland Security's Planned National Bio- and Agro-Defense Facility in Manhattan, Kansas (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Evaluation of Methods MAJOR MODELING ASSUMPTIONS AND ERRORS Risk of Release Assumed Low Frequency of Aerosol Release Considering the various sources of spillage and the resulting aerosolization, the site- specific risk assessment (SSRA) made a key assumption about the frequency of aerosol release of foot-and-mouth disease virus (FMDv) that may result in an underestimation of actual risk. The estimated rate of 2.6 laboratory-scale spills per year is low, but no confidence intervals were given for the estimate, which would have allowed some assessment of a maximum-credible risk scenario. The estimated spill rate is low for several reasons: (1) In some instances, a person responsible for the spill either would not report it or would be unaware of it. (2) The spill rate does not include the likelihood of spills from sample shipments or damage occurring during shipment. The National Bio- and Agro-Defense Facility (NBAF) may insist on the use of special shipping containers, but experience in other laboratories shows that not everyone will use them. Shipping containers will enter the NBAF from foreign countries where primary sample containers and shipping boxes, although required, may not meet U.S. regulations. The likelihood of spills caused by damage after a conveyance accident was not addressed in the SSRA (for example, a spill may not be immediately apparent on inspection of a container that is transported to the laboratory by a courier). (3) Of greatest concern is the omission of anticipated virus spillage that would occur routinely as part of regular cleaning of large animal rooms. The NBAF will be equipped for biosafety level 3 agriculture (BSL-3Ag) and BSL-4 research with large animals. Rooms housing infected large animals that shed large amounts of FMDv (or other foreign animal or zoonotic pathogens) are typically washed down with water at least once a day, depending on the accumulation of feces, urine, and feed . The cleaning process will deliver kinetic energy (both through manual labor and through 27 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 28 high-velocity water) that will impact indoor air mixing regimes and will generate both aerosol and fomite loads, which were not appropriately considered in the SSRA. Wash-down is likely to generate fine aerosol from shear, bursting bubble film, and jet droplets. The latter two mechanisms are known to generate an aerosol that contains a higher content of virions or bacteria per unit volume than what is found in the bulk water phase (Baylor et al., 1977; Hejkal et al., 1980; Blanchard and Syzdek, 1982). The wash-down process would aerosolize virus deposited in the room from animal secretions and excretions, and would result in removal of massive amounts of virus through the air filtration system. Even with the use of disinfectants,1 the committee feels that those sources offer more frequent (daily) opportunities and possibly higher viral loading than the laboratory-scale spills that were evaluated in the SSRA. If only one room were used for FMD experiments, it would be the equivalent of experiencing 365 necessary and anticipated spills per year. Such a spill rate would raise the risk estimates by a factor of more than 140 from what is given in the SSRA (365/2.6 = 140). An important factor that was neglected in the SSRA is the distinction between real and simulated conditions for viral disinfection and natural viral decay. The hosing of waste materials (such as secretions and excretions) would create a protective bioburden matrix for virus particles, and their aerosolization would lead to a severe underestimation of the amount and duration of potentially infectious material generated. The SSRA also did not address the effects of large amounts of aerosolized material (such as dust, dander, and other particles [such as fur, feed, vomit, cud, mucus, and hoof detritus]) on high-efficiency particle air (HEPA) filters in animal rooms and how it would affect filter performance over time. Laboratory-Release Risk Estimates Not Based on Real World Experiences The SSRA did not adequately consider case histories in arriving at risk estimates of laboratory leaks, and information from the documented cases of FMD releases were not fully taken into account. Although the NBAF will be engineered and constructed to a new level of safety, an examination of past FMD incidents from laboratory releases needed to have been considered in the SSRA. Between 1960 and 2007, there were 15 known escapes of FMDv from laboratories worldwide, with 13 occurring in Europe, 1 in Russia, and 1 in the United States (Anderson, 2008; GAO, 2008). Lessons learned from the escapes of FMDv from the Plum Island Animal Disease Center (PIADC) in 1978 (GAO, 2008) and Institute for Animal Health Pirbright Laboratory in the UK in 2007 (Anderson, 2008) appear not to have been applied in deriving reasonable expectations for risks of virus escape. When DHS was asked about this at the public session of the committee’s meeting, a somewhat confusing answer was provided: that the escape from Plum Island (Margasak, 2008) was irrelevant because livestock were being housed on the island, and this will not be the case for the NBAF, which will be in Manhattan, Kansas. The facts remain that an FMDv escape did take place and that there are many cattle near the proposed site 1 The protocol for the use of disinfectants at the NBAF has not been established nor was it discussed in the SSRA. Disinfectants would likely be used in the final cleaning of a vacated room, but routine use during an experiment would affect indoor air quality (and could impose human and animal health risks) and affect solid and liquid wastewater treatment (and could affect equipment performance). The use of disinfectants would still aerosolize particles and increase the risk of fomites. The complicated mix of organic solids and liquids present in animal rooms will serve as chemical and physical barriers to the intended microbial targets, and will thus reduce the predictability and efficacy of disinfection. This deserves careful consideration and a dedicated standard operating procedure to determine the appropriate cleaning and disinfection procedures and the dosing and delivery of disinfectants. 

EVALUATION OF METHODS 29 in Manhattan. In addition, the escape of FMDv from the Institute of Animal Health Pirbright Laboratory in 2007 has been reported to have occurred through a series of events that would seem highly improbable. The committee recognizes that the risk is not zero, and that unexpected events can result in an inadvertent release of FMDv. The SSRA states that the estimated frequency of failure of the liquid effluent decontamination system is once every 2.1 million years. However, such a failure was the cause of the release of FMDv from Pirbright in 2007 (UK-HSE, 2007); and in June 1999, just before the National Centre for Foreign Animal Disease in Winnipeg, Canada, became operational with infectious agents on board, a batch of untreated wastewater was accidentally released into Winnipeg’s sewage system (Löfstedt, 2002). Both incidents are well known and will raise questions about the SSRA’s estimated frequency of once every 2.1 million years. There have been many documented leaks of dangerous viruses from laboratories, including a leak of the Sabiá virus from a laboratory in Brazil (Lemonick and Park, 1994) and a leak of Venezuelan equine encephalitis virus from a laboratory in Colombia, which was cited as the likely cause of a human epidemic in 1995 (Brault et al., 2001). There have been leaks of FMDv from the Plum Island (GAO, 2008) and Pirbright laboratories, most notably the recent outbreak at Pirbright in 2007 (Derbyshire, 2007), which apparently was not considered in the SSRA. The committee believes that an assessment based on a plethora of information on case histories of escape of agents from laboratories would likely have provided a more realistic assessment of the case scenarios, likely frequencies, and confidence intervals of laboratory escapes projected for the NBAF. Risk Uncertainties and Variation of Risk over Time Not Considered The SSRA does not discuss or quantify uncertainties in the risk estimates. The Review of the Department of Homeland Security’s Approach to Risk Analysis noted that “there is little understanding of the uncertainties in DHS risk models… and in addition there is a tendency toward false precision” (NRC, 2010), which was also the case with the SSRA. The risk estimates included in the SSRA are presented as single point estimate numbers to two decimal places, which implies a level of precision in the risk estimates that is likely unrealistic. While sensitivity analyses of several case model components were undertaken, the SSRA does not provide a quantitative assessment of the uncertainty surrounding the case event risk estimates (for example, in the form of confidence intervals) nor is there a qualitative discussion of the sources and magnitude of the uncertainties associated with these scenario risk estimates. Furthermore, the SSRA neglected to consider how risk would vary over time. The annual risk estimates presented in the SSRA assume constant annual risks over the 50-year life span of the NBAF. However, annual risks will not necessarily remain constant over time because operating practices, experimental design and equipment, and staffing—among many other aspects—could result in either improvements or degradations that accordingly decrease or increase risks. The SSRA does not address the variation of risk over time in either a quantitative or qualitative manner. The potential for degradation and the subsequent means to address such risk were not considered in the SSRA. 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 30 Aggregate and Cumulative Risk The SSRA failed to provide an appropriately aggregated assessment of cumulative risk over the expected life span of the NBAF. Although previous NRC reports have highlighted the need for assessing overall risk (e.g., NRC, 1994, 2009), the SSRA did not provide a cumulative risk assessment2 for multiple agents and stressors by all routes and pathways to determine the overall risk of operating the NBAF in Manhattan, Kansas. At a minimum, the SSRA needed to have provided aggregate3 risk, which is an essential component of risk characterization (NRC, 1994). The need to include lifetime risk estimates is consistent with the mandate included in the Homeland Security Presidential Directive 9 to “develop a plan to provide safe, secure, and state- of-the-art agriculture biocontainment laboratories that research and develop diagnostic capabilities for foreign animal and zoonotic diseases,” and is also consistent with the recommendation included in the 2008 National Research Council report that the Department of Homeland Security (DHS) address the probabilities of a sequence of events that would lead to a pathogen release (NRC, 2008). When the committee prompted DHS to provide an estimate of overall risk based on the completed risk assessment, the agency responded by stating that “the NBAF operations in Manhattan, Kansas overall brings extremely low risk relative to the greater risk of the intentional or accidental introduction of FMDv by an external source” (Response to question 1, DHS follow-up letter, July 28, 2010). The committee finds that the comparison is misleading because the SSRA does not consider or quantify the risk of infection from an external source. DHS responded furthermore that it was beyond its congressional mandate to determine an overall estimate of risk, and that the scope of the SSRA was limited to determining release scenarios and integrating their economic impacts (Response to question 1, DHS follow-up letter, July 28, 2010). On the basis of the release scenarios that were each deemed as “low risk”, DHS ranked the scenarios by “risk dollar”. 4 There are several issues related to DHS’s attempt to present estimates of risk. It is inaccurate to conclude that risk is “low” without further definition that would make its meaning less subjective. Although DHS did calculate risk by case frequency as cases per year (see Table 6-3 of the SSRA), it chose to present “risk” on the basis of economic consequences for a series of independent scenarios (see Table 6-4 of the SSRA). As indicated in Chapter 2 of the present report, risk assessment metrics often include the probability of an event over a period of time, yet the SSRA focused on single-year risks. The committee believes that it is misleading to convey risk in dollar amounts. The SSRA should have presented the risks for each scenario over the expected 50-year life expectancy of the NBAF, the associated costs (taking into account interest and inflation), and the cumulative risk for the NBAF with its expected lifetime operations in Manhattan, Kansas. The committee found that the SSRA fell short of considering the cumulative effect of all independent scenarios and estimating the overall (cumulative) risk of release that would result in 2 The 2009 National Research Council report Science and Decisions: Advancing Risk Assessment defines cumulative risk as “the combination of risks posed by aggregate exposure to multiple agents or stressors in which the aggregate exposure is by all routes and pathways and from all sources of each given agent or stressor.” 3 According to the 1994 NRC report Science and Judgment in Risk Assessment: “An essential component of risk characterization is the aggregation of different measures and characteristics of risk: the risk assessor must communicate measures and characteristics of predicted risk in ways that are useful in risk management.” 4 The committee notes that the term “risk dollars” may be useful as a way of ranking risks to identify which actions have the highest payoff to mitigate the impact, but the term can easily be misinterpreted as a measure of annual mitigation costs or total dollars at risk. 

EVALUATION OF METHODS 31 infection with pathogens that will be handled in the NBAF. The case frequencies presented in Tables 6-3 and 6-4 of the SSRA are the products of the probability of pathogen release and the probability of infection in each scenario per year. The resulting frequencies of infection caused by laboratory release are difficult to interpret because they present individual risks without an overall risk of an index case over a period of 50 years.5 Given the expected 50-year operational life span of the NBAF, the question is what the likelihood of a release and a disease outbreak will be in that period. Overall risk for the events listed in Table 6-3 of the SSRA could have been estimated by aggregating the annual frequencies of the scenarios leading to each independent event for each agent. The SSRA failed to provide an overall risk estimate for the accidental release of even a single agent such as FMDv. A lower-bound estimate of the cumulative risk of release that results in FMD infection over the NBAF’s 50-year life expectancy can be obtained by using the sum of two of the SSRA’s scenarios with the greatest risk of FMDv release from Tables 6-3 and 6-4 (FMDv fomite personal contamination and FMDv worker with no respiratory protection). Figure 3-1 illustrates the results of the overall risk calculation6 for FMDv. Over the NBAF’s 50-year life span, the probability of an FMD infection from a laboratory release approaches 70% (100% would be absolute certainty), which the committee does not consider to be low. Even if the risk scenarios were not aggregated and if the top scenario of FMDv fomite release were the only risk considered, the SSRA estimates still show a probability that is not low. With the SSRA showing an accidental FMDv fomite release7 leading to an infection recurring every 77 years and costing a mean of $32 billion (Tables 6-3 and 6-4 of the June 2010 SSRA), such an event has a 48% probability of occurring over the expected 50-year life span of the NBAF, assuming that the SSRA estimates provided for the committee’s evaluation are accurate. The committee notes that these numbers probably represent conservative estimates because the SSRA overlooked other factors that would elevate risk. 5 The case summary tables for FMDv worker with and without respiratory protection (pages 117 and 118 of the SSRA) do contain assessments of the event frequency for the projected 50 year operations of NBAF. These two FMDv events are for fomite carryout and are important findings of the SSRA. However, the SSRA does not aggregate independent events such as these two examples to give an overall estimate of the operational risk by all scenarios for the operational life span of the NBAF. 6 For the two scenarios with the greatest risk of FMDv release provided in Tables 6-3 and 6-4 of the SSRA, the total probability of a release that results in an index case is 2.33% for any year. Figure 3-1 of the present report was generated by using a binomial distribution and assuming that pathogen release during any year is a Bernoulli trial with a probability of infection of 0.0233. A Poisson distribution based on an infection frequency of 0.0233 per year yielded similar results. Regardless of the distribution used to calculate the probability, the probability that at least one index case (infection) will occur in 50 years is about 70%; this is a lower-bound estimate and not the overall risk for all scenarios in the SSRA. This number assumes that risks are constant and it does not account for the variation of risk over time. As such, it is an approximation of the cumulative risk over 50 years of operation. Some risk factors will change over time: some likely will increase risk as the facility ages while others may lower risk due to improvements in technology and practice and planned mitigation strategies at the NBAF. 7 The SSRA states on pages 329-330: “For FMDv, the vector-to-susceptible animal transmission rates varied widely and many approached 100%. Thus, it was decided to use Pi = 1x100 for the probability of infection given an accidental FMDv release through the fomite/vector/pathway. It was recognized that this probability is high, but it represented the probability of the pathway and was appropriate for the vector case.” 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 32 80% 70% Cumulative Risk of an FMD Index Case 60% 50% 40% 30% 20% 10% 0% 1 10 20 30 40 50 Years of Operation Figure 3-1 Risk of release that results in FMD infection over the life span of the NBAF. The probability was estimated by summing the risk from the two scenarios in the SSRA’s Table 6-4 that had the highest risk (FMDv fomite personal contamination and FMDv worker with no respiratory protection). MODELING CRITICAL TORNADO AND AIR DISPERSION SCENARIOS Wind and Tornado Model Used by the Site-Specific Risk Assessment The SSRA analyzes the likelihood of tornadoes in Manhattan, Kansas, on the basis of tornado data on 1950-2009 archived at the Storm Prediction Center (SPC) of the National Oceanic and Atmospheric Administration, the only dataset available for assessing the risk of tornadic wind speed and associated effects. The SSRA shows tornado tracks from 1950-2009 covering a rectangular area of 29-43° latitude and 91-105° longitude; the NBAF is near a corner of the rectangle. In addition to the recorded tornadoes, the archived data are enhanced by potentially unrecorded tornadoes by using the method advanced by Ray et al. (2003). The enhancement is generally not performed to assess tornado risk although it provides a 

EVALUATION OF METHODS 33 conservative answer for tornado risk because it predicts additional tornadoes that are not part of the SPC dataset. The SSRA’s enhanced tornado data were analyzed statistically for frequency of tornado occurrence in an area surrounding the NBAF site. The analysis roughly predicted the occurrence of tornadoes of a particular F-scale intensity8 in a fairly large box of 10 km square and yielded a mean return period of 77 years for tornadoes with F2 or greater intensity and a mean return period of 300 years for tornadoes with F3 or greater intensity (Table 5 in Appendix H of the SSRA). It did not predict wind speed versus time (or the mean recurrence interval) for the Manhattan, Kansas, point location. The analysis failed to include the area-intensity relationship and failed to assess the probability that a point would experience tornado wind intensity. The SSRA states that design and construction will provide structural integrity and continued containment if the NBAF is in the direct path of a tornado of F2 intensity (or a wind speed range of 113-157 mph). The tornado scenario in the SSRA assesses the risk of a direct hit of the facility by a tornado of F3 or greater intensity, and it makes a poor assumption that the release would be minimal even if the facility were damaged and containment were lost. To justify that assumption, the SSRA is forced to make several subjective judgments so that it can consider a release or accident to be a rare event. Alternative Model for Determining Wind Speed A site-specific probabilistic tornado wind hazard model developed by the Lawrence Livermore National Laboratory (Boissonnade et al., 2000) has been used by the Department of Energy in developing design evaluation criteria for its facilities (DOE, 2002). The use of a tornado hazard model would provide a more accurate assessment that correlates tornadic wind speed with the annual probability of occurrence (or the mean return period) for Manhattan, Kansas. Its use would eliminate the need for subjective judgment on the low probability of overlap of tornadic wind speed and the NBAF site. In addition, a tornado hazard model could provide the specificity needed to determine specific wind speeds and a given time period rather than a range of wind speeds related to F-scale tornado categories. In response to Question 43 from the committee (follow-up letter, August 26, 2010), DHS outlined preliminary plans for conducting a tornado wind hazard model for the final edition of the SSRA. However, the results of the planned tornado hazard model to be pursued by DHS were not available for the committee’s review, thus it is not possible to comment on its efficacy. In wind climatology, there is an inverse relationship between wind speed value and the probability of tornadic wind speed occurrence, and that relationship is shown as a continuous plot of wind speed v. probability of occurrence. If a rare wind speed event of 1 x 10-5 per year or less is chosen on which to base building designs, the facility envelope can be designed to provide greater containment protection to prepare for a rare high-wind event. 8 As of February 2007, the National Weather Service replaced the F-scale with an EF-scale. The archived tornado data were recorded in terms of the F-scale. The development of the EF-scale includes correlation of wind speed on both F- and EF-scales, and this correlation is provided in the Enhanced Fujita scale report (McDonald and Mehta, 2008). If the tornado hazard model for the site is developed to yield wind speed on the F-scale, the wind speed for tornado risk and design can be translated to equivalent EF-scale wind speed. It should be noted that all tornadoes after February 2007 are categorized in terms of the EF-scale. Accommodation should be made in combining data before and after February 2007. The Nuclear Regulatory Commission has used the EF-scale in updating its requirements for nuclear power plants (Nuclear Regulatory Commission, 2007; Ramsdell and Rishel, 2007). 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 34 The facility envelope will need to be designed on the basis of the determined wind speed and not used merely for the facility’s structural integrity; if the envelope of the facility is properly designed, there will be little chance of pathogen release to the atmosphere. The design- basis wind speed obtained from the tornado hazard model may or may not be higher than that of the F4 tornado that occurred in 2008 in the Manhattan, Kansas, vicinity. Engineering best practices do not dictate that facilities be designed for the most severe event that may have occurred in the past or that has a chance of occurring in the future. Relying on information from a reliable tornado hazard model assessment and relying on the inherent resistance in building materials and the structural framework, the committee is confident that a facility can be built to withstand high-wind events and provide a safe environment in the rare event of an occurrence such as a high-intensity tornado. Aerosol Formation and Dispersion Estimating the risk of infection in animals through exposure to airborne virus particles consists of modeling the following processes: 1. Release of virus particles. 2. Transport, dispersion, and deposition of the released particles. 3. Survival of the virus particles. 4. Inhalation of air that contains ambient concentrations of virus particles that have undergone processes 2 and 3. 5. Infection associated with dose. Virus particles released over a short period are modeled as a “puff” of material that increases in size as it travels from the source to a receptor. The concentration of virus particles in the puff depends on meteorological variables (such as wind speed and turbulence) that govern the transport, dispersion, and deposition of the released material. It also depends on ambient temperature and relative humidity, which determine the viability of the virus. The concentration in the air also depends on the size distribution of the virus and on the properties of the surface on which the virus is deposited. For an aerosol that contains infectious virus particles, the infectious dose (ID) is the number of virus particles inhaled as the puff travels past an animal.9 The ID determines the risk of infection caused by the virus. The dose is calculated by summing the virus particles in the air that is inhaled as a puff passes over the animal. The ID depends on the breathing rate of the animal and the concentration of the virus particles in the air. For a highly contagious disease, such as FMD, the risk of infection corresponds to the collective dose to all the animals exposed to the puff of virus particles. Thus, the dose calculation requires summing the doses to all the animals exposed to the puff of virus particles. If enough animals are exposed, infection may be likely even if the doses are very small. 9 The SSRA uses the terminology “0.1 plaque forming unit (pfu)” to estimate the risk that escaped virus will cause infection in animals. The use of the term pfu is vague, unverifiable, and inconsistent with contemporary means for quantifying virus and estimating a minimal infectious dose. In the preliminary letter report, the committee recommended using a low infectious dose for determining the likelihood of infection. For the rest of its final letter report, the committee uses the more accurate terminology “0.1 virus particle”. 

EVALUATION OF METHODS 35 Appropriateness of the Second-Order Closure Integrated Puff Model In general, the SSRA follows accepted approaches to modeling the risk of airborne infection whereas the earlier environmental impact statement (EIS) did not. The EIS used a simple Gaussian dispersion model to estimate doses associated with a continuous source of virus particles at distance of 10 km. It did not go beyond suggesting that the dose at 10 km was less than the assumed minimum ID of 10 virus particles. The SSRA responds to the Government Accountability Office’s valid criticism of the EIS by recognizing that the releases occur over a short period compared with the travel time from the source of material to the receptor. The transport and dispersal of the resulting puff is modeled with the Second-Order Closure Integrated Puff (SCIPUFF) model (EPA, 2000; Sage Management, 2010). Although SCIPUFF is not the preferred model for regulatory applications, it is clearly a model with a solid scientific foundation and is one of the best models available. SCIPUFF is one of few models that incorporate realistic dispersion of instantaneous releases. It also incorporates the complex processes that govern transport of virus-containing aerosols, and this makes it a complex model with a large number of inputs, outputs, and options. That complexity can lead to errors in inputs that lead to errors in outputs. It is not clear from the SSRA that tests have been conducted to check for calculation errors. The SCIPUFF model is used to simulate a number of hypothetical releases. The effect of those releases is measured in terms of the area over which the inhaled dose exceeds 0.1 virus particle. The minimum value of 0.1 virus particle is based on the argument that in a group of animals, a small number might be exposed to one virus particle that results in infection whereas the majority would not inhale any pathogens. The assumption is made to decouple the dispersion calculations from the epidemiological modeling. The SCIPUFF model calculations provide contours within which the dose exceeds 0.1 virus particle. Those contours are then superimposed on livestock distribution maps for the epidemiological modeling. If a farm or facility where livestock can be infected lies within a contour, all the animals in the premises are taken to be infected. Criticism of the Method The probability of infection is sensitive to the size of puffs released and to the number of releases during a period of time. Small laboratory spills that are estimated to occur 2.6 times per year are much less important sources of risk than the daily washing of animal pens, which has the potential to aerosolize much larger numbers of virus particles. Even with the use of disinfectants during cleaning of the animal pens, the daily cleaning of animal pens will probably result in releases that far exceed the 103 virus particles that are estimated to be released when a container is dropped even when a HEPA filter is functioning properly; the “material available for release” is likely to be much larger than the 1012 assumed in the SSRA, and the aerosolized fraction will be larger than the 10-4 value assumed for an accidental spill from a container. Because the probability of infection depends on both the amount of release and the frequency of release, a binomial model for infection indicates that even a small infection probability of 0.01 for an individual release translates into a 63% probability of infection when there are 100 releases, which corresponds roughly to the number of releases resulting from pen cleanings per year. It is not clear from the SSRA that the arbitrary value of 0.1 virus particle represents a sharp cutoff that would exclude premises with a slightly smaller dose (such as 0.099 virus 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 36 particle) from the risk of infection. The assumption of a minimum dose could have been avoided by calculating the integrated dose over the group of animals exposed to a puff of virus particles. That approach would have required the dispersion calculations to explicitly account for livestock distribution in computing the integrated dose. The dose could then be related to the risk of infection at any specified location relative to the release location. DHS has added information on exposure distances to supplement the exposure areas presented in Tables 3-31 and 3-32. These distances have the same problem as the impact areas: they are based on the arbitrary cutoff of 0.1 virus particle. Furthermore, they do not reflect uncertainties in meteorological variables, surface parameters, and emission strength. The exposure distance is directly proportional to the emission of viruses, which could be much higher than the 103 FMDv particles assumed for routine releases. The largest doses are likely to occur when the boundary layer height is small, the wind speed is low, and the period over which the virus is viable is long, the dry-deposition velocity is low, and the relative humidity and temperature favor FMDv viability. That corresponds to stable, low-wind speed conditions in which meteorological models are notoriously unreliable (Luhar et al., 2009). Furthermore, several inputs cannot be specified objectively, such as the surface parameters (for example, roughness length). That suggests a need to conduct sensitivity studies to examine the effects of uncertainty in meteorological variables and values of model parameters on predicted doses. The SSRA assumes that the variation in meteorological inputs captures most of the dose variation associated with uncertainty in model inputs. That assumption implies that the distribution of modeled doses will be close to that of observed doses—an assumption that needs to be justified by showing that variations in values of surface parameters have only a small effect on computed doses. In response to how input uncertainty is accounted for in the models, DHS states that “due to the computation burden of fully exploring the entire parameter uncertainty space associated with these inputs, ‘reasonable worst case’ estimates were used” (Response to question 4, DHS follow-up letter, July 28, 2010). However, the SSRA does not provide information on the release and meteorological conditions that constitute a “reasonable worst case”, so thus the committee could not judge the validity of that approach for treating model uncertainty. The results presented in the SSRA suggest that the horizontal scale of the effect of accidental releases of FMDv is in tens of kilometers. Under these circumstances, it might have been appropriate to assimilate the Manhattan Regional Airport meteorological data into the database constructed by Rife et al. (2010), which has a relatively coarse resolution of 40 km. The appendix to the SSRA states that the Rife et al. database was constructed to “recreate the observed characteristics of the Great Plains Nocturnal Low Level Jet”; however, there are no references to support the SSRA’s claim that the dataset was “specifically developed and subsequently validated to support boundary layer aerosol transport and dispersion modeling applications” (SSRA Appendix J: Aerosol Fate and Transport (Plume) Modeling). The use of site-specific meteorology, such as that available from the airport, is likely to be important in developing emergency response plans for an accidental release of virus. Because the airport might be the only source of meteorological information during an actual emergency, it is important to run scenarios corresponding to airport data. If resources are available, the results will need to be evaluated with data from tracer studies at the NBAF location. 

EVALUATION OF METHODS 37 EPIDEMIOLOGICAL MODELING The SSRA applied the North American Animal Disease Spread Model (NAADSM), a stochastic model used to obtain multiple simulations of the hypothetical spread of FMD in a primary modeling region that included neighboring states after the establishment of index cases resulting from various FMDv escape scenarios. The simulation runs generated probability distributions for the number of animals destroyed and durations of FMD epidemics in each scenario. Sensitivity analysis was conducted to determine how the probability distributions changed in response to various initial conditions, assumptions, and parameter values, including some related to mitigation. Overall estimates of epidemic magnitude and duration were used as the bases of the SSRA’s economic analyses. The epidemiological modeling component of the SSRA provides key inputs to the economic assessment, thus any concerns about that component will affect the economic analyses and conclusions of the SSRA. The SSRA demonstrates substantial effort in modeling FMD spread, given the short period of time to conduct the assessment and the absence of critical data and well-documented assumptions for many parameter values. In particular, the SSRA team made a clear and constructive effort to collect data and estimate values for model parameters, including attempts to estimate locations of livestock herds throughout Kansas and nearby states. Overall Concerns The SSRA is unclear as to how the specific mitigation parameter values were determined, in part because of the inherent complexity of the NAADSM (Schoenbaum and Disney, 2003; Harvey et al., 2007) but also in part because of the SSRA’s focus on a more general analysis. The ambiguity could theoretically be partially alleviated by the existing documentation and openness of the NAADSM. In practice, however, applying parameter values and interpreting the NAADSM results for the risk assessment will require personnel who have NAADSM expertise and familiarity with veterinary epidemiology. In computational epidemiology, it is common to conduct sensitivity analyses of model outcomes in terms of model parameters. However, model parameters are usually phenomenological representations of subprocesses that the model makers have chosen not to include for reasons of insignificance, efficiency, uncertainty, or simplicity. In the case of this SSRA, realistic constraints on mitigation measures—such as supply, cost, and efficiency—have been lost because of particular modeling decisions. Without realistic constraints to provide context, outcomes from simulation of mitigation measures and associated sensitivity analyses cannot be turned into practical recommendations. The quantitative epidemiological study in the SSRA provides a systematic and reasonable description of the distribution of outcomes in light of variations in mitigation rates. However, it does not connect the general mitigation rates to the logistics of specific mitigation practices in a site-specific manner. For instance, the SSRA uses a baseline culling rate of 120 herds/day (page 230) but does not consider the logistical demands that culling would place on personnel and equipment, nor does it allow an emergency response manager to gauge the reasonableness and relevance of the baseline scenario. The report also observes that an outbreak’s risk can be “nearly completely mitigated” by active surveillance (page 225), but it does not discuss what implementation measures this would require. Those broad concerns about the overall validity of 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 38 the current quantitative epidemiological analysis of mitigation also apply to some of the results. Figure 4-20 of the SSRA, for instance, shows a bimodal distribution of epidemic outcomes. The bimodal effect may be due to random differences in initial condition, as suggested in Figure 4-14 of the SSRA, but bimodality over the given range of 3-6 million animals is surprising. It is well known that the probability an index infection leads to a widespread epidemic will depend on an initial chain of transmission events that is difficult to predict, and will depend strongly on location and the local environment. However, when an index case initiates an epidemic, modest transmission between locations is usually sufficient for final epidemic sizes to be independent of index-case location, particularly with respect to FMD. That suggests that the cumulative distribution in Figure 4-20 of the SSRA may be an artifact of specific model assumptions placed on contact patterns and mitigation. Evidence is presented (page 233, for instance) that epidemics are primarily limited by density and geographic characteristics rather than by mitigation measures, but this requires further elaboration by DHS inasmuch as many disease outbreaks are contained by mitigation strategies. Specific Concerns about Model Adequacy Exclusion of Species or Groups The committee is concerned about the exclusion of specific species or management-type groups, such as “backyard” operations, sheep herds, and feral swine (the committee recommended inclusion of the latter in its preliminary letter report, see Appendix B). Backyard operations involve relatively small numbers of animals and often sell off stock at auction yards and purchase animals from other herds. They typically do not use veterinarians, and their activities would fall outside the scope of a passive surveillance system. Whereas few data on backyard operations would be available, such operations would contribute substantially to the spread of FMD from a primary source, and the spread rate would be much higher than that associated with typical commercial livestock operations (Bates et al., 2003). Similarly, the time until detection in such settings would be longer than for commercial operations. Sheep make up only 1% of Kansas herds, but other states have a much higher percentage of sheep. Sheep and goats serve as maintenance hosts for FMD infection because they typically show few clinical signs of infection and can shed virus for extended periods. In the UK epidemic of 2001, sheep were a primary reservoir for infection and contributed substantially to the spread of the disease (Haydon et al., 2004). Feral swine would be important to consider in that they would probably become infected and be outside the passive surveillance system. In some of the states studied, feral swine are considered as posing a high risk for FMD spread. Thus, the exclusion of sheep and feral swine from the model leads to underestimation of the magnitude and duration of an FMD epidemic. Limited Number of States Considered The committee advised in its preliminary letter report that the SSRA consider the entire nation, or more specifically the 48 contiguous states, when modeling the epidemiological and economic impacts of FMD. However, the SSRA selects only seven “primary” states to consider in the model; the states were selected on the basis of their cattle and swine populations. A study limited to a few states is unlikely to reflect the impact of an FMD epidemic in the United States 

EVALUATION OF METHODS 39 as a whole, and assuming that the disease stays within the borders of seven states and within the borders of the United States, it is unlikely to reflect the complex geometric progression of the disease. The issue is not whether one should be concerned about states that have large or small livestock populations; rather the concern should be about how FMD is being transmitted within and among states and/or Canada and Mexico10, and how livestock management and transportation practices and policies among all states, Canada, and Mexico would retard, sustain, or speed FMD spread throughout the United States. States that have small animal populations might actually have more sales barns, more garbage-feeding operations, and less stringent animal inspection policies than states that have large animal populations; these characteristics would increase the likelihood that FMD would be acquired by states that have fewer animals, would be disseminated within those states, and would spread to other states. Failure of the model to include the other 41 of the 48 states as well as incursions in and out of Canada and Mexico would clearly be manifested as unrealistically low estimates of the overall impact of FMD in the United States. Exclusion of states that are at high risk for acquisition or dissemination would probably further bias expectations toward an epidemic of low magnitude and short duration. Lack of Long-Distance Transport Data and Method The SSRA notes that a key weakness of the NADDSM is its inability to consider secondary FMD transmission between states or even between regions within a state. The SSRA notes that efforts at “model enhancement” (page 175) or development of “secondary models” (page 168) were undertaken to address that shortcoming, but it is unclear whether these are the same. The enhanced model is described imprecisely, and the committee could not determine the complete details of its implementation from the material supplied. In addition, the enhanced model inherits most of the limitations of the standard NAADSM, which may introduce substantial biases when applied on larger spatial scales than NAADSM was intended to handle. Thus, the ability of the overall modeling effort to reasonably and adequately estimate and account for interstate transmission of FMD remains an important question and casts doubt on results and conclusions drawn from the model. Instead of developing a scenario-specific model, the SSRA addresses some of the contact-pattern issues across state boundaries by adjusting the NAADSM to include “sales barn” units, which facilitate “long-distance” dispersal. This adjustment is described in a general way that requires detailed understanding of the NAADSM design framework for implementation and replication. To determine the parameters for the long-distance dispersal mechanism, the model estimated cattle movements through sales barns in such a way that the sum of interstate movements among all sales barns equaled expectations for total interstate movement rates. Thus, sales barns were used as a surrogate for between-state transport livestock activities from all other possible sources, including large independent cattle operations; movements resulting from private sales appear not to have been considered. The primary justifications for assuming sales barn transport as a surrogate were concerns about model complexity and identifiability. While adding sales barns to the NAADSM is an improvement because it takes some interstate animal movement into account, sales barns are not the sole source of long-distance animal movement 10 The SSRA did not fully consider modeling disease spread across state and country borders. Modeling disease spread with artificial boundaries (such as politically-designated state and country lines) creates a dilemma about how to deal with spread across country borders (into Canada and Mexico) and how such spread could be transmitted back across the border to impact spread in the primary area of interest (the United States). 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 40 rates (USDA-ERS, 2003), and therefore the SSRA underestimates the long-distance transport of animals and equipment and in doing so also underestimates the rate and extent of FMD spread. Even if use of sales barns as surrogates for all interstate transport were sufficient for estimating total epidemic size, it introduces difficulties in assessing mitigation measures. There are differences in interests, regulation, and responsiveness between sales barns and large independent cattle operations that mitigation practices should consider. Specifically, the assumption leads to overestimation of the effectiveness of baseline mitigation measures, which therefore leads to an underestimation of the epidemic magnitude. Highly Optimistic Detection Times Both the time to develop clinical signs (lesions) and the time to detect (and report) clinical signs once they appear were considered in the model (pages 179 and 219 of the SSRA), but location-type-specific observation rates were not adequately documented. Some assumptions were made that probably resulted in an overoptimistic time to report a case of FMD, which would be manifested in limiting the simulated spread of disease and would bias results toward fewer cases and shorter duration. Many of the data used to compile distributions of time to lesion appearance were obtained from experimental studies that examined the oral cavity and feet of animals for evidence of lesions (erosions and vesicles) once or twice a day after exposure to large doses of FMDv. Because a broad spectrum of immunological, inflammatory, and secondary infection events follow lesion development, the actual signs of frothing or lameness typically may not appear for some days after the first oral or foot lesions appear, depending on exposure dose, strain, and so on. For some species, such as sheep and goats, lesions may never appear or may be so subtle that they are missed entirely, as was observed in the UK epidemic (Haydon et al., 2004; McLaws et al., 2007), so exclusion of these species from consideration would result in a low estimate of FMD spread. Consequently, the time to appearance of “gross” clinical signs sufficiently apparent to be observed by a layperson would extend beyond when lesions first appear, perhaps by several days. Furthermore, it is overoptimistic to assume that the probability of detection and reporting would be 100% within 6 days (page 265) of when lesions begin to appear. On some premises, livestock may not be observed more than once a week. There is reason to believe that detection and reporting times could be months or more in some cases, as was observed in the Canadian FMD epidemic in 1952-1953 (Sellers and Daggupaty, 1990). Anecdotal statements of U.S. veterinarians who participated in the control program in the UK in 2001 indicated that very long delays may be expected between lesion development and reporting. The assumption that “self-announcing” leaks from the NBAF (page 182) would result in even more rapid detection of cases around the laboratory is unrealistic given the process that would have to be involved in detection. It is unlikely that livestock owners would tolerate having their animals inspected (requiring lock up, oral inspections, swabbing, and so on) each day for weeks after a possible breach of containment in the NBAF. It is also unlikely that sufficient personnel would be available to conduct such investigations each time a breach were suspected. Overestimated Diagnostic Capabilities and Sensitivity Several important aspects of the clinical diagnostic sensitivity and of the laboratory diagnostic capability and sensitivity for FMDv were not considered in the modeling. In the case 

EVALUATION OF METHODS 41 of a suspected or actual release of FMDv from the NBAF, the laboratory will need to take special precautions and procedures that could adversely affect its ability to perform diagnostic testing. If the facility needed to shut down (for example, physical breach caused by a tornado or suspected pathogen leak from the laboratory), diagnostic support for necessary investigation and mitigation of a suspected escape may be compromised. When an FMDv release occurred in 2007 from the Institute of Animal Health Pirbright Laboratory in the UK, some FMD work at that facility was halted, including work on vaccines that could have been necessary had the FMD outbreak not been controlled (Anderson, 2008). The NBAF design plans include a pilot scale GMP manufacturing facility with BSL-3 capability (Figures 1-1 and 3-2 of the SSRA), which could produce very high concentrations of FMDv and other pathogens for vaccine production. As previously noted, there were 15 known escapes of FMDv from laboratories worldwide between 1960 and 2007 (Anderson, 2008; GAO, 2008), with most of those escapes occurring from vaccine manufacturing facilities that produce very large amounts of virus (Anderson, 2008). The SSRA does not provide contingency plans for auxiliary diagnostic support by other laboratories (such as state laboratories and the Winnipeg and Pirbright laboratories); such diagnostic contingency plans are critical for NBAF operations and should have been included in the SSRA. The NAADSM uses a sensitivity value of 1.0 (1.0 being perfect accuracy) in identifying FMD-affected premises and assumes that the clinical diagnostic processes involved in contact tracing would be reliable and accurate. However, the 2001 UK epidemic demonstrated that sensitivity is not perfect and that it may be around 0.947 (McLaws et al., 2007). On the basis of that estimate, clinical monitoring and declaration could miss about 5.3% of infected herds. The existence of infected but undetected premises, referred to as occult infection premises (Jewell et al., 2009), is in part related to the exhaustion of resources needed to undertake trace contact investigations (Ferguson et al., 2001a,b; Keeling et al., 2001); the efficacy of contact tracing can be diminished considerably by resource constraints (Eames and Keeling, 2003; Kao, 2003; Kiss et al., 2005). As a consequence of the poor efficacy of contact tracing experienced early in the 2001 UK epidemic, a “contiguous cull” policy was implemented whereby animals in high-risk premises were destroyed, without diagnostic confirmation, in order to rapidly eliminate potentially infected premises (The Royal Society, 2002; Haydon et al., 2004). The failure of the SSRA to consider less than perfect diagnostic efficacy in the contact tracing is not a trivial omission in modeling the spread of FMD, and would probably contribute substantially to underestimating the magnitude and duration of an epidemic and confuse assessment of mitigation strategies. The report also assumes that the laboratory diagnostic sensitivity and specificity values are 1.0. Those are distinct from the aforementioned clinical and contact tracing sensitivity. The assumption of perfection indicates that submitted specimens from truly infected animals would test positive every time, and that specimens from truly uninfected animals would test negative every time. That is not the case in reality when laboratory assays are compared (King et al., 2006; Tam et al., 2009). For some serotypes, such as those designated as the South African Territories serotypes (SAT1, SAT2, and SAT3), laboratory test sensitivity can be quite low. Even so, published values are probably much higher than what would be experienced in practice during an epidemic because they represent testing of tissue cultures with high titers of virus rather than testing of clinical samples that may have virus titers that are several orders of magnitude lower, which would consequently be more likely to result in false-negative results. The consequence of those inherent diagnostic assumptions would be that the model would incorrectly assume that all infected herds would be identified and declared with clinical 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 42 and contact tracing and investigations and then confirmed by laboratory testing. Because a percentage of infected herds would be missed in a real outbreak, the model has underestimated the number of cases and the duration of the epidemic. Delays in diagnostic testing resulting from loss of the NBAF diagnostic capability could result in continued spread of disease while awaiting orders to cull the animals of suspected positive herds. Initial Cases of Fomite Leaks Restricted to Manhattan, Kansas, Area As part of efforts to prevent FMD from entering the United States, the U.S. Department of Agriculture has established programs that prohibit travelers from carrying certain animal products that are potentially contaminated with FMDv into the United States, and it has specific inspection procedures at ports of entry to enforce the requirements. It is unclear, therefore, why a seemingly likely scenario was not considered for an FMDv fomite “walking out” of the NBAF via a person who became contaminated with the virus in the laboratory and who flew or drove from Manhattan, Kansas, to a livestock or wildlife site in some other state. The assumption that the first case of FMD resulting from fomite escape from the NBAF would appear only in the Manhattan area (pages 168-169) ignores the reality of human travel and disease movement, especially in a university town. By not including a scenario of escape to another state, the model allows the disease to be diagnosed and controlled only in a very small area around Manhattan that presumably has been bolstered, because of the presence of the NBAF, with an unusually high degree of passive surveillance and education, which would be expected to provide earlier detection and control than would occur in other states. Consequently, model results will be biased toward a low estimate of magnitude and duration of an FMD epidemic in the seven states studied. Additional Concerns about the Model The committee questioned the following aspects used in the model: 1. The ability to cull and bury or burn animals from 120 infected premises and to destroy all feed and clean and disinfect environments of these premises in only 1 day is too optimistic even if military intervention is involved. 2. No indication was provided as to whether contiguous cull was considered in the model. 3. No indication was provided as to specific vaccination strategies that were considered, how populations were selected for vaccination, and specific timeframes (and limits to the estimates). 4. An assumption inherent in the scenarios presented was that only one strain of one serotype of FMDv would be released. That assumption would not represent the situation if the laboratory were physically breached (by airplane, tornado, or other damage) or if filtration systems failed, given the presence of many strains of all seven serotypes of the virus in the laboratory. The model also assumes that none of the leaked strains would be cell or tissue-culture-adapted; if they were, that could result in different disease manifestations than those modeled. For example, infection with engineered or adapted strains, such as the one that escaped into Taiwan swine (Beard and Mason, 2000) might not be manifested in typical clinical signs expected for FMD or might not be detected by PCR testing, thus delaying detection and perhaps control. Several other scenarios could 

EVALUATION OF METHODS 43 have been considered, including the leak of engineered or adapted strains that do not behave like typical FMD viruses and the simultaneous leak of multiple serotypes and strains of the virus. Rift Valley Fever In the preliminary letter report, the committee advised DHS to include in the SSRA a pathogen that posed a potential risk to humans. DHS subsequently chose to investigate the potential risks posed by research on the Rift Valley fever virus (RVFV), which is a BSL-3 pathogen that will be studied at the NBAF. To conduct quantitative studies of the risks posed by a release of RVFV, the SSRA team developed a new, custom epidemic model, using the VenSim software package. FMD has been the focus of numerous modeling studies, but Rift Valley Fever (RVF) has not received similar attention. There were no suitable models of the quality of the NAADSM or the Davis Animal Disease Simulation (DADS) models for addressing community risk posed by a release of RVFV. Without a suitable existing model, the only available course for meeting the requirements of the SSRA was the development of a custom model, as described in Section 4.4 of the SSRA. The SSRA RVF epidemic model is a compartmental model that represents transmission dynamics between vertebrate hosts and mosquitoes, the arthropod vector responsible for RVF transmission in the wild. It includes parameters representing the course of infection in vertebrates and mosquitoes, biting rates, mosquito lifetimes, and some potential mitigation measures, but it does not incorporate spatial or geographic structure. Values were assigned to the parameters by using data from a variety of sources, and the SSRA specifically identifies some kinds of data that are absent and for which further basic research is needed. The RVF model is described in detail in the SSRA (pages 237-243), but in lay language. Although the model described in lay language seems reasonable and appropriate, its implementation remains opaque. The lay language is imprecise, making the independent replication of the model difficult. The algorithms used by the VenSim software package for simulating the model are not described; therefore, it is unclear what form the model takes (such as ordinary differential equations, stochastic differential equations, birth-death process, and individual-based simulation). Because the model is new, there has not been much opportunity for it to undergo peer review and independent validation, which would address those issues. In addition, the absence of basic research in some areas leaves large gaps in the model parameterization that are important in assessing the overall effect of an RVF outbreak. Even some aspects of the model that have been extensively studied, such as mosquito biting rates, still have substantial uncertainty. In addition, the RVF model in the SSRA assumed that the only infection mechanism in humans was primarily by mosquitoes and possibly by biting flies as well. The current view on risk factors for infection with RVF in humans is that direct contact with animals, particularly contact with body fluids, is one of the most important risk factors (Woods et al., 2002). In a country such as the United States, the consequences of such transmission routes create considerable uncertainties with respect to the risk of infection in the human population. Collectively, the issues noted above leave the committee with no confidence in the quantitative results of the SSRA’s RVF epidemic model. The issues also limit the committee’s confidence in the analysis of the economic consequences of an RVF outbreak to the extent that it depends on the epidemiological model. Numerical results obtained from simulations of the 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 44 SSRA’s RVF model should be treated as guesses and should not be relied on in assessment of risk or in the design of mitigation measures. The omission of direct transmission may significantly bias results. The RVF model does provide constructive content for the SSRA to the extent that it partially identifies factors—some known and others for which more will need to be known—that could contribute to the potential spread of an RVF outbreak. The identified factors provide qualitative guidance for potential risk pathways that require further attention and research as part of continuing risk assessment and mitigation practices. The committee expects that the development of an RVF model comparable with the NAADSM or DADS would require a year or more of work, and it may not be possible to complete and calibrate an RVF model without further epidemiological research. The committee acknowledges that the time frame given to DHS for conducting the SSRA was shorter than the time needed to further develop the model for assessing the risk of RVF. ECONOMIC MODELING The evaluation of the economic modeling portion of the SSRA is based on the revisions to Chapter 5 of the SSRA that were submitted to the committee on August 26, 2010. The report has several positive aspects, especially the integration of epidemiological modeling with economic modeling. Three economic models are used in concert with cost calculations external to the modeling. Methods Employed A partial equilibrium model of the U.S. agriculture sector, also referred to as an equilibrium displacement model, is used to estimate the national effects of FMD and RVF outbreaks. Such models estimate prices, quantities, and economic welfare in the United States in an open international trading system. The model used is documented as cited in the SSRA but is modified with more recent elasticity estimates. The SSRA correctly used the partial equilibrium U.S. agricultural sector model by considering the effects of three shocks caused by an outbreak. One shock is the reduction in the supply of animals determined by the epidemiological model discussed. The number of animals culled are reported in Table 5-5 of the SSRA and are converted to percent reductions compared to the U.S. inventory as shown in Table 5-10 of the SSRA. A second shock is a reduction in U.S. exports of meats and animals that is assumed on the basis of World Organisation for Animal Health (OIE) guidelines, observed experiences in other nations that have FMD and RVF, and the U.S. bovine spongiform encephalopathy (BSE) event. The assumed export reduction used in the SSRA is plausible: the SSRA analysis assumes U.S. exports of relevant products drop by 95% during the outbreak and the first quarter following the end of the outbreak. The 95% reduction reflects that the United States could export some cooked products. Export recovery occurs slowly over 9 quarters after the United States recovers its FMD-free status. The quarter-by-quarter percent reductions are shown in Table 5-15 of the SSRA, and these reductions are a substantial source of the economic losses. The December 2003 case of BSE resulted in reduced U.S. beef exports by 93% (USITC, 2009). Assuming there was no growth in U.S. beef exports, export quantity did not approach pre-BSE levels until the third quarter of 2008 (USITC, 2009). That pattern occurred even though Japan and Korea had earlier relaxed, but not ended, restrictions on 

EVALUATION OF METHODS 45 purchases of U.S. beef (USITC, 2009). The third shock is a potential reduction in U.S. consumption of meat and dairy products. Table 5-13 of the SSRA assumes reductions in consumer demand, and shows the largest reductions (of 5-10%) during the outbreak and a gradual recovery afterwards. The assumed reduction in demand is also based on observations of other nations and statistical estimates after the U.S. BSE cases. During the first month after the 2003 BSE case, sales decline was 21% (Schlenker and Villas-Boas, 2009). By day 90, the estimated sales reduction was 10% (Schlenker and Villas-Boas, 2009). Using the regression model of Schlenker and Villas-Boas (2009), consumer demand for U.S. beef appears to recover around day 150. The SSRA’s partial equilibrium U.S. agricultural sector model is an appropriate modeling framework, and the model has been used to estimate the national impacts of other livestock diseases. The values of the three shocks are reported in the SSRA. The second model is a regional input-output model constructed from the input-output model developed by the Bureau of Economic Analysis. Regional input-output modeling is an accepted means of estimating the economic impact of a localized shock. Typically, regional input-output modeling holds prices and costs constant because they are determined in national or global markets. The value of economic activity is changed to track its effects. Regional analysis relies on multipliers, and these can be different from the measures used in the partial equilibrium model. The partial equilibrium model treats prices as endogenously determined in global markets, so it becomes a means of recognizing price effects resulting from disease outbreaks. The economic estimates from the regional input-output model need to be compatible to be added to the estimates from the partial equilibrium model, as was done in the report. The report indicates that the regional effects are modified to avoid double counting of the impacts obtained with the partial equilibrium model. It does not report detailed sector impacts. The third model used in the economic analysis provides estimates of values attached to mortality and morbidity that result from an outbreak of RVF. The report generates estimates of marginal willingness to pay (WTP) to avoid RVF by using questions about vaccination options to elicit survey responses and using a value for a life updated from other estimates. Those techniques are often used to determine the value of non-market attributes, such as environmental amenities and disasters, recreation, and public goods. The methods have several problems in application. The SSRA recognizes the problems and discusses them. In addition to the three models, various costs are relevant to a disease outbreak that are external to the partial equilibrium and input-output models, such as quarantine, surveillance, destruction, and carcass disposal. It is correct to include such costs. Estimated values used in the analysis are taken from values reported in other studies. Concerns about Implementation of the Analysis Although the models are appropriate, the quality of the results depends on how the models are used. The estimated impacts from the partial equilibrium model are generated from the shocks introduced into that model and its parameter values. Except for the livestock products price elasticities, the parameter values are documented in the report cited in the SSRA; the price elasticity estimates for livestock products are from more recent published research. Although it is unknown what will occur in reality, the reductions in export and domestic demand are assumed on the basis of evidence cited in the SSRA. The values assumed are plausible and can be defended. The livestock depopulation and outbreak duration used in the partial equilibrium model are from the epidemiological model. As indicated above, the committee believes that the 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 46 magnitudes of livestock culls and the durations of outbreaks are underestimated. The estimated livestock depopulation ranges from 94,000 head to 23 million head in a susceptible population in the epidemiological model’s database of 61.9 million head. The longest estimated duration of an outbreak is no longer than 2 quarters (6 months); the 2001 outbreak in Britain lasted from the last week of February through the end of September (UK-MAFF, 2002), and about 6 million head were culled during that 7-month outbreak (UK-Defra, 2004). Export and domestic demand recovery times depend on the duration; OIE guidelines state that a nation can recover its FMD- free status 3-6 months after the end of an outbreak, depending on how the outbreak is managed (OIE, 2009). The same concern applies to the regional analysis and the government cost calculations. Both estimates are contingent on the magnitude of livestock depopulation and the duration of the outbreak. Many of the government costs are herd-based, so the size and number of herds depopulated affect the scale. Greater depopulation of animal numbers and herds would raise the costs. These economic effects are calculated only for outbreak duration, and longer durations would increase the effects. The SSRA description of how the regional analysis is conducted is inadequate in the text and leaves the reader to assume that it was done correctly unless proven otherwise. The specific shock introduced into the regional input-output model is not reported. The total dollar regional impact of each scenario is reported, but results for individual economic activities that would help to confirm the quality of the analysis are not reported. The SSRA recognizes the problems inherent in WTP estimates and describes steps undertaken to mitigate them. Comparisons with similar values from research on diseases other than RVF are used to validate the results. Because the values are unknown, it is difficult to determine the success of the efforts beyond noting that the proper methods were applied. 

EVALUATION OF METHODS 47 REFERENCES Anderson, I. 2008. Foot and Mouth Disease 2007: A Review and Lessons Learned. London: The Stationery Office. Bates, T.W., M.C. Thurmond, and T.E. Carpenter. 2003. Results of epidemic simulation modeling to evaluate strategies to control an outbreak of foot-and-mouth disease. Am J Vet Res 64(2):205-210. Baylor, E.R., M.B. Baylor, D.C. Blanchard, L.D. Syzdek, and C. Appel. 1977. Virus transfer from surf to wind. Science 198(4317):575-580. Beard, C.W., and P.W. Mason. 2000. Genetic determinants of altered virulence of Taiwanese foot-and-mouth disease virus. J Virology 74:987-991. Blanchard, D.C., and L.D. Syzdek. 1982. Water-to-air transfer and enrichment of bacteria in drops from bursting bubbles. Appl Environ Microbiol 43:1001-1005. Boissonade, A., Q. Hossain, and J. Kimball. 2000. Development of a probabilistic tornado wind hazard model for the continental United States, Volume 1: Main Report. Livermore, CA: Lawrence Livermore National Laboratory. Brault, A.C., A.M. Powers, G. Medina, E. Wang, W. Kang, R.A. Salas, J. De Siger, and S.G. Weaver. 2001. Potential sources of the 1995 Venezuelan Equine Encephalitis subtype IC epidemic. Virology 75(13):5823-5832. Derbyshire, D. 2007. New leak of foot and mouth disease discovered at Pirbright laboratory. Daily Mail, Associated Newspapers Ltd. 23 November. Available online at http://www.dailymail.co.uk/news/article-495608/New-leak-foot-mouth-disease- discovered-Pirbright-laboratory.html [accessed September 6, 2010]. DOE (U.S. Department of Energy). 2002. DOE standard: Natural phenomena hazards design and evaluation criteria for Department of Energy facilities. DOE-STD-1020-2002. Available online at http://www.hss.energy.gov/nuclearsafety/ns/techstds/standard/std1020/STD- 10202002.pdf [accessed September 6, 2010]. Eames, K.T.D., and M.J. Keeling. 2003. Contact tracing and disease control. Proc R Soc B 270:2565-2571. EPA (U.S. Environmental Protection Agency). 2000. Second-Order Closure Integrated PUFF Model (SCIPUFF). Available online at http://www.epa.gov/scram001/7thconf/information/scipuff.pdf [accessed September 6, 2010]. Ferguson, N.M., C. Donnelly, and R. Anderson. 2001a. Transmission intensity and impact of control policies on the foot and mouth epidemic in Great Britain. Nature 413:542-548. Ferguson, N.M., C.A. Donnelly, and R.M. Anderson. 2001b. The foot-and-mouth epidemic in Great Britain: pattern of spread and impact of interventions. Science 292:1155-1161. GAO (U.S. Government Accountability Office). 2008. High-containment biosafety laboratories: DHS lacks evidence to conclude that foot-and-mouth disease research can be done safely on the U.S. mainland. Washington, DC: GAO. Available online at http://www.gao.gov/products/GAO-08-821T [accessed March 4, 2010]. Harvey, N., A. Reeves, M.A. Schoenbaum, F.J. Zagmutt-Vergara, C. Dubé, A.E. Hill, B.A. Corso, W.B. McNab, C.I. Cartwright, and M.D. Salman. 2007. The North American Animal Disease Spread Model: A simulation model to assist decision making in evaluating animal disease incursions. Preventive Veterinary Medicine 82:176-197. 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 48 Haydon, D. T., R.R. Kao, and R.P. Kitching. 2004. The UK foot-and-mouth disease outbreak— the aftermath. Nat Rev Microbiol 2:675-681. Hejkal, T.W., P.A. LaRock, and J.W. Winchester. 1980. Water-to-air fractionation of bacteria. Appl Environ Microbiol 39(2):335-338. Jewell, C.P., M.J. Keeling, and G.O. Roberts. 2009. Predicting undetected infections during the 2007 foot-and-mouth disease outbreak. J R Soc Interface 6(41):1145-1151. Kao, R.R. 2003. The impact of local heterogeneity on alternative control strategies for foot-and- mouth disease. Proc R Soc B 270:2557–2564. Keeling, M.J., M.E.J. Woolhouse, D.J. Shaw, L. Matthews, M. Chase-Topping, D.T. Haydon, S.J. Cornell, J. Kappey, J. Wilesmith, and B.T. Grenfell. 2001. Dynamics of the 2001 UK foot and mouth epidemic: Stochastic dispersal in a heterogeneous landscape. Science 294(5543):813-817. King, D.P., N.P. Ferris, A.E. Shaw, S.M. Reid, G.H. Hutchings, A.C. Giuffre, J.M. Robida, J.D. Callahan, W.M. Nelson, and T.R. Beckham. 2006. Detection of foot-and-mouth disease virus: comparative diagnostic sensitivity of two independent real-time reverse transcription-polymerase chain reaction assays. J Vet Diagn Invest 18:93-97. Kiss, I.Z., D.M. Green, and R.R. Kao. 2005. Disease contact tracing in random and clustered networks. Proc R Soc B 272:1407. Lemonick, M.D., and A. Park. 1994. A deadly virus escapes. September 5. Time. Available online at http://www.time.com/time/magazine/article/0,9171,981359,00.html [September 6, 2010]. Löfstedt, R. 2002. Good and bad examples of siting and building biosafety level laboratories: a study of Winnipeg, Galveston and Etobicoke. Journal of Hazardous Materials 93(1):47- 66. Luhar, A.K., P.J. Hurley, and K.N. Rayner. 2009. Modelling near-surface low winds over land under stable conditions: Sensitivity tests, flux-gradient relationships, and stability parameters. Boundary Layer Meteorology 130(2):249-274. Margasak, L. 2008. Accidents at disease lab acknowledged. USA Today. April 11. Available online at http://www.usatoday.com/news/washington/2008-04-11-1445284742_x.htm [accessed September 6, 2010]. McDonald, J.R., and K.C. Mehta. 2008. A recommendation for an Enhanced Fujita scale (EF- scale) submitted to the National Weather Center and other interested users. Available online at http://www.depts.ttu.edu/weweb/EFScale.pdf [accessed September 6, 2010]. McLaws, M., C. Ribble, C. Stephen, B. McNab, and P.R. Barrios. 2007. Reporting of suspect cases of foot-and-mouth disease during the 2001 epidemic in the UK, and the herd sensitivity and herd specificity of clinical diagnosis. Prev Vet Med 78(1):12-23. NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, DC: National Academy Press. NRC. 2008. Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change. Washington, DC: The National Academies Press. NRC. 2009. Science and Decisions: Advancing Risk Assessment. Washington, DC: The National Academies Press. NRC. 2010. Review of the Department of Homeland Security’s Approach to Risk Analysis. Washington, DC: The National Academies Press. 

EVALUATION OF METHODS 49 Nuclear Regulatory Commission. 2007. Design Basis Tornado and Tornado Missiles for Nuclear Power plants, Nuclear Regulatory Commission, Regulatory Guide 1.76, Revision 1, March 2007. OIE (World Organisation for Animal Health). 2009. Terrestrial Animal Health Code (Article 8.5.8). Available online at www.oie.org [accessed October 20, 2009]. Ramsdell, J.V., and J.P. Rishel. 2007. Tornado Climatology of the Contiguous United States, NUREG/CR-4461, Rev.2, Pacific Northwest Laboratory. Ray, P. S., P. Bieringer, X. Niu, and B. Whissel. 2003. An improved estimate of tornado occurrence in the central plains of the United States. Mon Wea Rev 131:1026-1031. Rife, D.L., J.O. Pinto, A.J. Monaghan, C.A. Davis, and J.R. Hannan. 2010. Global distribution and characteristics of diurnally varying low-level jets. J Climate 23(19):5041-5064. Sage Management. 2010. SCIPUFF Dispersion Model. Available online at http://www.sage- mgt.net/services/modeling-and-simulation/scipuff-dispersion-model [accessed September 6, 2010]. Schlenker, W., and S.B. Villas-Boas. 2009. Consumer and market responses to mad cow disease. American Journal of Agricultural Economics 91(4):1140-1152. Schoenbaum, M.A., and W.T. Disney. 2003. Modeling alternative mitigation strategies for a hypothetical outbreak of foot-and-mouth disease in the United States. Preventive Veterinary Medicine 58:25-52. Sellers R.F., and S.M. Daggupaty. 1990. The epidemiology of foot-and-mouth disease in Saskatchewan Canada, 1952. Canadian J Vet Res 54:457-464. Tam, S., A. Clavijo, E.K. Engelhard, and M.C. Thurmond. 2009. Fluorescence-based multiplex real-time RT-PCR arrays for the detection and serotype determination of foot-and-mouth disease virus. J of Virol Methods 161:183-191. The Royal Society. 2002. Inquiry into infectious diseases in livestock, Tech. rep. London, UK: The Stationary Office. UK-Defra (United Kingdom, Department of Environment, Food, and Rural Affairs). 2004. Animal Health and Welfare: FMD Data Archive. 2001 Foot and mouth disease online database. Available online at http://footandmouth.csl.gov.uk/ [accessed September 3, 2010]. UK-HSE (United Kingdom Health and Safety Executive). 2007. Final report on potential breaches of biosecurity at the Pirbright site 2007. Available online at www.hse.gov.uk/news/archive/07aug/finalreport.pdf [accessed March 3, 2010]. UK-MAFF (United Kingdom, Ministry of Agriculture, Forestry and Fisheries). 2002. Foot and mouth disease daily situation report. Available online at www.maff.gov.uk [accessed weekly from March 29, 2001-January 14, 2002]. USDA-ERS (U.S. Department of Agriculture, Economic Research Service). 2003. Interstate livestock movements: Total inshipments. June 23. Available online at http://www.ers.usda.gov/Data/InterstateLivestockMovements/totalinshipments.htm [accessed September 25, 2010]. USITC (U.S. International Trade Commission). 2009. Trade DataWeb. Available online at http://dataweb.usitc.gov/ [accessed October 16, 20, and 23, 2009]. Woods, C.W., A.M. Karpati, T. Grein, N. McCarthy, P. Gaturuku, E. Muchiri, L. Dunster, A. Henderson, A.S. Khan, R. Swanepoel, I. Bonmarin, L. Martin, P. Mann, B.L. Smoak, M. Ryan, T.G. Ksiazek, R.R. Arthur, A. Ndikuyeze, N.N. Agata, C. J. Peters, and the World 

EVALUATION OF THE NBAF SITE-SPECIFIC RISK ASSESSMENT 50 Health Organization Hemorrhagic Fever Task Force. 2002. An Outbreak of Rift Valley Fever in Northeastern Kenya, 1997-98. Emerging Infectious Diseases 8(2):138-144.