The first step in quantitative risk assessment, as described in the updated site-specific risk assessment (uSSRA), is a description and analysis of the circumstances (accident events) that could lead to release of foot-and-mouth disease virus (FMDv) from the proposed National Bio- and Agro-Defense Facility (NBAF). The purpose of this step is to develop estimates of four critical inputs for the ultimate characterization of site-specific risk. The analysis focuses on the probability and frequency of pathogen release given particular circumstances related to loss of containment and on the amount of pathogen likely to be released. Questions related to the probability that a given release would result in infection and the consequences of such an infection are left to later sections of the uSSRA; the ultimate characterization of FMDv-related risk is presented in Section 8 of the uSSRA.
Section 4 of the uSSRA begins with a description and analysis of the pathways that could lead to pathogen release from each of three originating locations within containment and from non-containment areas outside the laboratories. The uSSRA created conceptual models of release pathways and provided estimates of the total amount of material available for release (MAR) from each originating location.
Accident sequences culminate in an event (defined as loss of containment in the uSSRA) that may or may not result in an infection outside the facility. As previously mentioned in Chapter 3, event trees (which are incorrectly called fault trees in the uSSRA) are used to describe the set of
circumstances leading to release of material. Each node of the event tree (Figure 4.5-1 of the uSSRA) indicates a point in the sequence of events at which a release mitigation system (including human action) either succeeds or fails. “Success” means that the system functions as expected, not that it is 100% effective. An event tree is developed for each originating location and for each of four possible mechanisms, called pathways, by which pathogens could be released (aerosol, liquid waste, solid waste, and transference). This is accompanied by a table that provides the following for each node of the event tree: failure probabilities and “reduction factors” that are to be applied to the MAR, one reduction factor that is assigned when the mitigation system at each event tree node fails, and another when it is fully functional.
The reliability of the ultimate risk estimates presented in Section 8 depends heavily upon the adequacy of the analyses and results from the accident event modeling of Section 4.
The method applied in Section 4 is a distinct improvement over that applied in the 2010 SSRA. The use of event tree analysis and probabilistic modeling is preferable to the scenario-based, semi-quantitative approach of the earlier assessment, and is consistent with current risk assessment science for facilities like the NBAF. The adoption of International Organization for Standardization (ISO) Standard 31000 terminology is also to be applauded, although the committee notes some concerns about the misuse of terms.
The committee identified a few significant omissions in the conceptual models used to describe containment and in the elucidation of system failures that could lead to a release (the system failures are summarized in the 24 circumstances presented in Table 4.3.1-1 of the uSSRA). The committee finds that the development of the 142 events that could lead to an infectious or non-infectious release is nearly complete and generally takes into account mitigation systems, including human action, identified in the conceptual models. However, critical issues remain that affect other aspects of the risk assessment, and these are discussed below.
The uSSRA generally adheres to ISO 31000 terminology. As previously mentioned in Chapter 3, the committee finds that use of the term Ploss is confusing, and the uSSRA should have adopted a different term. The uSSRA would be less confusing if the “loss” subscript were dropped, inasmuch as Ploss can be easily misread as the probability of loss of FMDv when it carries no such meaning. The term refers to the probability that,
given a specific opportunity, a particular pathway in the event tree will allow release. It includes pathways in the event tree in which all mitigation systems are assumed to be fully functional and pathways in which some mitigation systems fail.
Apart from asking subject matter experts for their opinions, no explanation was provided on how the uSSRA selected the 142 events (Appendix Tables A8-1 and A8-2) that were the bases of the event trees. In designing the event trees, several logical circumstances (as indicated by a node) were omitted (i.e., out-of-containment leaks, power systems failures). Thus, the committee is not confident in assuming that all the critical pathways for escape were considered in the uSSRA.
It is impossible in the time provided for this evaluation to review every event tree and accompanying table in Section 4 and to comment on the adequacy of the data and assumptions that are used to support all of the large number of estimates of probabilities and reduction factors associated with each of the many nodes of each event tree. Even with a sound method, the reliability of the assessment remains completely dependent on the scientific reliability of each of the hundreds of inputs used.
The committee selected a sampling of event trees and analyzed some of the assumptions and data used to develop estimates for Q, Ploss, R0, and Floss. For each node of at least three event trees, the committee examined the probabilities of failure and success of the mitigation systems and the resulting MAR reduction factors. Such analyses were conducted for events associated with ATR (transference of virus to the respiratory tracts of workers in biosafety level-3 [BSL-3] animal holding rooms), OTB (transference to the body in non-containment areas), and AA (aerosol release from BSL-3 animal holding rooms), and for selected other events.
Although the uSSRA provided detailed analysis of various risks, the use of questionable assumptions in the data inputs demonstrates that there was insufficient familiarity with the body of scientific literature or with institutional knowledge. The committee believes that this was manifested in citations’ being too limited to constitute compelling evidence of support. The uSSRA failed to validate some assumptions through multiple sources
and/or high-quality references. This is evident in how assumptions for data inputs were made; examples of the invalid assumptions are presented below in the case of the aerosol transference pathway and calculations for FMDv. The examples set out in the following are not exhaustive but illustrate the committee’s concerns.
Estimates of Material Available for Release
MAR for Aerosol
The committee is concerned that the MAR for aerosolized virus was not adequately estimated. Limited data were applied from two sources which examined exhaled virus of only two serotypes (Alexandersen et al., 2002; Gloster et al., 2008); the uSSRA did not apply data from at least four other studies that found much higher concentrations of aerosol virus (Donaldson et al., 1982; Donaldson and Alexandersen, 2001; Alexandersen and Donaldson, 2002; Alexanderson et al., 2003). Moreover, because of the methods used in the studies cited, the estimates did not account for virus aerosolization from urine, feces, saliva, vesicular fluid, feed dust, etc., and thus the MAR figures for airborne virus would further underestimate actual virus available in the air of animal rooms. Failure to sufficiently consider the broad natural variation in virulence among FMDv serotypes and among strains within a serotype (Beard and Mason, 2000; Mason et al., 2003; Grubman and Baxt, 2004) resulted in overly restrictive and likely unrepresentatively low aerosol MAR estimates for the repertoire of strain–host experiments expected for the NBAF.
MAR for Special Procedures
The MAR assumed for special procedures (e.g., shipment spills) was 3.6 × 104 plaque-forming units per millimeter (PFU/mL) (p. 130 of the uSSRA). The assumed value is low by at least a factor of 100 if one considers the most common type of procedure undertaken daily—namely, virus passage in cell cultures—whether as part of an experiment or simply to maintain the laboratory’s inventory of viruses. Specific cell lines are used to maximize the titer of virus, depending on the serotype and strain. Typical virus concentrations are around 105–107, and sometimes 108 for cell-adapted virus (Tam et al., 2009). For determining amounts of FMDv, the uSSRA states that it chose to use only references that involved primary bovine thyroid cells for input data “because the bovine thyroid cell assay is the most sensitive for determining the concentration of FMDv” (p. 110). The only reference cited for that optimal sensitivity dates to 1966 (Snowdon, 1966). However, investigators have been using other cell cultures to grow FMDv for the in-
tervening 45 years with excellent results, and FMDv is now usually grown in BHK or LK cells (of bovine origin) or PK or IBRS2 cells (of swine origin), depending on serotype and other factors. The committee finds that ignoring the intervening literature and basing future practices on a reference from 1966 is a critical oversimplification in stating that only one cell culture is the most sensitive for growing the virus.
More reasonable concentrations would be 105–107, and occasionally 108, depending on serotype and whether it is a primary isolation or a cell-adapted virus. Thus, the use of such a low titer of virus throughout the uSSRA as an estimate for MAR artificially diminishes the magnitude of a leak. Because factors in the model multiply to yield risks, an order-of-magnitude underestimation in each of several multiplied factors quickly reduces the final risk.
Low, Median, and High MAR Values
Accounting for variability in MAR is a valid objective, as is the use of Monte Carlo simulations. The committee notes that many of the input parameters appear skewed and that calculations were completed for the low, median, and high values for MAR (and other factors; see below). These were then assembled by weighting the 5th percentile by 5%, the median by 90%, and the 95th by 5% and taking a weighted sum. That is a purely ad hoc procedure and is inconsistent with the mathematics of probability. If the distribution of values were known, randomly sampling from the whole distribution (rather than separate runs for the 5th, median, and 95th percentiles) would have given more robust insights and allowed for rigorous sensitivity analyses.
Section 4.4.1 of the uSSRA notes that “1,000 runs were performed”; the committee assumes that this refers to the Monte Carlo simulation. Typically, moderately complex models require more than 1,000 or even 10,000 iterations to achieve stability in the output. Without sufficient iterations to give model stability, the results may be inaccurate compared with results based on adequate iterations. The uSSRA should have consistently established that the number of iterations used in Monte Carlo simulations were sufficient to give stable model results, as is standard practice in stochastic modeling. That is, the target error rate and confidence should be specified. Some carefully chosen examples would have improved transparency and clarity of the methods and results.
Assumption About Respirator Use
Regarding lack of sufficient institutional knowledge or practices, the uSSRA states that N95 respirators will always be worn by workers dealing
with animals (alive or dead) that are infected with FMDv (p. 99). The committee suspects that that change in standard operating procedure (SOP) to require respirator use may have been a result of the 2010 SSRA’s indicating that the scenario of FMDv transfer without respirators contributed about half the estimated 70% chance of release over 50 years. FMDv investigators at several institutes explored the option of using N95 respirators when working with infected animals and concluded that they were unnecessary (Donaldson, 2008). Inasmuch as animal caretakers routinely go from infected to uninfected rooms the next day, using routine shower and decontamination practices without respirators and without transmitting FMDv to control animals, readers familiar with FMDv would question why N95 respirators would be required and what base of institutional knowledge the uSSRA chose to build from.
The committee also has concerns about the documentation of the sources and reliability of the data used in this analysis. The scientific rationale for wearing respirators presented here is based on studies reported in 1969 and 1970 in which experiments were done to determine the amount of virus present in the nasal passages of humans after exposure to animals infected with FMDv and to determine potential transmission to naïve animals. Those studies are well known to the global FMD community and formed the basis of the 3- to 7-day quarantine period that has been observed at many research facilities working with FMD. However, these studies and assessments were brought into question after the 2001 outbreak of FMD in the United Kingdom: By implementing a policy for an outbreak situation that was intended for the laboratory setting, the mandatory quarantine period so severely restricted the availability of animal health personnel to visit farms that it prompted research after the outbreak. Subsequent experiments done at Plum Island and Pirbright (Amass et al., 2003, 2004; Wright et al., 2010) observed that for a couple of strains, routine biosafety measures, such as showering and changing clothing, were sufficient to keep operators from spreading FMD infection from one animal room to another. The uSSRA does refer to two of the publications dealing with assessment of FMDv carriage by animal health personnel (Amass et al., 2004; Wright et al., 2010) but chooses to compute risk based on much earlier data. Again, it is probably the conclusions from the 2010 SSRA that formed the uSSRA’s basis for wearing N95 masks, but it is in contrast to the policy at FMD laboratories worldwide.
The committee also questions the data used to model the efficiency of N95 respirators. On p. 157, the uSSRA notes that a 2.5% failure rate for N95 respirators is expected because of a published failure rate (Cummings et al., 2007). The cited reference deals with poor N95 efficiency in workers in the aftermath of Hurricane Katrina, when the failure rate was actually around 75%, so the reference is not appropriate. The uSSRA also cites considerable N95 experimental penetration data (p. 100), and this may
be what allows it to use a 2.5% failure rate; however, these data are from studies that dealt with very controlled laboratory studies and did not take into account head movement, facial abnormalities, and other human (but non-fit) issues. Therefore, the committee views the use of a 2.5% failure rate in a real-life setting as an under-representation of reality.
Factors Related to Respiratory Transference
With respect to transference to the respiratory tract, the uSSRA assumes that failure rates due to poor fit of N95 masks would be only one-tenth the rate identified in a published study, in which failure rates of about 25% were found when the mask fit was poor. The uSSRA justifies the much lower rate based on the purported NBAF requirement that masks will fit correctly. Such a requirement may exist, but the basis for tenfold reduction seems poorly supported, especially because only a single study of mask failure is cited. It is not clear that human error is taken into account in this estimate of failure. The assumed failure rate seems overly optimistic, particularly in light of the physical exertion required of personnel working with large animals.
Although the wearing of N95 respirators may be a moot point, inasmuch as the change in SOP to require respirator use for FMDv work is not supported by literature, the committee is concerned that there are so many errors in the analysis of the data and the computations surrounding these respirators.
What Constitutes a System Failure
The uSSRA appears to assume 100% function or 0% function—all or none. It does not address functional disabilities that would adversely affect the efficiency of systems (such as incineration, autoclaves, EDS, and HEPA filtration) when it is less than 100% operational but has not indicated “failure.” These may be subtle problems that do not appreciably affect sensors or monitors and thus would not detect partial loss of function. It is unlikely that all equipment and systems will operate at full 100% (perfect) function 100% of the time or that redundancy will always protect against such marginal failure conditions. It is also unlikely that systems that are not at 100% function will be at 0% function unless they are completely shut down. The question not addressed is how often systems would be less than 100% functional and how that would adversely affect, for example, efficiency of virus kill or reduction. Downtime for routine maintenance, repairs, and replacement when there is not likely to be redundancy was not addressed.
The uSSRA attempts to model natural disasters caused by extreme winds (including tornadoes and hurricanes) and seismic activity. That was done to determine engineering requirements for ensuring the integrity of the biocontainment areas and to assess the risk of an envelope breach. The Manhattan, Kansas, site is in the heart of “Tornado Alley,” and tornadoes are generally known as the most significant natural hazard threat for that area. Hurricanes and floods were discussed briefly but are not included as catastrophic events examined in the uSSRA. Earthquakes would not normally be considered a serious hazard in the area, but they are also assessed. It does not appear that the Riley County hazard and vulnerability assessment was reviewed as part of the uSSRA process; that could have informed the uSSRA on the highest risks perceived by those most familiar with the area.
Tornado Risk Method
The uSSRA substantially extended and refined its treatment of tornadoes by transitioning from tornado F-Scale to Enhanced F-Scale (EF-Scale), using the leveraging method developed by the Pacific Northwest National Laboratory, including provision for the facility size and considering spatial variation of wind speed along the damage path of a tornado. A site-specific tornado risk model that relates tornadic wind speed with the annual probability of occurrence (or the mean recurrence interval, commonly referred to as the mean return period) is the most critical component of any tornado risk assessment study.
The uSSRA provides an overall tornado risk analysis that is state of the art and that has been used by the Nuclear Regulatory Commission for power plant designs, and this analysis is an improvement over that provided in the 2010 SSRA. It uses an appropriate method, which includes additional data that are “event-based” rather than “segment-based”; the latter has inherent shortcomings. The uSSRA provides a continuous distribution of the strike probability of a tornado by wind speed and includes 5th and 95th percentiles. Due to insufficient sample size at higher wind speeds for large EF-Scale tornadoes, the estimated percentiles are influenced by this lack of data for high wind speeds which adds to the uncertainty for the estimates,
and the uSSRA noted that the estimates should be used with caution. The design wind speeds used in the uSSRA appear to be adequate for the design of such a facility with the prescribed probability of exceedance (POE).
Whereas the overall hazard analysis is state of the art, the results should have been analyzed by using more refined spatial techniques to observe tornado patterns, such as kernel density estimation (KDE). KDE is an interpolation scheme that emphasizes spatial patterns at a location rather than considering only locations where tornadoes were recorded. In light of the risks provided in the uSSRA, it is unlikely that any further refinement in analysis would yield changes that would affect the final cumulative risk across events.
Tornado Design Aspects
The uSSRA suggests that the current NBAF 65% designs provide a tornado-hardened zone to ensure protection against loss of containment in the event of a tornado and protection against envelope penetration and development of cracks up to wind speeds of 228 mph. The protection also includes defense against windborne missiles (such as projectiles and debris) that can become airborne in tornadoes and can result in serious damage.
Figures 2.4.5-2 and 4.6.3-3 of the uSSRA highlight the tornado-hardened sections of the NBAF to ensure the integrity of the containment and envelope. The uSSRA does not include any systematically derived fragility curve (e.g., conditional probability of failure or other adverse performance given the level of tornado loading) for each performance level to correspond with the established level of risk associated with tornado wind speeds. That would be necessary to demonstrate the efficacy of the designed performance levels of the containment system and its envelope. A fragility curve for a prescribed performance criterion would define a level of damage conditional on wind speed. When weighted with the corresponding probability density function of wind speed, it yields a probability of failure at the stated performance level. In the absence of a fragility analysis, it is assumed that no pathogens will be lost at the maximum design wind speed of 228 mph. The uSSRA also assumes that 100% of the MAR will be released if the winds reach 260–280 mph. That assumption is not backed up by a fragility analysis related to the integrity of the structural system or a breach of the containment. In the absence of a detailed catastrophic failure model for the NBAF, it has been further assumed that the releases between the design wind speed of 228 mph and the catastrophic wind speed of 260–289 mph follow a prescribed distribution. Those release levels should have been refined further with a fragility framework-based analysis, which would affect the annual probability of release due to tornado loading.
Assessment of Methods and Assumptions
The uSSRA does not include a systematically derived fragility analysis for different performance levels to correspond with the established level of risk associated with tornado wind speeds. In the absence of such information, it is not possible to assess the adequacy of the containment system performance under tornadic winds.
Seismic Risk Assessment
The uSSRA assesses the risk of earthquakes at the site by using U.S. Geological Survey spectral acceleration data to determine a 2% POE over 50 years at the NBAF site. Spectral acceleration is provided for two periods: 0.2 second and 1 second; these are appropriate starting points for the seismic analysis.
Those numbers are subsequently updated in the uSSRA with the NBAF design values. Rather than selecting a value of spectral acceleration commensurate with the dynamic features of the building’s containment system—which would be a short period of ground shaking (for example, the 0.2-second hazard)—the uSSRA uses a 1-second value. Selecting a long rather than short event resulted in a POE over 50 years that is 20 times higher than what would be expected. Because the NBAF would be a low-rise structure, the uSSRA should have selected a short period of acceleration, which would result in a lower POE and lower hazard across events. It is important to note that the short period of ground shaking with lower POE also results in a higher degree of damage. It is not possible to know how it will affect the overall risk without conducting a systematic analysis of structural fragility; however, it appears that the uSSRA predicts cumulative risk across events that is excessively high.
Earthquake Design Aspects
In the uSSRA, it appears that the selection of earthquake ground acceleration and the associated performance of the structure at a given ground acceleration have been treated in isolation. The selection of ground motion in the uSSRA was used to assess the risk of pathogen escape as a result of cracks and breaches in the building envelope, whereas the seismic performance of the structure has been the responsibility of the architects and engineers at the NBAF Design Partnership. The lower ground acceleration associated with modeling a 1-second period would yield a higher POE, but would result in a lower level of impact on the structural response, would
result in lower level of cracking and ductile behavior, and may lead to a smaller probability of pathogen escape. It has also been stipulated that the NBAF Design Partnership would conform to the most current codes of practice in designing and constructing the facility.
The uSSRA anticipates that hardening the facility for tornadoes also improves the containment system’s capacity to resist earthquakes and reduces the probable loss of containment caused by an earthquake. A performance-based multi-hazard analysis would allow complementary structural features to share load effects of different hazards. An integrated approach should have been used to appropriately account for hardened structural designs in assessing risk associated with multiple hazards (such as tornadoes and earthquakes).
A concern that arises with regard to the seismic analysis is the omission of the effect of vibrations on non-structural elements, including sensitive equipment necessary for filtering, ventilation, and control. To a large extent, this is a design issue and such lab appurtenances should be well secured and detuned from the main structure.
Assessment of Methods and Assumptions
The seismic risk analysis in the uSSRA fails to address fundamental issues in the selection of appropriate design spectral acceleration and the attendant performance of the containment system under design earthquake conditions. Therefore, the committee questions the estimated values of cumulative risk across events associated with seismic catastrophic events, and finds that the uSSRA overestimates the risk due to wind and seismic hazards.
Alexandersen, S., and A.I. Donaldson, 2002. Further studies to quantify the dose of natural aerosols of foot-and-mouth disease virus for pigs. Epidemiol Infect 128(2):313-323.
Alexandersen, S., Z. Zhang, S.M. Reid, G.H. Hutchings, and A.I. Donaldson. 2002. Quantities of infectious virus and viral RNA recovered from sheep and cattle experimentally infected with foot-and-mouth disease virus O UK 2001. J Gen Virol 83(8):1915-1923.
Alexandersen, S., Z. Zhang, A.I. Donaldson, and A.J. Garland. 2003. The pathogenesis and diagnosis of foot-and-mouth disease. J Comp Pathol 129(1):1-36.
Amass, S.F., J.M. Pacheco, J.L. Schneider, R.M. Alvarez, K.L. Clark, and D. Ragland. 2003. Procedures for preventing the transmission of foot-and-mouth disease virus to pigs and sheep by personnel in contact with infected pigs. Vet Rec 153(5):137-140.
Amass, S.F., P.W. Mason, J.M. Pacheco, C.A. Miller, A. Ramirez, K.L. Clark, B. Ragland, J.L. Schneider, and S.K. Kenyon. 2004. Procedures for preventing trasmission of foot-and-mouth diessase virus (O/TAW/97) by people. Vet Microbiol 103(3-4):143-149.
Beard, C.W., and P.W. Mason. 2000. Genetic determinants of altered virulence of Taiwanese foot-and-mouth disease virus. J Virol 74:987-991.
Cummings, K.J., J. Cox-Ganser, M.A. Riggs, N. Edwards, and K. Kreiss. 2007. Respirator donning in post-hurricane New Orleans. Emerg Infect Dis 13(5):700-707.
Donaldson, A.I., J. Gloster, L.D. Harvey, and D.H. Deans. 1982. Use of prediction models to forecast and analyse airborne spread during the foot-and-mouth disease outbreaks in Brittany, Jersey and the Isle of Wight in 1981. Vet Rec 110(3):53-57.
Donaldson, A.I., and S. Alexandersen. 2001. Relative resistance of pigs to infection by natural aerosols of FMD virus. Vet Rec 148(19):600-602.
Donaldson, A.I. 2008. Foot-and-mouth disease and the use of face masks. Vet Rec 162(5):163.
Gloster, J., C. Doel, S. Gubbins, and D.J. Paton. 2008. Foot-and-mouth disease: Measurements of aerosol emission from pigs as a function of virus strain and initial dose. Vet J 177(3):374-380.
Grubman, M.J., and B. Baxt. 2004. Foot-and-mouth disease. Microbiol Rev 17(2):465-493.
Mason, P.W., M.J. Grubman, and B. Baxt. 2003. Molecular basis of pathogenesis of FMDV. Virus Res 91(1):9-32.
Snowdon, W.A. 1966. Growth of foot-and-mouth disease virus in monolayer cultures of calf thyroid cells. Nature 210(5040):1079-1080.
Tam, S., A. Clavijo, E. Englehard, and M. Thurmond. 2009. Fluorescence-based multiplex real-time RT-PCR arrays for the detection and serotype determination of foot-and-mouth disease virus. J Virol Methods 161(2):183–191.
Wright, C.F., J. Gloster, L. Mazelet, D.J. Paton, and E.D. Ryan. 2010. Short-lived carriage of foot-and mouth disease virus in human nasal cavities after exposure to infected animals. Vet Rec 167(24):928-931.