Evaluation of the Updated Site-Specific Risk Assessment for the National Bio- and Agro-Defense Facility in Manhattan, Kansas (2012)

9

Overall Assessment, Findings,
and Conclusions

The updated site-specific risk assessment (uSSRA) for the proposed National Bio- and Agro-Defense Facility (NBAF) released in 2012 is a substantial improvement over the original 2010 site-specific risk assessment (SSRA), and the Department of Homeland Security (DHS) and its contractors should be commended for this effort. Many of the shortcomings identified by the previous National Research Council committee (NRC, 2010) have been addressed in the uSSRA, and this has resulted in a more quantitative and transparent analysis. The uSSRA uses more conventional risk assessment methods and better complies with standard practice than did the 2010 SSRA. In general, the descriptions of the approaches are clear, and the uSSRA uses appropriate conceptual models and methods. However, the committee finds that some questionable and inappropriate assumptions were made to develop estimates of the probabilities, frequencies, and amounts of release of foot-and-mouth disease virus (FMDv) and other pathogens. In general, one needs to distinguish between the use of appropriate methods and use of appropriate assumptions to produce estimates. In the uSSRA, the former are generally acceptable, but the latter in some cases are not.

OVERALL ASSESSMENT

The quantitative conclusions of the uSSRA differ dramatically from those of the 2010 SSRA. Data and methods of the previous risk assessment led to a conclusion that for the two scenarios with the greatest risk of FMDv release (fomite and worker without respiratory protection), there

would be a 70% probability that FMDv release would cause an infection resulting in an outbreak during the 50-year life span of the NBAF in Manhattan, Kansas. In contrast, the uSSRA concludes that the cumulative probability for 142 risk events (including catastrophic events such as tornadoes and earthquakes) leading to an accidental release of FMDv over 50 years is about 0.11% (or 1 in 46,000 per year), which is orders of magnitude lower than the first estimate. Improvements in the 65% design phase documents for the facility compared with the earlier and less complete design documents on which the 2010 SSRA was based may explain some of the risk reduction. However, the committee believes that questionable and inappropriate assumptions were used in the uSSRA that led to artificially lower estimates of the probabilities and amounts of pathogen released.

In contrast with the 2010 SSRA, which cited fomites and lack of respiratory protection as the most likely pathways of accidental FMDv release, the uSSRA concludes that the most likely release mechanisms are those associated with natural hazards, specifically earthquakes and tornadoes. The uSSRA concludes that these are about 20 times more likely than operational pathways.

Despite improvements, the committee finds that the uSSRA underestimates the risks of pathogen release and infection and inadequately characterizes the uncertainties in those risks. The committee finds that the extremely low probabilities of release are based on overly optimistic and unsupported estimates of human error rates, underestimates of infectious material available for release, and inappropriate treatment of dependencies, uncertainties, and sensitivities in calculating release probabilities.

The committee is concerned that the vanishingly small estimates of risk found throughout the uSSRA are inconsistent with most modern, complex industrial systems. In many instances, the committee could not verify uSSRA results, because methods and data were unevenly or poorly presented. The uSSRA also contains inconsistent information, which made it difficult to interpret data or to reconstruct risk scenarios and thereby made it difficult to determine the degree to which risks were underestimated.

The committee recognizes that significant complexities accompany a risk assessment of this nature, yet the practice of risk analysis is sufficiently mature to be able to treat such complexities (Kumamoto and Henley, 1996; NASA, 2011) and therefore the committee’s expectations for such a risk assessment are customary and attainable. The number of facilities comparable with the NBAF is small, so there is little empirical validation of the risk estimates. However, because a pathogen release from the NBAF could have devastating widespread agricultural, economic, and public health consequences, a risk assessment that reaches inappropriate conclusions could have substantial repercussions.

The committee has identified a number of deficiencies that lead to the conclusion that the uSSRA continues to be inadequate in characterizing the risks associated with operating the NBAF in Manhattan, Kansas.

FINDINGS

Congressional Mandate

Finding 1: The uSSRA addresses many, but not all, of the issues outlined in the congressional mandate. In 2010, a National Research Council committee found that the initial SSRA was not entirely adequate or valid, because of methodological limitations and assumptions that underestimated the risks and economic costs associated with an accidental FMDv release from the Manhattan, Kansas, site (NRC, 2010). The uSSRA attempts a quantitative risk assessment and attempts to model FMDv release and infection from the proposed NBAF in Manhattan, Kansas. However, it does not adequately include overall risks associated with the most dangerous pathogens in its biosafety level 4 (BSL-4) assessment.

65% Design Phase Plans for the National Bio- and Agro-Defense Facility

Finding 2: The 65% design phase plans for the facility appear to be sound. The NBAF design plans provided to the committee—which were at only 65% completion—appear to have been competently executed by architects and engineers experienced with modern biocontainment laboratories, and the designs appear to conform to international safety standards for similar facilities. Although DHS stated that those conducting the risk assessment consulted the building designs, the uSSRA does not seem to clearly reflect design changes or to incorporate such design provisions in the risk assessment (i.e., natural hazards assessments not reflecting 65% design plans that harden the structure against tornadoes). The committee recognizes that it is necessary and challenging to integrate design improvements to produce an informed risk assessment.

It is important to note that inadequacies in the uSSRA do not necessarily imply inadequacies in the design of the facility itself. Any conclusions about the adequacy and validity of the uSSRA should not be construed to imply similar conclusions with respect to the quality of the 65% design plans.

Use and Application of Risk Methods

Finding 3: The uSSRA misinterprets and misapplies some risk methods, which have implications for the entire risk assessment. The uSSRA adopts some risk assessment methods that are in line with current practice, including the application of event tree analysis and other methods of quantitative probabilistic risk assessment. The modeling framework used is a “scenario-based” approach that is a well-established approach to risk analysis of complex systems and processes. However, those risk assessment methods are inconsistently applied across the various sections of the report.

Lack of Independence Among Events and Parameter Values

Finding 4: The uSSRA ignores probabilistic dependencies in calculating risk scenarios, and this results in potentially serious underestimations of total risk and incorrect ranking of risk contributors. Use of questionable or erroneous methods and assumptions about probabilistic dependencies in portions of the uSSRA most likely results in an underestimation of the probability of accidental FMDv release.

In probabilistic modeling, it is fundamentally important to correctly characterize probabilistic dependencies among events and model parameters, and to account for dependencies in calculating probabilities of the joint occurrence of those events and parameters. The uSSRA assumes that all events are probabilistically independent of each other whereas in reality they can be dependent. Failure to consider the possibility that many event sequences could contribute to loss of containment and subsequent infections is an important deficiency and could lead to a serious underestimation of risk.

Uncertainty and Sensitivity Analysis

Finding 5: The characterization and assessment of uncertainties are incomplete and inconsistent, and this leads to a false sense of precision. Sensitivity analyses and a quantitative assessment of the uncertainties in key model assumptions, model parameters, and risk results are required for the uSSRA to be viewed as adequate. A thorough view of uncertainties is necessary for sound decision making. Except for the epidemic modeling section, the uSSRA does not consider that there are large differences in the quality and quantity of data and information used to support the analysis or how those differences could affect the reliability of subsequent risk estimates. The only measure of “uncertainty” identified in the uSSRA (Section 8) is “stochastic variability.” There appears

to be no comprehensive consideration of epistemic uncertainty related to lack of data or knowledge, and the cumulative effects of uncertainty on the overall assessment are not adequately discussed.

Sensitivity analyses, which would be a way to examine the effects of model assumptions and variations in parameter values, are not comprehensive and in most cases do not adequately test the effects of the range of variability of the model parameters. The lack of full assessment of uncertainties is a serious deficiency in the uSSRA and limits its utility for decision making.

Treatment of Human Reliability and Human Error

Finding 6: The uSSRA applies very low human error rates and uses methods that omit important error pathways, which likely resulted in low estimated probabilities of release. Probabilities of human error used in the uSSRA appear not to be based on published literature or empirical experience. The 2010 SSRA concluded that human error would be the most likely cause of release, and the previous committee agreed with that conclusion (Finding 9 of NRC, 2010). Little justification is provided for the uSSRA’s optimistic assumption that, in at least one pathway, the rate of human error for NBAF workers will be several times less than that of similarly skilled workers in similar facilities. Furthermore, not all pathways by which human error can be a significant factor in failure modes have been satisfactorily considered.

Human error is an important contributor to risk in facilities similar to the NBAF. The uSSRA should have explored possible sources of data and operating experience for human error in analogous research laboratory settings as the basis of reference error probabilities.

Modeling Parameters and Assumptions

Finding 7: The uSSRA appears overly optimistic in its assignment of parameter values to models, yet describes the values as conservative. Underestimates of parameter values by one or two orders of magnitude—when taken together in multiplicative estimation—can lead to extremely low compound estimates of risk. This is significant because while a bias of one order of magnitude in central tendency appears minor, the presence of several such biases in sequence will shift the final results by many orders of magnitude. While this problem is not apparent throughout the report, the lack of an adequate sensitivity analysis makes it difficult to ascertain the effects on estimated risks of the alternative assumptions used.

Finding 8: The uSSRA does not describe the approaches used to ensure thoroughness in the review of parametric inputs. Therefore, it is impossible to determine whether the scientific literature and other information used to support risk assessment assumptions have been thoroughly reviewed and evaluated. Many parameter estimates depend on outdated references or on only a single reference. In many places, the uSSRA appears to use unrealistic assumptions to determine model parameters, and the parameter values that were adopted appear to be inconsistent among sections and even within sections of the report. Within sections of the report, parameter values given in the text and in tables are in apparent conflict. At times, the uSSRA selects the lowest resulting risk input factors despite the availability of other data yielding a higher risk.

Finding 9: The uSSRA has improved its epidemic modeling to address previous criticisms; however, there continue to be significant limitations in model capabilities and available data, leaving large uncertainties in the numbers provided. Particular assumptions likely led to an underestimation of the magnitude of spread and the duration of an FMD outbreak that would result from an FMDv release from the NBAF in Manhattan, Kansas. The uSSRA did not consider the possibility that an FMD outbreak could quickly spread out of control in the United States and result in FMD becoming an endemic disease that requires a very long-term eradication program.

Finding 10: Although the methods used for the economic analysis are appropriate, the uSSRA does not provide sufficient information to replicate the results or to assess whether the analysis was properly executed. Underestimates of the magnitude, spread, and duration of an FMD outbreak were carried over into the economic model and led to a likely underestimate of the economic consequence of an FMDv escape from the NBAF.

Treatment of Natural Hazards

Finding 11: The committee questions the conclusion that catastrophic natural hazards pose the greatest risk for accidental release of FMDv and finds that the uSSRA overestimates their probabilities. Despite eastern Kansas being a region of relatively low seismicity, the uSSRA designates earthquakes as the hazard most likely to lead to an FMDv release from the Manhattan, Kansas, facility. Tornadoes are a more significant natural hazard, given the proposed location of the NBAF in an area with a disproportionately high frequency of tornadoes known as “Tornado Alley.”

One reason for DHS’s conclusion is that the uSSRA uses an annual exceedance probability of a 1-second ground shaking as the defining hazard, whereas a low-rise structure like the NBAF is mostly susceptible to a shorter period event. This results in an exceedance probability that is higher by perhaps a factor of 20 than what should have been used and inflates the earthquake hazard risk estimates. Furthermore, the uSSRA does not account for the low structural fragility (strong resistance) of the proposed design, so the probability of a release is overestimated by perhaps several orders of magnitude.

For tornado loading, the hazard is estimated (e.g., probability of exceedance of tornado winds above certain speeds), but again the facility’s low structural fragility to those winds is not considered. That leads to overestimation of the risk of releases due to tornadoes by implicitly assuming that any wind above a particular speed leads to a release of 100% of the material available for release. Thus, the committee considers the estimates of the probability of releases due to both natural hazards to be too high.

Surveillance, Response, and Mitigation Plans

Finding 12: The uSSRA is based on assumptions about surveillance, detection, response, and mitigation strategies that were not adequately comprehensive or science-based. Also it appears they were developed with insufficient input from stakeholders and federal, state, and local governments. Assumptions used to model mitigation, response, and detection were based in large part on DHS and USDA expectations for significantly improved plans, programs, and strategies that would be implemented by the time the NBAF opens in 2020. Surveillance, detection, and emergency response capabilities (such as vaccine availability) are critical for mitigating an outbreak, yet those tools and capabilities are currently limited or not in place. There was no indication from DHS or USDA that the necessary science-based capabilities noted by the previous committee (Finding 7 in NRC, 2010) would be implemented for FMD surveillance and response.

DHS and USDA still have significant gaps for providing these critical capabilities and for realistically carrying out plans that identify and incorporate agricultural, animal health, and public health sectors and major issues related to a potential pathogen release. DHS and USDA have experience operating similar facilities such as the National Biodefense Analysis and Countermeasures Center (NBACC) and Plum Island Animal Disease Center (PIADC), which could have helped to supplement the uSSRA in providing robust plans for the development of emergency response capabilities. Similarly, local and state emergency

response, healthcare, and related organizations have significant gaps that the uSSRA assumes will be filled by the time the facility opens, but for which a comprehensive timeline has not been provided. Therefore, it is difficult to know whether the calculations and conclusions in the uSSRA regarding spread and impact are adequate and valid. If these assumed plans, programs, and strategies are not fully developed, validated, and implemented by the time the NBAF opens in 2020, the risks and consequent impacts will likely be substantially greater than estimated in the uSSRA.

Personnel Preparedness and Training

Finding 13: The uSSRA does not adequately address plans for personnel preparedness and training at the NBAF. Although the training and preparedness requirements of the Federal Select Agent Program established under the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 are well documented, the uSSRA fails to include the DHS plans for personnel training in security, laboratory procedures, and emergency response as required by P.L. 112-10. Those plans are critical for ensuring safe operations at the NBAF and for mitigating an accidental FMDv release from the laboratory. As previously mentioned, DHS and its contractors could have drawn from experiences of operating similar facilities like NBACC and PIADC to inform plans for NBAF personnel training and standard operating procedures. Exclusion of such information from the uSSRA leads the committee to believe that preparations for the requirement have not been fully addressed by DHS.

BSL-4 Assessment

Finding 14: In the BSL-4 assessment, the uSSRA does not consider overall risk and presents a limited qualitative assessment of impact. Such an evaluation likely underestimates the overall risk related to the BSL-4 suite, and the potential impact of a release cannot be evaluated. The committee recognizes the inherent limitations in the available information regarding the henipaviruses that form the basis of the BSL-4 review and other agents that may be studied in the BSL-4 suite. However, the risk assessment focuses only on the unique risks of release from the BSL-4 suite associated with the use of large animals.

The uSSRA does not adequately consider the totality of risks in its BSL-4 assessment. An overall risk assessment of the NBAF BSL-4 would need to include an evaluation of the additive risks associated with the entire array of BSL-4 work. That includes the risk of a release

LIMITED APPLICABILITY OF THE UPDATED
SITE-SPECIFIC RISK ASSESSMENT

For any risk assessment, results apply only when the assumptions upon which they are based are consistent with practice. The uSSRA makes key assumptions about the physical design of the facility, maintenance and operation, and implementation of mitigation strategies. Any significant deviation from the assumed characteristics will modify risk factors and reduce the validity of the risk assessment.

It is critical to recognize that a sufficient level of funding for the NBAF and for risk mitigation activities is required to carry out the planned assumptions noted in the uSSRA. Operating BSL-3 and especially BSL-4 facilities is expensive because of equipment, personnel, operating costs, and maintenance and because of the need for systems for detection and active surveillance. Shortcomings in any of those areas will impact the risk profile of the facility. Without a long-term funding commitment that is sufficient to maintain the level and quality of NBAF operations and that can sustain planned mitigation strategies, the findings presented in the uSSRA are not assured.

Not all deviations from the planning assumptions would significantly alter risk. If the uSSRA had included a careful sensitivity analysis based on alternative assumptions and if the deviations had been captured in such a sensitivity analysis, the uSSRA might still be applicable. The uSSRA provided to the committee contains no such sensitivity analysis. The uSSRA has limitations in its applicability, and these limitations are not clearly stated in the uSSRA. Absent a thorough sensitivity analysis, the applicability of the uSSRA under alternative operational conditions cannot be ascertained.

CONCLUSIONS

It is important to note that research, diagnostic, and mitigation capabilities envisioned for the NBAF are critical for protecting the nation against known threat agents along with emerging and unknown disease threats. The present committee echoes the conclusions of previous NRC committees that the United States needs the capacity to support critical research and diagnostic programs for the study of foreign animal diseases and zoonotic diseases that are directly linked to securing the health and wealth of the nation (NRC, 2005a,b, 2010; IOM and NRC, 2009).

As required by P.L. 112-10, the committee was instructed to judge the adequacy and validity of the uSSRA. The committee has identified serious concerns about (1) the misapplication of methods used to assess risk, (2) the failure to make clear whether and how the evidence used to support risk assessment assumptions had been thoroughly reviewed and adequately evaluated, (3) the limited breadth of literature cited and the misinterpretation of some of the significant supporting literature, (4) the failure to explain the criteria used to select assumptions when supporting literature is conflicting, (5) the failure to consider important risk pathways, and (6) the inadequate treatment of uncertainty. Those deficiencies are not equally problematic, but they occur with sufficient frequency to raise doubts about the adequacy and validity of the risk results presented. In most instances (e.g., operational activities at the NBAF), the identified problems lead to an underestimation of risk; in other instances (e.g., catastrophic natural hazards), the risks may be overestimated. As a result, the committee concludes that the uSSRA is technically inadequate in critical respects and is an insufficient basis on which to judge the risks associated with the proposed NBAF in Manhattan, Kansas.

REFERENCES

IOM (Institute of Medicine) and NRC (National Research Council). 2009. Sustaining Global Surveillance and Response to Emerging Zoonotic Diseases. Washington, DC: The National Academies Press.

Kumamoto, H., and E.J. Henley. 1996. Probabilistic Risk Assessment and Management for Engineers and Scientists (2nd Ed.). New York: Wiley-IEEE Press. 597 pp.

NASA (National Aeronautics and Space Administration). 2011. Probabilistic Risk Assessment Procedures Guide for NASA Managers and Practitioners (2nd Ed.). NASA/ SP-2011-3421. Available online at http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120001369_2012001000.pdf (accessed May 3, 2012).

NRC (National Research Council). 2005a. Animal Health at the Crossroads: Preventing, Detecting, and Diagnosing Animal Diseases. Washington, DC: The National Academies Press.

NRC. 2005b. Critical Needs for Research in Veterinary Science. Washington, DC: The National Academies Press.

NRC. 2010. Evaluation of a Site-Specific Risk Assessment for the Department of Homeland Security’s Planned National Bio- and Agro-Defense Facility in Manhattan, Kansas. Washington, DC: The National Academies Press.

This page intentionally left blank.