Safeguarding U.S. agriculture from foreign animal diseases and protecting our food system require cutting-edge research and diagnostic capabilities. The Department of Homeland Security (DHS) and the U.S. Department of Agriculture (USDA) have embarked on an important mission to replace the aging Plum Island Animal Disease Center with a new facility, the National Bio- and Agro-Defense Facility (NBAF). When operational, this new facility would be the world’s fourth biosafety level 4 laboratory capable of large animal research. It would serve as a critical world reference laboratory for identifying emerging and unknown disease threats, and would thus be a critical asset in securing the future health, wealth, and security of the nation.
DHS selected Manhattan, Kansas, as the site for the new NBAF after an extensive site-selection process that involved an environmental impact statement. The Government Accountability Office raised concerns about DHS’s analysis of the potential spread of foot-and-mouth disease virus (FMDv), one of the most serious foreign animal disease threats. Congress subsequently directed DHS to conduct a site-specific risk assessment (SSRA) for the NBAF, instructed the National Research Council (NRC) to independently evaluate the SSRA, and prohibited obligation of NBAF construction funds until the NRC review was complete (P.L. 111-83).
The NRC review found that the 2010 SSRA was inadequate due to flawed methods and assumptions which potentially underestimated the risk of an accidental FMDv release from the NBAF in Manhattan, Kansas. In response, Congress mandated that DHS revise its SSRA to address shortcomings, directed the NRC to evaluate the updated SSRA (uSSRA), and
again prohibited obligation of construction funds until the completion of the second review (P.L. 112-10). The scope for both the 2010 SSRA and the 2012 uSSRA addressed accidental release of pathogens from the NBAF in Manhattan, Kansas, and excluded terrorist acts and malicious threats from its risk assessments.
The NRC convened a committee of experts to evaluate the adequacy and validity of the uSSRA. This report is the committee’s evaluation of the final uSSRA.
The committee evaluated the uSSRA provided by DHS in February 2012 and finds that it is a substantial improvement over the original 2010 SSRA. DHS and its contractors should be commended for this effort. Many of the shortcomings identified by the previous committee have been addressed in the uSSRA, and this has resulted in a more quantitative and transparent analysis. The uSSRA uses more conventional risk assessment methods and better complies with standard practice than did the 2010 SSRA. In general, the descriptions of the approaches are clear, and the uSSRA uses appropriate conceptual models and methods.
The quantitative conclusions of the uSSRA differ dramatically from those of the 2010 SSRA. Data and methods of the previous risk assessment led to a conclusion that for the two scenarios with the greatest risk of FMDv release (fomite and worker without respiratory protection), there would be a 70% probability that FMDv release would cause an infection resulting in an outbreak during the 50-year life span of the NBAF in Manhattan, Kansas. In contrast, the uSSRA concludes that the cumulative probability for 142 risk events (including catastrophic events such as tornadoes and earthquakes) leading to an accidental release of FMDv over 50 years is about 0.11% (or 1 in 46,000 per year), which is orders of magnitude lower than the first estimate. Improvements in the 65% design phase documents for the facility compared with the earlier and less complete design documents on which the 2010 SSRA was based may explain some of the risk reduction. However, the committee believes that questionable and inappropriate assumptions were used in the uSSRA that led to artificially lower estimates of the probabilities and amounts of pathogen released.
In contrast with the 2010 SSRA, which cited fomites and lack of respiratory protection as the most likely pathways of accidental FMDv release, the uSSRA concludes that the most likely release mechanisms are those associated with natural hazards, specifically earthquakes and tornadoes. The uSSRA concludes that these are about 20 times more likely than operational pathways.
Despite improvements, the committee finds that the uSSRA under-
estimates the risks of pathogen release and infection and inadequately characterizes the uncertainties in those risks. The committee finds that the extremely low probabilities of release are based on overly optimistic and unsupported estimates of human error rates, underestimates of infectious material available for release, and inappropriate treatment of dependencies, uncertainties, and sensitivities in calculating release probabilities.
The committee is concerned that the vanishingly small estimates of risk found throughout the uSSRA are inconsistent with most modern, complex industrial systems. In many instances, the committee could not verify uSSRA results, because methods and data were unevenly or poorly presented. The uSSRA also contains inconsistent information, which made it difficult to interpret data or to reconstruct risk scenarios, and thereby made it difficult to determine the degree to which risks were underestimated.
The committee recognizes that significant complexities accompany a risk assessment of this nature, yet the practice of risk analysis is sufficiently mature to be able to treat such complexities and therefore the committee’s expectations for such a risk assessment are customary and attainable. The number of facilities comparable with the NBAF is small, so there is little empirical validation of the risk estimates. However, because a pathogen release from the NBAF could have devastating agricultural, economic, and public health consequences, a risk assessment that reaches inappropriate conclusions could have substantial repercussions.
The committee has identified a number of deficiencies that lead to the conclusion that the uSSRA continues to be inadequate in characterizing the risks associated with operating the NBAF in Manhattan, Kansas.
Finding 1: The uSSRA addresses many, but not all, of the issues outlined in the congressional mandate. The uSSRA attempts a quantitative risk assessment and attempts to model FMDv release and infection from the proposed NBAF in Manhattan, Kansas. However, it does not adequately include overall risks associated with the most dangerous pathogens in its biosafety level 4 (BSL-4) assessment.
Finding 2: The 65% design phase plans for the facility appear to be sound. The NBAF design plans provided to the committee—which were at only 65% completion—appear to have been competently executed by architects and engineers experienced with modern biocontainment laboratories, and the designs appear to conform to international safety standards for similar facilities. Although DHS stated that those conducting the risk assessment consulted the building designs, the uSSRA does not seem to clearly reflect design changes or to incorporate design
provisions in the risk assessment (i.e., natural hazards assessments not reflecting 65% design plans that harden the structure against tornadoes). The committee recognizes that it is necessary and challenging to integrate design improvements to produce an informed risk assessment. It is important to note that inadequacies in the uSSRA do not necessarily imply inadequacies in the design of the facility itself. Any conclusions about the adequacy and validity of the uSSRA should not be construed to imply similar conclusions with respect to the quality of the 65% design plans.
Finding 3: The uSSRA misinterprets and misapplies some risk methods, which have implications for the entire risk assessment. The uSSRA adopts some risk assessment methods that are in line with current practice, including the application of event tree analysis and other methods of quantitative probabilistic risk assessment. The modeling framework used is a “scenario-based” approach that is a well-established approach to risk analysis of complex systems and processes. However, those risk assessment methods are inconsistently applied across the various sections of the report.
Finding 4: The uSSRA ignores probabilistic dependencies in calculating risk scenarios, and this results in potentially serious underestimations of total risk and incorrect ranking of risk contributors. Use of questionable or erroneous methods and assumptions about probabilistic dependencies in portions of the uSSRA most likely results in an underestimation of the probability of accidental FMDv release.
Finding 5: The characterization and assessment of uncertainties are incomplete and inconsistent, and this leads to a false sense of precision. Sensitivity analyses and a quantitative assessment of the uncertainties in key model assumptions, model parameters, and risk results are required for the uSSRA to be viewed as adequate. The lack of full assessment of uncertainties is a serious deficiency in the uSSRA and limits its utility for decision making.
Finding 6: The uSSRA applies very low human error rates and uses methods that omit important error pathways, which likely resulted in low estimated probabilities of release. Probabilities of human error used in the uSSRA appear not to be based on published literature or empirical experience. The 2010 SSRA concluded that human error would be the most likely cause of release, and the previous committee agreed with that conclusion (see Finding 9 of 2010 NRC report Evaluation of a Site-Specific Risk Assessment for the Department of
Homeland Security’s Planned National Bio- and Agro-Defense Facility in Manhattan, Kansas). Little justification is provided for the uSSRA’s optimistic assumption that the rate of human error for NBAF workers will be several times less than that of similarly skilled workers in similar facilities.
Finding 7: The uSSRA appears overly optimistic in its assignment of parameter values to models, yet describes those values as conservative. Underestimates of parameter values by one or two orders of magnitude—when taken together in multiplicative estimation—can lead to extremely low compound estimates of risk. This is significant because while a bias of one order of magnitude in central tendency appears minor, the presence of several such biases in sequence will shift final results by many orders of magnitude.
Finding 8: The uSSRA does not describe the approaches used to ensure thoroughness in the review of parametric inputs. It is impossible to determine whether the scientific literature and other information used to support risk assessment assumptions have been thoroughly reviewed and evaluated. Many parameter estimates depend on outdated references or on only a single reference. At times, the uSSRA selects the lowest resulting risk input factors despite the availability of other data yielding higher risk.
Finding 9: The uSSRA has improved its epidemic modeling to address previous criticisms; however, there continue to be significant limitations in model capabilities and available data, leaving large uncertainties in the numbers provided. Particular assumptions likely led to an underestimation of the magnitude of spread and the duration of an FMD outbreak that would result from an FMDv release from the NBAF in Manhattan, Kansas.
Finding 10: Although the methods used for the economic analysis are appropriate, the uSSRA does not provide sufficient information to replicate the results or to assess whether the analysis was properly executed. Underestimates of the magnitude, spread, and duration of an FMD outbreak were carried over into the economic model and led to a likely underestimate of the economic consequence of an FMDv escape from the NBAF.
Finding 11: The committee questions the conclusion that catastrophic natural hazards pose the greatest risk for accidental release of FMDv and finds that the uSSRA overestimates their probabilities. Despite
eastern Kansas being a region of relatively low seismicity, the uSSRA designates earthquakes as the hazard most likely to lead to an FMDv release from the Manhattan, Kansas, facility. One reason for DHS’s conclusion is that the uSSRA uses an annual exceedance probability of a 1-second ground shaking as the defining hazard, whereas a low-rise structure like the NBAF is mostly susceptible to a shorter period event. This results in an exceedance probability that is higher by perhaps a factor of 20 than what should have been used and inflates the earthquake hazard risk estimates. Furthermore, the uSSRA does not account for the low structural fragility (strong resistance) of the proposed design, so the probability of a release is overestimated by perhaps several orders of magnitude.
For tornado loading, the hazard is estimated (e.g., probability of exceedance of tornado winds above certain speeds), but again the facility’s low structural fragility to those winds is not considered. That leads to overestimation of the risk of releases due to tornadoes by implicitly assuming that any wind event above a particular speed leads to a release of 100% of the material available for release. Thus, the committee considers the estimates of the probability of releases due to both natural hazards to be too high.
Finding 12: The uSSRA is based on assumptions about surveillance, detection, response, and mitigation strategies that were not adequately comprehensive or science-based. Also, it appears they were developed with insufficient input from stakeholders and federal, state, and local governments. Assumptions used to model mitigation, response, and detection were based in large part on DHS and USDA expectations for significantly improved plans, programs, and strategies that would be implemented by the time the NBAF opens in 2020. Surveillance, detection, and emergency response capabilities (such as vaccine availability) are critical for mitigating an outbreak; yet those tools and capabilities are currently limited or not in place. There was no indication from DHS or USDA that the necessary science-based capabilities noted by the previous committee (see Finding 7 of 2010 NRC report Evaluation of a Site-Specific Risk Assessment for the Department of Homeland Security’s Planned National Bio- and Agro-Defense Facility in Manhattan, Kansas) would be implemented for FMD surveillance and response.
DHS and USDA still have significant gaps for providing these critical capabilities and for realistically carrying out plans that identify and incorporate agricultural, animal health, and public health sectors and major issues related to a potential pathogen release. If these assumed plans, programs, and strategies are not fully developed, validated, and implemented by the time the NBAF opens in 2020, the risks and con-
sequent impacts will likely be substantially greater than estimated in the uSSRA.
Finding 13: The uSSRA does not adequately address plans for personnel preparedness and training at the NBAF. Although the training and preparedness requirements of the Federal Select Agent Program established under the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 are well documented, the uSSRA fails to include the DHS plans for personnel training in security, laboratory procedures, and emergency response as required by P.L. 112-10. Those plans are critical for ensuring safe operations at the NBAF and for mitigating an accidental FMDv release from the laboratory. Exclusion of such information from the uSSRA leads the committee to believe that preparations for the requirement have not been fully addressed by DHS.
Finding 14: In the BSL-4 assessment, the uSSRA does not consider overall risk and presents a limited qualitative assessment of impact. Such an evaluation likely underestimates the overall risk related to the BSL-4 suite, and the potential impact of a release cannot be evaluated. The committee recognizes the inherent limitations in the available information regarding the henipaviruses that form the basis of the BSL-4 review and other agents that may be studied in the BSL-4 suite. However, the risk assessment presented focuses only on the unique risks of release from the BSL-4 suite associated with use of large animals. As a result, the approach used for the BSL-4 evaluation understates the range of potential risks in the BSL-4 environment.
LIMITED APPLICABILITY OF THE UPDATED
SITE-SPECIFIC RISK ASSESSMENT
For any risk assessment, results apply only when the assumptions upon which they are based are consistent with practice. The uSSRA makes key assumptions about the physical design of the facility, maintenance and operation, and implementation of mitigation strategies. Any significant deviation from the assumed characteristics will modify risk factors and reduce the validity of the risk assessment.
It is critical to recognize that a sufficient level of funding for the NBAF and for risk mitigation activities is required to carry out the planned assumptions noted in the uSSRA. Operating BSL-3 and especially BSL-4 facilities is expensive because of equipment, personnel, operating costs, and maintenance and because of the need for systems for detection and active surveillance. Shortcomings in any of those areas will impact the risk profile
of the facility. Without a long-term funding commitment that is sufficient to maintain the level and quality of NBAF operations and that can sustain planned mitigation strategies, the findings presented in the uSSRA are not assured.
Not all deviations from the planning assumptions would significantly alter risk. If the uSSRA had included a careful sensitivity analysis based on alternative assumptions and if the deviations had been captured in such a sensitivity analysis, the uSSRA might still be applicable. The uSSRA provided to the committee contains no such sensitivity analysis. The uSSRA has limitations in its applicability, and these limitations are not clearly stated in the uSSRA. Absent a thorough sensitivity analysis, the applicability of the uSSRA under alternative operational conditions cannot be ascertained.
It is important to note that research, diagnostic, and mitigation capabilities envisioned for the NBAF are critical for protecting the nation against known threat agents along with emerging and unknown disease threats. The present committee echoes the conclusions of previous NRC committees that the United States needs the capacity to support critical research and diagnostic programs for the study of foreign animal diseases and zoonotic diseases that are directly linked to securing the health and wealth of the nation.
As required by P.L. 112-10, the committee was instructed to judge the adequacy and validity of the uSSRA. The committee has identified serious concerns about (1) the misapplication of methods used to assess risk, (2) the failure to make clear whether and how the evidence used to support risk assessment assumptions had been thoroughly reviewed and adequately evaluated, (3) the limited breadth of literature cited and the misinterpretation of some of the significant supporting literature, (4) the failure to explain the criteria used to select assumptions when supporting literature is conflicting, (5) the failure to consider important risk pathways, and (6) the inadequate treatment of uncertainty. Those deficiencies are not equally problematic, but they occur with sufficient frequency to raise doubts about the adequacy and validity of the risk results presented. In most instances (e.g., operational activities at the NBAF), the identified problems lead to an underestimation of risk; in other instances (e.g., catastrophic natural hazards), the risks may be overestimated. As a result, the committee concludes that the uSSRA is technically inadequate in critical respects and is an insufficient basis on which to judge the risks associated with the proposed NBAF in Manhattan, Kansas.