Summary

Safeguarding U.S. agriculture from foreign animal diseases and protecting our food system require cutting-edge research and diagnostic capabilities. The Department of Homeland Security (DHS) and the U.S. Department of Agriculture (USDA) have embarked on an important mission to replace the aging Plum Island Animal Disease Center with a new facility, the National Bio- and Agro-Defense Facility (NBAF). When operational, this new facility would be the world’s fourth biosafety level 4 laboratory capable of large animal research. It would serve as a critical world reference laboratory for identifying emerging and unknown disease threats, and would thus be a critical asset in securing the future health, wealth, and security of the nation.

DHS selected Manhattan, Kansas, as the site for the new NBAF after an extensive site-selection process that involved an environmental impact statement. The Government Accountability Office raised concerns about DHS’s analysis of the potential spread of foot-and-mouth disease virus (FMDv), one of the most serious foreign animal disease threats. Congress subsequently directed DHS to conduct a site-specific risk assessment (SSRA) for the NBAF, instructed the National Research Council (NRC) to independently evaluate the SSRA, and prohibited obligation of NBAF construction funds until the NRC review was complete (P.L. 111-83).

The NRC review found that the 2010 SSRA was inadequate due to flawed methods and assumptions which potentially underestimated the risk of an accidental FMDv release from the NBAF in Manhattan, Kansas. In response, Congress mandated that DHS revise its SSRA to address shortcomings, directed the NRC to evaluate the updated SSRA (uSSRA), and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary Safeguarding U.S. agriculture from foreign animal diseases and pro- tecting our food system require cutting-edge research and diagnostic capabilities. The Department of Homeland Security (DHS) and the U.S. Department of Agriculture (USDA) have embarked on an important mis- sion to replace the aging Plum Island Animal Disease Center with a new facility, the National Bio- and Agro-Defense Facility (NBAF). When opera- tional, this new facility would be the world’s fourth biosafety level 4 labo- ratory capable of large animal research. It would serve as a critical world reference laboratory for identifying emerging and unknown disease threats, and would thus be a critical asset in securing the future health, wealth, and security of the nation. DHS selected Manhattan, Kansas, as the site for the new NBAF after an extensive site-selection process that involved an environmental impact statement. The Government Accountability Office raised concerns about DHS’s analysis of the potential spread of foot-and-mouth disease virus (FMDv), one of the most serious foreign animal disease threats. Congress subsequently directed DHS to conduct a site-specific risk assessment (SSRA) for the NBAF, instructed the National Research Council (NRC) to indepen- dently evaluate the SSRA, and prohibited obligation of NBAF construction funds until the NRC review was complete (P.L. 111-83). The NRC review found that the 2010 SSRA was inadequate due to flawed methods and assumptions which potentially underestimated the risk of an accidental FMDv release from the NBAF in Manhattan, Kansas. In response, Congress mandated that DHS revise its SSRA to address short- comings, directed the NRC to evaluate the updated SSRA (uSSRA), and 1

OCR for page 1
2 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT again prohibited obligation of construction funds until the completion of the second review (P.L. 112-10). The scope for both the 2010 SSRA and the 2012 uSSRA addressed accidental release of pathogens from the NBAF in Manhattan, Kansas, and excluded terrorist acts and malicious threats from its risk assessments. The NRC convened a committee of experts to evaluate the adequacy and validity of the uSSRA. This report is the committee’s evaluation of the final uSSRA. OVERALL ASSESSMENT The committee evaluated the uSSRA provided by DHS in February 2012 and finds that it is a substantial improvement over the original 2010 SSRA. DHS and its contractors should be commended for this effort. Many of the shortcomings identified by the previous committee have been addressed in the uSSRA, and this has resulted in a more quantitative and transparent analysis. The uSSRA uses more conventional risk assessment methods and better complies with standard practice than did the 2010 SSRA. In general, the descriptions of the approaches are clear, and the uSSRA uses appropriate conceptual models and methods. The quantitative conclusions of the uSSRA differ dramatically from those of the 2010 SSRA. Data and methods of the previous risk assess- ment led to a conclusion that for the two scenarios with the greatest risk of FMDv release (fomite and worker without respiratory protection), there would be a 70% probability that FMDv release would cause an infection resulting in an outbreak during the 50-year life span of the NBAF in Man- hattan, Kansas. In contrast, the uSSRA concludes that the cumulative prob- ability for 142 risk events (including catastrophic events such as tornadoes and earthquakes) leading to an accidental release of FMDv over 50 years is about 0.11% (or 1 in 46,000 per year), which is orders of magnitude lower than the first estimate. Improvements in the 65% design phase docu- ments for the facility compared with the earlier and less complete design documents on which the 2010 SSRA was based may explain some of the risk reduction. However, the committee believes that questionable and inap- propriate assumptions were used in the uSSRA that led to artificially lower estimates of the probabilities and amounts of pathogen released. In contrast with the 2010 SSRA, which cited fomites and lack of respi- ratory protection as the most likely pathways of accidental FMDv release, the uSSRA concludes that the most likely release mechanisms are those as- sociated with natural hazards, specifically earthquakes and tornadoes. The uSSRA concludes that these are about 20 times more likely than operational pathways. Despite improvements, the committee finds that the uSSRA under-

OCR for page 1
3 SUMMARY estimates the risks of pathogen release and infection and inadequately characterizes the uncertainties in those risks. The committee finds that the extremely low probabilities of release are based on overly optimistic and unsupported estimates of human error rates, underestimates of infectious material available for release, and inappropriate treatment of dependencies, uncertainties, and sensitivities in calculating release probabilities. The committee is concerned that the vanishingly small estimates of risk found throughout the uSSRA are inconsistent with most modern, com- plex industrial systems. In many instances, the committee could not verify uSSRA results, because methods and data were unevenly or poorly pre- sented. The uSSRA also contains inconsistent information, which made it difficult to interpret data or to reconstruct risk scenarios, and thereby made it difficult to determine the degree to which risks were underestimated. The committee recognizes that significant complexities accompany a risk assessment of this nature, yet the practice of risk analysis is sufficiently mature to be able to treat such complexities and therefore the committee’s expectations for such a risk assessment are customary and attainable. The number of facilities comparable with the NBAF is small, so there is little empirical validation of the risk estimates. However, because a pathogen release from the NBAF could have devastating agricultural, economic, and public health consequences, a risk assessment that reaches inappropriate conclusions could have substantial repercussions. The committee has identified a number of deficiencies that lead to the conclusion that the uSSRA continues to be inadequate in characterizing the risks associated with operating the NBAF in Manhattan, Kansas. FINDINGS Finding 1: The uSSRA addresses many, but not all, of the issues out- lined in the congressional mandate. The uSSRA attempts a quantitative risk assessment and attempts to model FMDv release and infection from the proposed NBAF in Manhattan, Kansas. However, it does not adequately include overall risks associated with the most dangerous pathogens in its biosafety level 4 (BSL-4) assessment. Finding 2: The 65% design phase plans for the facility appear to be sound. The NBAF design plans provided to the committee—which were at only 65% completion—appear to have been competently executed by architects and engineers experienced with modern biocontainment laboratories, and the designs appear to conform to international safety standards for similar facilities. Although DHS stated that those con- ducting the risk assessment consulted the building designs, the uSSRA does not seem to clearly reflect design changes or to incorporate design

OCR for page 1
4 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT provisions in the risk assessment (i.e., natural hazards assessments not reflecting 65% design plans that harden the structure against torna- does). The committee recognizes that it is necessary and challenging to integrate design improvements to produce an informed risk assess- ment. It is important to note that inadequacies in the uSSRA do not necessarily imply inadequacies in the design of the facility itself. Any conclusions about the adequacy and validity of the uSSRA should not be construed to imply similar conclusions with respect to the quality of the 65% design plans. Finding 3: The uSSRA misinterprets and misapplies some risk methods, which have implications for the entire risk assessment. The uSSRA adopts some risk assessment methods that are in line with current prac- tice, including the application of event tree analysis and other methods of quantitative probabilistic risk assessment. The modeling framework used is a “scenario-based” approach that is a well-established approach to risk analysis of complex systems and processes. However, those risk assessment methods are inconsistently applied across the various sec- tions of the report. Finding 4: The uSSRA ignores probabilistic dependencies in calculating risk scenarios, and this results in potentially serious underestimations of total risk and incorrect ranking of risk contributors. Use of ques- tionable or erroneous methods and assumptions about probabilistic dependencies in portions of the uSSRA most likely results in an under- estimation of the probability of accidental FMDv release. Finding 5: The characterization and assessment of uncertainties are incomplete and inconsistent, and this leads to a false sense of precision. Sensitivity analyses and a quantitative assessment of the uncertainties in key model assumptions, model parameters, and risk results are required for the uSSRA to be viewed as adequate. The lack of full assessment of uncertainties is a serious deficiency in the uSSRA and limits its utility for decision making. Finding 6: The uSSRA applies very low human error rates and uses methods that omit important error pathways, which likely resulted in low estimated probabilities of release. Probabilities of human error used in the uSSRA appear not to be based on published literature or empirical experience. The 2010 SSRA concluded that human error would be the most likely cause of release, and the previous commit- tee agreed with that conclusion (see Finding 9 of 2010 NRC report Evaluation of a Site-Specific Risk Assessment for the Department of

OCR for page 1
5 SUMMARY Homeland Security’s Planned National Bio- and Agro-Defense Facility in Manhattan, Kansas). Little justification is provided for the uSSRA’s optimistic assumption that the rate of human error for NBAF work- ers will be several times less than that of similarly skilled workers in similar facilities. Finding 7: The uSSRA appears overly optimistic in its assignment of parameter values to models, yet describes those values as conservative. Underestimates of parameter values by one or two orders of magni- tude—when taken together in multiplicative estimation—can lead to extremely low compound estimates of risk. This is significant because while a bias of one order of magnitude in central tendency appears minor, the presence of several such biases in sequence will shift final results by many orders of magnitude. Finding 8: The uSSRA does not describe the approaches used to ensure thoroughness in the review of parametric inputs. It is impossible to determine whether the scientific literature and other information used to support risk assessment assumptions have been thoroughly reviewed and evaluated. Many parameter estimates depend on outdated refer- ences or on only a single reference. At times, the uSSRA selects the lowest resulting risk input factors despite the availability of other data yielding higher risk. Finding 9: The uSSRA has improved its epidemic modeling to address previous criticisms; however, there continue to be significant limitations in model capabilities and available data, leaving large uncertainties in the numbers provided. Particular assumptions likely led to an under- estimation of the magnitude of spread and the duration of an FMD outbreak that would result from an FMDv release from the NBAF in Manhattan, Kansas. Finding 10: Although the methods used for the economic analysis are appropriate, the uSSRA does not provide sufficient information to replicate the results or to assess whether the analysis was properly executed. Underestimates of the magnitude, spread, and duration of an FMD outbreak were carried over into the economic model and led to a likely underestimate of the economic consequence of an FMDv escape from the NBAF. Finding 11: The committee questions the conclusion that catastrophic natural hazards pose the greatest risk for accidental release of FMDv and finds that the uSSRA overestimates their probabilities. Despite

OCR for page 1
6 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT eastern Kansas being a region of relatively low seismicity, the uSSRA designates earthquakes as the hazard most likely to lead to an FMDv release from the Manhattan, Kansas, facility. One reason for DHS’s conclusion is that the uSSRA uses an annual exceedance probability of a 1-second ground shaking as the defining hazard, whereas a low-rise structure like the NBAF is mostly susceptible to a shorter period event. This results in an exceedance probability that is higher by perhaps a factor of 20 than what should have been used and inflates the earth- quake hazard risk estimates. Furthermore, the uSSRA does not account for the low structural fragility (strong resistance) of the proposed de- sign, so the probability of a release is overestimated by perhaps several orders of magnitude. For tornado loading, the hazard is estimated (e.g., probability of exceedance of tornado winds above certain speeds), but again the facil- ity’s low structural fragility to those winds is not considered. That leads to overestimation of the risk of releases due to tornadoes by implicitly assuming that any wind event above a particular speed leads to a re- lease of 100% of the material available for release. Thus, the committee considers the estimates of the probability of releases due to both natural hazards to be too high. Finding 12: The uSSRA is based on assumptions about surveillance, detection, response, and mitigation strategies that were not adequately comprehensive or science-based. Also, it appears they were developed with insufficient input from stakeholders and federal, state, and local governments. Assumptions used to model mitigation, response, and detection were based in large part on DHS and USDA expectations for significantly improved plans, programs, and strategies that would be implemented by the time the NBAF opens in 2020. Surveillance, detec- tion, and emergency response capabilities (such as vaccine availability) are critical for mitigating an outbreak; yet those tools and capabilities are currently limited or not in place. There was no indication from DHS or USDA that the necessary science-based capabilities noted by the previous committee (see Finding 7 of 2010 NRC report Evaluation of a Site-Specific Risk Assessment for the Department of Homeland Secu- rity’s Planned National Bio- and Agro-Defense Facility in Manhattan, Kansas) would be implemented for FMD surveillance and response. DHS and USDA still have significant gaps for providing these criti- cal capabilities and for realistically carrying out plans that identify and incorporate agricultural, animal health, and public health sectors and major issues related to a potential pathogen release. If these assumed plans, programs, and strategies are not fully developed, validated, and implemented by the time the NBAF opens in 2020, the risks and con-

OCR for page 1
7 SUMMARY sequent impacts will likely be substantially greater than estimated in the uSSRA. Finding 13: The uSSRA does not adequately address plans for person- nel preparedness and training at the NBAF. Although the training and preparedness requirements of the Federal Select Agent Program estab- lished under the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 are well documented, the uSSRA fails to include the DHS plans for personnel training in security, laboratory procedures, and emergency response as required by P.L. 112-10. Those plans are critical for ensuring safe operations at the NBAF and for mitigating an accidental FMDv release from the laboratory. Exclusion of such information from the uSSRA leads the committee to believe that preparations for the requirement have not been fully addressed by DHS. Finding 14: In the BSL-4 assessment, the uSSRA does not consider overall risk and presents a limited qualitative assessment of impact. Such an evaluation likely underestimates the overall risk related to the BSL-4 suite, and the potential impact of a release cannot be evaluated. The committee recognizes the inherent limitations in the available in- formation regarding the henipaviruses that form the basis of the BSL-4 review and other agents that may be studied in the BSL-4 suite. How- ever, the risk assessment presented focuses only on the unique risks of release from the BSL-4 suite associated with use of large animals. As a result, the approach used for the BSL-4 evaluation understates the range of potential risks in the BSL-4 environment. LIMITED APPLICABILITY OF THE UPDATED SITE-SPECIFIC RISK ASSESSMENT For any risk assessment, results apply only when the assumptions upon which they are based are consistent with practice. The uSSRA makes key assumptions about the physical design of the facility, maintenance and op- eration, and implementation of mitigation strategies. Any significant devia- tion from the assumed characteristics will modify risk factors and reduce the validity of the risk assessment. It is critical to recognize that a sufficient level of funding for the NBAF and for risk mitigation activities is required to carry out the planned as- sumptions noted in the uSSRA. Operating BSL-3 and especially BSL-4 fa- cilities is expensive because of equipment, personnel, operating costs, and maintenance and because of the need for systems for detection and active surveillance. Shortcomings in any of those areas will impact the risk profile

OCR for page 1
8 NBAF UPDATED SITE-SPECIFIC RISK ASSESSMENT of the facility. Without a long-term funding commitment that is sufficient to maintain the level and quality of NBAF operations and that can sustain planned mitigation strategies, the findings presented in the uSSRA are not assured. Not all deviations from the planning assumptions would significantly alter risk. If the uSSRA had included a careful sensitivity analysis based on alternative assumptions and if the deviations had been captured in such a sensitivity analysis, the uSSRA might still be applicable. The uSSRA pro- vided to the committee contains no such sensitivity analysis. The uSSRA has limitations in its applicability, and these limitations are not clearly stated in the uSSRA. Absent a thorough sensitivity analysis, the applicability of the uSSRA under alternative operational conditions cannot be ascertained. CONCLUSIONS It is important to note that research, diagnostic, and mitigation capabil- ities envisioned for the NBAF are critical for protecting the nation against known threat agents along with emerging and unknown disease threats. The present committee echoes the conclusions of previous NRC committees that the United States needs the capacity to support critical research and diagnostic programs for the study of foreign animal diseases and zoonotic diseases that are directly linked to securing the health and wealth of the nation. As required by P.L. 112-10, the committee was instructed to judge the adequacy and validity of the uSSRA. The committee has identified seri- ous concerns about (1) the misapplication of methods used to assess risk, (2) the failure to make clear whether and how the evidence used to support risk assessment assumptions had been thoroughly reviewed and adequately evaluated, (3) the limited breadth of literature cited and the misinterpreta- tion of some of the significant supporting literature, (4) the failure to ex- plain the criteria used to select assumptions when supporting literature is conflicting, (5) the failure to consider important risk pathways, and (6) the inadequate treatment of uncertainty. Those deficiencies are not equally problematic, but they occur with sufficient frequency to raise doubts about the adequacy and validity of the risk results presented. In most instances (e.g., operational activities at the NBAF), the identified problems lead to an underestimation of risk; in other instances (e.g., catastrophic natural hazards), the risks may be overestimated. As a result, the committee con- cludes that the uSSRA is technically inadequate in critical respects and is an insufficient basis on which to judge the risks associated with the proposed NBAF in Manhattan, Kansas.