Interim Report on Methodological Improvements to the Department of Homeland Security's Biological Agent Risk Analysis (2007)

In recognition of potential bioterrorist threats, President George W. Bush issued Homeland Security Presidential Directive 10 (HSPD10), “Biodefense for the 21st Century,”¹ on April 28, 2004. The directive requires assessments of the biological weapons threat to the nation:

The first Department of Homeland Security bioterrorism risk assessment was completed on January 31, 2006, and the report documenting the analysis was published on October 1, 2006.³ This assessment and report implemented the requirement of the National Strategy for Homeland Security,⁴ issued in July 2002 by the Office of Homeland Security, and of HSPD10 for DHS to assess the biological weapons threat in coordination with other appropriate federal departments and agencies. At DHS’s request, the National Research Council (NRC) established the Committee on Methodological Improvements to the Department of Homeland Security’s Biological Agent Risk Analysis to provide a review, via two reports (interim and final), of the methodology used in DHS’s report.

The committee’s first meeting was held at the National Academies’ Keck Center in Washington, D.C., on August 28-29, 2006. The appendix contains the agenda for that meeting. The committee heard and discussed presentations regarding risk analysis for biological pathogens by representatives of DHS, its National Biodefense Analysis and Countermeasures Center (NBACC), Battelle Memorial Institute, the White House Homeland Security Council, and the Homeland Security Center for Risk and Economic Analysis of Terrorism Events (CREATE). The recommendations and discussion below are based solely on those briefings; DHS’s bioterrorism risk assessment was not made available to the committee in time for this interim report; however, the committee believes that these briefings included sufficient detail to adequately present the methodology used in the risk analysis.

NBACC has contracted with Battelle to produce a computational engine that assesses the “normalized risk” of 28 pathogens as that risk relates to death, morbidity, and direct economic costs.⁵ In federal Fiscal Year 2007 (FY2007), DHS intends to improve and refine its probabilistic risk assessment (PRA). The committee has been asked to recommend possible directions for improvement, as well as to comment on the technical aspects of DHS’s technique and the broader suitability of PRA. These

comments are intended to provide guidance to DHS for its work during FY2007. Specifically, the committee has been given the following charge for this interim report:

• Assess the adequacy of the DHS’s current methodology as a foundation for the desired risk analysis capabilities;

• Identify any other risk analyses that rely on the major components of the existing methodology, probabilistic risk analysis and multi-attribute risk analysis, and which could guide DHS’s future developments;

• Assess the feasibility of incorporating models of second-order economic effects into the methodology during FY07;

• Identify better methods, if any, for handling the high degrees of uncertainty associated with the risk analyses of biological agents;

• Recommend near-term improvements to enhance the transparency of the method and its usefulness to decision makers;

• Discuss how the methodology could be extended to risks associated with classes of agents, including enhanced or engineered agents that have yet to be developed; and

• Discuss the feasibility of extending the methodology to also serve as a framework for risk analysis of chemical or radioactive threats.

For this interim report, the committee was not able to address the last of these tasks—to examine risk analysis for chemical or radioactive threats—because the breadth of this task exceeds the information that could be provided during briefings to the committee in one meeting. That task, however, will be addressed in the committee’s final report.

The committee’s charge for its final report is as follows:

• Recommend how the methodology can incorporate changing probability distributions that reflect how various actors (e.g., terrorists, first responders, public health community) adjust their choices over time or in different contexts;

• Recommend further improvements to the consequence analysis component of the methodology, including its models of economic effects;

• Identify any emerging methods for handling large degrees of uncertainty (e.g., fuzzy logic, possibility analysis) that merit consideration for future incorporation;

• Recommend further improvements to the transparency and usability of the methodology;

• Discuss in more detail beyond the first report how the methodology could be extended to risks associated with classes of agents, including enhanced or engineered agents that have yet to be developed; and

• Discuss in more detail beyond the first report the feasibility of extending the methodology to also serve as a framework for risk analysis of chemical or radioactive threats.

This charge will require study of the issues addressed here in greater depth and with a view toward the longer term. The committee is confident, however, that the recommendations included in this interim report are appropriate and necessary in the near term. The committee’s recommendations that follow address the general goal of improving methodology. Each recommendation relates to multiple elements of the charge, as noted in the accompanying text.

This interim report frequently refers to “risk” and activities surrounding its manipulation. For purposes of clarity, several definitions are given:

• Risk—the potential for realization of unwanted, adverse consequences to human life, health, property, or the environment, computed as the product of the probability of an event and the consequence of that event.

• Risk analysis—the overall process that involves risk assessment, risk perception, risk communication, and risk management. The hazards to be analyzed (e.g., physical, chemical, nuclear, radiological, and biological agents) may result from natural events (e.g., earthquakes and hurricanes), technological events (e.g., chemical accidents), and human activity (e.g., design and operation of engineered systems or attack by terrorists).

• Risk assessment—the scientific process of identifying hazards and quantifying their potential adverse consequences (magnitude, spatial scale, duration, and intensity) and associated probabilities including the uncertainties surrounding these estimates. Risk assessment may include a description of the cause-and-effect links among hazards and the nature of the interdependencies, vulnerabilities, and consequences.

• Risk perception—beliefs held by individuals or organizations about the risks of a hazard. Risk perception is concerned with psychological and emotional factors, which have been shown to have an enormous impact on behavior. Risk perception can be influenced by personal knowledge, experience, and beliefs; it can be affected by changing perceptions of the threat, the vulnerabilities, and/or the consequences; it may be influenced by information about hazards, risk assessments, risk policies, and risk management decisions.

• Risk communication—the process used by risk analysts, decision makers, policy makers, and intelligent adversaries to provide data, information, and knowledge to change the risk perceptions of individuals and organizations and enable them to assess the risk differently than they otherwise might. Risk communication needs must be considered when developing strategies for managing risk; thus any risk analysis methodology must take into account how affected individuals perceive and understand risk.

• Risk management—the process of constructing and evaluating strategies for reducing losses from future hazards and dealing with the recovery process should a disaster occur. Risk management strategies include a combination of options, such as providing information (i.e., risk communication), economic incentives (e.g., subsidies, fines), insurance, compensation, regulations, and standards. These strategies enable individuals and private-sector or public-sector organizations to transfer, mitigate, or accept their perceived risks. Risk management strategies can be evaluated by undertaking cost-benefit analyses to determine the trade-off between the reduction of risk and the costs of undertaking such measures. In evaluating a risk management strategy, one needs to be concerned with the way resources are allocated (i.e., efficiency considerations) as well as with the impact of these measures on different stakeholders (i.e., distribution or equity considerations).

The model used for the DHS bioterrorism risk assessment is a computer-based tool used for assessing the relative risk of terrorist use of each of 28 specific pathogens, identified in other sources. The methodology described below is an instance of probabilistic risk assessment, which is particularly well adapted for low-frequency, high-potential-consequence events for which there is no database sufficient to assess risk using statistical analysis of historical data.

The PRA used by DHS divides the spectrum of possible attacks into a discrete set of scenarios, or sequences of events, and for each scenario it provides an estimate of the scenario’s probability of occurrence, consequences, and risk. Owing to the extremely large size of the sample space, Battelle sampled the events in the scenarios involving a particular pathogen, estimated the risk associated with that pathogen, and compared it with the risk of other pathogens in order to obtain risk relative to that of other pathogens.

Each scenario involves a chain of as many as 17 events, which can be partitioned into those characterizing the terrorist group’s motivations and goals; those involving its methods and ability to

acquire, produce, and transport the given bioagent; and those surrounding the attack and response to it. Each event is further given discrete characteristics. For instance, the event of target selection can be further decomposed into the selection of a large, open building; a small enclosure; a large, divided building; a large outdoor space; a water pathway; a food pathway; or a contact target such as a letter. The event tree⁶ generated thus has millions of scenarios, or paths through the tree, for which the probabilities and consequences must be explicitly or implicitly calculated.

For each scenario, a range of consequences—measured in terms of illnesses, fatalities, and economic losses—must be computed, with a probability distribution over the range. The “consequence engine” used for these computations consists of a series of equations whose variables are derived from the properties of the pathogen, the details of the scenario, and the hypothesized U.S. response to the terrorist event. DHS is developing improved means to estimate the first- and second-order economic effects (as discussed later in this report). In addition, it is developing systems dynamics models of the ways in which the scenarios might unfold. The committee will review this systems dynamics approach in its final report.

Even from this brief description, it can be seen that the DHS model requires a large amount of information, much of which is uncertain. This information includes the known properties of the pathogens, estimates of the propensities of terrorists to take different actions, and estimates of the reactions of the affected population and of the timeliness and effectiveness of the government response. With the exception of known scientific information, the parameters are either estimated from historical experience or elicited from experts, often in the form of probability distributions.

For the most part, the analysis described in the previous section follows approaches considered technically sound and useful in other areas of risk analysis such as nuclear reactor safety and chemical safety. In validation of risk, PRA avoids many of the practical problems and difficulties that arise from other alternative methods such as fuzzy logic, the analytic hierarchy process, or worst-case analysis (Banks and Anderson, 2006; Laviolette et al., 1995).

Event-tree analysis, which is the basis of PRA, is a well-developed risk tool in nuclear reactor safety and many other, usually engineering, contexts (Lindley and Singpurwalla, 1986). The main concern of the committee is that the current PRA event-tree paradigm does not fully support any of the components of risk analysis. It does not include consideration of the actions of an intelligent and reactive adversary, which is required for a complete risk analysis. It makes no provision for risk perception. It does not allow the exploration by decision makers of “what-if” questions, which is needed for risk management.⁷ DHS needs to provide analyses for a variety of purposes to a variety of customers, and all within the context of competing security demands in the short run, while taking into account the longer-run concerns that may change over time. Therefore, a necessary first step is to clarify the longer-term goals and objectives of bioterrorism risk analysis.

In order to justify the current methodology as a foundation for future analyses, a clear statement of the long-term purposes of the bioterrorism risk analysis is needed to enunciate how it will support risk assessment, risk perception, and especially risk management decision making. Criteria and measures should be specified for measuring how well these purposes are achieved. Key issues to be addressed by such a statement should include the following: who the key stakeholders are; what their short- and long-term values, goals, and objectives are; how these values, goals, and objectives change over time; how the stakeholders perceive the risks; how they can communicate these risks more effectively; what they need from the risk assessment in order to make better (more effective, confident, rational, and defensible) resource-allocation decisions; and who should perform the analyses (contractors, government, both). Another important operational consideration is the determination of how DHS should incorporate new information in its analyses. The pace of change in biotechnology will require frequent and systematic updates of information used by the model. DHS issues “tailored assessments” to respond to unscheduled requirements, in addition to its biennial report, and it must be able to incorporate new intelligence information or technological change, for instance, in these analyses.

DHS’s purposes for its bioterrorism risk assessment must be supported by its customers, by the U.S. Congress, and by the scientific community, among others; thus, DHS should actively solicit the opinions of its stakeholders to ensure that communication on issues of risk analysis is two-way. To that end, the language and analyses used must be precise. The technical presentations given to the committee suggest that the model documentation does not always use standard and consistent terminology. For example, several speakers at the committee’s first meeting used the term “relative risk” to refer to what should be called “normalized risk,” and “likelihood” was sometimes used as a synonym for “probability.” The terms “risk,” “expected risk,” and “expected consequences” were often casually interchanged, and the computation of “normalized risk” was flawed.⁸ The terms “illness” and “morbidity” should be clarified and defined more precisely (i.e., illness would need to be defined as either “infected” or “symptomatic”).

Other terms used in the presentations to the committee were not precisely defined, and functional notation was confusing. DHS should define and use a standard lexicon, clarify concepts, and align with contemporary literature in order to improve the transparency of its models and results. DHS’s operational definition of “risk” should be refined to include time explicitly—for example, by indicating how many events with various degrees of severity of adverse consequences can be expected over what time intervals if different risk management interventions are implemented. Attention also needs to be given to the uncertainty and ambiguity associated with these risks. Use of outside peer reviews may help in this regard. The issues raised here are not minor concerns; this lack of precision can lead to internal inconsistencies in the model and to communication problems at all levels.

DHS’s risk assessment currently encompasses what are mainly traditional bioagents. However, it seems logical that the DHS vision for risk analysis should be broad enough to include risks posed by other significant future biological threats. Traditional bioagents are “naturally occurring microorganisms or toxin products with the potential to be weaponized and disseminated to cause mass casualties.”.⁹ Testing the methodology by using existing biological agent threat lists, as has been done to date, is a prudent and logical way to start, given the very large number of pathogens that could possibly be used as weapons. Existing threat lists (e.g., from the Centers for Disease Control and Prevention¹⁰) reflect

extensive experience and the judgment of the intelligence and scientific communities. However, many bioterrorism experts would agree that the “logic behind biowarfare programs of the past will not necessarily guide the life sciences as new technology rapidly emerges; biowarfare programs of the past predated current knowledge of molecular biology” (Relman, 2006, pp. 113-115). Therefore, future iterations of the methodology should also consider enhanced, emerging, and advanced agents in addition to traditional bioagents:

• Enhanced agents are those that are modified to circumvent current countermeasures—for example, microorganisms that are purposefully manipulated to be resistant to multiple antibiotics, thus complicating a public health response in the aftermath of an attack.

• Emerging agents are those that occur naturally but are newly recognized or anticipated to pose a public health threat—for example, a highly lethal and readily transmissible influenza strain that may cause a pandemic.

• Advanced agents are novel microorganisms that may be created by employing laboratory methods.

The results of such an extended risk assessment would be useful in determining the appropriate allocation of resources to develop flexible defenses—those that may be useful against a wide range of microorganisms that may share common processes in causing disease. Such an assessment would require information that is not currently available—estimates of likely developments in biotechnology that would enable new capabilities that could be used by terrorists. The committee believes that, for the near term, the elicitation of expert opinion, similar to what was undertaken in DHS’s assessment of traditional bioagents, would be a useful starting point. This could be the first step in establishing the risk imposed by agents not yet in the environment and in broadening the analysis to include classes of agents rather than individual agents. The committee will examine this difficult problem in more depth in its final report.

Event trees were not originally developed to model intelligent adversaries who adapt their attacks in response to (or in anticipation of) their opponents’ defensive actions and/or in response to their own initial successes or failures in plan execution. Alternative risk analysis techniques, including attack-preference, decision-tree, attack-tree, or attack-graph models,¹¹ can complement or replace probability elicitation. There have been recent advances in dealing with interdependent and coordinated adversary actions, called interdependent security (Heal and Kunreuther, 2005), which may improve the fidelity of DHS models.

To use a PRA event-tree risk assessment in the analysis of intelligent adversaries, the tree must include all realistic threats that adversaries may pursue. The committee believes that the DHS PRA tree is reasonably complete, although DHS should examine this further in light of the expectation that adversaries will adapt to any defensive decisions made by the United States. A small number of well-chosen red teams (i.e., individuals including both technologists and those with experience in targeting and strategy, whose purpose is to simulate adversarial decision making) to provide input for “what-if” scenarios can help to confirm and expand the current state of understanding and model validation and can complement expert opinion.

The probabilities in the event tree must be of sufficient quality to produce trustworthy results. Most of the event probabilities have been generated using expert opinion. DHS is keenly aware that this approach may be unreliable, and the committee is pleased that DHS intends to use CREATE’s expertise to improve elicitation of the views of subject-matter experts. But the reliability of these probability assessments will always be problematic, requiring careful attention to the elicitation methods as well as needing well-designed sensitivity analyses (Kahneman and Tversky, 2000; Meyer and Booker, 2001). Moreover, strictly probabilistic analysis should also be supplemented with other methods, such as attack-preference models and attack-tree models, in order to ascertain any severe contradictions in the resulting risk management (or mitigation) recommendations.

The Mission Oriented Risk and Design Analysis (MORDA) model, used in several Department of Defense risk assessment studies, is an example of the use of subject-matter expert teams from various disciplines to collect data and incorporate expert knowledge about adversaries. The MORDA model uses this collected information in adversary models and attack-tree models (Buckshaw et al., 2005).

In order to better understand the sources of uncertainty and to plan for their reduction, any analysis resulting from the PRA model should include a data-quality matrix with a qualitative assessment of the sources and quality of the data and perhaps quantitative indications of the confidence and precision associated with current estimates (e.g., plausible range of values for model inputs) for the 28 bioagents and the 17 steps in the event tree developed by Battelle.

The committee believes that static probabilities, as they are currently used by DHS, are insufficient to model the behavior of intelligent adversaries. Static probabilities may be appropriate when dealing with nuclear reactors, but not for an intelligent adversary who adapts an attack on the basis of the actions of the defenders and on information that it acquires as planning and execution progress. Although classical game theory is a formal way to handle such situations, there is now a growing literature that may be more relevant for dealing with the adversarial nature of the bioterrorism problem (Bier et al., 2005; Enders and Sandler, 2006; Heal and Kunreuther, 2005). Studies have been conducted by the Navy Postgraduate School in which the defender computed a strategy that would minimize the maximum damage that could be caused by an attacker (terrorist) who was aware of that strategy. These “attacker-defender” studies, which have been undertaken in various contexts to determine how best to protect U.S. infrastructure, might serve to complement the static probability analyses currently used by DHS (Brown et al., in press).

Any analysis of adversarial actions, as well as of mitigation strategies and responses, will require accurate estimates of the real damages that the United States would experience. Currently, the PRA computes measures of mortality, morbidity, and direct economic costs. But indirect economic costs (e.g., of business interruption) must also be included to avoid underestimating true financial consequences. If these indirect costs are large, it may be necessary to evaluate their impact, taking into account risk aversion and/or loss aversion.¹²

Evaluation of these costs will require that DHS more carefully consider its consequence measures and modeling, which should be augmented to include indirect economic effects. DHS is planning to use input-output models and CREATE-developed general equilibrium models to improve its estimates of the direct economic consequences of terrorist events in its FY08 risk assessment. Both of these techniques can be used to estimate the indirect costs. The committee agrees that their use is appropriate for the next stage of model development.

DHS is planning, however, to pursue consequence modeling that is of higher fidelity and resolution than that of the modeling being used now. Such a path is not clearly justified by either data availability or currently articulated decision needs. More fine-grained and detailed consequence models of targets should only be pursued if such granularity directly supports improved risk management decision making.

The committee is concerned about the use of too fine a granularity in the simulation. It could result in false precision that might be mistaken for accuracy in a model that is, by necessity, not particularly well validated, affecting both risk assessment and risk management. In addition, too fine a granularity decreases the transparency of the model. The committee is concerned that merely increasing the number of parameters that need to be elicited may not increase the real or useful precision of the model.

Individuals’ perceptions of risks can have a major influence on indirect economic consequences, resulting in a need to develop strategies to manage risk perception and to deal with these perceptions. DHS should consider decision-analytic methods for dealing with issues such as attitudes toward probabilities and consequences (the components of risk), the role of affect and emotion, biases in judgment, and the types of rules used by individuals and groups in choosing between alternatives.

Risk managers should be able to explore the impact of different investment strategies on the effects they might have on the attacker. Typical trade-offs facing U.S. risk managers might involve allocating resources among human intelligence versus vaccine development or deployment of biohazard sensors. A given resource allocation may drive a corresponding set of decisions by potential terrorists, which in turn changes risks. The current DHS event-tree PRA is not adequate for such risk management purposes. This is so because the event-tree PRA cannot determine which portfolio of investments is most effective and how potential attackers are likely to respond, although it does provide value in giving a coarse look at relative risks. This inadequacy highlights the importance of improving the current risk analysis with red teaming, attack-preference models, attack-tree models, and perhaps, game-theoretic analyses or alternatives. All of these techniques will serve to mitigate the high degree of uncertainty associated with the risk analysis of biological agents.

It is unclear to the committee how the current PRA approach supports DHS’s design and evaluation of alternative risk management strategies. The computational engine does not permit, let alone encourage, risk managers to explore scenarios of “if resource allocation, then probable consequence.” DHS needs to determine how alternative risk management strategies, involving specific resource investments in attack prevention, consequence mitigation, or other forms of protection, translate to changes in the overall level of risk. An interface and visualization component is needed to display results and limitations of this very complex model and to improve transparency.

In evaluating alternative risk management strategies, DHS should take into account all significant benefits that result from any strategy, beyond just those benefits that directly impact the risks of bioterrorism attacks. For instance, investment in intelligence might include all homeland security risks, and the risk management trade-offs should be considered in that larger context. This last conclusion has ramifications for all of DHS’s risk analysis and directly addresses the committee’s final charge. It will be more fully explored in this study’s final report.

DHS should develop a targeted research program to develop risk analysis methods that take into account the decision maker’s risk perception and risk management strategies. Such a program would include the following, for example: consideration of how constraints on resources available to the decision maker might affect terrorist decisions, and an understanding of how attackers who encounter failures or setbacks in executing an initial plan will respond—including the realistic possibility that they will implement contingency plans or adaptively replan to achieve goals that still appear feasible and worthwhile.¹³ Methods for modeling multiple coordinated attacks by teams of adversaries should also be considered.¹⁴ These changes should all be incorporated into the next generation of DHS’s bioterrorism risk assessment and management technologies. The committee believes that these extensions can be achieved by expanding the models rather than by increasing the fidelity of existing models.

As previously noted, each of the committee’s recommendations relates to multiple elements of its charge. Here, responses to each element of the charge, in order, are summarized.

• DHS’s current methodology is adequate but incomplete. A statement of purpose is needed, as well as methods to handle intelligent adversaries. Red teaming, attack-preference models, attack-tree models, and game-theoretic analyses should all be examined for the purpose of supplementing the existing methodology.

• The analyses cited, by Buckshaw et al. (2005) and by Brown et al. (in press), are examples of other types of risk analysis that would be appropriate for DHS’s future development.

• DHS’s current plans for the incorporation of second-order indirect economic effects into its methodology are appropriate, as long as the model’s level of granularity is carefully considered.

• High degrees of uncertainty can be addressed by the incorporation of red teaming, attack-preference models, attack-tree models, and game-theoretic analyses. The incorporation of data-quality matrices in DHS’s analyses will lead to a better understanding of the sources of uncertainty.

• In order to improve transparency, DHS should define and use a standard lexicon, clarify concepts, and align with the contemporary literature.

• In order to extend the methodology to risks associated with classes of agents, careful elicitation of expert opinion is the best starting point. This issue will be further examined in the committee’s final report.

• No examination was made in this interim report of the feasibility of extending the methodology to serve as a framework for risk analysis of chemical or radioactive threats.

Banks, D., and S. Anderson. 2006. “Combining Game Theory and Risk Analysis in Counterterrorism: A Smallpox Example.” Pp. 9-12 in A. Wilson, G. Wilson, and D. Olwell, eds., Statistical Methods in Counterterrorism. New York: Springer.

Bier, Vicki, Santiago Oliveros, and Larry Samuelson. 2005. “Choosing What to Protect: Strategic Defense Allocation Against an Unknown Attacker.” University of Wisconsin Working Paper.

Brown, G., W. Matthew Carlyle, Javier Salmeron, and Kevin Wood. In press. “Defending Critical Infrastructure.” Interfaces.

Buckshaw, Donald L., Gregory S. Parnell, Willard L. Unkenhotz, Donald L. Parks, James M. Wallner, and O. Sami Saydjari. 2005. “Mission Oriented Risk and Design Analysis of Critical Information Systems.” Military Operations Research 10(2): 19-38.

Enders, Walter, and Todd Sandler. 2006. The Political Economy of Terrorism. Cambridge: Cambridge University Press.

Heal, Geoffrey, and Howard Kunreuther. 2005. “You Only Die Once: Interdependent Security in an Uncertain World.” Pp. 35-36 in H.W. Richardson, P. Gordon, and J.E. Moore II, eds., The Economic Impacts of Terrorist Attacks. Cheltenham, U.K.: Edward Elgar.

Kahneman, Daniel, and Amos Tversky. 2000. Choices, Values and Frames. New York: Cambridge University Press.

Laviolette, Michael, John W. Seamon, Jr., J. Douglas Barrett, and William H. Woodall. 1995. “A Probabilistic and Statistical View of Fuzzy Methods.” Technometrics 37: 249-261.

Lindley, Dennis V., and Nozer D. Singpurwalla. 1986. “Reliability and Fault Tree Analysis Using Expert Opinions.” Journal of the American Statistical Association 81: 87-90.

Meyer, M.A., and J.M. Booker. 2001. Eliciting and Analyzing Expert Judgment: A Practical Guide. ASA-SIAM Series on Statistics and Applied Probability, Vol. 7. Philadelphia, Pa.: Society for Industrial and Applied Mathematics.

Relman, D.A. 2006. “Bioterrorism—Preparing to Fight the Next War.” New England Journal of Medicine 354: 113-115.