Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 20
3 Description and Analysis of the Department of Homeland Security’s Biological Threat Risk Assessment of 2006 [T]he United States requires a continuous, formal process for conducting routine capabilities assess- ments to guide prioritization of our on-going investments in biodefense-related research, develop- ment, planning, and preparedness. —Homeland Security Presidential Directive 10: Biodefense for the 2st Century, 2004 The Department of Homeland Security’s (DHS’s) sys- DHS credits seminal work on nuclear reactor safety as tem for Biological Threat Risk Assessment (BTRA) is a the basis for its risk assessment, citing “NUREG-1150” computer-based tool that has been applied by DHS to assess (case studies of probabilistic risk assessment) (U.S. Nuclear the risk associated with the intentional release of each of Regulatory Commission, 1991) and “NUREG-1489” (a tuto- the 28 biological agents listed in Figure 3.1. The methodol- rial on probabilistic risk assessment) (U.S. Nuclear Regula- ogy, an instance of probabilistic risk assessment (PRA), is tory Commission, 1994) as basic references. The committee described in Bioterrorism Risk Assessment, a report from also found valuable an earlier foundation work, “NUREG the DHS Biological Threat Characterization Center of the 75/014” (U.S. Nuclear Regulatory Commission, 1975), National Biodefense Analysis and Countermeasures Center widely known as the Rasmussen Report, which establishes (DHS, 2006). the theoretical and policy foundations on which the 1991 and FIGURE 3.1 Biological threat agents as categorized by the Centers for Disease Control and Prevention (CDC). High-priority, Category A agents include organisms that pose a risk to national security because they can be easily disseminated or transmitted from person to per- son, they result in high mortality rates and have the potential for major public health impacts, they might cause social disruption, and they require special action for public R01268, Figure 3-1 B, the second-highest priority, includes agents that are moderately easy to health preparedness. Category disseminate, that result in moderate morbidity rates not low mortality rates, and that require specific enhancements of CDC’s diagnostic Fixed image, and changeable capacity and enhanced disease surveillance. Category C agents include emerging pathogens that could be engineered for mass dissemination in the future because of availability, ease of production and dissemination, and potential for high morbidity and mortality rates and for major health impact. A later CDC-categorized list (CDC, 2007) features the same categories, but with agent entries revised. SOURCE: Available at www.bt.cdc.gov/Agent/Agentlist.asp. 20

OCR for page 20
2 DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 200 FIGURE 3.2 Ranking the risk of bioagents—the principal product of the Biological Threat Risk Assessment (BTRA) of 2006. In this figure, biological agents versus normalized risk, a sample display is based on fictitious data that represents only the general appearance of a key BTRA result. One of the vertical bars in this sample display represents anthrax; the dot shows the mean expected fatalities, and the horizontal bars show the 5th and 95th percentiles. However, as is done in the analyses included in DHS (2006), the vertical scale has been normalized so that the sum of the mean risks over all agents is 1. The committee does not know the normalization constant applied by BTRA and so R01268, Figure 3-2 cannot recover the actual expected risks. Fixed image, not changeable 1994 U.S. Nuclear Regulatory Commission reports and later For each agent, the estimate of the 5th percentile and applications depend. of the 95th percentile of expected fatalities is displayed The principal product of the BTRA of 2006 was a ranking as a tick mark on a vertical line on a logarithmic ordinate of the risk posed by bioagent use based on calculated prob- scale of (normalized) consequences. The mean of expected abilities of expected fatalities. DHS chose to assess threat fatalities is displayed as a dot. A typical display shows by ranking bioagents because government stakeholders had 28 parallel vertical lines, one for each agent. The specific advised DHS that they “expected the primary assessments to numbers and rankings of agents by risk are functions of be in the form of risk-prioritized groups of biological threat the assumptions underlying each of the many steps in the agents” (DHS, 2006, Ch. 1). Although a terrorist’s choice model’s execution. of agent is just one step in a sequence of events leading to Before results are presented in DHS (2006), a normal- a potential attack, for practical purposes the BTRA of 2006 izing constant is computed by multiplying, for each agent, evaluates each agent separately. A probability is computed the conditional expected consequence of the agent’s use by for each scenario involving that agent. Risk is then calcu- the probability of its use, and then summing over all the lated as the product of these probabilities and the associated agents. All statistics are divided by this constant to force consequences. The overall risk associated with each agent the normalized means to sum to 1. This critical normaliza- is the integrated risk distribution over all possible scenarios tion constant is not displayed in the DHS (2006) report, so involving that agent. no absolute (versus relative) consequence can be recovered The product of the analysis by the BTRA of 2006 is from the analysis presented there. Therefore, the normal- displayed in a figure (such as Figure 3.2) that shows, for ization method cannot be verified by the committee. The each agent, a normalization (whose normalization constant normalization step is a curious one, in that it damages the is not defined) of three estimated parameters of the distribu- results irreparably for purposes of decision making about, for tion of consequences of agent attack in terms of expected instance, risk management. The committee conjectures that fatalities:1 the normalization may reflect a well-intentioned but nonethe- less an unfortunate effort to mitigate the stark nature of the • The 5th percentile, estimated risks reported. • The expected value (or mean), and DHS (2006) also contains some qualitative analysis • The 95th percentile. distinguishing between most-, less-, and least-“worrisome” bioagents. As for the quantitative analyses, consequences in- 1The analyses presented in DHS (2006) are based entirely on estimated clude only immediate numbers of expected fatalities. Future fatalities. However, DHS has conducted assessments based on illnesses and assessments have been promised with estimated casualties direct economic consequences as well.

OCR for page 20
22 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT Dissemination Agent Release Mitigation Agent Disease Efficiency Modeling Response Mass Spread RISK Target Selection Scenario AGENT Agent Dispersion Mitigation Frequency Consequences RELEASE Production of Initiation Bioagent Selection Scenario Agent Probability Risk Ranking Selection Selection Selection Event Initiation Probability Probability Probability Detection Frequency Event- Tree Quantification FIGURE 3.3 Biological Threat Risk Assessment (BTRA) event-tree risk assessment (left-to-right sequence) and consequence evaluation (at the right) are loosely coupled components. SOURCE: Tracy Hale, Battelle Memorial Institute, “2008 DHS Bioterrorism Risk Assessment: Planned Improvements,” presented to this committee on February 10, 2007, Washington, D.C. R01268, Figure 3-3 and indirect economic consequences. The committee does 2006 is presented in this chapter using its own technical not know whether such estimates will also be normalized, lexicon (Appendix A), which cross-references the terms used but it hopes not. by the committee and those used in the DHS lexicon, when relevant. Readers interested more in the policy implications and potential uses of BTRA than in the technical details DETAILS OF THE MODEL USED TO PRODUCE might want to skim the text of this chapter and read the four THE DEPARTMENT OF HOMELAND recommendations interspersed in the text below. SECURITY’S BTRA OF 2006 The process that produced the estimates in the BTRA of The BTRA of 2006 Uses a Probabilistic 2006 consists of two loosely coupled analyses: (1) a PRA Risk Assessment Event Tree event-tree evaluation and (2) a consequence analysis (Fig- ure 3.3). DHS has conducted “Material Threat Assessments” A PRA event tree represents a sequence of random vari- for single bioagents. “These are plausible, high consequence ables, called events, or nodes. Each random-event branching scenarios used to estimate the potential number of exposed node is followed by the possible random-variable realiza- individuals, their exposure levels, contaminated areas, and tions, called outcomes, or arcs, with each arc leading from other collateral effects.”2 Presumably, the results of these the branching, predecessor node, to the next, successor-event assessments were used to inform the BTRA of 2006, but node (and it can be said without ambiguity that the predeces- the committee was not briefed on them. DHS (2006) does sor event selects this outcome, or, equivalently, selects the not contain mathematical definitions of all of the parameters successor event). With the exception of the first event, or root and variables used in the BTRA and does not present a node, each event is connected by exactly one outcome of a complete mathematical model. (A complete mathematical preceding event. A node with no successor event is called a model would show how each input is used to produce each final event, or leaf. From each event, it is possible to trace a output.) In response to the committee’s request for that unique path back through alternating predecessor outcomes information, DHS has developed a lexicon and a mathemati- and events to the root event. The path from the root to a cal model. Informed by discussions with DHS analysts, the particular leaf is called a scenario. Each successive random committee’s understanding of the details of the BTRA of event in a scenario path has a probability depending on all preceding outcomes in the path, and the probability of this scenario is the joint probability of the intersection of the 2 John Vitko, Jr., Director, Chemical and Biological Division, DHS, Sci- outcomes on the path and is the product of these outcome ence and Technology Directorate, briefing to the BioShield Stakeholders probabilities. A natural way to construct an event tree is to Workshop, December 26, 2006.

OCR for page 20
2 DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 200 place events in the chronological order in which they occur, in column 3. The number of possible outcomes for each event if this order is known (e.g., Paté-Cornell, 1984). in a stage is shown in column 4. The maximum cumula- This committee’s concise mathematical definition of the tive number of paths into each stage is shown in column 5. BTRA event tree and associated computations are given in Because outcome probabilities are conditional upon some Appendix B. preceding outcomes, column 6 shows the maximum number Figure 3.3 shows some of the events in the BTRA tree. of such dependencies—this helps convey the complexity The “Frequency of Initiation” box at the extreme left consists and sheer number of probabilities that must be reckoned for of only one event—the beginning of a terrorist attack, which the BTRA. includes the terrorist’s choice of frequency of attack, a ran- In practice in the BTRA, the event tree is not actually dom variable with four possible outcomes. Each frequency evaluated as shown in Figure 3.4; each of the 28 agents selected leads to a new event in the “Target Selection” box, (outcomes of events in Stage 3) is analyzed in isolation, as shown in Figure 3.3, and each of these four events is a yielding 28 sets of, in theory, as many as 350 million paths random variable with eight possible outcomes, leading to a based on as few as 5,448 distinct probabilities for each agent. total of 32 events in the “Bioagent Selection” box. Each of Although the maximum number of possible scenario paths these events is a random variable with 28 possible outcomes, is large (i.e., exponential in problem size), agent-by-agent, depending on which of the 28 agents is used. Although not the event tree has many paths terminated early with no at- shown in Figure 3.3, there is a sequence of 17 such boxes in tack (e.g., by failure to manufacture an agent, by successful the BTRA event tree, enumerated and named in Figure 3.4, interdiction, and so on), while others continue to completion. with each box corresponding to a different stage in the Among the 28 event trees, each corresponding to the selec- chronology of a terrorist attack. A complete listing of all tion of a different agent, DHS (2006) reports one agent with the possible outcomes for these random variables is given only 1,184 scenarios, and another, the largest agent tree, with in the BTRA documentation but not in this report of the 192,928 scenarios. committee. In the remainder of this chapter, the committee The individual agent results are merged a posteriori into a uses the term “stage” to mean all of the possible events at distribution using probabilities for the selection of each agent each step. As can be inferred from the names given to the and target. With the exception of this separation of event stages (see Figure 3.4), each corresponds either to a terror- trees by agent, BTRA treats each of these successive events ist decision (e.g., Bioagent Selection), or to a U.S. decision in ascending order of the stage in which it occurs. (e.g., Mitigation). It is a fundamental property of the BTRA For example, Figure 3.5 shows the outcomes for each of 2006 that every event, whether representing a terrorist event in Stage 2. After the frequency of attack has been decision or a U.S. decision, has a probability of occurrence chosen, the terrorist can choose among eight types of tar- associated with it. get to pursue. The BTRA represents the selection of each As indicated above, Figure 3.4 displays the succession such outcome as an arc chosen randomly, with a selection of 17 stages of the BTRA event tree. The BTRA represents probability that may depend on outcomes of events in prior epistemic uncertainty (uncertainty due to incomplete knowl- stages. In this example, the outcome probabilities from edge) by using a distribution of event probabilities from events in Stage 2 may depend on the outcomes chosen for which a particular probability is sampled; that is, adopting prior events in Stage 1. the convention that from a node, each branching outcome The BTRA analyzes each of the 28 agents as follows: “selects” a successor event, each such event leading to an outcome has a probability distribution over its probability 1. The selection probability of the agent under study is set of selection. For events in all but the first stage, each event to 1 for each event in Stage 3. All other probabilities leading to an outcome is chosen with a probability drawn for events in Stage 3 are set to 0. (Stage 3 consists of from a distribution of probabilities for that outcome. The agent-selection events; there are 32 events that result in selection of outcomes from the only event in the first stage, agent-selection outcomes.) It is important to note that “Frequency of Initiation by Terrorist Group,” is the rate at no attack using multiple agents is considered. which terrorists are anticipated to make attempts during a 2. The tree for this agent is Monte Carlo generated, with time horizon over which this rate applies; each such rate and outcome probability distributions conditioned upon time horizon has an associated probability.3 outcomes from events in Stages 1 and 2 as well as An 18th stage has been added to Figure 3.4 by the com- on the knowledge of which agent is being modeled. mittee to represent the “Consequences” random variable. If BTRA represents epistemic uncertainty by using a the probability of an outcome depends on outcomes from an distribution of outcome probabilities from which a event in a preceding stage, the prior stage number is shown particular probability is sampled. These epistemic probability distributions over outcome probabilities are elicited from subject-matter experts for each indi- 3 Given that the number of opportunities for such attempts is huge and vidual possible outcome, although there are thousands the probability that any particular opportunity will be pursued is tiny, this of such conditional outcomes. is a Poisson rate.

OCR for page 20
2 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT Depends Maximum Number Maximum Stage on Cumulative Event Type of Possible Number of Phase No. Stage Number of Paths Outcomes Dependencies No. into Stage Frequency of 1 Initiation by 4 4 4 Terrorist Group 2 Target Selection 1 8 32 32 Agent/Target/Dissemination 3 Bioagent Selection 2 28 896 224 Selection Mode of Dissemination 4 (also determines 1, 2, 3 9 8,064 8,064 wet or dry dispersal form) Mode of Agent 5 3 4 32,256 112 Acquisition Acquisition Interdiction during (6) 1, 3, 5 2 64,512 896 Acquisition Location of 7 Production and 1 2 129,024 8 Processing Mode of Agent 8 1, 3 3 387,072 336 Production Preprocessing and 1, 2, 3, 9 3 1,161,216 72,576 Concentration 4, 8 Production and Processing Drying and 10 1, 2, 3, 4 3 3,483,648 24,192 Processing 11 Additives 1, 2, 3, 4 2 6,967,296 16,128 Interdiction (12) During Production 2 13,934,592 2 and Processing Mode of Transport 13 1, 2, 3, 4 3 41,803,776 24,192 and Storage Transport and Storage Interdiction (14) During Transport 7 2 83,607,552 4 and Storage Interdiction (15) 2 167,215,104 2 During Attack Attack Potential for 16 1 2 334,430,208 8 Multiple Attacks (17) Event Detection 2, 3, 4 3 1,003,290,624 6,048 Response 18 Consequences tbd 10 10,032,906,240 tbd Final Outcome FIGURE 3.4 Successive stages in the Biological Threat Risk Assessment (BTRA) event tree. A BTRA event tree consists of 17 stages clas- sified into six successive phases. The committee has emphasized Stages 6, 12, 14, 15, and 17 by inserting parentheses around these stage numbers in the left-hand column, to distinguish interdiction opportunities. Outcomes of events in all other stages are chosen by the bioter- rorist. The committee added the columns labeled “Number of Possible Outcomes,” “Maximum Cumulative Number of Paths into Stage,” and “Maximum Number of Dependencies,” as well as an 18th stage representing “Consequences.” NOTE: tbd, to be determined. SOURCE: Adapted from DHS (2006, Table 5.1).

OCR for page 20
2 DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 200 Stage No. Event Type Possible Outcomes Depends on Stage No. 2.1 Large Open Building 2.2 Small Enclosure 2.3 Large “Divided” Building 2.4 Large Outdoor Spaces 2 Target Selection 1 2.5 Water Pathway 2.6 Food Pathway 2.7 Human Vectors 2.8 Contact (letters) FIGURE 3.5 Each event offers the terrorist one choice of a number of alternate outcomes. Here, Stage 2, “Target Selection,” is amplified into eight outcomes. The Biological Threat Risk Assessment represents the choice of each outcome with a probability and refers to this as a “split fraction” (i.e., conditional arc probability). The number at the right shows that the probability distribution on outcomes from events in Stage 2 is dependent on outcomes from events in Stage 1, “Frequency of Initiation by Terrorist Group.” SOURCE: Adapted from DHS (2006, Table 5.2). 3. A set of outcome probabilities is generated, and the a host of other considerations for subject-matter experts resulting probabilistic risk assessment event tree is to consider when rendering opinions about consequences. solved. That is, each leaf (terminal event) with nonzero Event by event, outcomes that support this scenario can be probability is associated with a consequence distribu- identified, although many nuances (e.g., steps to concentrate, tion, from which the leaf-probability-weighted conse- process, and introduce additives to “weaponize” the anthrax quence distributions are sampled to produce a sample spores for better dispersal) may be hard to unambiguously unconditional consequence distribution. The BTRA identify (i.e., the attackers have either weaponized a lot of an- does this 500 times, thus generating a random sample thrax, or they have produced an even larger quantity of crude of 500 PRA trees and associated consequence distri- anthrax to use). Regardless, the base mission of the BTRA is butions. For each of these trees, the resulting 5th and to automatically generate hosts of scenarios, including ones 95th percentiles and the average of the consequences that resemble the committee’s hypothetical scenario, and are computed. This sampling of multiple realizations rank them in terms of expected risk (i.e., fatalities). from the same starting conditions represents aleatory Three short papers (DHS, 2007a,b,c) presented to the uncertainty—the influence of pure randomness. committee give details on and contain versions of Figure 3.6. 4. The outcome of each random-sample scenario is cap- In this tree, the starting event is at the extreme left, followed tured by the distribution of expected consequences; the by two stages of events representing the terrorist choice of expectation is over purely aleatory randomness. agent and then choice of target. A complete scenario in this 5. The 28 agent statistics are merged, after the fact, using reduced example is characterized by a left-to-right scenario the agent-selection probabilities. path from starting event to final event and is documented by the successive outcomes, or arcs, in this scenario path. The committee’s hypothetical scenario, introduced in For instance, a path with arcs labeled “PA1, 1-PT1” leads to scenario s2 with consequence distribution c(x | s2), where x Chapter 1, may be approximately described by a number of possible sequences of outcomes in the BTRA event tree. represents fatalities. The notation “PA1” represents, at once, The type of terrorist group here would not likely be a de- the selection of Agent 1 and its probability of selection. Al- ranged individual or even a small cell, because the volume though not shown in Figure 3.6, each successive probability of anthrax hypothesized for this large-scale, outdoor aerosol could depend on everything that precedes it in its scenario attack exceeds that of the attacks following 9/11 by several path. So, in example scenario path “PA1, 1-PT1,” the prob- orders of magnitude, and thus the terrorists are evidently well ability PT1 can depend on the prior choice of event PA1. funded, perhaps even state-sponsored. Target selection (Fig- As noted above, a fundamental property of the event trees ure 3.5) would be a “Large Outdoor Space.” That this space used in the BTRA is that every decision by a bioattacker is hypothetically filled with commuters conditions the con- (e.g., choice of an agent) or by a defender (e.g., choice of sequences, but it is not clear where these commuters would a countermeasure) is considered to be an uncertain event— appear in the BTRA; they are evidently rolled up along with hence associated with an outcome selection probability. In

OCR for page 20
2 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT FIGURE 3.6 A simplified event tree for two successive stages (events), each with two alternate outcomes. The initiation frequency and succession of dependent probabilities in each scenario path lead to a “scenario frequency” that is multiplied by the consequence distribution at the leaf of that scenario. The consequence distribution c(x | si) is expressed , for scenario si in units of number of fatalities, although any convenient units would suf- fice. At the right, a distribution of consequences is shown for each scenario. At the lower right, the unconditional distribution is shown—the path-probability-weighted distribution of expected consequences for one sampled event tree. (The histogram collects outcomes in discrete “bins,” that is, intervals; for convenience, the consequences for all scenarios share the same 10 intervals, bounded above by integral powers of 10.) SOURCE: DHS (2007c, Figure 1). R01268, Figure 3-6 fact, the BTRA uses pure probability not changeable Fixed image, trees, and no decision probability solicitations is converted into a marginal prob- tree at all. A statement in a presentation to the committee: ability density of probabilities for selecting the particular “An event tree (decision tree) is a visual tool . . .”4 indicates outcome. Documentation indicates that most subject-matter confusion on this point. The distinction between event and experts for the BTRA of 2006 were experts from Battelle decision trees is fundamental, not semantic. In event trees, Memorial Institute, Columbus, Ohio, but that subsequent all outcomes are modeled as random events determined by work will draw from a much wider pool of experience. some probability distribution; decision trees allow the possi- Some observations by the committee about the details in bility that outcomes are chosen by the defender or attacker to these steps follow. In step 2 above, the Monte Carlo simu- achieve some objective. Decision trees as tools for modeling lation generates probabilities for each event one outcome terrorist threats are discussed in Chapter 7. (arc) at a time in some fixed sequence of outcomes. For In step 3 above, for each outcome from each event, the each successive outcome, a marginal probability distribu- probability of selection has been elicited as the consensus of tion over the probability for selecting this outcome is used. a group of subject-matter experts in the form of an expected The probability distribution for each successive outcome probability, and reportedly some additional guidance (such as is conditioned on the probabilities already realized for this the 5th and 95th percentile of this outcome probability) that event. Because the outcome probabilities must sum to 1, has been transformed by some unspecified means into a vari- the marginal distributions for each should be constrained ance for each probability. Each of these outcome selection to have their expectations sum to 1. Although the original marginal distributions are, for instance, beta densities, the successive conditioning by sampled outcomes means that 4 Richard S. Denning, Battelle Memorial Institute, “DHS 2006 Bioterror- outcomes are really sampled from some multivariate density ism Risk Assessment Methodology,” presentation to the committee, August for which the marginals are not beta, and in fact are not 28, 2006, Slide 8.

OCR for page 20
2 DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 200 characterized at all in closed form, and not mentioned at all The BTRA consequence analysis is qualitatively different in DHS (2006). from its event-tree analysis. Subject-matter expert opinions The DHS procedure selects the last outcome probability are developed much like case studies, and there is less clear so that the sum of outcome probabilities emanating from this dependence on specific events leading to each consequence. event is 1 (i.e., the last marginal probability distribution is Thus, each consequence distribution should be viewed as not used at all). However, the outcome probabilities should being dependent on eery event leading to its outcome. How- have a joint distribution that captures their dependencies (the ever, an examination of the underlying analysis in the DHS most important being that they sum to 1). Even if the present (2006) report suggests that there is really only a single con- method were not technically superfluous (as is shown below), sequence distribution for each scenario: one that depends not subject-matter experts typically cannot assess such high-di- on the complete scenario but only on a subset of parameter mensional distributions (Moskowitz and Sarin, 1983). values. (Indeed, “Consequence uncertainty was omitted due to the overwhelming processing requirements.”5) A Monte In step 3, the 500 sets of outcome probabilities for each agent event tree are obtained using a Latin Hypercube Sam- Carlo simulation of 1,000 samples was used to estimate each pling design (Stein, 1987), a sampling technique applied in consequence distribution in the BTRA of 2006. The commit- earlier years to probabilistic risk analysis of nuclear safety. tee has no details about how this was accomplished. However, the committee notes that this sampling design produces unbiased estimates of the mean and quantiles with THE EVENT TREE CAN BE IMPROVED asymptotic sample size. Further, see Stein (1987, p. 144, Equation (3) and Section 5) and McKay et al. (1979, Section The Approach to Determining the Probabilities 8.3). Moreover, the variance may be decreased or increased of Terrorist Decisions Is Incomplete by this design, depending on the covariance structure of the distributions sampled. Note that the proofs of unbiasedness The BTRA of 2006 uses probabilities to represent ad- for quantiles are for independent random variables. There is versarial decisions. These are conditional probabilities, but no evidence that the efficacy of the particular BTRA sample the conditioning is retrospective, rather than prospective. design has been established. Consider that if the consequence model for a bioagent is completely changed to reflect some new discovery about the efficacy of the bioagent, this would have no influence at The BTRA of 2006 Does Not Use Event all on the BTRA probabilities; neither the terrorist nor the Trees for Consequence Analysis United States would change probabilities in response. Consequence models characterize the probability dis- When dealing with an intelligent, goal-oriented, and tribution of consequences for each scenario. The BTRA resourceful adversary, not with a force such as nature that employs a mass-release model that assesses the production randomly determines whether unwanted events occur, this of each bioagent, beginning with time to grow and produce, committee believes that the use of probabilities to represent preprocess and concentrate, dry, store and transport, and bioterrorism decisions must be tempered by a thorough un- dispense. The net result is a biological agent dose that is derstanding of how these probabilities have been assessed input to a consequence model to assess casualties. One equa- (whether by means of formal game-theoretical models, tion from the model is produced here to give a flavor of the elicitation of subject-matter experts, of other means). For computations. decision problems as complex as those motivating BTRA, the assessment of the probabilities that adversaries will MR = MT × QF1 × QF2 × QF3 × QF4 × QF5 choose courses of action should be the outputs of analysis, not required input parameters. The BTRA has reversed this where MR is bioagent mass release, MT is target mass, and preferred approach by requiring that subject-matter experts QFi are factors to explain production, processing, storage, predict, a priori, how adversaries will behave. For this ap- and so on and are random variables conditioned on the sce- proach to make sense, the subject-matter experts must grasp nario whose consequences are being evaluated. nuances of alternatives and outcomes and render opinions The complete model computes, for an attack with a given founded on an analysis of the entire decision process, which agent on a given target, how much agent has been used, how would be very difficult for a process this complex. The com- efficiently it has been dispersed (and, for an infectious agent, mittee saw no evidence that this level of analysis was used. how far it spreads in the target population), and the potential Moreover, the static probabilities used are not appropriate effects of mitigation efforts. For the BTRA of 2006, all of when terrorists can observe and react dynamically to any these factors were assigned values by eliciting opinions of earlier decisions made by the United States. subject-matter experts in the form of subjective discrete probability distributions of likely outcomes, and by some 5Traci Hale, Battelle Memorial Institute, “2008 DHS Bioterrorism Risk application of information on the spread of infectious agent, Assessment: Planned Improvements,” presented to this committee on Febru- atmospheric dispersion, and so on. ary 10, 2007, Washington, D.C.

OCR for page 20
2 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT quences of multiple attacks. For l′ = 0.9, ƒ1(0.9 = 1.25, Recommendation: To assess the probabilities of terror- ƒ2(0.9) = 91, and the correct expectation ƒ3(0.9) = 10. For ist decisions, DHS should use elicitation techniques and decision-oriented models that explicitly recognize terror- this numerical example, the two expectations respectively ists as intelligent adversaries who observe U.S. defensive would underestimate and overestimate consequences by an preparations and seek to maximize the achievement of order of magnitude. their own objectives. If each repeated attack is independent, the distribution of total consequences across all attacks will presumably be It should be noted that this recommendation does not require additive. The distribution of this sum is characterized by a the prediction of terrorist actions, a difficult task at best. Its statistical conolution, not by mere multiplication by the intent is to evaluate risk on the basis of hypothetical terror- expected number of re-attacks. ist attacks against U.S. defenses that have been designed to It is not very realistic to assume an infinite supply of thwart terrorist goals. Thus, its implementation will produce potential attacks that all have equal probabilities of success. a conservative estimate of risk. In Chapter 7, the committee Judging from U.S. actions taken after the 9/11 terrorist at- offers alternate modeling techniques to accomplish this more tacks, the committee believes that all of the probabilities complex assessment. assessed in the event tree will change following any attack. Thus, the implicit, homogeneous steady-state Poisson process underlying the rate used for “Frequency of Initiation The Mathematics Used by the BTRA in by Terrorist Group” will almost surely be rendered invalid by Modeling Multiple Attacks Has Errors any detected attack, whether successful or not, and whether Given a successful attack, the PRA tree’s Stage 16, Po- interdicted or not. Subsequent to any such event, the BTRA tential for Multiple Attacks6 (see Figure 3.4) presents an analysis would be rendered inapplicable until a host of key opportunity for the terrorist to mount more such attacks. The parameters could be reestimated and the BTRA then repeated probability for succeeding at each additional attack is given from scratch. as l′ (implying that the attacks that are attempted first are The BTRA multiple-attack feature is an embellishment no more likely to succeed than those postponed until the first that has been incorrectly implemented both mathematically attempts have failed), and the expected number of attacks and statistically, and even if correctly implemented would before interdiction is given in the DHS (2006) report and be based on a questionable underlying model. presentation to the committee as λ′ The 2006 BTRA’s Assessment of Outcome f1 (λ ′) = 1 + . Probabilities Is Unnecessarily Complex (1 + λ ′)2 This expectation is multiplied by the consequence distribu- Each node in the PRA tree offers two or more outcomes tion for such attacks. leading to successor events, each selected with an epistemic During a site visit to Battelle in Columbus, Ohio, in Octo- probability density that is used to generate an aleatory out- ber 2006, the committee pointed out that this equation must come probability to be used to solve the event tree. Each of be in error (e.g., if l′ = 1, the expected number of re-attacks these outcome densities is typically a beta density function, should go to infinity, but ƒ1(l′ = 1) = 1.25). formed somehow from averages elicited from subject-matter Subsequent briefing materials. (Battelle Columbus Opera- experts, whose means sum to 1. It is straightforward to show tion, 2007) featured a new expectation: that, when given a distribution over outcome probabilities, the means of these distributions suffice to completely capture λ′ f2 (λ ') = 1 + . the unconditional distributions oer any consequence. For (1 - λ ′)2 example, suppose that This expectation is also wrong. Given one successful attack, xi = probability that outcome i will occur, i = 1,2, …, n the total number of successful attacks before an interdiction with probability of success for each additional attack l′ is → ξ = [x1,x2,…,xn] f3 (λ ′) = 1 + E[n | λ ′] = ƒi(x) = probability distribution over outcome X, given that outcome i occurs. λ′ ∞ 1 1 + ∑ n(λ ′)n (1 - λ ′) = 1 + = . 1 - λ′ 1 - λ′ If the values of xi are known, then the unconditional distribu- n= 0 Figure 3.7 shows these expressions as a function of λ′. tion over consequences X is This has a significant influence on the expected conse- n → f ( x | ξ ) = ∑ ξi fi ( x ). i =1 6 “Multiple attacks” refer to attacks in sequence. “Simultaneous multiple attacks” are considered by the BTRA to be a single attack.

OCR for page 20
2 DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 200 10 9 8 7 6 5 f f 2 (λ ′) f 3 (λ ′) 4 3 2 f 1 (λ ′) 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 λ′ FIGURE 3.7 Expected number of attacks before interdiction, given that a first attack is successful and that continued attacks each evade interdiction with probability λ′. ƒ3(λ′) is the expected number of attacks before interdiction. ƒ1 is the BTRA expression, and ƒ2 is the ex- pression offered with a complete numerical example (Battelle Columbus Operation, 2007). For λ′ = 0.9, ƒ1 underestimates by an order of magnitude, and ƒ2 overestimates by an order of magnitude. This expectation is multiplied by the single-attack distribution of consequences, so these errors have major influence. The value λ′ = .9 has been chosen by the committee for expository purposes. The committee does not represent that this value occurs in any scenario analyzed by DHS or in the example provided by Battelle. The interested reader my substitute any other value for λ′ to assess its effect. However, the epistemic approach considers xi to be a no analysis in the BTRA of 2006 and no improvement in → random variable, and therefore ξ a random vector. Letting analysis recommended by the committee can make mean- → ϕ(ξ ) be the (joint) probability distribution over the elements ingful use of the information available in the family of risk → xi of the random vector ξ , the unconditioned distribution over curves, beyond that provided by their expectation. Further, consequences becomes given the planned improvements to the BTRA incorporat- ing additional consequence measures and utility functions, → → → f ( x ) = ∫ f ( x | ξ )ϕ (ξ )d ξ = the committee does not anticipate analyses that require the family of risk curves. n n n → → → → ∫ ∑ ξ f ( x )ϕ (ξ )d ξ = ∑ f ( x ) ∫ ξ ϕ (ξ )d ξ = ∑ f ( x )E (ξ ) . If the conditional consequence distributions are given in ii i i i i parametric form, or in numerical lookup tables, calculation i =1 i =1 i =1 of the risk distribution can be done exactly, without resorting Therefore, even when using a general (and possibly highly to estimating these distributions from the outputs of Monte dependent) joint distribution, all that is needed is the expec- Carlo simulations. This computation is easy and fast, and tation E(xi), which is the mean of the epistemic distribution; the result is the distribution—not merely an estimate of its the rest of the distribution is irrelevant to determining the features. unconditional distribution of consequences (and, in particu- For these reasons, the committee’s finding is that the lar, its moments, percentiles, and so on). epistemic features of the BTRA probabilistic risk assess- Because of this, the consequence distribution can be ment are unnecessary and that they increase computation calculated without sampling from the outcome probability time and complicate exposition, analysis, and understanding distributions. Appendix C provides a self-contained, simpli- of results. fied example of this point. For an event tree the size of the one used in the BTRA of 2006, this represents a significant Recommendation: The event-tree probability elicitation computational simplification and would also significantly should be simplified by assessing probabilities instead simplify the BTRA exposition; both of these results are desir- of probability distributions for the outcomes of each able. What is lost in the simplification is the family of risk event. curves—i.e., one curve for each possible outcome. However,

OCR for page 20
0 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT BTRA Results Should Not Be Normalized in the fixed hierarchy of the BTRA event sequence, and that by an Unspecified Constant this selection depends on both prior stages, this approach has complicated the analysis and exposition of results. Fixed The absence of a normalization constant in DHS docu- adherence to the 17 sequential stages in the BTRA event mentation and presentation irretrievably obscures those tree leads to large PRA trees that have had to be separated BTRA results where normalization is employed, rendering by 28 individual agents. That the choice of agent is not a those results essentially useless for further analyses, espe- first-stage event, or even a second-stage one, but rather a cially for risk management. As an illustrative example, sup- third-stage event, causes some difficulty in recovering results pose that the United States discovers how to make a reliable after the fact. biological agent alarm the size and cost of a smoke detector and how to connect such detectors to local area networks; Recommendation: Two significant simplifications should educates the U.S. populace to shelter in place on alarm; be made to the BTRA of 2006 event tree: implements effective, immediate cordoning and quarantine • DHS should eliminate Stage 1, Frequency of Initia- procedures; and thus attains an estimated threefold reduction tion [of an attack] by Terrorist Group, and Stage 16, in expected consequences from terrorist use of all biological Potential for Multiple Attacks; and agents. This improed capability of detection and response • DHS should seek opportunities to aggregate some would not change a single normalized result presented in stages of the tree to only those essential to calcu- the BTRA of 200. late probabilities and consequences with realistic The committee wonders how senior leadership has in- fidelity. terpreted a normalized fatality scale (with no units) in the DHS (2006) report and presentation materials: the com- The elimination of probability elicitation for terrorist mittee does not know why this normalization was applied, decisions will greatly simplify the model. Additional sim- and especially why its essential details are absent from all plifications are also possible. For instance, Stages 7 through underlying documentation. The normalized results are clas- 11 (successively: Location of Production and Processing, sified, as would be the non-normalized results, and this one Mode of Agent Production, Preprocessing and Concentra- step—normalization—has made it impossible for anyone to tion, Drying and Processing, and Additives) appear to reflect reproduce any BTRA result or for anyone to use independent a somewhat artificial taxonomy and permutation of decisions means to assess the accuracy of any BTRA result. in a proliferation effort. Similarly Stages 14 and 15 (Interdic- Most important, risk management deals with risk, not tion During Transport and Storage, and Interdiction During normalized risk. The BTRA needs to report risk, not nor- Attack) might be aggregated. It should be noted, however, malized risk. that the level of detail shown in Figure 3.4 may coordinate with the steps that the FBI and the National Counterterror- Recommendation: Normalization of BTRA risk assess- ism Center consider in evaluating an attack, because in many ment results obscures information that is essential for cases these steps can be associated with specific technical risk-informed decision making. BTRA results should not capabilities and as a result can be tied to intelligence assess- be normalized. ments of what capabilities and activities have occurred. The committee has received no briefings on this aspect of the The BTRA Event Tree Can Be Simplified evaluation of the bioterrorism threat. The BTRA is a risk assessment. (The committee argues ADDITIONAL OBSERVATIONS REGARDING in Chapter 4 that mere risk assessment is inadequate, but THE DEPARTMENT OF HOMELAND for purposes of this chapter the committee adopts the purely SECURITY’S BTRA OF 2006 probabilistic BTRA view with the objective of improving the exposition of the methodology used.) The committee Reporting Results thinks that the entire BTRA analysis can be envisioned, implemented, carried out, and documented more simply and According to DHS (2006), given the use of a particular clearly as a single, unified probabilistic risk analysis with agent, the probability of a typical scenario on the BTRA a single PRA tree that includes conditional consequence event tree may be on the order of 10–10. A typical end-state distributions. consequence may be on the order of tens of thousands of The fixed sequence of 17 stages (or 18, including the fatalities or more. The product of this probability and con- committee’s additional stage) drives the BTRA. The analy- sequence represents a particular scenario’s contribution to sis has been frustrated by the sheer size of the PRA tree for the total expected consequence associated with the use of all biological threat agents, and as a practical matter, the that agent. The sum of these represents that agent’s “relative BTRA separates the 28 agents and solves each PRA tree in importance,” given that it is selected for use. The result of isolation. But given that selection of agent is the third stage multiplying these infinitesimal probabilities by large num-

OCR for page 20
 DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 200 bers of casualties yields risk, which is represented with the • Prioritization of agents for purposes of research. This 5th percentile, mean, and 95th percentile of this risk, normal- analysis seeks to identify particular discoveries that ized by the total expected risk of all agents. might have a large influence on risk. DHS need not focus exclusively on using the rank of • Prioritization of agents for purposes of deelopment relative (expected) risk (and presents only 3 statistics, i.e., of medical countermeasures. This analysis seeks to 5th percentile, mean, and 95th percentile, per agent) as the identify improvements in medical countermeasures final result of all this analysis. With a pure event tree, more that could impact expected fatalities. “The metric for information and insight can be obtained by a more thorough assessing the potential impact of countermeasure de- analysis. velopment research is based on two criteria: baseline It would be easy to illustrate on the same plot (similar to fatality risk and current countermeasure efficacy. This Figure 3.2, with a probability above or below each vertical prioritization does not consider the current state of bar representing an agent) that (for the BTRA of 2006), for research on each agent, i.e. how close current counter- example, the agent selection probability estimated by one measure research is to a countermeasure breakthrough subject-matter expert was greater than 40 percent, by another on individual agents” (DHS, 2006, Ch. 3, p. 11). almost 30 percent, by a third greater than 10 percent, by a handful of others about 5 percent, with the rest much smaller. Analysis of Sensitivity and Risk That is, one of the “most likely” three agents was selected with 80 percent probability. The committee believes that The BTRA of 2006 offers an additional set of results that presenting the prior probability of agent selection—a key investigate (1) how much key assumptions contribute to the subject-matter expert opinion—on the same plot with the results of the risk analysis and (2) how much alternative risk level of risk associated with the use of each particular agent mitigation strategies might reduce overall risk. Sets of runs would help determine whether these estimates by subject- systematically vary the epistemic outcome probabilities. matter experts are credible and would help interpret whether Key assumptions are examined by varying parameters for agent-selection probability is a significant factor leading to agent selection and acquisition, production, and utilization; agent risk. This is extremely valuable information that is not risk mitigations are examined by varying parameters related easy for the reader to recover from analyses presented in to interdiction and medical mitigation. In Chapter 4, the DHS (2006) (and cannot be recovered at all from the Execu- committee discusses the importance of sensitivity analysis tive Summary of DHS [2006]). and the difficulty of accomplishing sensitivity analysis with Similarly, it would be easy to show the number of sce- BTRA. narios (i.e., successful attack paths) associated with each of the 28 agents. This and other simple gauges would lend Critical Knowledge Gaps and Biodefense Vulnerabilities insight into the robustness of each PRA tree with respect to either U.S. or terrorist decision represented in the tree. Critical knowledge gaps provide the greatest opportuni- “Prioritization” with a strict ranking by specific agent ties for the reduction of uncertainty in risk analysis, while may not be the best way to present results. For instance, if critical biodefense vulnerabilities provide the greatest areas one simple, cheap action can remediate the consequences of for the reduction of risk. The BTRA of 2006 identifies three a number of infectious agents, none of which appears in the areas of critical knowledge gaps: (1) intelligence and terrorist top tier of qualitatively identified “worst” ones, the rank- organization preferences, (2) event detection and response, ordering would not reveal this. The BTRA of 2006 does not and (3) biological threat agent properties. The possibility of anticipate prescriptive covering of multiple-agent risks by using these threat agent properties to aggregate current and a single action or set of actions. In subsequent chapters the potential biological agents is discussed in Chapter 5. committee recommends the pursuit of a resource-constrained Two areas of critical biodefense vulnerabilities are exam- optimization of DHS investments to maximize total risk ined: (1) threat-related vulnerabilities and (2) consequence- mitigation, and suggests some examples in Chapter 7. and/or mitigation-related vulnerabilities (Figure 3.8). Tailored Risk Assessments Planned Improvement for the BTRA of 2008 The BTRA of 2006 conducts a series of “tailored” risk Homeland Security Presidential Directive 18: Medical assessments that address, in particular: Countermeasures Against Weapons of Mass Destruction (The White House, 2007) states: • High-consequence (i.e., high-fatality) eents. Because The Secretary of Homeland Security shall develop a strate- these events are of keen concern to decision mak- gic, integrated all-CBRN risk assessment that integrates the ers, consequence distributions are truncated below a findings of the intelligence and law enforcement communi- threshold number of fatalities, and the conditioned risk ties with input from the scientific, medical, and public health rankings are presented. communities. Not later than June 1, 2008, the Secretary of

OCR for page 20
2 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT Closing Critical Biodefense Vulnerability Closing Critical Knowledge Gap Risk Risk Risk Agent X Agent X Agent X FIGURE 3.8 Closing a critical biodefense vulnerability reduces the overall risk but may not affect the uncertainties associated with that risk. Closing a critical knowledge gap does not reduce risk, but does lower the uncertainty associated with the risk. SOURCE: Adapted from DHS (2006, Figure 4.1). use EPA’s existing models of waterborne contamination Homeland Security shall submit a report to the President through the Assistant to the President for Homeland Security (EPA, 2007). and Counterterrorism, which shall summarize the key find- • DHS is developing a detailed susceptible, exposed, ings of this assessment, and shall update those findings when infected, and recovered (SEIR) model for the spread of appropriate, but not less frequently than every 2 years. infectious agents, using STELLA,8 to simulate disease transmission and medical mitigation measures through With this guidance, the following BTRA activity was the solution of systems of differential equations. In undertaken in 2007 to support the future DHS report on the Chapter , the committee cautions that there may be BTRA of 2008. The committee offers a few comments, not insufficient scientific knowledge to erify or alidate anticipated in its charge, shown in italics. these models. • In addition to indoor aerosol dispersion models, DHS • The consequence models will employ epistemic sam- is particularly interested in modeling an agent release pling, and there will be more than 10 consequence bins and spread in a subway system. in the discrete consequence distributions. • DHS plans to cooperate with the Lawrence Livermore • A library of consequence models will include a Leon- National Laboratory (2006) and the National Center for tief model of indirect economic consequences, a water Foreign Animal and Zoonotic Disease (FAZD) at Texas contamination model, agricultural disease models, a A&M University.9 differential equation model of the spread of infection • BTRA plans to incorporate more agents, including and the effects of medical countermeasures, atmo- anti-agricultural, engineered, and emerging agents. spheric dispersion forecasts, air circulation models within buildings, and others. The specific means by Although the committee agrees that some additional which outputs from these models will be converted human-threatening agents and agricultural agents may into consequence distributions has not been presented warrant attention, the committee recommends less detail to the committee. Chapter  of the present report cau- in future BTRA analyses, rather than more. Chapter 5 sug- tions against including excessie detail in these models gests aggregate categorization of agents. Such simplification where there are insufficient supporting data. would not materially damage model credibility or fidelity, • DHS plans to develop its own model of food supply given the enormous volume of assumptions and estimates contamination in cooperation with various other agen- required to instantiate any given event tree. Simplification cies and BTSafety, LLC,7 and anticipates cooperating with the Environmental Protection Agency (EPA) to 8 For further information, see www.iseesystems.com. Accessed February 23, 2007. 7 For 9 For further information, see fazd.tamu.edu. Accessed February 23, further information, see www.btsafety.com/software.htm. Accessed February 23, 2007. 2007.

OCR for page 20
 DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 200 would yield more insights, accessible results, faster computa- EPA (Environmental Protection Agency). 2007. A Water Security Handbook: Planning for and Responding to Drinking Water Contamination Threats tion, and thus better responsiveness to requests for informa- and Incidents. Available at www.watersc.org/pdf/water_security_hand- tion, as addressed in Chapter 4. Insightful analysis explaining book_rptb.pdf. Accessed February 23, 2007. the “why” of results is much more important than additional Lawrence Livermore National Laboratory. 2006. “Protecting Our Nation’s detail cluttering the “what” of results. Livestock.” Science and Technology Reiew. Available at www.llnl. gov/str/May06/Lenhoff.htm. Accessed February 23, 2007. McKay, M., R. Beckman, and W. Conover. 1979. “A Comparison of Three REFERENCES Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code.” Technometrics 21(2):239-245. Battelle Columbus Operation. 2007. “Detailed Single Scenario Analysis.” Moskowitz, H., and R. Sarin. 1983. “Improving the Consistency of Condi- Prepared for National Biodefense Analysis and Countermeasures Cen- tional Probability Assessments for Forecasting and Decision Making.” ter. Columbus, Ohio, March 27. Management Science 29(6):735-749. CDC (Centers for Disease Control and Prevention). 2007. “Emergency Paté-Cornell, E. 1984. “Fault Trees vs. Event Trees in Reliability Analysis.” Preparedness and Response—Bioterrorism Agents/Diseases.” Avail- Risk Analysis 4(3):177-186. able at www.bt.cdc.gov/agent/agentlist-category.asp. Accessed April Stein, M. 1987. “Large Sample Properties of Simulation Using Latin Hy- 13, 2007. percube Sampling.” Technometrics 29(2):143-151. DHS (Department of Homeland Security). 2006. Bioterrorism Risk Assess- U.S. Nuclear Regulatory Commission. 1975. Reactor Safety Study: Assess- ment. Biological Threat Characterization Center of the National Biode- ment of Accident Risk in U.S. Commercial Nuclear Plants. WASH-1400, fense Analysis and Countermeasures Center. Fort Detrick, Md. NUREG-75/014. Washington, D.C. DHS. 2007a. “A Lexicon of Risk Terminology and Methodological Descrip- U.S. Nuclear Regulatory Commission. 1991. Seere Accident Risks: An As- tion of the DHS Bioterrorism Risk Assessment.” Draft document dated sessment for Fie U.S. Nuclear Power Plants, NUREG-1150, January. February 2, 2007. Washington, D.C. Available at www.nrc.gov/reading-rm/doc-collections/nuregs/staff/. DHS. 2007b. “Example of Risk Assessment Calculations.” Document Accessed February 22, 2007. provided to the Committee on Methodological Improvements to the U.S. Nuclear Regulatory Commission. 1994. A Reiew of NRC Staff Uses Department of Homeland Security’s Biological Agent Risk Analysis. of Probabilistic Risk Assessment, NUREG-1489, March. Washington, February 26, 2007. Washington D.C. D.C. DHS. 2007c. “A Lexicon of Risk Terminology and Methodological De- The White House. 2007. Homeland Security Presidential Directive 18 scription of the DHS Bioterrorism Risk Assessment.” Written com- [HSPD-18]: Medical Countermeasures Against Weapons of Mass munication to the Committee on Methodological Improvements to the Destruction. Available at www.fas.org/irp/offdocs/nspd/hspd-18.html. Department of Homeland Security’s Biological Agent Risk Analysis. Accessed January 16, 2008. April 14, 2007.