Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.

Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 20

3
Description and Analysis of the Department of Homeland
Security’s Biological Threat Risk Assessment of 2006
[T]he United States requires a continuous, formal process for conducting routine capabilities assess-
ments to guide prioritization of our on-going investments in biodefense-related research, develop-
ment, planning, and preparedness.
—Homeland Security Presidential Directive 10: Biodefense for the 21st Century, 2004
The Department of Homeland Security’s (DHS’s) sys- DHS credits seminal work on nuclear reactor safety as
tem for Biological Threat Risk Assessment (BTRA) is a the basis for its risk assessment, citing “NUREG-1150”
computer-based tool that has been applied by DHS to assess (case studies of probabilistic risk assessment) (U.S. Nuclear
the risk associated with the intentional release of each of Regulatory Commission, 1991) and “NUREG-1489” (a tuto-
the 28 biological agents listed in Figure 3.1. The methodol- rial on probabilistic risk assessment) (U.S. Nuclear Regula-
ogy, an instance of probabilistic risk assessment (PRA), is tory Commission, 1994) as basic references. The committee
described in Bioterrorism Risk Assessment, a report from also found valuable an earlier foundation work, “NUREG
the DHS Biological Threat Characterization Center of the 75/014” (U.S. Nuclear Regulatory Commission, 1975),
National Biodefense Analysis and Countermeasures Center widely known as the Rasmussen Report, which establishes
(DHS, 2006). the theoretical and policy foundations on which the 1991 and
FIGURE 3.1 Biological threat agents as categorized by the Centers for Disease Control and Prevention (CDC). High-priority, Category A
agents include organisms that pose a risk to national security because they can be easily disseminated or transmitted from person to per-
son, they result in high mortality rates and have the potential for major public health impacts, they might cause social disruption, and they
require special action for public R01268, Figure 3-1 B, the second-highest priority, includes agents that are moderately easy to
health preparedness. Category
disseminate, that result in moderate morbidity rates not low mortality rates, and that require specific enhancements of CDC’s diagnostic
Fixed image, and changeable
capacity and enhanced disease surveillance. Category C agents include emerging pathogens that could be engineered for mass dissemination
in the future because of availability, ease of production and dissemination, and potential for high morbidity and mortality rates and for major
health impact. A later CDC-categorized list (CDC, 2007) features the same categories, but with agent entries revised. SOURCE: Available
at www.bt.cdc.gov/Agent/Agentlist.asp.
20

OCR for page 20

DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 2006 21
FIGURE 3.2 Ranking the risk of bioagents—the principal product of the Biological Threat Risk Assessment (BTRA) of 2006. In this figure,
biological agents versus normalized risk, a sample display is based on fictitious data that represents only the general appearance of a key
BTRA result. One of the vertical bars in this sample display represents anthrax; the dot shows the mean expected fatalities, and the horizontal
bars show the 5th and 95th percentiles. However, as is done in the analyses included in DHS (2006), the vertical scale has been normalized
so that the sum of the mean risks over all agents is 1. The committee does not know the normalization constant applied by BTRA and so
R01268, Figure 3-2
cannot recover the actual expected risks.
Fixed image, not changeable
1994 U.S. Nuclear Regulatory Commission reports and later For each agent, the estimate of the 5th percentile and
applications depend. of the 95th percentile of expected fatalities is displayed
The principal product of the BTRA of 2006 was a ranking as a tick mark on a vertical line on a logarithmic ordinate
of the risk posed by bioagent use based on calculated prob- scale of (normalized) consequences. The mean of expected
abilities of expected fatalities. DHS chose to assess threat fatalities is displayed as a dot. A typical display shows
by ranking bioagents because government stakeholders had 28 parallel vertical lines, one for each agent. The specific
advised DHS that they “expected the primary assessments to numbers and rankings of agents by risk are functions of
be in the form of risk-prioritized groups of biological threat the assumptions underlying each of the many steps in the
agents” (DHS, 2006, Ch. 1). Although a terrorist’s choice model’s execution.
of agent is just one step in a sequence of events leading to Before results are presented in DHS (2006), a normal-
a potential attack, for practical purposes the BTRA of 2006 izing constant is computed by multiplying, for each agent,
evaluates each agent separately. A probability is computed the conditional expected consequence of the agent’s use by
for each scenario involving that agent. Risk is then calcu- the probability of its use, and then summing over all the
lated as the product of these probabilities and the associated agents. All statistics are divided by this constant to force
consequences. The overall risk associated with each agent the normalized means to sum to 1. This critical normaliza-
is the integrated risk distribution over all possible scenarios tion constant is not displayed in the DHS (2006) report, so
involving that agent. no absolute (versus relative) consequence can be recovered
The product of the analysis by the BTRA of 2006 is from the analysis presented there. Therefore, the normal-
displayed in a figure (such as Figure 3.2) that shows, for ization method cannot be verified by the committee. The
each agent, a normalization (whose normalization constant normalization step is a curious one, in that it damages the
is not defined) of three estimated parameters of the distribu- results irreparably for purposes of decision making about, for
tion of consequences of agent attack in terms of expected instance, risk management. The committee conjectures that
fatalities: the normalization may reflect a well-intentioned but nonethe-
less an unfortunate effort to mitigate the stark nature of the
• The 5th percentile, estimated risks reported.
• The expected value (or mean), and DHS (2006) also contains some qualitative analysis
• The 95th percentile. distinguishing between most-, less-, and least-“worrisome”
bioagents. As for the quantitative analyses, consequences in-
The analyses presented in DHS (2006) are based entirely on estimated clude only immediate numbers of expected fatalities. Future
fatalities. However, DHS has conducted assessments based on illnesses and assessments have been promised with estimated casualties
direct economic consequences as well.

OCR for page 20

22 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
Dissemination Agent Release Mitigation
Agent Disease
Efficiency Modeling
Mass Spread Response
RISK
Target Selection AGENT Scenario
Frequency Agent Dispersion Mitigation Consequences
RELEASE
of Initiation Production
Bioagent Selection
Scenario
Probability Agent
Risk
Ranking
Initiation Selection Selection Selection Event
Frequency Probability Probability Probability Detection
Event-
Tree
Quantification
FIGURE 3.3 Biological Threat Risk Assessment (BTRA) event-tree risk assessment (left-to-right sequence) and consequence evaluation (at
the right) are loosely coupled components. SOURCE: Tracy Hale, Battelle Memorial Institute, “2008 DHS Bioterrorism Risk Assessment:
Planned Improvements,” presented to this committee on February 10, 2007, Washington, D.C.
R01268, Figure 3-3
and indirect economic consequences. The committee does 2006 is presented in this chapter using its own technical
not know whether such estimates will also be normalized, lexicon (Appendix A), which cross-references the terms used
but it hopes not. by the committee and those used in the DHS lexicon, when
relevant. Readers interested more in the policy implications
and potential uses of BTRA than in the technical details
DETAILS OF THE MODEL USED TO PRODUCE
might want to skim the text of this chapter and read the four
THE DEPARTMENT OF HOMELAND
recommendations interspersed in the text below.
SECURITY’S BTRA OF 2006
The process that produced the estimates in the BTRA of
The BTRA of 2006 Uses a Probabilistic
2006 consists of two loosely coupled analyses: (1) a PRA
Risk Assessment Event Tree
event-tree evaluation and (2) a consequence analysis (Fig-
ure 3.3). DHS has conducted “Material Threat Assessments” A PRA event tree represents a sequence of random vari-
for single bioagents. “These are plausible, high consequence ables, called events, or nodes. Each random-event branching
scenarios used to estimate the potential number of exposed node is followed by the possible random-variable realiza-
individuals, their exposure levels, contaminated areas, and tions, called outcomes, or arcs, with each arc leading from
other collateral effects.” Presumably, the results of these the branching, predecessor node, to the next, successor-event
assessments were used to inform the BTRA of 2006, but node (and it can be said without ambiguity that the predeces-
the committee was not briefed on them. DHS (2006) does sor event selects this outcome, or, equivalently, selects the
not contain mathematical definitions of all of the parameters successor event). With the exception of the first event, or root
and variables used in the BTRA and does not present a node, each event is connected by exactly one outcome of a
complete mathematical model. (A complete mathematical preceding event. A node with no successor event is called a
model would show how each input is used to produce each final event, or leaf. From each event, it is possible to trace a
output.) In response to the committee’s request for that unique path back through alternating predecessor outcomes
information, DHS has developed a lexicon and a mathemati- and events to the root event. The path from the root to a
cal model. Informed by discussions with DHS analysts, the particular leaf is called a scenario. Each successive random
committee’s understanding of the details of the BTRA of event in a scenario path has a probability depending on all
preceding outcomes in the path, and the probability of this
John Vitko, Jr., Director, Chemical and Biological Division, DHS, Sci-
scenario is the joint probability of the intersection of the
ence and Technology Directorate, briefing to the BioShield Stakeholders outcomes on the path and is the product of these outcome
Workshop, December 26, 2006. probabilities. A natural way to construct an event tree is to

OCR for page 20

DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 2006 23
place events in the chronological order in which they occur, in column 3. The number of possible outcomes for each event
if this order is known (e.g., Paté-Cornell, 1984). in a stage is shown in column 4. The maximum cumula-
This committee’s concise mathematical definition of the tive number of paths into each stage is shown in column 5.
BTRA event tree and associated computations are given in Because outcome probabilities are conditional upon some
Appendix B. preceding outcomes, column 6 shows the maximum number
Figure 3.3 shows some of the events in the BTRA tree. of such dependencies—this helps convey the complexity
The “Frequency of Initiation” box at the extreme left consists and sheer number of probabilities that must be reckoned for
of only one event—the beginning of a terrorist attack, which the BTRA.
includes the terrorist’s choice of frequency of attack, a ran- In practice in the BTRA, the event tree is not actually
dom variable with four possible outcomes. Each frequency evaluated as shown in Figure 3.4; each of the 28 agents
selected leads to a new event in the “Target Selection” box, (outcomes of events in Stage 3) is analyzed in isolation,
as shown in Figure 3.3, and each of these four events is a yielding 28 sets of, in theory, as many as 350 million paths
random variable with eight possible outcomes, leading to a based on as few as 5,448 distinct probabilities for each agent.
total of 32 events in the “Bioagent Selection” box. Each of Although the maximum number of possible scenario paths
these events is a random variable with 28 possible outcomes, is large (i.e., exponential in problem size), agent-by-agent,
depending on which of the 28 agents is used. Although not the event tree has many paths terminated early with no at-
shown in Figure 3.3, there is a sequence of 17 such boxes in tack (e.g., by failure to manufacture an agent, by successful
the BTRA event tree, enumerated and named in Figure 3.4, interdiction, and so on), while others continue to completion.
with each box corresponding to a different stage in the Among the 28 event trees, each corresponding to the selec-
chronology of a terrorist attack. A complete listing of all tion of a different agent, DHS (2006) reports one agent with
the possible outcomes for these random variables is given only 1,184 scenarios, and another, the largest agent tree, with
in the BTRA documentation but not in this report of the 192,928 scenarios.
committee. In the remainder of this chapter, the committee The individual agent results are merged a posteriori into a
uses the term “stage” to mean all of the possible events at distribution using probabilities for the selection of each agent
each step. As can be inferred from the names given to the and target. With the exception of this separation of event
stages (see Figure 3.4), each corresponds either to a terror- trees by agent, BTRA treats each of these successive events
ist decision (e.g., Bioagent Selection), or to a U.S. decision in ascending order of the stage in which it occurs.
(e.g., Mitigation). It is a fundamental property of the BTRA For example, Figure 3.5 shows the outcomes for each
of 2006 that every event, whether representing a terrorist event in Stage 2. After the frequency of attack has been
decision or a U.S. decision, has a probability of occurrence chosen, the terrorist can choose among eight types of tar-
associated with it. get to pursue. The BTRA represents the selection of each
As indicated above, Figure 3.4 displays the succession such outcome as an arc chosen randomly, with a selection
of 17 stages of the BTRA event tree. The BTRA represents probability that may depend on outcomes of events in prior
epistemic uncertainty (uncertainty due to incomplete knowl- stages. In this example, the outcome probabilities from
edge) by using a distribution of event probabilities from events in Stage 2 may depend on the outcomes chosen for
which a particular probability is sampled; that is, adopting prior events in Stage 1.
the convention that from a node, each branching outcome The BTRA analyzes each of the 28 agents as follows:
“selects” a successor event, each such event leading to an
outcome has a probability distribution over its probability 1. selection probability of the agent under study is set
The
of selection. For events in all but the first stage, each event to 1 for each event in Stage 3. All other probabilities
leading to an outcome is chosen with a probability drawn for events in Stage 3 are set to 0. (Stage 3 consists of
from a distribution of probabilities for that outcome. The agent-selection events; there are 32 events that result in
selection of outcomes from the only event in the first stage, agent-selection outcomes.) It is important to note that
“Frequency of Initiation by Terrorist Group,” is the rate at no attack using multiple agents is considered.
which terrorists are anticipated to make attempts during a 2.
The tree for this agent is Monte Carlo generated, with
time horizon over which this rate applies; each such rate and outcome probability distributions conditioned upon
time horizon has an associated probability. outcomes from events in Stages 1 and 2 as well as
An 18th stage has been added to Figure 3.4 by the com- on the knowledge of which agent is being modeled.
mittee to represent the “Consequences” random variable. If BTRA represents epistemic uncertainty by using a
the probability of an outcome depends on outcomes from an distribution of outcome probabilities from which a
event in a preceding stage, the prior stage number is shown particular probability is sampled. These epistemic
probability distributions over outcome probabilities
Given that the number of opportunities for such attempts is huge and
are elicited from subject-matter experts for each indi-
the probability that any particular opportunity will be pursued is tiny, this vidual possible outcome, although there are thousands
is a Poisson rate. of such conditional outcomes.

OCR for page 20

24 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
Depends Maximum
Number Maximum
Stage on Cumulative
Event Type of Possible Number of Phase
No. Stage Number of Paths
Outcomes Dependencies
No. into Stage
Frequency of
1 Initiation by 4 4 4
Terrorist Group
2 Target Selection 1 8 32 32
3 Bioagent Selection 2 28 896 224 Agent/Target/Dissemination
Selection
Mode of
Dissemination
4 (also determines 1, 2, 3 9 8,064 8,064
wet or dry
dispersal form)
Mode of Agent
5 3 4 32,256 112
Acquisition
Acquisition
Interdiction during
(6) 1, 3, 5 2 64,512 896
Acquisition
Location of
7 Production and 1 2 129,024 8
Processing
Mode of Agent
8 1, 3 3 387,072 336
Production
Preprocessing and 1, 2, 3,
9 3 1,161,216 72,576
Concentration 4, 8 Production and Processing
Drying and
10 1, 2, 3, 4 3 3,483,648 24,192
Processing
11 Additives 1, 2, 3, 4 2 6,967,296 16,128
Interdiction
(12) During Production 2 13,934,592 2
and Processing
Mode of Transport
13 1, 2, 3, 4 3 41,803,776 24,192
and Storage
Interdiction Transport and Storage
(14) During Transport 7 2 83,607,552 4
and Storage
Interdiction
(15) 2 167,215,104 2
During Attack
Attack
Potential for
16 1 2 334,430,208 8
Multiple Attacks
(17) Event Detection 2, 3, 4 3 1,003,290,624 6,048 Response
18 Consequences tbd 10 10,032,906,240 tbd Final Outcome
FIGURE 3.4 Successive stages in the Biological Threat Risk Assessment (BTRA) event tree. A BTRA event tree consists of 17 stages clas-
sified into six successive phases. The committee has emphasized Stages 6, 12, 14, 15, and 17 by inserting parentheses around these stage
numbers in the left-hand column, to distinguish interdiction opportunities. Outcomes of events in all other stages are chosen by the bioter-
rorist. The committee added the columns labeled “Number of Possible Outcomes,” “Maximum Cumulative Number of Paths into Stage,”
and “Maximum Number of Dependencies,” as well as an 18th stage representing “Consequences.” NOTE: tbd, to be determined. SOURCE:
Adapted from DHS (2006, Table 5.1).

OCR for page 20

DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 2006 25
Stage No. Event Type Possible Outcomes Depends on Stage No.
2.1 Large Open Building
2.2 Small Enclosure
2.3 Large “Divided” Building
2.4 Large Outdoor Spaces
2 Target Selection 1
2.5 Water Pathway
2.6 Food Pathway
2.7 Human Vectors
2.8 Contact (letters)
FIGURE 3.5 Each event offers the terrorist one choice of a number of alternate outcomes. Here, Stage 2, “Target Selection,” is amplified
into eight outcomes. The Biological Threat Risk Assessment represents the choice of each outcome with a probability and refers to this as
a “split fraction” (i.e., conditional arc probability). The number at the right shows that the probability distribution on outcomes from events
in Stage 2 is dependent on outcomes from events in Stage 1, “Frequency of Initiation by Terrorist Group.” SOURCE: Adapted from DHS
(2006, Table 5.2).
3. set of outcome probabilities is generated, and the
A a host of other considerations for subject-matter experts
resulting probabilistic risk assessment event tree is to consider when rendering opinions about consequences.
solved. That is, each leaf (terminal event) with nonzero Event by event, outcomes that support this scenario can be
probability is associated with a consequence distribu- identified, although many nuances (e.g., steps to concentrate,
tion, from which the leaf-probability-weighted conse- process, and introduce additives to “weaponize” the anthrax
quence distributions are sampled to produce a sample spores for better dispersal) may be hard to unambiguously
unconditional consequence distribution. The BTRA identify (i.e., the attackers have either weaponized a lot of an-
does this 500 times, thus generating a random sample thrax, or they have produced an even larger quantity of crude
of 500 PRA trees and associated consequence distri- anthrax to use). Regardless, the base mission of the BTRA is
butions. For each of these trees, the resulting 5th and to automatically generate hosts of scenarios, including ones
95th percentiles and the average of the consequences that resemble the committee’s hypothetical scenario, and
are computed. This sampling of multiple realizations rank them in terms of expected risk (i.e., fatalities).
from the same starting conditions represents aleatory Three short papers (DHS, 2007a,b,c) presented to the
uncertainty—the influence of pure randomness. committee give details on and contain versions of Figure 3.6.
4.
The outcome of each random-sample scenario is cap- In this tree, the starting event is at the extreme left, followed
tured by the distribution of expected consequences; the by two stages of events representing the terrorist choice of
expectation is over purely aleatory randomness. agent and then choice of target. A complete scenario in this
5.
The 28 agent statistics are merged, after the fact, using reduced example is characterized by a left-to-right scenario
the agent-selection probabilities. path from starting event to final event and is documented
by the successive outcomes, or arcs, in this scenario path.
The committee’s hypothetical scenario, introduced in For instance, a path with arcs labeled “PA1, 1-PT1” leads to
Chapter 1, may be approximately described by a number scenario s2 with consequence distribution c(x | s2), where x
of possible sequences of outcomes in the BTRA event tree. represents fatalities. The notation “PA1” represents, at once,
The type of terrorist group here would not likely be a de- the selection of Agent 1 and its probability of selection. Al-
ranged individual or even a small cell, because the volume though not shown in Figure 3.6, each successive probability
of anthrax hypothesized for this large-scale, outdoor aerosol could depend on everything that precedes it in its scenario
attack exceeds that of the attacks following 9/11 by several path. So, in example scenario path “PA1, 1-PT1,” the prob-
orders of magnitude, and thus the terrorists are evidently well ability PT1 can depend on the prior choice of event PA1.
funded, perhaps even state-sponsored. Target selection (Fig- As noted above, a fundamental property of the event trees
ure 3.5) would be a “Large Outdoor Space.” That this space used in the BTRA is that every decision by a bioattacker
is hypothetically filled with commuters conditions the con- (e.g., choice of an agent) or by a defender (e.g., choice of
sequences, but it is not clear where these commuters would a countermeasure) is considered to be an uncertain event—
appear in the BTRA; they are evidently rolled up along with hence associated with an outcome selection probability. In

OCR for page 20

26 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
FIGURE 3.6 A simplified event tree for two successive stages (events), each with two
alternate outcomes. The initiation frequency and succession of dependent probabilities in
each scenario path lead to a “scenario frequency” that is multiplied by the consequence
distribution at the leaf of that scenario. The consequence distribution c(x | si) is expressed ,
for scenario si in units of number of fatalities, although any convenient units would suf-
fice. At the right, a distribution of consequences is shown for each scenario. At the lower
right, the unconditional distribution is shown—the path-probability-weighted distribution
of expected consequences for one sampled event tree. (The histogram collects outcomes
in discrete “bins,” that is, intervals; for convenience, the consequences for all scenarios
share the same 10 intervals, bounded above by integral powers of 10.) SOURCE: DHS
(2007c, Figure 1).
R01268, Figure 3-6
fact, the BTRA uses pure probability not changeable
Fixed image, trees, and no decision probability solicitations is converted into a marginal prob-
tree at all. A statement in a presentation to the committee: ability density of probabilities for selecting the particular
“An event tree (decision tree) is a visual tool . . .” indicates outcome. Documentation indicates that most subject-matter
confusion on this point. The distinction between event and experts for the BTRA of 2006 were experts from Battelle
decision trees is fundamental, not semantic. In event trees, Memorial Institute, Columbus, Ohio, but that subsequent
all outcomes are modeled as random events determined by work will draw from a much wider pool of experience.
some probability distribution; decision trees allow the possi- Some observations by the committee about the details in
bility that outcomes are chosen by the defender or attacker to these steps follow. In step 2 above, the Monte Carlo simu-
achieve some objective. Decision trees as tools for modeling lation generates probabilities for each event one outcome
terrorist threats are discussed in Chapter 7. (arc) at a time in some fixed sequence of outcomes. For
In step 3 above, for each outcome from each event, the each successive outcome, a marginal probability distribu-
probability of selection has been elicited as the consensus of tion over the probability for selecting this outcome is used.
a group of subject-matter experts in the form of an expected The probability distribution for each successive outcome
probability, and reportedly some additional guidance (such as is conditioned on the probabilities already realized for this
the 5th and 95th percentile of this outcome probability) that event. Because the outcome probabilities must sum to 1,
has been transformed by some unspecified means into a vari- the marginal distributions for each should be constrained
ance for each probability. Each of these outcome selection to have their expectations sum to 1. Although the original
marginal distributions are, for instance, beta densities, the
successive conditioning by sampled outcomes means that
Richard S. Denning, Battelle Memorial Institute, “DHS 2006 Bioterror- outcomes are really sampled from some multivariate density
ism Risk Assessment Methodology,” presentation to the committee, August
28, 2006, Slide 8.
for which the marginals are not beta, and in fact are not

OCR for page 20

DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 2006 27
characterized at all in closed form, and not mentioned at all The BTRA consequence analysis is qualitatively different
in DHS (2006). from its event-tree analysis. Subject-matter expert opinions
The DHS procedure selects the last outcome probability are developed much like case studies, and there is less clear
so that the sum of outcome probabilities emanating from this dependence on specific events leading to each consequence.
event is 1 (i.e., the last marginal probability distribution is Thus, each consequence distribution should be viewed as
not used at all). However, the outcome probabilities should being dependent on every event leading to its outcome. How-
have a joint distribution that captures their dependencies (the ever, an examination of the underlying analysis in the DHS
most important being that they sum to 1). Even if the present (2006) report suggests that there is really only a single con-
method were not technically superfluous (as is shown below), sequence distribution for each scenario: one that depends not
subject-matter experts typically cannot assess such high-di- on the complete scenario but only on a subset of parameter
mensional distributions (Moskowitz and Sarin, 1983). values. (Indeed, “Consequence uncertainty was omitted due
In step 3, the 500 sets of outcome probabilities for each to the overwhelming processing requirements.”) A Monte
agent event tree are obtained using a Latin Hypercube Sam- Carlo simulation of 1,000 samples was used to estimate each
pling design (Stein, 1987), a sampling technique applied in consequence distribution in the BTRA of 2006. The commit-
earlier years to probabilistic risk analysis of nuclear safety. tee has no details about how this was accomplished.
However, the committee notes that this sampling design
produces unbiased estimates of the mean and quantiles with
THE EVENT TREE CAN BE IMPROVED
asymptotic sample size. Further, see Stein (1987, p. 144,
Equation (3) and Section 5) and McKay et al. (1979, Section
The Approach to Determining the Probabilities
8.3). Moreover, the variance may be decreased or increased
of Terrorist Decisions Is Incomplete
by this design, depending on the covariance structure of the
distributions sampled. Note that the proofs of unbiasedness The BTRA of 2006 uses probabilities to represent ad-
for quantiles are for independent random variables. There is versarial decisions. These are conditional probabilities, but
no evidence that the efficacy of the particular BTRA sample the conditioning is retrospective, rather than prospective.
design has been established. Consider that if the consequence model for a bioagent is
completely changed to reflect some new discovery about
the efficacy of the bioagent, this would have no influence at
The BTRA of 2006 Does Not Use Event
all on the BTRA probabilities; neither the terrorist nor the
Trees for Consequence Analysis
United States would change probabilities in response.
Consequence models characterize the probability dis- When dealing with an intelligent, goal-oriented, and
tribution of consequences for each scenario. The BTRA resourceful adversary, not with a force such as nature that
employs a mass-release model that assesses the production randomly determines whether unwanted events occur, this
of each bioagent, beginning with time to grow and produce, committee believes that the use of probabilities to represent
preprocess and concentrate, dry, store and transport, and bioterrorism decisions must be tempered by a thorough un-
dispense. The net result is a biological agent dose that is derstanding of how these probabilities have been assessed
input to a consequence model to assess casualties. One equa- (whether by means of formal game-theoretical models,
tion from the model is produced here to give a flavor of the elicitation of subject-matter experts, of other means). For
computations. decision problems as complex as those motivating BTRA,
the assessment of the probabilities that adversaries will
MR = MT × QF1 × QF2 × QF3 × QF4 × QF5 choose courses of action should be the outputs of analysis,
not required input parameters. The BTRA has reversed this
where MR is bioagent mass release, MT is target mass, and preferred approach by requiring that subject-matter experts
QFi are factors to explain production, processing, storage, predict, a priori, how adversaries will behave. For this ap-
and so on and are random variables conditioned on the sce- proach to make sense, the subject-matter experts must grasp
nario whose consequences are being evaluated. nuances of alternatives and outcomes and render opinions
The complete model computes, for an attack with a given founded on an analysis of the entire decision process, which
agent on a given target, how much agent has been used, how would be very difficult for a process this complex. The com-
efficiently it has been dispersed (and, for an infectious agent, mittee saw no evidence that this level of analysis was used.
how far it spreads in the target population), and the potential Moreover, the static probabilities used are not appropriate
effects of mitigation efforts. For the BTRA of 2006, all of when terrorists can observe and react dynamically to any
these factors were assigned values by eliciting opinions of earlier decisions made by the United States.
subject-matter experts in the form of subjective discrete
probability distributions of likely outcomes, and by some
Traci Hale, Battelle Memorial Institute, “2008 DHS Bioterrorism Risk
application of information on the spread of infectious agent, Assessment: Planned Improvements,” presented to this committee on Febru-
atmospheric dispersion, and so on. ary 10, 2007, Washington, D.C.

OCR for page 20

28 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
Recommendation: To assess the probabilities of terror- quences of multiple attacks. For l′ = 0.9, ƒ1(0.9 = 1.25,
ist decisions, DHS should use elicitation techniques and ƒ2(0.9) = 91, and the correct expectation ƒ3(0.9) = 10. For
decision-oriented models that explicitly recognize terror- this numerical example, the two expectations respectively
ists as intelligent adversaries who observe U.S. defensive would underestimate and overestimate consequences by an
preparations and seek to maximize the achievement of order of magnitude.
their own objectives. If each repeated attack is independent, the distribution
of total consequences across all attacks will presumably be
It should be noted that this recommendation does not require additive. The distribution of this sum is characterized by a
the prediction of terrorist actions, a difficult task at best. Its statistical convolution, not by mere multiplication by the
intent is to evaluate risk on the basis of hypothetical terror- expected number of re-attacks.
ist attacks against U.S. defenses that have been designed to It is not very realistic to assume an infinite supply of
thwart terrorist goals. Thus, its implementation will produce potential attacks that all have equal probabilities of success.
a conservative estimate of risk. In Chapter 7, the committee Judging from U.S. actions taken after the 9/11 terrorist at-
offers alternate modeling techniques to accomplish this more tacks, the committee believes that all of the probabilities
complex assessment. assessed in the event tree will change following any attack.
Thus, the implicit, homogeneous steady-state Poisson
process underlying the rate used for “Frequency of Initiation
The Mathematics Used by the BTRA in
by Terrorist Group” will almost surely be rendered invalid by
Modeling Multiple Attacks Has Errors
any detected attack, whether successful or not, and whether
Given a successful attack, the PRA tree’s Stage 16, Po- interdicted or not. Subsequent to any such event, the BTRA
tential for Multiple Attacks (see Figure 3.4) presents an analysis would be rendered inapplicable until a host of key
opportunity for the terrorist to mount more such attacks. The parameters could be reestimated and the BTRA then repeated
probability for succeeding at each additional attack is given from scratch.
as l′ (implying that the attacks that are attempted first are The BTRA multiple-attack feature is an embellishment
no more likely to succeed than those postponed until the first that has been incorrectly implemented both mathematically
attempts have failed), and the expected number of attacks and statistically, and even if correctly implemented would
before interdiction is given in the DHS (2006) report and be based on a questionable underlying model.
presentation to the committee as
λ′ The 2006 BTRA’s Assessment of Outcome
f1 (λ ′) = 1 + .
(1 + λ ′)2 Probabilities Is Unnecessarily Complex
This expectation is multiplied by the consequence distribu- Each node in the PRA tree offers two or more outcomes
tion for such attacks. leading to successor events, each selected with an epistemic
During a site visit to Battelle in Columbus, Ohio, in Octo- probability density that is used to generate an aleatory out-
ber 2006, the committee pointed out that this equation must come probability to be used to solve the event tree. Each of
be in error (e.g., if l′ = 1, the expected number of re-attacks these outcome densities is typically a beta density function,
should go to infinity, but ƒ1(l′ = 1) = 1.25). formed somehow from averages elicited from subject-matter
Subsequent briefing materials. (Battelle Columbus Opera- experts, whose means sum to 1. It is straightforward to show
tion, 2007) featured a new expectation: that, when given a distribution over outcome probabilities,
the means of these distributions suffice to completely capture
λ′
f2 (λ ') = 1 + . the unconditional distributions over any consequence. For
(1 - λ ′)2 example, suppose that
This expectation is also wrong. Given one successful attack,
the total number of successful attacks before an interdiction xi = probability that outcome i will occur, i = 1,2, …, n
with probability of success for each additional attack l′ is →
ξ = [x1,x2,…,xn]
f3 (λ ′) = 1 + E[n | λ ′] = ƒi(x) = probability distribution over outcome X, given that
∞
λ′ 1 outcome i occurs.
1 + ∑ n(λ ′)n (1 - λ ′) = 1 + = .
1 - λ′ 1 - λ′
n= 0
If the values of xi are known, then the unconditional distribu-
Figure 3.7 shows these expressions as a function of λ′. tion over consequences X is
This has a significant influence on the expected conse- → n
f ( x | ξ ) = ∑ ξi fi ( x ).
i =1
“Multiple attacks” refer to attacks in sequence. “Simultaneous multiple
attacks” are considered by the BTRA to be a single attack.

OCR for page 20

DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 2006 29
10
9
8
7
6
f 5
f 2 (λ ′)
4 f 3 (λ ′)
3
2
f 1 (λ ′)
1
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
λ′
FIGURE 3.7 Expected number of attacks before interdiction, given that a first attack is successful and that continued attacks each evade
interdiction with probability λ′. ƒ3(λ′) is the expected number of attacks before interdiction. ƒ1 is the BTRA expression, and ƒ2 is the ex-
pression offered with a complete numerical example (Battelle Columbus Operation, 2007). For λ′ = 0.9, ƒ1 underestimates by an order of
magnitude, and ƒ2 overestimates by an order of magnitude. This expectation is multiplied by the single-attack distribution of consequences,
so these errors have major influence. The value λ′ = .9 has been chosen by the committee for expository purposes. The committee does not
represent that this value occurs in any scenario analyzed by DHS or in the example provided by Battelle. The interested reader my substitute
any other value for λ′ to assess its effect.
However, the epistemic approach considers xi to be a
→
no analysis in the BTRA of 2006 and no improvement in
random variable, and therefore ξ a random vector. Letting
→
analysis recommended by the committee can make mean-
ϕ(ξ ) be the (joint) probability distribution over the elements
→
ingful use of the information available in the family of risk
xi of the random vector ξ , the unconditioned distribution over curves, beyond that provided by their expectation. Further,
consequences becomes given the planned improvements to the BTRA incorporat-
→ → → ing additional consequence measures and utility functions,
f ( x ) = ∫ f ( x | ξ )ϕ (ξ )d ξ = the committee does not anticipate analyses that require the
n → → n → → n family of risk curves.
∫ ∑ ξ f ( x )ϕ (ξ )d ξ = ∑ f ( x ) ∫ ξ ϕ (ξ )d ξ = ∑ f ( x )E (ξ ) .
i i i i i i
If the conditional consequence distributions are given in
i =1 i =1 i =1 parametric form, or in numerical lookup tables, calculation
Therefore, even when using a general (and possibly highly of the risk distribution can be done exactly, without resorting
dependent) joint distribution, all that is needed is the expec- to estimating these distributions from the outputs of Monte
tation E(xi), which is the mean of the epistemic distribution; Carlo simulations. This computation is easy and fast, and
the rest of the distribution is irrelevant to determining the the result is the distribution—not merely an estimate of its
unconditional distribution of consequences (and, in particu- features.
lar, its moments, percentiles, and so on). For these reasons, the committee’s finding is that the
Because of this, the consequence distribution can be epistemic features of the BTRA probabilistic risk assess-
calculated without sampling from the outcome probability ment are unnecessary and that they increase computation
distributions. Appendix C provides a self-contained, simpli- time and complicate exposition, analysis, and understanding
fied example of this point. For an event tree the size of the of results.
one used in the BTRA of 2006, this represents a significant
computational simplification and would also significantly Recommendation: The event-tree probability elicitation
simplify the BTRA exposition; both of these results are desir- should be simplified by assessing probabilities instead
able. What is lost in the simplification is the family of risk of probability distributions for the outcomes of each
curves—i.e., one curve for each possible outcome. However, event.

OCR for page 20

30 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
BTRA Results Should Not Be Normalized in the fixed hierarchy of the BTRA event sequence, and that
by an Unspecified Constant this selection depends on both prior stages, this approach
has complicated the analysis and exposition of results. Fixed
The absence of a normalization constant in DHS docu-
adherence to the 17 sequential stages in the BTRA event
mentation and presentation irretrievably obscures those
tree leads to large PRA trees that have had to be separated
BTRA results where normalization is employed, rendering
by 28 individual agents. That the choice of agent is not a
those results essentially useless for further analyses, espe-
first-stage event, or even a second-stage one, but rather a
cially for risk management. As an illustrative example, sup-
third-stage event, causes some difficulty in recovering results
pose that the United States discovers how to make a reliable
after the fact.
biological agent alarm the size and cost of a smoke detector
and how to connect such detectors to local area networks;
Recommendation: Two significant simplifications should
educates the U.S. populace to shelter in place on alarm;
be made to the BTRA of 2006 event tree:
implements effective, immediate cordoning and quarantine
• DHS should eliminate Stage 1, Frequency of Initia-
procedures; and thus attains an estimated threefold reduction
tion [of an attack] by Terrorist Group, and Stage 16,
in expected consequences from terrorist use of all biological
Potential for Multiple Attacks; and
agents. This improved capability of detection and response
• DHS should seek opportunities to aggregate some
would not change a single normalized result presented in
stages of the tree to only those essential to calcu-
the BTRA of 2006.
late probabilities and consequences with realistic
The committee wonders how senior leadership has in-
fidelity.
terpreted a normalized fatality scale (with no units) in the
DHS (2006) report and presentation materials: the com-
The elimination of probability elicitation for terrorist
mittee does not know why this normalization was applied,
decisions will greatly simplify the model. Additional sim-
and especially why its essential details are absent from all
plifications are also possible. For instance, Stages 7 through
underlying documentation. The normalized results are clas-
11 (successively: Location of Production and Processing,
sified, as would be the non-normalized results, and this one
Mode of Agent Production, Preprocessing and Concentra-
step—normalization—has made it impossible for anyone to
tion, Drying and Processing, and Additives) appear to reflect
reproduce any BTRA result or for anyone to use independent
a somewhat artificial taxonomy and permutation of decisions
means to assess the accuracy of any BTRA result.
in a proliferation effort. Similarly Stages 14 and 15 (Interdic-
Most important, risk management deals with risk, not
tion During Transport and Storage, and Interdiction During
normalized risk. The BTRA needs to report risk, not nor-
Attack) might be aggregated. It should be noted, however,
malized risk.
that the level of detail shown in Figure 3.4 may coordinate
with the steps that the FBI and the National Counterterror-
Recommendation: Normalization of BTRA risk assess-
ism Center consider in evaluating an attack, because in many
ment results obscures information that is essential for
cases these steps can be associated with specific technical
risk-informed decision making. BTRA results should not
capabilities and as a result can be tied to intelligence assess-
be normalized.
ments of what capabilities and activities have occurred. The
committee has received no briefings on this aspect of the
The BTRA Event Tree Can Be Simplified evaluation of the bioterrorism threat.
The BTRA is a risk assessment. (The committee argues
in Chapter 4 that mere risk assessment is inadequate, but ADDITIONAL OBSERVATIONS REGARDING
for purposes of this chapter the committee adopts the purely THE DEPARTMENT OF HOMELAND
probabilistic BTRA view with the objective of improving SECURITY’S BTRA OF 2006
the exposition of the methodology used.) The committee
thinks that the entire BTRA analysis can be envisioned, Reporting Results
implemented, carried out, and documented more simply and
According to DHS (2006), given the use of a particular
clearly as a single, unified probabilistic risk analysis with
agent, the probability of a typical scenario on the BTRA
a single PRA tree that includes conditional consequence
event tree may be on the order of 10–10. A typical end-state
distributions.
consequence may be on the order of tens of thousands of
The fixed sequence of 17 stages (or 18, including the
fatalities or more. The product of this probability and con-
committee’s additional stage) drives the BTRA. The analy-
sequence represents a particular scenario’s contribution to
sis has been frustrated by the sheer size of the PRA tree for
the total expected consequence associated with the use of
all biological threat agents, and as a practical matter, the
that agent. The sum of these represents that agent’s “relative
BTRA separates the 28 agents and solves each PRA tree in
importance,” given that it is selected for use. The result of
isolation. But given that selection of agent is the third stage
multiplying these infinitesimal probabilities by large num-

OCR for page 20

DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 2006 31
bers of casualties yields risk, which is represented with the • Prioritization of agents for purposes of research. This
5th percentile, mean, and 95th percentile of this risk, normal- analysis seeks to identify particular discoveries that
ized by the total expected risk of all agents. might have a large influence on risk.
DHS need not focus exclusively on using the rank of • Prioritization of agents for purposes of development
relative (expected) risk (and presents only 3 statistics, i.e., of medical countermeasures. This analysis seeks to
5th percentile, mean, and 95th percentile, per agent) as the identify improvements in medical countermeasures
final result of all this analysis. With a pure event tree, more that could impact expected fatalities. “The metric for
information and insight can be obtained by a more thorough assessing the potential impact of countermeasure de-
analysis. velopment research is based on two criteria: baseline
It would be easy to illustrate on the same plot (similar to fatality risk and current countermeasure efficacy. This
Figure 3.2, with a probability above or below each vertical prioritization does not consider the current state of
bar representing an agent) that (for the BTRA of 2006), for research on each agent, i.e. how close current counter-
example, the agent selection probability estimated by one measure research is to a countermeasure breakthrough
subject-matter expert was greater than 40 percent, by another on individual agents” (DHS, 2006, Ch. 3, p. 11).
almost 30 percent, by a third greater than 10 percent, by a
handful of others about 5 percent, with the rest much smaller.
Analysis of Sensitivity and Risk
That is, one of the “most likely” three agents was selected
with 80 percent probability. The committee believes that The BTRA of 2006 offers an additional set of results that
presenting the prior probability of agent selection—a key investigate (1) how much key assumptions contribute to the
subject-matter expert opinion—on the same plot with the results of the risk analysis and (2) how much alternative risk
level of risk associated with the use of each particular agent mitigation strategies might reduce overall risk. Sets of runs
would help determine whether these estimates by subject- systematically vary the epistemic outcome probabilities.
matter experts are credible and would help interpret whether Key assumptions are examined by varying parameters for
agent-selection probability is a significant factor leading to agent selection and acquisition, production, and utilization;
agent risk. This is extremely valuable information that is not risk mitigations are examined by varying parameters related
easy for the reader to recover from analyses presented in to interdiction and medical mitigation. In Chapter 4, the
DHS (2006) (and cannot be recovered at all from the Execu- committee discusses the importance of sensitivity analysis
tive Summary of DHS [2006]). and the difficulty of accomplishing sensitivity analysis with
Similarly, it would be easy to show the number of sce- BTRA.
narios (i.e., successful attack paths) associated with each
of the 28 agents. This and other simple gauges would lend
Critical Knowledge Gaps and Biodefense Vulnerabilities
insight into the robustness of each PRA tree with respect to
either U.S. or terrorist decision represented in the tree. Critical knowledge gaps provide the greatest opportuni-
“Prioritization” with a strict ranking by specific agent ties for the reduction of uncertainty in risk analysis, while
may not be the best way to present results. For instance, if critical biodefense vulnerabilities provide the greatest areas
one simple, cheap action can remediate the consequences of for the reduction of risk. The BTRA of 2006 identifies three
a number of infectious agents, none of which appears in the areas of critical knowledge gaps: (1) intelligence and terrorist
top tier of qualitatively identified “worst” ones, the rank- organization preferences, (2) event detection and response,
ordering would not reveal this. The BTRA of 2006 does not and (3) biological threat agent properties. The possibility of
anticipate prescriptive covering of multiple-agent risks by using these threat agent properties to aggregate current and
a single action or set of actions. In subsequent chapters the potential biological agents is discussed in Chapter 5.
committee recommends the pursuit of a resource-constrained Two areas of critical biodefense vulnerabilities are exam-
optimization of DHS investments to maximize total risk ined: (1) threat-related vulnerabilities and (2) consequence-
mitigation, and suggests some examples in Chapter 7. and/or mitigation-related vulnerabilities (Figure 3.8).
Tailored Risk Assessments Planned Improvement for the BTRA of 2008
The BTRA of 2006 conducts a series of “tailored” risk Homeland Security Presidential Directive 18: Medical
assessments that address, in particular: Countermeasures Against Weapons of Mass Destruction
(The White House, 2007) states:
• High-consequence (i.e., high-fatality) events. Because The Secretary of Homeland Security shall develop a strate-
these events are of keen concern to decision mak- gic, integrated all-CBRN risk assessment that integrates the
ers, consequence distributions are truncated below a findings of the intelligence and law enforcement communi-
threshold number of fatalities, and the conditioned risk ties with input from the scientific, medical, and public health
rankings are presented. communities. Not later than June 1, 2008, the Secretary of

OCR for page 20

32 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
Closing Critical Biodefense Vulnerability
Closing Critical Knowledge Gap
Risk Risk Risk
Agent X Agent X Agent X
FIGURE 3.8 Closing a critical biodefense vulnerability reduces the overall risk but may not affect the uncertainties associated with that
risk. Closing a critical knowledge gap does not reduce risk, but does lower the uncertainty associated with the risk. SOURCE: Adapted from
DHS (2006, Figure 4.1).
Homeland Security shall submit a report to the President use EPA’s existing models of waterborne contamination
through the Assistant to the President for Homeland Security (EPA, 2007).
and Counterterrorism, which shall summarize the key find- • DHS is developing a detailed susceptible, exposed,
ings of this assessment, and shall update those findings when infected, and recovered (SEIR) model for the spread of
appropriate, but not less frequently than every 2 years.
infectious agents, using STELLA, to simulate disease
With this guidance, the following BTRA activity was transmission and medical mitigation measures through
undertaken in 2007 to support the future DHS report on the the solution of systems of differential equations. In
BTRA of 2008. The committee offers a few comments, not Chapter 6, the committee cautions that there may be
anticipated in its charge, shown in italics. insufficient scientific knowledge to verify or validate
these models.
• The consequence models will employ epistemic sam- • In addition to indoor aerosol dispersion models, DHS
pling, and there will be more than 10 consequence bins is particularly interested in modeling an agent release
in the discrete consequence distributions. and spread in a subway system.
• A library of consequence models will include a Leon- • DHS plans to cooperate with the Lawrence Livermore
tief model of indirect economic consequences, a water National Laboratory (2006) and the National Center for
contamination model, agricultural disease models, a Foreign Animal and Zoonotic Disease (FAZD) at Texas
differential equation model of the spread of infection A&M University.
and the effects of medical countermeasures, atmo- • BTRA plans to incorporate more agents, including
spheric dispersion forecasts, air circulation models anti-agricultural, engineered, and emerging agents.
within buildings, and others. The specific means by
which outputs from these models will be converted Although the committee agrees that some additional
into consequence distributions has not been presented human-threatening agents and agricultural agents may
to the committee. Chapter 4 of the present report cau- warrant attention, the committee recommends less detail
tions against including excessive detail in these models in future BTRA analyses, rather than more. Chapter 5 sug-
where there are insufficient supporting data. gests aggregate categorization of agents. Such simplification
• DHS plans to develop its own model of food supply would not materially damage model credibility or fidelity,
contamination in cooperation with various other agen- given the enormous volume of assumptions and estimates
cies and BTSafety, LLC, and anticipates cooperating required to instantiate any given event tree. Simplification
with the Environmental Protection Agency (EPA) to
For
further information, see www.iseesystems.com. Accessed February
23, 2007.
For further information, see www.btsafety.com/software.htm. Accessed For further information, see fazd.tamu.edu. Accessed February 23,
February 23, 2007. 2007.

OCR for page 20

DESCRIPTION AND ANALYSIS OF HOMELAND SECURITY’S BIOLOGICAL THREAT RISK ASSESSMENT OF 2006 33
would yield more insights, accessible results, faster computa- EPA (Environmental Protection Agency). 2007. A Water Security Handbook:
tion, and thus better responsiveness to requests for informa- Planning for and Responding to Drinking Water Contamination Threats
and Incidents. Available at www.watersc.org/pdf/water_security_hand-
tion, as addressed in Chapter 4. Insightful analysis explaining book_rptb.pdf. Accessed February 23, 2007.
the “why” of results is much more important than additional Lawrence Livermore National Laboratory. 2006. “Protecting Our Nation’s
detail cluttering the “what” of results. Livestock.” Science and Technology Review. Available at www.llnl.
gov/str/May06/Lenhoff.htm. Accessed February 23, 2007.
McKay, M., R. Beckman, and W. Conover. 1979. “A Comparison of Three
REFERENCES Methods for Selecting Values of Input Variables in the Analysis of
Output from a Computer Code.” Technometrics 21(2):239-245.
Battelle Columbus Operation. 2007. “Detailed Single Scenario Analysis.”
Moskowitz, H., and R. Sarin. 1983. “Improving the Consistency of Condi-
Prepared for National Biodefense Analysis and Countermeasures Cen-
tional Probability Assessments for Forecasting and Decision Making.”
ter. Columbus, Ohio, March 27.
Management Science 29(6):735-749.
CDC (Centers for Disease Control and Prevention). 2007. “Emergency
Paté-Cornell, E. 1984. “Fault Trees vs. Event Trees in Reliability Analysis.”
Preparedness and Response—Bioterrorism Agents/Diseases.” Avail-
Risk Analysis 4(3):177-186.
able at www.bt.cdc.gov/agent/agentlist-category.asp. Accessed April
Stein, M. 1987. “Large Sample Properties of Simulation Using Latin Hy-
13, 2007.
percube Sampling.” Technometrics 29(2):143-151.
DHS (Department of Homeland Security). 2006. Bioterrorism Risk Assess-
U.S. Nuclear Regulatory Commission. 1975. Reactor Safety Study: Assess-
ment. Biological Threat Characterization Center of the National Biode-
ment of Accident Risk in U.S. Commercial Nuclear Plants. WASH-1400,
fense Analysis and Countermeasures Center. Fort Detrick, Md.
NUREG-75/014. Washington, D.C.
DHS. 2007a. “A Lexicon of Risk Terminology and Methodological Descrip-
U.S. Nuclear Regulatory Commission. 1991. Severe Accident Risks: An As-
tion of the DHS Bioterrorism Risk Assessment.” Draft document dated
sessment for Five U.S. Nuclear Power Plants, NUREG-1150, January.
February 2, 2007. Washington, D.C.
Available at www.nrc.gov/reading-rm/doc-collections/nuregs/staff/.
DHS. 2007b. “Example of Risk Assessment Calculations.” Document
Accessed February 22, 2007.
provided to the Committee on Methodological Improvements to the
U.S. Nuclear Regulatory Commission. 1994. A Review of NRC Staff Uses
Department of Homeland Security’s Biological Agent Risk Analysis.
of Probabilistic Risk Assessment, NUREG-1489, March. Washington,
February 26, 2007. Washington D.C.
D.C.
DHS. 2007c. “A Lexicon of Risk Terminology and Methodological De-
The White House. 2007. Homeland Security Presidential Directive 18
scription of the DHS Bioterrorism Risk Assessment.” Written com-
[HSPD-18]: Medical Countermeasures Against Weapons of Mass
munication to the Committee on Methodological Improvements to the
Destruction. Available at www.fas.org/irp/offdocs/nspd/hspd-18.html.
Department of Homeland Security’s Biological Agent Risk Analysis.
Accessed January 16, 2008.
April 14, 2007.