PAPERBACK
\$39.75

### Appendix IReview of BTRA Modeling

Alan R. Washburn, Ph.D.

Distinguished Professor Emeritus of Operations Research

July 10, 2007

MEMORANDUM FOR THE NATIONAL ACADEMY OF SCIENCES (NAS)

Review of the Department of Homeland Security (2006) work on bioterrorism.

Background. The Department of Homeland Security (DHS) has produced a 2006 bioterrorism study, and is working on subsequent versions. DHS has asked NAS to assess the 2006 work, which I will refer to hereafter as “the 2006 work.” I have become acquainted with the work through contacts with the NAS committee, and have been invited to provide a review. This is the review. It is intended for a scientific audience, so I will not hesitate to use the language of probability in describing what I think was done in 2006, or in how things might be handled differently in the future. Random variables are uppercase symbols, P() and E() are the probability and expected value functions, respectively.

My Qualifications. After working five years for the Boeing Company, I joined the Operations Research faculty at the Naval Postgraduate School in 1970, where I did the usual academic things until retiring in 2006. My teaching includes probability and decision theory, which are relevant here. See my resume at http://www.nps.navy.mil/orfacpag/resumePages/washbu.htm for details. I have no biological or medical qualifications. My acquaintance with the work is mainly through the references listed at the end of this review.

Event Trees. The fundamental idea behind the 2006 work is an event tree. As I will use the term in this review, an event tree is a branching structure whose root corresponds to the assertion that some event has occurred, the event in this case being what I will call an “incident.” The tree branches repeatedly until a “scenario” is encountered, at which point one will find a probability distribution that determines the consequence of the incident, a random variable that I will call Y. I think of consequences as being “lives lost,” but any other scalar measure would do. Each node of the tree has a set of successor arcs, and there is a given probability distribution over these arcs. One can imagine starting at the root and randomly selecting an arc at each node encountered until finally the consequence is determined. In addition to Y, the event tree involved in the 2006 work is such that every path from root to consequence also defines two other random variables:

• A, the biological agent, one of 28 possibilities, and

• S, the scenario.

The scenario might be null in the sense that Y is 0 because the incident is terminated prematurely, but is nonetheless always defined.

DHS determines the consequence distributions through Monte Carlo simulation based on expert input. The results are collected into decade-width histograms. I will not comment further on the methodology for producing the consequence distributions, since I have not examined it in detail.

DHS has modified the above definition of an event tree in three senses. One is that the initial branches from the root are rates, rather than probabilities. Call the rate on branch i λi, and let the sum of all of these rates be λ. If one interprets these rates as independent Poisson rates of the various kinds of incident, then it is equivalent to think of incidents as occurring in a Poisson process with rate λ, with each incident being of type i with probability λi/λ. These ratios can be the first set of branch probabilities, so this is all equivalent to the standard event tree definition, except that we must remember that incidents occur at the given rate λ. This first modification is thus of little import.

The second modification is that an incident might involve multiple attacks, each with separate consequences. This is a more significant modification, and will be discussed separately below.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 122
Appendix I Review of BTRA Modeling Alan R. Washburn, Ph.D. Distinguished Professor Emeritus of Operations Research Naval Postgraduate School, Monterey, California July 10, 2007 consequence of the incident, a random variable that I will call Y. I think of consequences as being “lives lost,” but MEMORANDUM FOR THE NATIONAL ACADEMY OF any other scalar measure would do. Each node of the tree SCIENCES (NAS) has a set of successor arcs, and there is a given probability distribution over these arcs. One can imagine starting at the Review of the Department of Homeland Security (2006) root and randomly selecting an arc at each node encountered work on bioterrorism. until finally the consequence is determined. In addition to Y, the event tree involved in the 2006 work is such that every Background. The Department of Homeland Security (DHS) path from root to consequence also defines two other random has produced a 2006 bioterrorism study, and is working on variables: subsequent versions. DHS has asked NAS to assess the 2006 work, which I will refer to hereafter as “the 2006 work.” I • A, the biological agent, one of 28 possibilities, and have become acquainted with the work through contacts • S, the scenario. with the NAS committee, and have been invited to provide a review. This is the review. It is intended for a scientific audi- The scenario might be null in the sense that Y is 0 because ence, so I will not hesitate to use the language of probability the incident is terminated prematurely, but is nonetheless in describing what I think was done in 2006, or in how things always defined. might be handled differently in the future. Random variables DHS determines the consequence distributions through are uppercase symbols, P() and E() are the probability and Monte Carlo simulation based on expert input. The results expected value functions, respectively. are collected into decade-width histograms. I will not com- ment further on the methodology for producing the conse- My Qualifications. After working five years for the Boeing quence distributions, since I have not examined it in detail. Company, I joined the Operations Research faculty at the DHS has modified the above definition of an event tree Naval Postgraduate School in 1970, where I did the usual aca- in three senses. One is that the initial branches from the root demic things until retiring in 2006. My teaching includes prob- are rates, rather than probabilities. Call the rate on branch i λi, and let the sum of all of these rates be λ. If one interprets ability and decision theory, which are relevant here. See my resume at http://www.nps.navy.mil/orfacpag/resumePages these rates as independent Poisson rates of the various kinds /washbu.htm for details. I have no biological or medical quali- of incident, then it is equivalent to think of incidents as oc- curring in a Poisson process with rate λ, with each incident fications. My acquaintance with the work is mainly through being of type i with probability λi/λ. These ratios can be the the references listed at the end of this review. first set of branch probabilities, so this is all equivalent to the Event Trees. The fundamental idea behind the 2006 work is standard event tree definition, except that we must remember that incidents occur at the given rate λ. This first modification an event tree. As I will use the term in this review, an event tree is a branching structure whose root corresponds to the is thus of little import. assertion that some event has occurred, the event in this The second modification is that an incident might involve case being what I will call an “incident.” The tree branches multiple attacks, each with separate consequences. This is repeatedly until a “scenario” is encountered, at which point a more significant modification, and will be discussed sepa- one will find a probability distribution that determines the rately below. 22

OCR for page 122
2 APPENDIX I The third and most significant modification is that the However, summing to 1 is not sufficient for the SME marginals to be meaningful. This is most obvious when N = branching probabilities (DHS on occasion also calls them “branch fractions”) are not fixed, but are instead themselves 2. If the first branch has probability A, then the second must have probability 1 - A, and therefore the second probability determined by sampling from beta distributions provided indirectly by Subject Matter Experts (SMEs). Let θ be the distribution has no choice but to be the mirror image of the first. If the experts feel that the first marginal has α = 1 and collection of branching probabilities. In each incident we therefore observe (θ, A, S, Y), with θ determining the event b = 1, while the second has α = 2 and b = 2, then we must tree for the other three random variables. This modification explain to the experts that what they are saying is meaning- will also be discussed separately below. less, even though both marginals have a mean of 0.5. The second marginal has no choice but to be the mirror image The Second Modification: Repeated Attacks per Incident. of the first, and must therefore be the first, by symmetry. The vision is that a cell or group of terrorists will not plan Any other possibility is literally meaningless, since there is a single attack, but will plan to continue to attack until no pair of random variables (A1, A2) such that Ai has the ith marginal distribution and also A1 + A2 is always exactly 1. interrupted, with the entire group of attacks constituting I think DHS recognizes the difficulty when N = 2, and has an incident. The effect of this is to change the distribution of consequences of an incident, since a successful attack basically fixed it in that case by asking the SMEs for only one marginal, but the same difficulty is present for N > 2, will be accompanied by afterattacks, the number of which I will call X. I believe that the formula used for calculating and has not been fixed. The sampling procedure offered on E(X) is incorrect. Specifically, let λ′ be the probability that page C-81 of Department of Homeland Security (2006) will any one of the afterattacks will succeed, assume that after- reliably produce probabilities A1, …, AN that sum to 1, and attacks continue until one of them fails, and assume that the which are correct on the average, but they do not have the failed afterattack terminates the process and itself has no marginal beta distributions given by the SMEs. This is most consequences. Then the average value of X is E(X) = λ′/(1 - obvious in the case of the last branch, since the Nth marginal λ′), the mean of a geometric-type random variable. This is is never used in the sampling process, but I believe that the not the formula in use. Using the correct formula would be marginal distribution is correct only for the first branch. a simple enough change, but I believe the numerical effect There is a multivariable distribution (the Dirichlet distri- might be significant. bution) whose marginals are all beta distributions, but the Dirichlet distribution has only N + 1 parameters. The SME Other changes may also be necessary to implement the original vision. If the afterattacks all have independent con- marginals require 2N, in total, so the Dirichlet distribution is sequences, then the distribution of total consequences is the not a satisfactory joint distribution for A1, …, AN. (1 + X)-fold convolution of the consequence distribution, a Estimation of the Spread in Agent-Damage Charts. I have complicated operation that I see no evidence of. The docu- mentation is mute on what is actually assumed about the defined Y to be the consequence and A to be the agent. Define Ya to be the consequence if A = a, or otherwise 0, so that the independence of after attacks, and on how the E(X) computa- tion is actually used. Simply scaling up the consequences of 28 random variables Ya sum to Y. Most of the DHS output one attack by the factor (1 + E(X)) is correct on the average, deals with the random variable E(Ya | θ), the expected conse- regardless of independence assumptions, but will not give quence contribution from agent a, given the sampled branch probabilities θ. This quantity is random only because of its the correct distribution of total consequences. dependence on θ, the natural variability of Ya having been averaged out. A sample E(Ya | θj), j = 1,…, 500 is produced The Third Modification: “Random Probabilities.” DHS has accommodated SME uncertainty by allowing the branch by Latin Hypercube Sampling (LHS) of the branch prob- probabilities themselves to be random quantities, with the abilities, each sample including the standard average risk ˆ SMEs merely agreeing to a distribution for each probability, computations for the event tree. A sample mean estimate Ya of 500 rather than a specific number. I will refer to each of these ∑ ˆ E(Y ) is then made by Y = (1 / 500 ) E (Y | θ ) . The agents a a a j probability distributions as a “marginal” for its branch. If a j =1 are then sorted in order of decreasing sample mean, and node has N branches, the experts contribute N marginals, one displayed in what I will call “agent-damage” charts showing for each branch. Except at the root, these marginals are all the expected values and spreads as a function of agent. The beta distributions on the interval [0 1], and each therefore sample means are normalized before being displayed, prob- has two parameters, alpha (α) and beta (b). Each of these ably by forcing them to sum to 1. The normalization destroys distributions has a mean, and since the probabilities them- information that is relevant to the decisions being made. I do selves must sum over the branches to 1, the same thing must not know the motivation for doing so. logically be true of the means. The same need not be true The spreads display the epistemic variability due to SME of the SME inputs, but DHS seems to have disciplined the uncertainty about θ, but suppress all of the aleatoric vari- elicitation process so that the SME marginal means actually ability implied by the event tree. If there were no uncertainty do sum to 1. That is true in all of the data that I have seen.

OCR for page 122