National Academies Press: OpenBook

Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change (2008)

Chapter: Appendix I: Review of BTRA Modeling

« Previous: Appendix H: Game Theory and Interdependencies
Suggested Citation:"Appendix I: Review of BTRA Modeling." National Research Council. 2008. Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change. Washington, DC: The National Academies Press. doi: 10.17226/12206.
×
Page 122
Suggested Citation:"Appendix I: Review of BTRA Modeling." National Research Council. 2008. Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change. Washington, DC: The National Academies Press. doi: 10.17226/12206.
×
Page 123
Suggested Citation:"Appendix I: Review of BTRA Modeling." National Research Council. 2008. Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change. Washington, DC: The National Academies Press. doi: 10.17226/12206.
×
Page 124
Suggested Citation:"Appendix I: Review of BTRA Modeling." National Research Council. 2008. Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change. Washington, DC: The National Academies Press. doi: 10.17226/12206.
×
Page 125

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix I Review of BTRA Modeling Alan R. Washburn, Ph.D. Distinguished Professor Emeritus of Operations Research Naval Postgraduate School, Monterey, California July 10, 2007 consequence of the incident, a random variable that I will call Y. I think of consequences as being “lives lost,” but MEMORANDUM FOR THE NATIONAL ACADEMY OF any other scalar measure would do. Each node of the tree SCIENCES (NAS) has a set of successor arcs, and there is a given probability distribution over these arcs. One can imagine starting at the Review of the Department of Homeland Security (2006) root and randomly selecting an arc at each node encountered work on bioterrorism. until finally the consequence is determined. In addition to Y, the event tree involved in the 2006 work is such that every Background.  The Department of Homeland Security (DHS) path from root to consequence also defines two other random has produced a 2006 bioterrorism study, and is working on variables: subsequent versions. DHS has asked NAS to assess the 2006 work, which I will refer to hereafter as “the 2006 work.” I • A, the biological agent, one of 28 possibilities, and have become acquainted with the work through contacts • S, the scenario. with the NAS committee, and have been invited to provide a review. This is the review. It is intended for a scientific audi- The scenario might be null in the sense that Y is 0 because ence, so I will not hesitate to use the language of probability the incident is terminated prematurely, but is nonetheless in describing what I think was done in 2006, or in how things always defined. might be handled differently in the future. Random variables DHS determines the consequence distributions through are uppercase symbols, P() and E() are the probability and Monte Carlo simulation based on expert input. The results expected value functions, respectively. are collected into decade-width histograms. I will not com- ment further on the methodology for producing the conse- My Qualifications.  After working five years for the Boeing quence distributions, since I have not examined it in detail. Company, I joined the Operations Research faculty at the DHS has modified the above definition of an event tree Naval Postgraduate School in 1970, where I did the usual aca- in three senses. One is that the initial branches from the root demic things until retiring in 2006. My teaching includes prob- are rates, rather than probabilities. Call the rate on branch i ability and decision theory, which are relevant here. See my λi, and let the sum of all of these rates be λ. If one interprets resume at http://www.nps.navy.mil/orfacpag/resumePages these rates as independent Poisson rates of the various kinds /washbu.htm for details. I have no biological or medical quali- of incident, then it is equivalent to think of incidents as oc- fications. My acquaintance with the work is mainly through curring in a Poisson process with rate λ, with each incident the references listed at the end of this review. being of type i with probability λi/λ. These ratios can be the first set of branch probabilities, so this is all equivalent to the Event Trees.  The fundamental idea behind the 2006 work is standard event tree definition, except that we must remember an event tree. As I will use the term in this review, an event that incidents occur at the given rate λ. This first modification tree is a branching structure whose root corresponds to the is thus of little import. assertion that some event has occurred, the event in this The second modification is that an incident might involve case being what I will call an “incident.” The tree branches multiple attacks, each with separate consequences. This is repeatedly until a “scenario” is encountered, at which point a more significant modification, and will be discussed sepa- one will find a probability distribution that determines the rately below. 122

APPENDIX I 123 The third and most significant modification is that the However, summing to 1 is not sufficient for the SME branching probabilities (DHS on occasion also calls them marginals to be meaningful. This is most obvious when N = “branch fractions”) are not fixed, but are instead themselves 2. If the first branch has probability A, then the second must determined by sampling from beta distributions provided have probability 1 - A, and therefore the second probability indirectly by Subject Matter Experts (SMEs). Let θ be the distribution has no choice but to be the mirror image of the collection of branching probabilities. In each incident we first. If the experts feel that the first marginal has α = 1 and therefore observe (θ, A, S, Y), with θ determining the event β = 1, while the second has α = 2 and β = 2, then we must tree for the other three random variables. This modification explain to the experts that what they are saying is meaning- will also be discussed separately below. less, even though both marginals have a mean of 0.5. The second marginal has no choice but to be the mirror image The Second Modification: Repeated Attacks per Incident. of the first, and must therefore be the first, by symmetry. The vision is that a cell or group of terrorists will not plan Any other possibility is literally meaningless, since there is a single attack, but will plan to continue to attack until no pair of random variables (A1, A2) such that Ai has the ith interrupted, with the entire group of attacks constituting marginal distribution and also A1 + A2 is always exactly 1. an incident. The effect of this is to change the distribution I think DHS recognizes the difficulty when N = 2, and has of consequences of an incident, since a successful attack basically fixed it in that case by asking the SMEs for only will be accompanied by afterattacks, the number of which one marginal, but the same difficulty is present for N > 2, I will call X. I believe that the formula used for calculating and has not been fixed. The sampling procedure offered on E(X) is incorrect. Specifically, let λ′ be the probability that page C-81 of Department of Homeland Security (2006) will any one of the afterattacks will succeed, assume that after­ reliably produce probabilities A1, …, AN that sum to 1, and attacks continue until one of them fails, and assume that the which are correct on the average, but they do not have the failed afterattack terminates the process and itself has no marginal beta distributions given by the SMEs. This is most consequences. Then the average value of X is E(X) = λ′/(1 − obvious in the case of the last branch, since the Nth marginal λ′), the mean of a geometric-type random variable. This is is never used in the sampling process, but I believe that the not the formula in use. Using the correct formula would be marginal distribution is correct only for the first branch. a simple enough change, but I believe the numerical effect There is a multivariable distribution (the Dirichlet distri- might be significant. bution) whose marginals are all beta distributions, but the Other changes may also be necessary to implement the Dirichlet distribution has only N + 1 parameters. The SME original vision. If the afterattacks all have independent con- marginals require 2N, in total, so the Dirichlet distribution is sequences, then the distribution of total consequences is the not a satisfactory joint distribution for A1, …, AN. (1 + X)-fold convolution of the consequence distribution, a complicated operation that I see no evidence of. The docu- Estimation of the Spread in Agent-Damage Charts.  I have mentation is mute on what is actually assumed about the defined Y to be the consequence and A to be the agent. Define independence of after attacks, and on how the E(X) computa- Ya to be the consequence if A = a, or otherwise 0, so that the tion is actually used. Simply scaling up the consequences of 28 random variables Ya sum to Y. Most of the DHS output one attack by the factor (1 + E(X)) is correct on the average, deals with the random variable E(Ya | θ), the expected conse- regardless of independence assumptions, but will not give quence contribution from agent a, given the sampled branch the correct distribution of total consequences. probabilities θ. This quantity is random only because of its dependence on θ, the natural variability of Ya having been The Third Modification: “Random Probabilities.”  DHS averaged out. A sample E(Ya | θj), j = 1,…, 500 is produced has accommodated SME uncertainty by allowing the branch by Latin Hypercube Sampling (LHS) of the branch prob- probabilities themselves to be random quantities, with the abilities, each sample including the standard average risk SMEs merely agreeing to a distribution for each probability, ˆ computations for the event tree. A sample mean estimate Ya of 500 rather than a specific number. I will refer to each of these probability distributions as a “marginal” for its branch. If a ˆ a ∑ E(Y ) is then made by Y = (1 / 500 ) E (Y | θ ) . The agents a j =1 a j node has N branches, the experts contribute N marginals, one are then sorted in order of decreasing sample mean, and for each branch. Except at the root, these marginals are all displayed in what I will call “agent-damage” charts showing beta distributions on the interval [0 1], and each therefore the expected values and spreads as a function of agent. The has two parameters, alpha (α) and beta (β). Each of these sample means are normalized before being displayed, prob- distributions has a mean, and since the probabilities them- ably by forcing them to sum to 1. The normalization destroys selves must sum over the branches to 1, the same thing must information that is relevant to the decisions being made. I do logically be true of the means. The same need not be true not know the motivation for doing so. of the SME inputs, but DHS seems to have disciplined the The spreads display the epistemic variability due to SME elicitation process so that the SME marginal means actually uncertainty about θ, but suppress all of the aleatoric vari- do sum to 1. That is true in all of the data that I have seen. ability implied by the event tree. If there were no uncertainty

124 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT about θ, all of the spreads would collapse to a single point long as i and j are not the same, which is all that is required (the mean) for each agent. I am not sure how the variability for my conclusion to be true. While it is certainly true that displayed in agent-damage charts is supposed to relate to the branches chosen at nodes i and j are in general dependent, decision making, but I guess that the graphs are intended to the branch probabilities are not.) support conclusions such as the following: “I know that the mean damage for agent 1 is larger then the mean damage for Use of SMEs.  It is inevitable in a project like this that agent 2, but I still think that we ought to spend our money probabilities will have to be obtained from Subject Matter defending against agent 2 because of its high associated vari- Experts, rather than experimentation. The important thing is ability. Even a small prospect of the high damages associated that the SMEs at least know what they are estimating, and with agent 2 is not acceptable.” If that is the kind of logic that that estimates be used correctly once they are obtained. I the agent-damage charts are intended to support, then they have already mentioned that SME estimates of the marginal should include aleatoric variability. Without it, the spreads branch distributions are not reproduced by the sampling pro- associated with each agent are too small. This issue affects cedure. Another concern is at the third stage of the event tree, infectious agents more than the other kind, since infectious where SMEs are asked to deal with agent selection. At that diseases will have especially high damage variances. stage there are 4 × 8 = 32 nodes in the event tree where an The agent-damage charts are intended for a high level of agent might be selected, each of which has 28 branches. I can decision-making audience, and devote considerable space certainly understand DHS’s reluctance to conduct 896 inter- (one of the two available dimensions) to showing the spread views with SMEs, each to determine one of the needed beta associated with each agent. Without the need to show spread, distributions. Some kind of a shortcut is needed, but I wonder they could be replaced by bar charts or simple tables. If whether the one adopted is a good one. The SMEs are first spread is important enough to be displayed, then it ought to asked to determine an “input regarding known preferences of be displayed in a manner that facilitates good decisions. I terrorists” for each agent. If I were an SME and somebody doubt that that is currently the case. asked me to determine the quoted expression for agent a, I Even without the aleatoric issue, I still have concerns would announce my estimate of P(A = a), the probability about the spread that is displayed. The object ought to be that agent a is actually selected in an incident. Given all of to display the mean and fractiles (the spread) of the random these SME inputs, DHS then goes over the 896 branches, variable E(Ya | θ) for each value of a. The mean of E(Ya | θ) some of which have a logical 0 for the agent, and assigns is simply E(Ya) by the conditional expectation theorem, and probabilities using the rule that the probability is either 0 or ˆ is estimated by Ya. DHS claims graphically that the LHS else proportional to the SME’s agent input, the proportional- sample fractiles are also the fractiles of the random variable ity constant being selected in each of the 32 cases so that the E(Ya | θ). I suspect that this claim is false. LHS is basically probabilities sum to 1. My objections are that a variance reduction technique that makes the variance of ˆ Ya smaller than it would be with ordinary sampling. While • The quoted expression above does not make it clear that this effect is welcome, LHS also has an unpredictable effect the SME input is supposed to be P(A = a). There is a on variability. The spread that is shown for each agent may danger of every SME making a different interpretation not be a good estimate of the spread of the random variable of what is being asked for. E(Ya | θ). • If the SME does input the probabilities P(A = a), and if One final point on estimation. As long as there is no de- DHS applies the shortcut procedure to fill out the third pendence between the branch probabilities at different nodes, stage of the event tree, and if the probabilities of the 28 as there is not in the 2006 work, it is characteristic of an event agents are then computed from the tree, they will not tree that P(Ya ≤ y) = E(P(Ya ≤ y | θ)) = P(Ya ≤ y | E(θ)). The first necessarily agree with the SME’s inputs. This would equality is due to the conditional expectation theorem, and be true even without my next objection. the second is because no event tree probability enters more • The SME’s inputs are subsequently modified by vari- than once into calculating the probability of any scenario. ous formulas involving agent lethality, etc. What is an In other words, all information pertinent to the distribution SME who is already acquainted with agent lethality of Ya could be obtained without sampling error by simply to think of this? Should he adjust his input so that the replacing the marginal branch distributions by their means. net result of all this computation is the number that he This information includes E(Ya), which is currently being wanted in the first place? If one is going to elicit SME ˆ estimated (with sampling error) by Ya. (Note added in June inputs on probabilities, then it seems to me that one 2007. Let me expand the notation to clarify this final point, ought to use them as they are intended. since it has caused some confusion. Let θ = (Q1, …, Qn), where n is the number of nodes and Qi is the collection of Given that the agent probabilities strongly influence the branch probabilities at node i. Also let Qij be the jth branch agent-damage charts, the procedure for eliciting and using probability at node i. In the sampling procedure used by DHS them should be an object of concern in future work. to obtain θ, Qij and Qkl are independent random variables as

APPENDIX I 125 Tree Flipping?  The process described earlier for generat- software is sometimes used diagnostically, which might be ing agent-damage charts may not be a correct statement of of use in bioterrorism. One might observe that the agent is what DHS actually did in 2006. The DHS documentation in known to be anthrax, for example, and instantly recompute several places, after describing a single event tree with 17 the target probabilities based on that known condition. ranks, states that a separate analysis was actually done for Another suggestion is to examine the potential for op- each agent (paragraph C.3.4.2 of Department of Homeland timization. Given that the basic problem is how to spend Security [2006], for example). Now, it is possible to end money to reduce risk, it is too bad that a problem that simple up with the single-tree analysis described earlier by doing in structure cannot be posed formally. It is possible that that. The essential step is to first calculate P(A = a) for each some actions that we might take would be effective for all agent, and then make a new tree where the agent is selected contagious diseases. This should make them attractive, but at the root, with the agent selection probabilities on the 28 the low rank of most contagious diseases individually in the branches from the root. The second and third ranks of the agent-damage charts tends to suppress their attractiveness. tree would then be what were originally the first and second, My last suggestion is to report future results in a scientific with new probabilities as computed by Bayes’ theorem, and fashion that can be reviewed by scientists. English is a notori- the rest of the tree would be unchanged. Since the agent is ously imprecise language for describing operations involving at the root of the resulting “flipped” tree, using the flipped chance, so I have repeatedly struggled to understand what tree is in effect doing a separate analysis for each agent. The was actually done in making my way through the references. flipped tree would lead to the same earlier described agent- As a result, I may well have misinterpreted something above damage charts—the two trees are stochastically equivalent. that I hope DHS will correct. If I were reviewing the 2006 But I don’t see the motivation for doing all this extra work work for a journal, my first act would be to send the material in flipping the tree, and I have some concerns about whether back to the authors with a request that it be written up using the flipping operation was actually done correctly, or done mathematics embedded in English, instead of just English. I at all. know that DHS has to communicate complicated ideas about One concern is that the thing being manipulated is not an risk to laypeople. That task should be in addition to reporting ordinary event tree, and there is no reason to expect that beta the results scientifically, not a replacement for it. distributions will remain beta distributions in the flipping In summary, my opinion is that the 2006 DHS methodol- process. Of course, the flipping could occur after the tree is ogy is not yet the “rigorous and technically sound methodol- instantiated in each of the 500 replications, but that would get ogy” demanded by the 2004 Homeland Security Presidential to be a lot of work. I doubt if that has been the case. Directive 10: Biodefense for the 21st Century. Let me also The documentation is mute about the tree flipping pro- add that I consider the report as a whole to be a remarkable cess. I can only hope that the method actually used for pro- accomplishment, given the magnitude of the task and the ducing agent-damage charts is equivalent to analyzing the time available to do it. single event tree as described above. References.  Materials that I have examined before writing Suggestions.  My main suggestion for future work is that this review include the following: distributions for branch probabilities be abandoned in favor of direct branch probabilities, as in a standard event tree. In Department of Homeland Security. 2006. Bioterrorism Risk other words, keep it simple. SMEs will not be comfortable Assessment. Biological Threat Characterization Center of the expressing definite values for the probabilities, but then they National Biodefense Analysis and Countermeasures Center. are probably not comfortable with expressing definite values Fort Detrick, Md. for α and β, either. Most people are simply not comfortable quantifying uncertainty. There is very little to be gained by I have also examined various drafts of the following: including epistemic uncertainty about the branch probabili- ties in an analysis like this, and much to be lost in terms of Department of Homeland Security. 2007. “A Lexicon of complication. Epistemic uncertainty is not even discussed in Risk Terminology and Methodological Description of the most decision theory textbooks. Standard software for han- DHS Bioterrorism Risk Assessment.” April 16. dling decision trees would become applicable (event trees are just a special case where there are no decisions) if epistemic Of all the documents, this last one comes closest to the tech- uncertainty were not present. There is also standard software nical appendix that I recommend. It has been of considerable for handling influence diagrams, which ought to be consid- use to me, but even it does not address tree flipping. ered as an alternative to decision trees. Influence diagram

Next: Appendix J: Reprinted Interim Report »
Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The mission of Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change, the book published in December 2008, is to independently and scientifically review the methodology that led to the 2006 Department of Homeland Security report, Bioterrorism Risk Assessment (BTRA) and provide a foundation for future updates.

This book identifies a number of fundamental concerns with the BTRA of 2006, ranging from mathematical and statistical mistakes that have corrupted results, to unnecessarily complicated probability models and models with fidelity far exceeding existing data, to more basic questions about how terrorist behavior should be modeled.

Rather than merely criticizing what was done in the BTRA of 2006, this new NRC book consults outside experts and collects a number of proposed alternatives that could improve DHS's ability to assess potential terrorist behavior as a key element of risk-informed decision making, and it explains these alternatives in the specific context of the BTRA and the bioterrorism threat.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!