Tree Flipping? The process described earlier for generating agent-damage charts may not be a correct statement of what DHS actually did in 2006. The DHS documentation in several places, after describing a single event tree with 17 ranks, states that a separate analysis was actually done for each agent (paragraph C.3.4.2 of Department of Homeland Security , for example). Now, it is possible to end up with the single-tree analysis described earlier by doing that. The essential step is to first calculate P(A = a) for each agent, and then make a new tree where the agent is selected at the root, with the agent selection probabilities on the 28 branches from the root. The second and third ranks of the tree would then be what were originally the first and second, with new probabilities as computed by Bayes’ theorem, and the rest of the tree would be unchanged. Since the agent is at the root of the resulting “flipped” tree, using the flipped tree is in effect doing a separate analysis for each agent. The flipped tree would lead to the same earlier described agent-damage charts—the two trees are stochastically equivalent. But I don’t see the motivation for doing all this extra work in flipping the tree, and I have some concerns about whether the flipping operation was actually done correctly, or done at all.
One concern is that the thing being manipulated is not an ordinary event tree, and there is no reason to expect that beta distributions will remain beta distributions in the flipping process. Of course, the flipping could occur after the tree is instantiated in each of the 500 replications, but that would get to be a lot of work. I doubt if that has been the case.
The documentation is mute about the tree flipping process. I can only hope that the method actually used for producing agent-damage charts is equivalent to analyzing the single event tree as described above.
Suggestions. My main suggestion for future work is that distributions for branch probabilities be abandoned in favor of direct branch probabilities, as in a standard event tree. In other words, keep it simple. SMEs will not be comfortable expressing definite values for the probabilities, but then they are probably not comfortable with expressing definite values for α and β, either. Most people are simply not comfortable quantifying uncertainty. There is very little to be gained by including epistemic uncertainty about the branch probabilities in an analysis like this, and much to be lost in terms of complication. Epistemic uncertainty is not even discussed in most decision theory textbooks. Standard software for handling decision trees would become applicable (event trees are just a special case where there are no decisions) if epistemic uncertainty were not present. There is also standard software for handling influence diagrams, which ought to be considered as an alternative to decision trees. Influence diagram software is sometimes used diagnostically, which might be of use in bioterrorism. One might observe that the agent is known to be anthrax, for example, and instantly recompute the target probabilities based on that known condition.
Another suggestion is to examine the potential for optimization. Given that the basic problem is how to spend money to reduce risk, it is too bad that a problem that simple in structure cannot be posed formally. It is possible that some actions that we might take would be effective for all contagious diseases. This should make them attractive, but the low rank of most contagious diseases individually in the agent-damage charts tends to suppress their attractiveness.
My last suggestion is to report future results in a scientific fashion that can be reviewed by scientists. English is a notoriously imprecise language for describing operations involving chance, so I have repeatedly struggled to understand what was actually done in making my way through the references. As a result, I may well have misinterpreted something above that I hope DHS will correct. If I were reviewing the 2006 work for a journal, my first act would be to send the material back to the authors with a request that it be written up using mathematics embedded in English, instead of just English. I know that DHS has to communicate complicated ideas about risk to laypeople. That task should be in addition to reporting the results scientifically, not a replacement for it.
In summary, my opinion is that the 2006 DHS methodology is not yet the “rigorous and technically sound methodology” demanded by the 2004 Homeland Security Presidential Directive 10: Biodefense for the 21st Century. Let me also add that I consider the report as a whole to be a remarkable accomplishment, given the magnitude of the task and the time available to do it.
References. Materials that I have examined before writing this review include the following:
Department of Homeland Security. 2006. Bioterrorism Risk Assessment. Biological Threat Characterization Center of the National Biodefense Analysis and Countermeasures Center. Fort Detrick, Md.
I have also examined various drafts of the following:
Department of Homeland Security. 2007. “A Lexicon of Risk Terminology and Methodological Description of the DHS Bioterrorism Risk Assessment.” April 16.
Of all the documents, this last one comes closest to the technical appendix that I recommend. It has been of considerable use to me, but even it does not address tree flipping.