National Academies Press: OpenBook

Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters (2011)

Chapter: Appendix C: Committee Evaluation of Statistical Analysis Report

« Previous: Appendix B: The Forensics Potential of Stable Isotope Analysis
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

Appendix C

Committee Evaluation of Statistical Analysis Report

The Statistical Analysis Report (B2M10) was submitted in response to a contract with the FBI, to analyze the results of the assays on the 1,070 FBI Repository (FBIR) samples and determine whether the results of the assays could be related to those obtained on the evidentiary material. If such a relationship could be identified, a secondary issue was the development of a measure of its “statistical strength.” Two of the attack letters assayed positive for the four mutations A1, A3, D, and E. Results on 1,059 of the 1,070 samples were tabulated in the Statistical Analysis Report. Eight samples tested positive for all four mutations; seven of these eight samples came from one institution (USAMRIID) and the remaining sample came from a different institution (Battelle Memorial Institute [BMI]). A table of documented transfers of samples from one institution to another showed a transfer of sample material from the first institution (USAMRIID) to the second institution (BMI). This Appendix discusses the validity of the inferences and calculations in the Statistical Analysis Report submitted to the FBI.

As noted in Chapter 6, the statistical analyses used in the report (e.g., 95 percent confidence interval for the proportion of samples with four mutations, chi-squared tests of independence) require two key assumptions to be valid:

1.    Representativeness: The 1,059 samples are assumed to be a representative and random collection of samples from some well-defined population of samples.

2.    Independence: The 1,059 samples are assumed to be independent of one another (i.e., have no connection with each other, beyond that they all come from the same population).

The Statistical Analysis Report acknowledges that neither assumption can be validated from these data. The committee agrees with this assessment. As a consequence, many of the statistical methods applied to these data cannot be

Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

validated. The consequences of the violation of these assumptions and their impacts are listed below.

1.    FBIR is not a representative and random collection of samples from a well-defined population of B. anthracis samples.

1,059 samples do not appear to satisfy assumption 1. They were obtained in response to a request from the FBI. No information is available on samples in the population that were not submitted. In fact, the “target population’’ seems not to have been defined. It could be the population of all unique preparations of B. anthracis Ames in the United States, or in the world, or from selected institutions. The absence of a definition of “well-defined population” makes it difficult to assess representativeness of the collection. The elimination of samples that had “inconclusive” results on assays also appears to be nonrandom, as some institutions had many more “inconclusive” assays than others.

2.    The 1,059 samples in the FBIR are not independent.

FBI submitted to the committee a table of known transfers of samples between institutions. Hence, the second assumption is violated. Thus, the results of the chi-squared tests for independence of the mutations that are calculated in the report are not meaningful. Further, the confidence interval for the proportion 8/947 is not appropriate. The correct denominator for this proportion is likely not 947. A more accurate numerator and denominator might refer to the number of known independent preparations rather than the number of samples, but such information may not be possible to obtain.

3.    Violation of assumptions renders invalid the inferences from the statistical analyses.

the FBIR is not a representative and random collection of independent samples, the results on the assays from the repository may be biased. Virtually all statistical procedures assume that the units on which measurements are made comprise a random, representative collection from the target population. (The effects of biased sampling on inferences have been well documented; see, e.g., Freedman et al., 2007). Without an appropriate model that characterizes the nonrepresentativeness and the degree of dependence among the samples, it is not possible to calculate a meaningful measure of “statistical significance” in the results.

Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

4.    Results on 112 samples beyond the 947 samples

Statistical Analysis Report eliminated from most of its tables the results of the assays on 112 samples that showed “inconclusive” for A1, A3, MRI-D, or E. Twenty-one of these 112 sample that were eliminated from the statistical analysis assayed positive for 1, 2, or 3 mutations. Table C-1 lists these samples. (Five samples—05-022, 49-014, 53-014, 53-068, 54-008—are listed twice because they were reported as “inconclusive” or “variant” on two assays.)

TABLE C-1 Samples with Positive and “Inconclusive” or “Variant” Assays

FBIR Number A1 A3 MRI-D IITRI-D E +Mutations
039-010 inc + A3
044−034 var + IITRI−D
049−014 inc var + + MRI−D, IITRI−D
053−004 var + + A3, E
053−010 var + + + + A3, MRI−D, IITRI−D, E
053−014 var inc + E
053−068 inc inc + MRI−D
054−008 inc + + inc + A3, MRI−D, E
061−030 inc + + MRI−D, IITRI−D
066−015 inc inc + + MRI−D, IITRI−D
005−022 + var inc + A1, E
017−006 var + + MRI−D, IITRI−D
049−014 inc var + + MRI−D, IITRI−D
049−018 var + + MRI−D, ITRI−D
053−014 var inc + E
053−068 inc inc + MRI−D
054−066 + inc + + + A1, MRI−D, IITRI−D, E
054−068 var + + MRI−D, E
005−020 + inc MRI−D
005−022 + var inc + A1, E
043−016 + inc MRI−D
044−020 + inc + A3, E
052−026 + + + inc A1, A3, MRI−D
054−008 inc + + inc + A3, MRI−D, E
054−022 + inc MRI−D
057−036 + inc MRI−D
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

inc = inconclusive

IITRI = Illinois Institute for Technology Research Institute

MRI = Midwest Research Institute

var = variant

In addition to the two 3-positive samples (+++) among the 947 samples, the four samples below also tested positive for 3 mutations (ordered by FBIR number):

052-026 + + + inc A1, A3, MRI-D
053-010 var + + + + A3, MRI-D, IITRI-D, E
054-008 inc + + inc + A3, MRI-D, E
054-066 + Inc + + + A1, MRI-D, IITRI-D, E

The following four samples revealed positive assays for 2 of the 4 mutations, in addition to the 11 samples noted among the 947 samples (ordered by FBIR number):

005-022 + var inc + A1, E
044-020 + inc + A3, E
053-004 var + + A3, E
054-068 var + + MRI-D, E

DILUTION EXPERIMENTS

Dilution experiments were conducted to assess the sensitivity of the assays to various concentrations. Thirty samples were prepared from RMR-1029 at dilution 10.0. As with the other samples, some of the assays were “inconclusive.” Genotype E tested positive in all 30 samples; all 4 mutations tested positive for 16 samples. But in the remaining 14 samples, assays for one or more of the genotypes were negative. In fact, one sample tested negative for A1, A3, and D; it was positive for only E. Five samples were positive for two mutations only (A3 and E), and eight samples were positive for only three of the four mutations (7 for A3, D, E; 1 for A1, A3, E). Thus, 6 of the 30 replicate samples (20 percent) tested positive for only 1 or 2 of the mutations. Given that 50 of the 947 FBIR samples showed only 1 positive, and 11 of the 947 showed only 2 positives, this variation indicates that some of the samples may have harbored mutations that went undetected. Absent any repeat testing of these samples, however, it is difficult to know how such false negatives might have affected the inferences.

Additional experiments were conducted on RMR-1029 and another sample, “SPS.266 Tube#5,” at 10 dilutions levels (10.1, …, 10.10). The results of the

Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

three replicates at each dilution level, for each of the five genotypes, for samples from both RMR-1029 and SPS.266 Tube#5 were reported in Chapter 6. Variability in the results on replicates, even from the same sample at the same dilution level, demonstrates the value, and need for, replicate testing. For example, the results on the three replicates from RMR-1029 at dilution 10.1, ordered as A1, A3, MRI-D, IITRI-D, E, were: (- + + + +), (- + + + -), (+ + + + -). Clearly, dilution affects the assay result: the greater the dilution, the more likely the assay is negative. Moreover, it is perhaps unexpected that greater dilutions sometimes give positive results when not all replicates at lesser dilutions did so.

CONCORDANCE OF TESTS FROM IITRI-D AND MRI-D

The FBI retained both the Illinois Institute for Technology Research Institute (IITRI) and Midwest Research Institute (MRI) to conduct the D assays. Because the assays on the 1,059 samples can be considered to be independent between IITRI and MRI, the Statistical Analysis Report (Table 3, p. 7, as presented below) tabulates the results of the D assays from the two facilities:

IITRI-D MRI-D
Inconclusive Negative No growth Pending Positive Total
Inconclusive 0 22 12 0 0 34
Negative 17 909 1 1 12 940
Negative-u 1 20 0 0 0 21
Positive 6 12 0 0 46 64
TOTAL 24 963 13 1 58 1,059

The Statistical Analysis Report combined the “negative-u” results with the “negative” results, and eliminated the 12 samples that showed “no-growth” by IITRI-D and “inconclusive” by MRI-D as well as the one “pending” sample, to yield the following table:

IITRI-D MRI-D
Inconclusive Negative Positive Total
Inconclusive 0 22 0 22
Negative 18 929 12 959
Positive 6 12 46 64
TOTAL 24 963 58 1,045

Eliminating the 14 “no-growth” and “pending” samples, the concordance rate is 975/1045 = 0.933, with a 95 percent confidence interval (0.916, 0.947). Thus, the agreement between the facilities is unlikely to be lower than 91.6 per-

Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

cent and likely does not exceed 94.7 percent. Of greater interest, however, are the 12 samples that were positive by IITRI-D but negative for MRI-D, the 12 samples that were negative by IITRI-D but positive by MRI-D, and the six samples that were positive by MRI-D but inconclusive by IITRI-D. While concordance is informative, these 30 samples with discordant results might provide increased information about the samples and the assay process. On the other hand, we also know from the repeated assays of the dilution series that some discordance also arises owing to variation even when using the same assay procedure.

In any case, because genotype D is the only one of the four genotypes that was subjected to independent testing by a second organization one cannot say whether the results on the other genotypes might have been different if they also had been subjected to independent testing.

“SIGNIFICANCE” OF SEVEN (++++) SAMPLES FROM INSTITUTION F

The Statistical Analysis Report notes in its conclusions:

“In summation, though the random chance of occurrence of the sample type (++++) is 8 out of 947 (i.e., 0.84%) with exact 95% confidence interval of 0.0037 to 0.0166 (I.e., from 1 in 270 to 1 in 60), this sample type has been found in only two institutions thus far sampled (USAMRIID and BMI), and its occurrence in BMI is explained by a recent sample transfer from USAM to BMI, since there is no documented record of sample transfers in the other direction.’’ (p. 2)

As noted in Chapter 6, 598 of the 947 samples (63 percent) came from Institution F. (Twelve of the institutions submitted 6 or fewer samples; 4 institutions submitted 15-31 samples, and 4 institutions submitted 49-74 samples.) Therefore, one would not be surprised to find more “mutation-positive” samples from Institution F than, say, from Institution B (which contributed only one sample). One might naturally ask: How unusual is the occurrence of seven “4-mutation” samples—or even all eight—from Institution F? Given that Institution F contributed almost 2/3 of the 947 samples, how many of the 4-positive (++++) samples would Institution F receive if the 4-mutation samples were distributed completely at random?

The answer to this question is given by the probabilities of observing 0 or 1 or 2 or ... or 8 of the eight (++++) samples from Institution F, given that Institution F submitted 598 of the 947 samples that yielded definitive results on the A1, A3, MRI-D, and E assays. These probabilities (from the hypergeometric probability distribution) (Johnson et al., 2005) are shown in Table C-2.

Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

TABLE C-2 Probabilities of k 4-Mutation Samples in Institution F

k = 0 1 2 3 4 5 6 7 8
Probability 0.0003 0.0045 0.0276 0.0955 0.2058 0.2826 0.2415 0.1174 0.0248

This table shows that the chance of Institution F having ended up with seven or eight of the eight (++++) samples is (0.1174 + 0.0248) = 0.1422, or about 1/7. Therefore, while the observed data showing that seven of the eight (++++) samples appeared in Institution F is not completely typical, it also could hardly be considered extreme.

Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×

This page intentionally left blank.

Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 185
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 186
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 187
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 188
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 189
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 190
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 191
Suggested Citation:"Appendix C: Committee Evaluation of Statistical Analysis Report." National Research Council. 2011. Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters. Washington, DC: The National Academies Press. doi: 10.17226/13098.
×
Page 192
Next: Appendix D: Biographical Information of Committee and Staff »
Review of the Scientific Approaches Used During the FBI's Investigation of the 2001 Anthrax Letters Get This Book
×
Buy Paperback | $49.00 Buy Ebook | $39.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Less than a month after the September 11, 2001 attacks, letters containing spores of anthrax bacteria (Bacillus anthracis, or B. anthracis) were sent through the U.S. mail. Between October 4 and November 20, 2001, 22 individuals developed anthrax; 5 of the cases were fatal.

During its investigation of the anthrax mailings, the FBI worked with other federal agencies to coordinate and conduct scientific analyses of the anthrax letter spore powders, environmental samples, clinical samples, and samples collected from laboratories that might have been the source of the letter-associated spores. The agency relied on external experts, including some who had developed tests to differentiate among strains of B. anthracis. In 2008, seven years into the investigation, the FBI asked the National Research Council (NRC) of the National Academy of Sciences (NAS) to conduct an independent review of the scientific approaches used during the investigation of the 2001 B. anthracis mailings.

Review of the Scientific Approaches Used During the FBI's Investigation of the Anthrax Letters evaluates the scientific foundation for the techniques used by the FBI to determine whether these techniques met appropriate standards for scientific reliability and for use in forensic validation, and whether the FBI reached appropriate scientific conclusions from its use of these techniques. This report reviews and assesses scientific evidence considered in connection with the 2001 Bacillus anthracis mailings.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!