Forensic Analysis: Weighing Bullet Lead Evidence (2004)

It is the conclusion of the committee that, in many cases, CABL is a reasonably accurate way of determining whether two bullets could have come from the same compositionally indistinguishable volume of lead. It may thus in appropriate cases provide additional evidence that ties a suspect to a crime, or in some cases evidence that tends to exonerate a suspect. CABL does not, however, have the unique specificity of techniques such as DNA typing to be used as standalone evidence. It is important that criminal justice professionals and juries understand the capabilities as well as the significant limitations of this forensic technique. The value and reliability of CABL will be enhanced if the recommendations set forth in this report are followed.

The major findings and recommendations made by the committee in Chapters 2 through 4 are collected here.

Finding: The current analytical technology used by the FBI—inductively coupled plasma-optical emission spectroscopy (ICP-OES)—is appropriate and is currently the best available technology for the application.

Recommendation: The FBI Laboratory’s analytical protocol should be revised to contain all details of the inductively coupled plasma-optical emission spectroscopy (ICP-OES) procedure and to provide a better basis for the statistics of bullet comparison. Revisions should include:

1. Determining and documenting the precision and accuracy of the ICP-OES method and the concentration range of all seven elements to which the method is applicable.

2. Adding data on the correlation of older neutron activation analysis and more recent ICP-OES results and any additional data that address the accuracy or precision of the method.

3. Writing and documenting the unwritten standard practice for the order of sample analysis.

4. Modifying and validating the digestion procedure to assure that all of the alloying elements and impurities in all samples (soft lead and hard lead) are dissolved without loss.

5. Using a more formal control-chart system to track trends in the procedure’s variability.

6. Defining a mechanism for validation and documentation of future changes.

Recommendation: Because an important source of measurement variation in quality-assurance environments may be the analyst who makes the actual measurements, measurement repeatability (consistency of measurements made by the same analyst) and reproducibility (consistency of measurements made by different analysts) need to be quantified through Gage R & R studies. Such studies should be conducted for Federal Bureau of Investigation (FBI) comparison procedures.

Recommendation: The FBI’s documented analytical protocol should be applied to all samples and should be followed by all examiners for every case.

Recommendation: A formal and documented comprehensive proficiency test of each examiner needs to be developed by the FBI. This proficiency testing should ensure the ability of the analyst to distinguish bullet fragments that are compositionally indistinguishable from fragments with similar but analytically distinguishable composition. Testing could be internal or external (for example, conducted by the National Institute of Standards and Technology), and test results should be maintained and provided as appropriate. Proficiency should be tested regularly.

Recommendation: The FBI should publish the details of its CABL procedure and the research and data that support it in a peer-reviewed journal or at a minimum make its analytical protocol available through some other public venue.

Recommendation: The conclusions in laboratory reports should be expanded to include the limitations of compositional analysis of bullet lead evidence. In particular, a further explanatory comment should accompany the laboratory conclusions to readily portray the limitations of the evidence. Moreover, a section of the laboratory report translating the technical conclusions into language that a jury could understand would greatly facilitate the proper use of this evidence in the criminal

justice system. Finally, measurement data (means and standard deviations) for all of the crime scene bullets and those deemed to match should be included.

Recommendation: The FBI should continue to measure the seven elements As, Sb, Sn, Cu, Bi, Ag, and Cd as stated in the current analytical protocol.

Recommendation: The FBI should evaluate the potential gain from the use of high-performance inductively coupled plasma-optical emission spectroscopy because improvement in analytical precision may provide better discrimination.

Recommendation: The committee recommends that the FBI estimate within-bullet standard deviations on separate elements and correlations for element pairs, when used for comparisons among bullets, through use of pooling over bullets that have been analyzed with the same ICP-OES measurement technique. The use of pooled within-bullet standard deviations and correlations is strongly preferable to the use of within-bullet standard deviations that are calculated only from the two bullets being compared. Further, estimated standard deviations should be charted regularly to ensure the stability of the measurement process; only standard deviations within control-chart limits are eligible for use in pooled estimates.

Recommendation: The committee recommends that the FBI use either the T² test statistic or the successive t-test statistics procedure in place of the 2-SD overlap, range overlap, and chaining procedures. The tests should use pooled standard deviations and correlations, which can be calculated from the relevant bullets that have been analyzed by the FBI Laboratory. Changes in the analytical method (protocol, instrumentation, and technique) will be reflected in the standard deviations and correlations, so it is important to monitor these statistics for trends and, if necessary, to recalculate the pooled statistics.

Recommendation: To confirm the accuracy of the values used to assess the measurement uncertainty (within-bullet standard deviation) in each element, the committee recommends that a detailed statistical investigation using the FBI’s historical dataset of over 71,000 bullets be conducted. To confirm the relative accuracy of the committee’s recommended approaches to those used by the FBI, the cases that match using the committee’s recommended approaches should be compared with those obtained with the FBI approaches, and causes of discrepancies between the two approaches—such as excessively wide intervals from larger-than-expected estimates of the standard deviation, data from specific time periods, or examiners—should be identified. As the FBI adds new bullet data to its 71,000+ data set, it should note matches for future review in the data set, and the statistical procedures used to assess match status.

Recommendation: The FBI’s statistical protocol should be properly documented and followed by all examiners in every case.

Finding: Variations among and within lead bullet manufacturers make any modeling of the general manufacturing process unreliable and potentially misleading in CABL comparisons.

Finding: CABL is sufficiently reliable to support testimony that bullets from the same compositionally indistinguishable volume of lead (CIVL) are more likely to be analytically indistinguishable than bullets from different CIVLs. An examiner may also testify that having CABL evidence that two bullets are analytically indistinguishable increases the probability that two bullets come from the same CIVL, versus no evidence of match status.

Recommendation: Interpretation and testimony of examiners should be limited as described above, and assessed regularly.

Recommendation: Expert witnesses should define the range of “compositionally indistinguishable volumes of lead” (CIVL) that could make up the source of analytically indistinguishable bullets, because of variability in the bullet manufacturing process.

Finding: The committee’s review of the literature and discussions with manufacturers indicates that the size of a CIVL ranges from 70 lbs in a billet to 200,000 lbs in a melt. That is equivalent to 12,000 to 35 million 40-grain, .22 caliber longrifle bullets from a CIVL compared with a total of 9 billion bullets produced each year.

Finding: Although it has been demonstrated that there are a large number of different compositionally indistinguishable volumes of lead (CIVLs), there is evidence that bullets from different CIVLs can sometimes coincidentally be analytically indistinguishable.

Recommendation: The possible existence of coincidentally indistinguishable CIVLs should be acknowledged in the laboratory report and by the expert witness on direct examination.

Finding: Compositional analysis of bullet lead data alone does not permit any definitive statement concerning the date of bullet manufacture.

Finding: Detailed patterns of distribution of ammunition are unknown, and as a result, an expert should not testify as to the probability that a crime scene bullet

came from the defendant. Geographic distribution data on bullets and ammunition are needed before such testimony can be given.

Finding: The available data do not support any statement that a crime bullet came from, or is likely to have come from, a particular box of ammunition, and references to “boxes” of ammunition in any form are seriously misleading under Federal Rule of Evidence 403. Testimony that the crime bullet came from the defendant’s box or from a box manufactured at the same time, is also objectionable because it may be understood as implying a substantial probability that the bullet came from defendant’s box.

This page intentionally left blank.