When a crime involves gunfire, examination of physical evidence derived from ammunition often yields key pieces of evidence used in the investigation of that crime. Firearms examination focuses on characteristic marks left on fired bullets and expended cartridge cases by the weapon from which the cartridge is discharged. With bullets, this involves matching the striations on a bullet caused by its passage through the barrel of a gun with marks on test bullets fired through the barrel of a gun found in the possession of a suspect. However, frequently, no gun is recovered, or a bullet fragment is too small or mangled to observe adequate striations. In such instances, a different approach must be explored to evaluate the possibility of a link between the crime scene bullet(s)1 and the suspect.
One such approach is compositional analysis of bullet lead (CABL), which has been used by the law-enforcement community to provide circumstantial evidence for criminal investigation and prosecution since the 1960s. Crime scene investigators and autopsy pathologists collect bullet fragments (and sometimes a bullet in its entirety) from a crime scene or the body of a victim in order to compare them with unused cartridges in the possession of a suspect (suspect’s bullets) that investigators may have collected.
The FBI examiner takes three samples from each bullet or bullet fragment and analyzes them by a process known as inductively coupled plasma-optical emission spectroscopy (ICP-OES). This process is used to determine the concentrations of seven selected elements—arsenic (As), antimony (Sb), tin (Sn), copper (Cu), bismuth (Bi), silver (Ag), and cadmium (Cd)—in the bullet lead alloy of both the
crime-scene and the suspect’s bullets. The FBI examiner applies statistical tests to compare the elements in each crime-scene fragment with the elements in each of the suspect’s bullets. If any of the fragments and suspect’s bullets are determined statistically to be analytically indistinguishable for each of the elemental concentration means, the examiner’s expert court testimony currently will indicate that the fragments and bullets probably came from the same “source.”
The Federal Bureau of Investigation (FBI) asked the National Research Council to conduct an impartial scientific assessment of the soundness of the principles underlying CABL, the optimal manner for conducting an examination with CABL, and the scientifically valid conclusions that can be reached with CABL. In particular, the FBI asked the National Research Council to address the following three subjects and specific questions:
Analytical method. Is the method analytically sound? What are the relative merits of the methods currently available? Is the selection of elements used as comparison parameters appropriate? Can additional useful information be gained by measurement of isotopic compositions?
Statistics for comparison. Are the statistical tests used to compare two samples appropriate? Can known variations in compositions introduced in manufacturing processes be used to model specimen groupings and provide improved comparison criteria?
Interpretation issues. What are the appropriate statements that can be made to assist the requester in interpreting the results of compositional bullet lead comparison, for both indistinguishable and distinguishable compositions? Can significance statements be modified to include effects of such factors as the analytical technique, manufacturing process, comparison criteria, specimen history, and legal requirements?
The committee’s assessment of these questions and its overarching recommendations are summarized below. Its complete recommendations are found in the body of the report and collected in Chapter 5. The full report provides clear comments on the validity of the chemical and statistical analyses utilized in CABL, and on what can and cannot validly be stated in court regarding CABL evidence. It is up to prosecutors and judges to use the conclusions of this report to decide whether CABL evidence has enough value to be introduced in any specific case.
The current analytical instrumentation used by the FBI is appropriate and is the best available technology with respect to both precision and accuracy for the elements analyzed in a lead matrix. No other technique for this application provides as good or better quantitative, multi-element capability; wide linear dynamic range; limited interferences; and low (parts per billion) detection and quantitative limits. Furthermore, the elements selected by the FBI for analysis
(As, Sb, Sn, Cu, Bi, Ag, and Cd) are appropriate in the sense that they are quantifiable through the use of ICP-OES. Measurements of Sb, Sn, Cd, As, and Cu provide the best discrimination between bullets, and although measurements of Bi and Ag have less probative value, their measurement offers no disadvantage relative to the time and effort needed for analysis by ICP-OES. Recommendation: The FBI should continue to measure the seven elements As, Sb, Sn, Cu, Bi, Ag, and Cd through ICP-OES as stated in the current analytical protocol. Also, the FBI should evaluate the potential gain from the use of high-performance ICP-OES because improvement in analytical precision may provide better discrimination.
The committee also considered the use of approaches other than CABL to improve the ability to compare crime-scene evidence with a suspect’s bullets. For example, it has been reported that lead isotope determination can provide the high-precision analysis necessary to differentiate and identify bullet samples made from ores from different mines. At this time the method in its most practical form has not been shown to be particularly effective for differentiating among United States-based sources of lead. However the method may prove useful in conjunction with the ICP-OES method should the amount of foreign ammunition in use in the United States increase.
Although the current analytical technique is sound, the FBI Laboratory’s practices in quality assurance must be improved significantly to ensure the validity of its results. Chapter 2 includes detailed recommendations for how the FBI’s analytical practices should be improved. For example, the laboratory’s analytical protocol should be revised to contain all details of the procedure and to provide a better basis for the statistics of bullet comparison. The laboratory also needs to develop a more comprehensive formal and documented proficiency test of each examiner and carry out studies to quantify measurement repeatability and reproducibility. After they have been revised based on the recommendations in Chapter 2, the details of the FBI’s CABL procedure and the research and data that supports it should be published in a peer-reviewed journal or at a minimum its analytical protocol should be made available through some other public venue. The revised procedures also must be used consistently within the FBI Laboratory. Recommendation: The FBI’s documented analytical protocol should be applied to all samples and should be followed by all examiners for every case.
STATISTICS FOR COMPARISON
The FBI’s documented statistical protocol for matching CABL evidence2 describes a statistical procedure known as “chaining.” The chaining process
compares each evidence bullet (both from the crime scene and from the suspect, and which cannot be eliminated based on physical comparison) to the next sequentially to identify compositional groups in which all bullets and fragments are analytically indistinguishable within 2 standard deviations of each element’s average concentration. The standard deviation (SD) of each elemental concentration is determined on the basis of the variation found among all bullets and fragments analyzed for the particular case under investigation. If all seven of the concentration intervals (from mean − 2SD to mean + 2SD) of any of the crimescene fragments fall within one of the compositional groups formed by the suspect’s bullets, the fragments and matching suspect’s bullets are stated to be “analytically indistinguishable.”
In the committee’s assessment, chaining may lead to artificially large compositional groups of analytically indistinguishable bullets, thus causing a crimescene fragment and a suspect’s bullet to fall within the same analytically indistinguishable compositional group when this would not be true if other statistical methods were used. In addition, because of the small amount of data in any one study, the standard deviation from the evidence in the case will most likely be larger, less reliable, and more variable than the standard deviation of the analytical method when calculated over many studies (with pooled data).
Although the chaining method is the FBI’s documented statistical protocol, discussions with FBI staff led the committee to believe that the FBI is no longer using it. Instead, the unwritten protocol compares each of the crime-scene fragments with each individual suspect’s bullet (not with a compositional group). This method, 2-standard deviation overlap, deems bullets to be analytically indistinguishable if the intervals (from mean − 2SD to mean + 2SD) for the seven elemental concentrations for a crime-scene bullet and a suspect’s bullet overlap. The FBI claims based on analysis of historical data that this current procedure for bullet comparison will result in a false match probability (FPP) of 1 in 2,500. This report provides better methods for estimating false match and false nonmatch probabilities due to measurement error.
The full report examines the FBI’s current statistical protocol and provides detailed recommendations about how it should be revised in order to provide a sound basis for determining whether crime-scene evidence and suspects’ bullets are analytically indistinguishable. For example, within-bullet measurement standard deviations should be estimated using a pooled standard deviation over many bullets that have been analyzed with the same ICP-OES technique. In addition, a detailed statistical investigation of the FBI’s historical data set containing 71,000 bullets should be conducted to confirm the validity of the revised statistical protocol and the accuracy of the values used to assess the measurement uncertainty in each element. The revised procedures also must be used consistently within the FBI Laboratory. Recommendation: The committee recommends that the FBI use either the T2 test statistic or the successive
t-test statistics procedure described in this report in place of the 2-SD overlap, range overlap, and chaining procedures. Recommendation: The FBI’s statistical protocol should be properly documented and followed by all examiners in every case.
SIGNIFICANCE OF THE MANUFACTURING PROCESS IN THE INTERPRETATION OF EVIDENCE
The committee reviewed the lead bullet manufacturing process to determine whether known variations in lead compositions introduced in the manufacturing process can be used to improve CABL comparison data. In the United States, lead recycled primarily from car batteries is melted and refined at a secondary lead smelter to produce an intermediate lead ingot or billet. The ingot or billet is purchased by a bullet manufacturer and extruded into a large wire roll, which is cut to produce lead slugs whose length and diameter depend on the caliber of ammunition. Slugs are pressed into the form of a bullet and are stored in bins according to caliber. The slugs are sometimes molded into a thimble-shaped copper alloy cup to form a jacketed bullet and then loaded into a cartridge. Cartridges are boxed immediately by some manufacturers. Other manufacturers may store the cartridges in bins by caliber until a customer order must be filled, at which time boxes are filled with cartridges, stamped with a lot number, and collected in cases or pallets for shipment.
In practice, the detailed process followed by each manufacturer varies, and the process can vary even within a single manufacturer to meet demand. For example, many bullet manufacturers add scrap lead from the bullet production to the melt at random times, sporadically changing the composition of the original melt. Likewise, the binning of bullets and cartridges may introduce more mixing of bullets from different melts. In fact, the FBI’s own research has shown that a single box of ammunition can contain bullets from as many as 14 distinct compositional groups. Finding: Variations among and within lead bullet manufacturers make any modeling of the general manufacturing process unreliable and potentially misleading in CABL comparisons.
The committee also reviewed testimony from the FBI regarding the identification of the “source” of crime-scene fragments and suspects’ bullets. Because there are several poorly characterized processes in the production of bullet lead and ammunition, as well as ammunition distribution, it is very difficult to define a “source” and interpret it for legal purposes. It is evident to the committee that in the bullet manufacturing process there exists a volume of material that is compositionally indistinguishable, referred to by the committee as a “compositionally indistinguishable volume of lead” or CIVL. That volume could be the melt, sows, or billets, which vary greatly in size, or some subpart of these. One CIVL yields a number of bullets that are analytically indistinguishable. Those
bullets may be packed in boxes with bullets from other similar (but distinguishable) volumes or in boxes with bullets from the same compositionally indistinguishable volume of lead.
The committee attempted to obtain information on the distribution of ammunition and bullets in the United States. Such distribution information would assist with determining the probability of finding a large number of analytically indistinguishable bullets in one geographic region. Thus, the probability that a crime scene bullet which matches a suspect’s bullet actually came from the suspect might be vastly different in an isolated small town vs a major metropolitan area. But, distribution information on bullets and on loaded ammunition either does not exist or is considered proprietary, and the committee was unable to assess regional distribution patterns. For these reasons, unlike the situation with some forms of evidence such as DNA typing of bloodstains, it is not possible to obtain accurate and easily understood probability estimates that are directly applicable.
In legal proceedings, the interpretation of CABL results depends on the quality of the chemical analysis of the evidence bullets and bullet fragments, the statistical comparison of those bullets, and determination of the significance of the comparison. The committee found the analytical technique used is suitable and reliable for use in court, as long as FBI examiners apply it uniformly as recommended. The recommended changes in the statistical procedures would provide a sound basis for whether crime-scene evidence and a suspect’s bullets “match,” that is, whether they are analytically indistinguishable. However for legal proceedings, the probative value of these findings and how that probative value is conveyed to a jury remains a critical issue.
Despite the variations in manufacturing processes that make it difficult to determine whether bullets come from the same compositionally indistinguishable volume of lead (CIVL), CABL analysis can have value in some court cases. Finding: The committee found that CABL is sufficiently reliable to support testimony that bullets from the same CIVL are more likely to be analytically indistinguishable than bullets from different CIVLs. An examiner may also testify that having CABL evidence that two bullets are analytically indistinguishable increases the probability that two bullets came from the same CIVL, versus no evidence of match status. Recommendation: Interpretation and testimony of examiners should be limited as described above, and assessed regularly.
However, the committee’s review of the literature and discussions with manufacturers indicate that, because of variabilities in the manufacturing process, the amount of lead from a CIVL can range from the equivalent of as few as 12,000 to as many as 35 million 40-grain, .22 caliber longrifle bullets compared
with a total of 9 billion bullets produced each year. Further, there is the possibility that bullets from different CIVLs may be analytically indistinguishable. Recommendation: Expert witnesses should define the range of CIVLs that could make up the source of analytically indistinguishable bullets because of variability in the bullet manufacturing process. The possible existence of coincidentally indistinguishable CIVLs should be acknowledged in the laboratory report and by the expert witness on direct examination. The frequency with which coincidentally identical CIVLs occur is unknown.
Chapter 4 includes findings and recommendations about appropriate statements that can be made in laboratory reports or by expert witnesses based on the committee’s findings on analytical methods and statistical procedures and its knowledge of the bullet manufacturing process, including the following:
The available data do not support any statement that a crime bullet came from a particular box of ammunition. In particular, references to “boxes” of ammunition in any form should be avoided as misleading under Federal Rule of Evidence 403.
Compositional analysis of bullet lead data alone also does not permit any definitive statement concerning the date of bullet manufacture.
Detailed patterns of the distribution of ammunition are unknown, and as a result, experts should not testify as to the probability that the crime scene bullet came from the defendant. Geographic distribution data on bullets and ammunition are needed before such testimony can be given.
It is the conclusion of the committee that, in many cases, CABL is a reasonably accurate way of determining whether two bullets could have come from the same compositionally indistinguishable volume of lead. It may thus in appropriate cases provide additional evidence that ties a suspect to a crime, or in some cases evidence that tends to exonerate a suspect. CABL does not, however, have the unique specificity of techniques such as DNA typing to be used as standalone evidence. It is important that criminal justice professionals and juries understand the capabilities as well as the significant limitations of this forensic technique. The value and reliability of CABL will be enhanced if the recommendations set forth in this report are followed.