PART II
Current Ballistic Imaging and Databases



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 89
PART II Current Ballistic Imaging and Databases

OCR for page 89

OCR for page 89
4 Current Ballistic Imaging Technology “It has been common practice for firearms examiners to maintain an  ‘open-case file’ of physical evidence from unsolved crimes, sorted by cali- ber,” Thompson et al. (2002:8) note of the traditional approach to generat- ing  investigative  leads  through  firearms  identification.  “When  faced  with  a  crime  on  which  little  evidence  was  available,  the  examiner  would  then  go to the storage area for evidence from unsolved cases and choose some  potentially similar cases for examination of the originals.” This process can  be extremely time consuming—not only the direct examination of evidence,  but also the steps of retrieval, filing, and reporting. “Because of the time  required  for  the  manual  comparison  of  evidence,  the  effectiveness  of  this  method can be severely limited by the staffing and workload of an agency’s  examiners (which determines how much time examiners have to search the  open-case file).” In this chapter, we briefly review the background of imaging technology  in  firearms  identification  (Section  4–A),  the  basic  structure  of  Integrated  Ballistics Identification System (IBIS) equipment (4–B), and the manner in  which the IBIS equipment is used to acquire images (4–C). Section 4–D dis- cusses what is publicly known about IBIS procedures for scoring, ranking,  and analysis, crucial to assessing the technical capability of this technical  platform to “scale up” to meet the demands of a much larger database. Sec- tion 4–E reviews the major studies that have been conducted to date on IBIS  performance, particularly with large-scale databases or datasets consisting  of test fires from new weapons. Section 4–F presents basic assessments of  the current technology (specific recommendations related to IBIS usage are  in Chapter 6). An appendix to the chapter, Section 4–G, summarizes and  9

OCR for page 89
9 BALLISTIC IMAGING elaborates on technical evaluation tests performed on IBIS by the state of  California. Because it is important to consider IBIS and the National Inte- grated Ballistic Information Network (NIBIN) together, a summary and our  conclusions on the evidence in this chapter are in Chapter 6, together with  those from Chapter 5. 4–A bACkgROuND Contemporary ballistic imaging technology is the latest step in a grad- ual move over several decades to use technology to make it easier to main- tain and search open case files of ballistics evidence, including cases distant  in  time.  During  the  1970s,  calls  were  made  to  develop  automated  index  systems to assist examiners in search, as well as to explore new directions  for  the  imaging  of  ballistics  evidence.  Biasotti  (1970:12)  made  an  early  call  for  a  computer-based  open  case  file  that  would  permit  examiners  to  describe observed class characteristics “for all rifled weapons [and] uniden- tified bullets and cartridge cases” in a central repository. However, in this  early vision, imaging was not considered; instead, characteristics were to be  expressed using an alphanumeric string (e.g., FW105-100-1357-20-0102- 001-001),  coding  such  factors  as  the  measured  caliber  and  land  widths  of  bullet  evidence.  When  new  evidence  arrived,  a  query  on  the  database  could  then  determine  whether  cases  with  similar  class  characteristics  or  modi operandi were on file. On the technical side, other researchers sug- gested  the  utility  of  more  high-powered  microscopy  techniques  for  the  comparison of ballistics evidence, including several papers arguing for the  use of scanning electron microscopy (Gardner, 1979; Goebel et al., 1980;  Grove et al., 1972).1 Grove et al. (1972:20) considered scanning electron  microscopy “ideally suited for firing pin impression examination because  of its ability to reveal topographical features at the base of the impression.”  The  researchers  examined  “series  of  up  to  50  rounds”  from  “numerous  .32 caliber semi-automatic pistols,” analyzing the first, second, tenth, “and  in some instances the fiftieth firing pin impression.” “In all the firing pin  impressions examined, a match could be made using a criteria of 4 or more  points  of  identification”  whereas  “no  points  of  identification”  could  be  found for firings from different guns; moreover, they concluded that “the  first and fiftieth impressions can be matched.” 1  s summarized by Grove et al. (1972:20), scanning electron microscopy “consists basically  A of  a  finely  focused  beam  of  electrons  which  sweeps  over  the  sample  surface.  This  primary  electron  beam  causes  the  formation  of  low  energy  electrons  (secondary  electrons)  due  to  i  nteraction with the sample surface. These secondary electrons are then collected and displayed  on a cathode ray oscilloscope producing an image that gives extremely good topographical  information with great depth of field.”

OCR for page 89
9 CURRENT BALLISTIC IMAGING TECHNOLOGY In the 1960s, the Los Angeles Police Department (LAPD) developed a  “Balliscan” camera designed specifically to photograph the exterior surface  of a bullet, using a rotated slit to expose the film as a drum turned the bullet  at the same speed.2 Blackwell and Framan (1980) suggested an Automated  Firearms Identification System for the analysis of bullet evidence, based on  the consecutive matching striations methodology of Biasotti (1959) and uti- lizing scanned versions of Balliscan images as the image data. Though they  sketched a schematic diagram for such a system and did some preliminary  analysis  of  bullets  used  in  the  Biasotti  (1959)  study,  no  apparent  further  action on developing the system was taken.  In 1989 the Federal Bureau of Investigation (FBI) announced a program  called DRUGFIRE, which used a system for acquiring images from cartridge  evidence.  A  few  years  later,  the  Bureau  of  Alcohol,  Tobacco,  Firearms,  and  Explosives  (ATF)  adopted  the  BULLETPROOF  system  for  imaging  bullet  evidence,  marketed  by  what  is  now  Forensic  Technology  WAI,  Inc.  (FTI),  of  Montréal,  Canada,  as  the  basis  for  its  CEASEFIRE  network.  As  described in more detail in Chapter 5, the two databases operated in par- allel for several years until CEASEFIRE evolved into the NIBIN program,  using as its platform the IBIS formed by combining BULLETPROOF with a  BRASSCATCHER apparatus for imaging cartridge casings.  IBIS was made the technical base for the new NIBIN database, and the  major ballistic image databases in operation today (including NIBIN and  the state reference ballistic image databases in Maryland and New York)  use IBIS. IBIS is also in use by law enforcement agencies in several foreign  countries;  through  IBIS,  FTI  is  essentially  the  only  provider  of  ballistic  imaging  technology.  At  root,  the  IBIS  platform  combines  a  microscope  with a camera that acquires two-dimensional greyscale images of bullet and  cartridge case evidence; features of the traditional comparison microscope  can then be emulated using the images, and the images can be compared  with each other to assess similarity. Box 4-1 makes an important note about  current usage of the term “IBIS.” 4–b IbIS EquIPMENT Formally, IBIS represents the integration of two separate systems. The  BULLETPROOF  microscope  and  comparison  apparatus  for  acquiring  images  from  bullets  was  developed  first,  beginning  in  1991.  It  was  aug- 2   subsidiary of McDonnell-Douglas, Corp., later produced the Balliscan camera based on  A the LAPD design (Blackwell and Framan, 1980). Balliscan images became prominent in later  years because images made following the assassination of Robert F. Kennedy were reexam- ined by firearms examiners in the mid-1970s, in support of the work of the U.S. House Select  Committee on Assassinations.

OCR for page 89
9 BALLISTIC IMAGING BOX 4-1 “IBIS” Terminology As of January 2007, Forensic Technology WAI, Inc. (FTI), repositioned its line of products to emphasize its existing BulletTRAX-3D platform and developing BrassTRAX-3D platform for the acquisition of three-dimensional measurements from bullets and cartridge cases, respectively. Both of these are said to constitute the “IBIS-TRAX 3D” line, and as such has begun referring to these as IBIS (e.g., they formally refer to the product as “IBIS BulletTRAX-3D”). The IBIS described in this chapter—based on two-dimensional photography—has now been designated the “IBIS Heritage Series” on the firm’s Web site (http://www.fti-ibis.com), and FTI suggests that the two-dimensional product is no longer actively marketed. Though the name has now been linked with the new three-dimensional products, we use the term “IBIS” throughout this report to refer exclusively to the two-dimensional photography system, dating from the combination of the separate BRASSCATCHER and BULLETPROOF components and running through ver- sion 3.4 of the IBIS software. We do so because of the context of our study, which includes offering advice on the existing National Integrated Ballistic Information Network (NIBIN) and suggesting enhancements to it: the entire infrastructure of NIBIN is built on the two-dimensional photography IBIS. What is now dubbed the “IBIS Heritage Line” is in fact the current platform deployed to NIBIN partners; ac- cordingly, it is the appropriate benchmark of comparison for our study. Likewise, the experimental research conducted by the National Institute of Standards and Technology (NIST) in support of the committee’s work—described in Chapter 8—compared the current IBIS two-dimensional to a prototype three- dimensional acquisition system. This is because we consider three-dimensional topographic measurement as a possible enhancement within the current NIBIN system, so that it is appropriate to get a sense of how well the three-dimensional measurements and scores compare with the IBIS two-dimensional currently used in NIBIN. mented  in  1995  by  BRASSCATCHER,  which  adapted  the  apparatus  to  work with cartridge case evidence (McLean, 1999). Most  of  the  IBIS  installations  under  the  NIBIN  program  take  the  form  of  Remote  Data  Acquisition  Stations  (RDASs).  One  component  of  an  RDAS  is  the  Data  Acquisition  Station  (DAS),  a  microscope  with  two  built-in cameras mounted to it (one for bullets and one for cartridge cases).  The RDAS also includes a computer so that demographic data3 associated  3  hese  T auxiliary  data  might  more  accurately  be  described  as  metadata,  but  we  retain  “   demographic data” as common usage in the field.

OCR for page 89
9 CURRENT BALLISTIC IMAGING TECHNOLOGY with a case (e.g., gun  caliber, date of crime, and firing  pin  shape)  can be  entered by an operator; the microscope cameras display their output on the  computer monitor, so that the operator can determine how the image will  be acquired, as described below. In an RDAS, the computer also serves as  a Signature Analysis Station (SAS), where the results of comparisons with  other images can be reviewed. However, the key component that an RDAS  lacks is the “correlation server” that processes results from acquired images  and compares them against other cases in a database. An RDAS alone must  transmit  its  images  to  a  correlation  server  for  processing  and  await  the  results from the server. Standalone systems that include a correlation server  along with the base RDAS equipment are referred to as hubs. As  discussed  in  Chapter  5,  the  NIBIN  program  also  makes  use  of  three other related FTI products in addition to the base IBIS RDAS. As it  is currently structured, all comparisons of images are routed through cor- relation servers (separate from an IBIS hub) located in ATF’s three national  laboratories. To ease the task of reviewing results from image comparisons,  FTI also markets Matchpoint systems—essentially, the computer hardware  and software of a SAS, except that they are not built into the same physi- cal cabinet as the DAS in an RDAS. Finally, several NIBIN sites make use  of Rapid Brass Identification (RBI) units, portable suitcase-size microscope  setups that allow technicians to acquire breech face and firing pin images in  the field, including at crime scenes. RBI units are meant only for acquisition  of images (and transmittal, through an RDAS, to a correlation server), and  not the result of image comparisons. 4–C DATA ACquISITION The  obvious  first  step  in  working  with  IBIS  (and  NIBIN,  using  the  IBIS platform) is to have bullet or cartridge case evidence to enter into the  system. This evidence may be bullets or casings recovered at crime scenes,  or it may be test firings from weapons obtained by the police in the course  of investigations. In the first case, casings and (particularly) bullets present  challenges because they may be damaged and may require cleaning prior  to examination and entry.4 In the case of test firings, the ammunition used  in  the  firings—typically  done  into  a  water  tank,  to  facilitate  capture  of  the undamaged bullet—is a critical choice. To the greatest extent possible,  4  ector (2002) considers the effect of one cleaning process on IBIS performance for match- R ing bullet evidence. An ultrasonic bath—in which high-frequency sound waves produce vapor  bubbles in a liquid—can be used to dislodge some foreign materials that can prove stubborn  to  conventional  means,  including  soil  and  drywall.  However,  Rector  (2002)  observed  that  immersion  in  an  ultrasonic  cleaner  for  longer  periods  of  time  (up  to  30  minutes)  generally  reduced IBIS scores and that the surface etching done by the cleaner was directly visible on  lead bullets.

OCR for page 89
9 BALLISTIC IMAGING examiners prefer to match the test fire ammunition to the ammunition used  in crimes involving a suspected gun; in order of suitability, from most to  least,  De  Kinder  (2002b:9)  characterizes  the  typical  preference  hierarchy  for test firing: •  ammunition from the same lot as the recovered bullets and casings; •  ammunition of the same brand and make as the recovered bullets  and casings; •  ammunition from the same manufacturer as the recovered bullets  and  cartridge  cases,  having  the  same  primer  or  bullet  jacket  composition  but not necessarily being exactly the same type; and •  ammunition having the same primer or bullet jacket composition,  but not necessarily being from the same manufacturer. Often,  however,  no  such  information  is  possible—and  in  the  context  of  creating  an  reference  ballistic  image  database  (RBID),  it  can  never  be  known what type or lot of ammunition will be used with a new firearm.  To address ambiguous cases, ATF recommends certain “protocol ammuni- tion”  for  particular  calibers  to  its  NIBIN  agencies  in  the  hopes  of  “[giv- ing] the best chance overall for [test-fire] items to find matching evidence  b   ullets and casings in a database.” The protocol ammunition is chosen to  be “intermediate in recording toolmarks and impression hardness,” having  bullet metal and primer surfaces that are neither too hard nor too soft for  registration of marks (Thompson et al., 2002:15). 4–C.1 Mounting of Evidence and Demographic Data Entry To  begin  an  IBIS  entry,  operators  open  a  “case,”  which  can  contain  one or more constituent “exhibits,” bullets or cartridge casings. A case can  also  include  information  about  a  firearm,  if  it  has  been  recovered.  Links  suggested by IBIS in comparing exhibits are made between exhibits and not  cases as a whole. Although the case identification number is displayed in a  column when comparison results are returned for analysis, the system does  not  make  it  readily  apparent  where  individual  exhibits  from  a  particular  case fall in the list of rankings reported by IBIS. IBIS training materials emphasize the importance of correct entry of aux- iliary, context data about evidence and exhibits, the “demographic data.”  For cartridge case markings, the training guide indicates that “automatic  correlation requests use all of the following demographic information”— occurrence  date,  caliber,  firing  pin  shape,  and  event  type—“to  select  the  test candidates from the database,” and all these pieces of information are  described  as  “crucial  for  the  correlation  process”  (Forensic  Technology  WAI,  Inc.,  2002a:2-10,  3-2).  IBIS  defines  six  basic  event  types,  four  for 

OCR for page 89
9 CURRENT BALLISTIC IMAGING TECHNOLOGY exhibits related to crime and two for test firings. The crime-related event  codes are homicide (HOM), assaults with a deadly weapon (ADW), other  crime (OTH), and unknown (UNK). The distinction between the two test  fire events is whether the firearm is retained by police (and hence is out of  circulation on the street) or whether it is returned to the owner after firing;  these are coded as TF and TFR, respectively. (The basic manner in which  the  demographic  data  are  used  as  filters  is  described  in  Section  4–D.1.)  B   ullet exhibits are linked to operator-entered information on certain general  rifling characteristics that can be derived from the bullet and can narrow  down the database search. These include caliber, twist (the orientation of  the land and groove impressions, left or right, when looking from the base  of  the  bullet),  and  the  number  of  lands  and  grooves  on  the  bullet.  The  composition  and  type  (e.g.,  jacketed  or  hollow  point)  of  the  bullet  may  also be recorded. Although  accurate  demographic  data  entry  is  essential  to  the  IBIS  comparison process, the physical positioning of bullet or cartridge evidence  under the microscope (and camera) is crucial to the acquisition of quality,  comparable images. Indeed, Tontarski and Thompson (1998:644) observe  that “the greatest initial concern using this technology was whether or not  different examiners could enter projectile and cartridge casing images in a  sufficiently consistent way for the database to be able to locate a match.”  Though they go on to assert that “the equipment’s image capturing system  and its robust algorithm have all but eliminated operator variability as a  concern,” proper positioning of exhibits is still emphasized in IBIS training,  and some studies (e.g., Chan, 2000) suggest that substantial misalignment  can still cause problems in comparison. In its documentation, FTI suggests  standardized protocols for orienting evidence that have also been adopted  as  standards  by  the  NIBIN  program.  For  instance,  a  cartridge  bearing  roughly horizontal breech face marks across the primer surface is supposed  to be oriented so that the marks are as flat (not at an angle) as possible,  rotated  so  that  the  ejector  mark  on  the  cartridge  rim  is  in  the  southern  hemisphere  of  the  image.  If  the  cartridge  shows  evidence  of  a  firing  pin  drag mark, where the pin has scraped against the surface, the cartridge is  supposed to be rotated so that the drag mark is at or around the 3 o’clock  position.  4–C.2 Specification of Regions of Interest IBIS  allows  technicians  to  designate  regions  of  interest  on  an  image.  Because  these  regions  are  circular  for  the  markings  left  on  cartridge  cas- ings,  the  regions  of  interest  are  also  known  as  ring  limits.  For  a  breech  face  impression,  the  region  of  interest  is  indicated  based  on  two  circles.  The computer derives an automatic, “default” placement of the rings, but 

OCR for page 89
9 BALLISTIC IMAGING they can be adjusted directly by the operator. The outer (blue) circle is to  be  set  to  the  edge  of  the  primer  surface  of  the  stamp  and  the  inner  (red)  circle  marks  off  the  firing  pin  impact  region;  the  image  used  in  compar- ing breech face marks is based on the doughnut-shaped area between the  two  circles.  Though  marks  on  the  cartridge  case  area  may  be  irregularly  shaped—the  pit  of  the  firing  pin  impression  and  areas  where  the  primer  metal has been pushed back out of the firing pin impression—the region of  interest rings are strictly circular. Hence, the IBIS operator must make some  judgment  about  exact  placement  of  the  circle,  assessing  the  potential  for  “washout” areas (reflected light off of the jagged edge of the pit of the firing  pin impression) to show up in the final image. Operators may also adjust  procedures  to  accommodate  specific  firing  pin  types;  for  example,  Glock  firearms have a distinctive rectangular firing pin, and therefore technicians  place the inner circle so that it circumscribes the four corners of the impres- sion. Figure 4-1(a) shows an IBIS breech face image with the two circular  delimiters superimposed. Once  the  regions  of  interest  are  set  for  acquiring  a  breech  face,  the  image  is  taken  using  the  IBIS  standard  ring  lighting,  intended  to  provide  uniform  illumination,  and  the  system  automatically  suggests  a  lighting  intensity “to provide optimum lighting for acquisition.” However, the IBIS  training materials note (Forensic Technology WAI, Inc., 2002a:2-18): In  numerous  cases  the  suggested  lighting  may  not  appear  optimal  (for  e   xample, with smooth surfaces or uncommon metal primer compositions).  In these cases, you will need to manually adjust the light setting with the  light  scroll  bar  in  order  to  minimize  washout.  Eliminating  the  washed  out  (white  halo)  area  surrounding  the  firing  pin  impression  improves  correlation accuracy as this area is sometimes a common feature between  cartridge cases. This will increase score results on marks of lesser value.  Always keep in mind that your goal is to find the lighting intensity that  will provide the best contrast with the least washout. After acquiring the breech face image using the center light, the user has  the option of taking a second picture using alternate lighting, a side light  located at the 6 o’clock position relative to the mounted cartridge, while  holding the cartridge fixed in the same orientation. Figure 4-1(b) illustrates  a side light image of a cartridge breech face impression, side by side with the  standard center light image, Figure 4-1(a). The side light image is better for  seeing  some  impression  of  three-dimensional  detail,  though  it  necessarily  also casts shadows on other parts of the  image. If the side light image  is  acquired, it is filed with the case and remains available for viewing later on  (including the “Multiviewer” interface for viewing multiple exhibits simul- taneously, as when reviewing comparison scores). However, the side light 

OCR for page 89
99 CURRENT BALLISTIC IMAGING TECHNOLOGY a b c FIguRE 4-1  IBIS breech face images. 4-1.eps NOTES: The three images are (a) breech face image using the standard ring, center  light; (b) breech face image using the side light; and (c) firing pin image using the  standard ring, center light, acquired from the same cartridge casing. Although they  are  difficult  to  see  in  this  reproduction,  circular  region-of-interest  delimiters  are  indicated on images (a) and (c). The area between the outer circle and inner circle  (a) defines the breech face impression, and the area inside the single circle (c) defines  the firing pin impression.

OCR for page 89
 BALLISTIC IMAGING cleaned  with  acetone  prior  to  firing  to  remove  the  lacquer.  Two  lacquer- coated and two lacquer-stripped casings fired from each of three different  guns  were  entered  into  the  New  York  City  Police  Department’s  IBIS  and  scores were generated; at the time, the number of 9mm Luger, circular fir- ing pin exhibits (the base set for this comparison) in the New York system  was  estimated  at  5,700  images.  Generally,  guns  known  to  produce  clear  characteristic  breech  face  marks  performed  consistently  regardless  of  the  presence  of  lacquer,  which  is  to  say  that  pairs  of  lacquer-coated  exhibits  from the same gun were returned in the top ranks as were pairs of lacquer- stripped exhibits; guns known to produce fainter  breech face  marks pro- duced  lower-ranked  matches,  yet  still  generally  in  the  top  10.  However,  matching  lacquer-coated  to  lacquer-stripped  exhibits  from  the  same  gun  proved more problematic, apparently failing to clear the coarse correlation  and 20 percent threshold steps for guns with weaker propensity to generate  breech face marks (score reported as 0 and rank as “none”; Hayes et al.,  2004:Table 1). The IBIS function for comparing bullet evidence plays a prominent role  in a multi-part examination of criteria for identifying bullet matches, and  in particular standards for the number of groups of consecutive matching  striations  that  can  be  said  to  define  a  match  (Miller  and  McLean,  1998;  Miller, 2000, 2004; see also Miller, 2001). The  committee’s  own  experimentation,  conducted  by  NIST  under  a  separate contract with the National Institute of Justice, involved reanalysis  of some of the De Kinder et al. (2004) cartridge casings as well as construc- tion  of  a  new  144-exhibit  set  of  test-fired  casings,  varying  ammunition  brand and gun manufacturer. These casings were processed using both IBIS  and  three-dimensional  metrology  techniques,  and  were  also  run  through  IBIS  waiving  the  coarse  comparison  and  20  percent  threshold  steps.  We  also  performed  limited  IBIS  experimentation  using  the  New  York  CoBIS  RBID and the independent IBIS database of the New York Police Depart- ment. We discuss the full details in Chapter 8; in brief summary, our own  investigation  corroborated  the  major  findings  of  the  predecessor  studies  described in this chapter. 4–F ASSESSMENT The committee was charged to offer advice on the options of maintain- ing the current NIBIN program (limited to crime gun evidence) or enhanc- ing  it,  and  since  NIBIN  uses  IBIS  as  its  technical  base,  the  evaluation  of  one requires evaluation of the other. Yet focusing too much on assessment  of current IBIS is also somewhat unfair in light of the charge to our com- mittee to evaluate the feasibility of a national RBID. As De Kinder et al.  (2004:208) note, “currently, no technology has been perfected to deal spe-

OCR for page 89
 CURRENT BALLISTIC IMAGING TECHNOLOGY cifically with very large databases of images of marks made by firearms.”  IBIS  was  developed  to  deal  with  smaller,  regional  “open  case  files”  of  images, and it is unreasonable to expect that the full system used to imple- ment a national RBID would follow exactly the same lines as the current  IBIS platform. However, an RBID system—perhaps streamlining the image  acquisition process, allowing for mass entry of exhibits, and continuing to  refine  comparison  procedures—would  likely  be  based  on  IBIS,  if  only  to  maintain compatibility with NIBIN data. As noted above, a subgroup of our committee discussed the IBIS com- parison algorithm in detail with FTI staff under a confidential agreement.  It is our judgment that the algorithm is generally quite sound, novel, and  appropriate to the task of comparing images of ballistics evidence. Based on  the era in which it was developed, IBIS is a valuable system that is funda- mentally a vast improvement over relying on either human memory or the  posting of Polaroids on the forensic laboratory bulletin board for deriving  matches  to  evidence  in  open  case  files.  Properly  used—as  we  describe  in  Chapter  6—we  believe  that  IBIS  provides  an  adequate  investigative  tool  for local and regional searches of ballistics evidence images. However, as  we explain in fuller detail in Chapter 8, the review of past studies of IBIS  performance  and  our  own  experimental  work  suggest  that  IBIS  does  not  operate at the precision needed for a national RBID. In its structure and implementation, the IBIS platform is a computerized  version of the comparison microscope. This is beneficial in certain respects,  in that it provides a familiar (albeit not exactly identical) interface for fire- arms examiners to review image data. Yet it is also, fundamentally, a limi- tation of the technology. Since its origins in the early 1990s, the progress  in  developing  the  existing  IBIS  platform  for  ballistic  imaging  has  been  evolutionary rather than revolutionary, in that it has remained anchored to  the premise of emulating the functions of a comparison microscope. Direct  pairwise comparisons of exhibits remain the heart of the process; IBIS was  not  designed  to  perform  as  a  true  image  “search  engine,”  indexing  and  comparing across large sets of images, as would be desirable in a national  RBID implementation. In its form and function, IBIS functions as a quick sorting and ranking  mechanism:  a  tool  for  search,  but  not  verification.  There  is  great  value  in  the sorting that is performed with relative ease and speed by IBIS. However,  major problems arise when higher expectations are placed on the system than  it was designed to accommodate. Users and policy makers bear a large part  of the responsibility for “overselling” the system; it is unrealistic to expect  “hits”  on  every  database  search,  as  effective  use  of  the  system  depends  as  much  or  more  on  the  timely  entry  of  evidence  into  the  system  as  on  the  ability of the system to detect a possible match. The system is also ill-served  by  the  expectations  of  instantaneous  and  utterly  definitive  verification  of 

OCR for page 89
 BALLISTIC IMAGING evidence matches created by portrayals in popular media; Box 4-2 presents  an example. Overly high expectations and inaccurate portrayals have the unfortu- nate consequence of fueling the perception of ballistic imaging technology  as a test—a source of verification—rather than a search tool. Most recently,  this perception arose in litigation in Illinois (People . Pursley, 341 Ill. App.  3d 230; 2003 Ill. App. LEXIS 784, 2003). In 2000, in light of exonerations  due to DNA evidence, Illinois code was amended to give convicts the right  to make a “motion for fingerprint or forensic testing not available at trial  regarding actual innocence”—that is, to permit appeals for DNA testing.  Invoking  this  provision,  a  man  convicted  in  1993  of  first  degree  murder  (and  sentenced  to  life),  largely  on  the  basis  of  firearms  identification  evi- dence,  “filed  a  motion  .  .  .  seeking  an  order  requiring  that  his  handgun  be tested under the Integrated Ballistics Identification System (IBIS).” The  appeals court ruled against the convict’s motion for IBIS “testing,” holding  that the relevant statute was intended only to apply to fingerprint and DNA  testing. Nowhere in the ruling (or, presumably, the motion) is it indicated  what a “test under IBIS” might entail, how a comparison score might be  interpreted, or against what database images should be searched. Only once  (summarizing the state’s motion to dismiss the convict’s appeal) is it noted  that “IBIS is not a new test but a new system for cataloging for ballistics  information”  and  that  “application  of  the  IBIS  would  not  produce  new,  noncumulative  evidence.”7  Following  the  Pursley  decision,  Carso  (2007)  argued that the Illinois statute should be amended to include “ballistics test- ing” using IBIS but also does not describe what such a test would involve. IBIS  developers  and  proponents  also  bear  responsibility  for  “over- 7  udge  Gertner’s  ruling  in  United States . Green  (405  F.Supp.  2d  104;  2005  U.S.  Dist.  J LEXIS 34273), described in Box 3-4, is also of interest because IBIS was used in the course  of the investigation. It suggests that some basic concepts of IBIS scope and operation can be  misconstrued. Section G of the ruling notes: [The sergeant/examiner] also used the Integratable [sic] Ballistic [sic] Identification  System (IBIS) in his comparison, although the government represented that it would  not  offer  IBIS  results  [as  testimony].  A  national  computer  database,  IBIS  allows  examiners to identify the most likely matches for the evidence in a given case. IBIS  uses a laser measuring device to evaluate shell casings and provides the examiner  with a list of possible matches. . . . In fact, the IBIS system has been widely criticized.  Its efficacy is limited by the detail with which police departments have scanned old  shell  casings  into  the  computer  and  the  accuracy  of  the  mathematical  algorithms  used  to  compare  casings.  As  with  the  individual  examinations,  no  evidence  was  presented about the accuracy of the IBIS matches. . . . In any event, [the sergeant]  acknowledged  that  even  if  the  computer  suggests  numerous  possible  matches,  he  will not bother to check them all. That is, once he decides he has found a match,  he will not eliminate all other alternatives by exhausting the IBIS-generated list of  potential matches.

OCR for page 89
 CURRENT BALLISTIC IMAGING TECHNOLOGY BOX 4-2 CSI Ballistic Imaging Firearms identification concepts and the use of ballistic imaging have peri- odically been referenced on forensic science-themed television shows. One such example is episode 307 (“Fight Night”) of CSI: Crime Scene Investigation; this particular episode won an Emmy award for best writing. One scene finds a Las Vegas investigator talking with a firearms examiner who is peering intently into the microscope of what—externally—is a complete IBIS RDAS unit. The investigator asks, “Three guns found at the crime scene, none match the bullets recovered from the victim. What does that tell us?” “Shooter kept his weapon,” the examiner replies. “Means he likes his gun, and may have used it before,” says the investi- gator, as some part of the machinery makes a loud whirring noise. “Which is where the shell case and IBIS come in,” says the examiner cheerfully. “I’ll run it against the national database.” He wheels from the microscope to the keyboard and, off-camera, types a short sequence of characters. “Firing pin impressions and breech face marks—a closer look,” muses the investigator; instantaneously, the system makes a loud shuf- fling sound and several beeps. The camera now shows the “IBIS” screen, which prominently shows a single image of the entire base of a cartridge, headstamp and all; some text indicating “Halo On” and “Magnification 150X,” among other things, is superimposed over the corner of the image. Beside it is a four-column listing of “Case ID,” “Exhibit Number,” “Site Number,” and “Firing Pin;” the entries are obviously not sorted in descending order by the purported firing pin score (that is, three digit “scores” are interspersed with two digit scores). The middle entry (clearly not the highest legible score, albeit close) flashes blue several times as the system beeps; at no point does a second, comparison casing image appear. “Got us a hit,” the examiner intones, now reading off of a new window that has popped up on screen. “Los Angeles County Sheriff’s Department found . . . shell casings from the same gun . . . used in a gang murder two years ago.” The investigator interjects, “They get a conviction on the suspect?” “No. Guy beat the rap,” the examiner continues. “Timothy Fontaine, aka ‘Tiny Tim’ . . . member of the Snakebacks . . . current residence unknown.” The “Criminal Records” window that appears on the screen also includes entries for a vehicle license number and the name of an arresting officer; unfortunately, the space clearly reserved for a photo of the person is labeled “NP AVAILABLE.” The investigator says, “I bet I could find where he stays in Vegas,” and the scene ends. The total elapsed time of the scene is 44 seconds. promising”  the  system,  in  at  least  two  crucial  and  related  respects.  The  first  is  the  pervasive  mythology  that  has  come  to  surround  the  “top  10”  results  in  an  IBIS  search.  The  current  IBIS  provides  as  its  default  printed  report a listing of the 10 highest scores by each type of marking, and IBIS  training materials undercut guidance to consider gaps and features in the 

OCR for page 89
 BALLISTIC IMAGING distribution of comparison scores by promoting the examination of the top  10 suggested matches. However, the implied physical or cognitive restric- tion to the top 10 results is not likely to be appropriate in all searches or  all  database  sizes,  and  the  focus  on  the  top  10  results  is  inadequate  for  assessing  the  system’s  performance  and  for  understanding  the  variability  of scores by demographic characteristics (e.g., gun make and model). We  know of no substantiated rationale for the ad hoc cutoff at rank 10; the  resulting assumption that nothing outside of the top 10 ranked is valuable  puts unduly high expectations on the system.  The second basic flaw is the use of the term “correlation” to describe  the  IBIS  comparison  process,  which  imputes  to  the  system  an  unjustified  air of technical exactness. The common, statistical use of the term implies  a  particular  type  of  relationship  and  quantifies  the  strength  of  that  rela- tionship.  In  comparison,  IBIS  scores  are  described  by  the  system’s  own  training materials as having no intrinsic value, severely limiting the ability  to express the strength of similarity between two exhibits and to compare  results across different runs of the system. As we suggest in Chapter 6, we  believe that the usefulness of IBIS is compromised unless some meaning can  be imputed to its “correlation” scores—to make them function more like  true statistical correlations. 4–g APPENDIx: SuMMARy OF PERFORMANCE TESTS IN THE CALIFORNIA EvALuATION OF A REFERENCE bALLISTIC IMAgE DATAbASE This  appendix  describes  the  tests  performed  by  Tulleners  (2001)  in  response to the California legislature’s directive that the state’s Department  of Justice study the feasibility of a reference ballistic image database. We  begin by profiling those tests that were actually completed; these summa- ries  extract  additional  information  from  spreadsheet  printouts  that  were  included as an appendix to Tulleners (2001). We also describe those tests  that  were  planned  for  the  evaluation  but  were  unable  to  be  completed,  and summarize the formal responses to and independent assessment of the  California evaluation. 4–g.1 Completed Performance Tests The Tulleners (2001) technical evaluation was based on the completion  of five performance tests.

OCR for page 89
 CURRENT BALLISTIC IMAGING TECHNOLOGY Test 1—basic System Correlation Two  cartridges  were  fired  from  each  of  the  792  CHP  pistols,  one  to  be entered into a “test” database and the other retained as an “evidence”  exhibit. All of these firings used the same Federal brand ammunition. The  basic goals of this test were to assess the time required to enter specimens  into  a  database  and  to  test  the  accuracy  of  comparison  as  database  size  increases.  The first component of this test considered the basic ability of the sys- tem to find exhibits for guns known to be in the database. A sample of 50  test cartridges (the same-gun pairs of “evidence” entries already in the data- base) was drawn, and queries were made against the full database. Twenty- four (48 percent) of these test casings matched to their sister evidence casing  as the first-ranked entry in either breech face or firing pin mark. However, a  surprisingly high 19 of the comparisons (38 percent) did not find the sister  casing within the top 10 ranked items in either breech face or firing pin,8  of which 9 (18 percent) of these known-match comparisons failed to clear  IBIS’ coarse comparison and 20 percent threshold.9 It does not appear that  one mark was superior to the other in terms of generating possible matches:  the 31 instances where the known sister was found in the top 10 by either  mark are fairly evenly divided between cases where both marks were in the  top 10 (10), only the breech face was in the top 10 (9), and only the firing  pin was in the top 10 (12). A second component of the test selected five of the “evidence” casings  used  in  the  first  test  that  had  low  ranks  on  one  or  both  markings;  these  were  reacquired  by  a  second  IBIS  operator  and  matched  against  smaller  subsets of the data to see if those changes affected the rankings. In terms  of comparisons to the full database, the rankings changed using the image  from the second operator but not grossly so; no very low-ranked exhibits  were  converted  to  high  ranks,  although  two  of  the  casings  apparently  failed to clear the 20 percent threshold in the reacquisition.10 The entries  were compared against database subsets of size 100, 200, 300, 400, 500,  600,  700,  and  792;  generally,  rankings  degraded  with  the  larger  sample  8  he main text of Tulleners (2001) indicates these figures as being searches for matches in  T the top 15 ranked items, but it can be verified from the “raw data” spreadsheets in Appendix C  of the technical evaluation that the statements hold for the stronger (and more conventional)  top 10 filter. 9  ailure to clear the 20 percent threshold is assumed from the “Not in Selection” entry for  F both score types (breech face and firing pin) in the technical evaluation spreadsheets. 10  ne  of  these,  labeled  E44  in  the  first  test  and  E152A  in  the  reacquisition,  appears  to  O have had a significant difference in the acquisition of the firing pin image. The exhibit was  ranked  45  on  breech  face  and  1  on  firing  pin  in  the  first  analysis,  but  apparently  failed  to  clear the 20 percent threshold and was excluded from listing in the reanalysis (Tulleners, 2001: Appendix C).

OCR for page 89
 BALLISTIC IMAGING sizes,  though  very  high-ranked  exhibits  tended  to  stay  very  high  (e.g.,  a  one-ranked exhibit on firing pin remained the number one rank for all the  sizes, and a two-ranked exhibit for the 100-entry database slipped to rank  six in the full 792 set).11 Test 2—Cartridges Not in Database Ten cartridges were fired using the same Federal brand cartridges but  using 10 pistols of the same make and model not from the new CHP order.  The comparison scores for the best match on both marks were recorded and  were judged to be consistent with the range of scores registered in Test 1,  several with the high end of that range. However, the evaluation accepted  FTI’s advice that “a score is only relevant within a particular correlation”  and that “the score cannot be used to compare the ranking of two correla- tions.” The test was found to be inconclusive. Test 3—Different Ammunition During  the  test  firing  of  the  CHP  pistols,  22  of  the  pistols  were  also  used  to  fire  rounds  using  batches  of  five  different  ammunition  brands:  PMC-Eldorado  (.40  S&W  180  grain),  CORBON  (.40  S&W  165  grain),  ARMSCOR (.40 S&W 180 grain), Remington (.40 S&W 180 grain), and  Winchester  (.40  S&W  180  grain).  Not  all  of  the  ammunition  types  were  fired from each of the guns; 72 cartridges were acquired in total. Each of  these casings was then compared with the 792-exhibit set to test the ability  of the system to find the Federal-brand test fire from the same gun in the  database. The  test  found  poor  results  in  finding  matches  to  the  images  from  F   ederal-brand ammunition using images from the other five brands. Sixteen  of the 72 comparisons (22 percent) matched to the image from the same  gun as the top-ranked result on either the breech face or firing pin impres- sions;  in  total,  21  of  the  comparisons  (29  percent)  had  the  known  sister  image occur in the top 10 ranks on either mark. Neither mark was better  at generating matches; 13 top-10 matches were found on the breech face  mark and 14 on the firing pin. No match was found in the top 10 ranks  by either mark in 26 of the comparisons (36 percent), and 25 of the com- parisons (35 percent) failed to clear the coarse comparison and 20 percent  threshold. 11   third part of the test timed the comparison times for three selected exhibits for data- A base  subsets  of  different  sizes  (100,  250,  500,  792).  From  the  results,  Tulleners  (2001:8-5)  concluded that “correlation times are not a significant issue for a large database” although he  assumed a strict linear interpolation in processing times.

OCR for page 89
9 CURRENT BALLISTIC IMAGING TECHNOLOGY Tabulations  from  Tulleners  (2001:Appendix  C)  suggest  that  the  ARMSCOR and Corbon ammunition proved particularly difficult to match.  Out of 14 matches of ARMSCOR rounds to the Federal-brand images in  the database by breech face, 6 failed to clear the coarse comparison step,  and 8 ranked lower than 25; 13 Corbon comparisons were attempted, with  8 being rejected at the coarse comparison stage and only one ranking better  than 25 (but out of the top 10). Results by firing pin were similar, with a  few more rankings in the 11–25 range and one ARMSCOR round finding  its  Federal-brand  sister  as  the  top-ranked  result.  The  Winchester  rounds  proved most amenable to matches in the 18 comparisons that were made: 7  found the Federal sister round in the top-ranked slot, with 1 ranking 11–25,  6 ranking below 25, and 4 missing the 20 percent threshold (see also De  Kinder’s [2002b:11] analysis of the same spreadsheet). Test 4—Altered breech Face After firing the two Federal-brand cartridges for the “test” and “evi- dence” sets, the firing pin tip and breech face of one of the CHP pistols was  subjected to “minimum file and sandpaper efforts” to attempt to change the  firearm’s individualizing marks (Tulleners, 2001:B-5). “This filing alteration  took  about  three  minutes  using  a  standard  file”  (Tulleners,  2001:7-3).  A  second set of two test fires with Federal ammunition was then performed,  one for entry in the database and the other used as an “evidence” query.  The two sets of exhibits, before and after alteration, matched to each other  well:  the  pre-alteration  casings  matched  to  each  other  in  the  top-ranked  position on both firing pin and breech face and the post-alteration casings  matched as the top-ranked pairing on firing pin (however, the rank was 35  on breech face). However, no match was possible from the pre-alteration  to the post-alteration exhibits; in both cases, the technical evaluation’s data  appendix  lists  the  matches  as  “not  in  selection  list,”  suggesting  that  the  deliberate  alteration  prevented  the  exhibits  from  clearing  the  IBIS  coarse  comparison pass. Test 7—breech Face Longevity Study In a test intended to determine whether a breech face maintains individ- ual marks over repeated firings, an independent laboratory was contracted  to perform 600 test fires from each of two .40 caliber pistols; the make of  one was described as a Glock type and the other as unknown (Tulleners,  2001:8-11). The Glock-type pistol was fired using CCI brand ammunition,  and IMI ammunition was used in the unknown-make pistol. For each pis- tol,  casings  in  the  intervals  1–6,  101–106,  201–206,  301–306,  401–406,  501–506,  and  595–600  were  retained  for  analysis;  one  casing  from  each 

OCR for page 89
0 BALLISTIC IMAGING interval was used as the “test” database and the other as “evidence” entries.  Ultimately, the Glock-type firings turned out to be unusable due to the lack  of a larger database for comparison; none of the other CHP weapons have  Glock type firing pins, so they could not be compared to the Glock firings  due  to  IBIS’  demographic  filtering.  Tulleners  (2001:8-11)  concluded  that  there  were  signs  of  “definitive  ranking  degradation”  as  the  firings  from  later intervals were tended to rank lower than those from the earlier firings  among the IMI cartridges. However, the evaluation suggested that “further  tests need to be conducted in this area.” 4–g.2 Incomplete Performance Tests Tulleners (2001) was unable to carry out tests 5, 6, and 8 in his original  slate of experiments. Test 5 was intended to assess IBIS performance using  cartridges fired from SIG Sauer firearms, which are known among examin- ers for having minimal breech face characteristics. An extensive set of SIG  Sauer test fires was subsequently used in Tulleners’ joint study, De Kinder et  al. (2004), described in the next section. Test 8 was meant to test the system  using firearms known to have strong subclass characteristic carry-over, such  as some Heckler and Koch and Lorcin firearms (see Section 3–B.1). Test 6 “would have taken some test-fired cartridge cases from selected  weapons,  buried  one  of  the  cartridge  cases  in  a  large  database  and  then  observe  the  correlation  on  these  cartridge  cases”  (Tulleners,  2001:7-3,  7-4).  The  California  Department  of  Justice  was  unable  to  complete  the  test as planned, though it arranged for a limited test along the same lines  to  be  conducted  by  the  New  York  City  Police  Department  (NYPD).  The  C   alifornia Criminalistics Institute submitted eight casings each fired from  two  9mm  SIG  Sauer  pistols.  In  each  set,  two  rounds  used  Remington- Peters  ammunition,  and  the  other  firings  used  Winchester,  Federal,  Hor- nady Vector, Fiocchi, CCI, and Sellier and Bellot ammunition. One of the  Remington rounds was retained as the “evidence” casing, so that for each  of the two pistols, seven sister images were mixed into the NYPD’s 9mm  database, which then contained 3,673 items. For both pistols, four of the  seven  sister  images  were  found  in  the  top  15  ranks  by  either  breech  face  or firing pin, and the second Remington round generally turned up as the  top-ranked entry by either mark. The Hornady Vector, CCI, and Sellier and  Bellot rounds “seemed to be the most difficult for comparison” (Tulleners,  2001:8-10). 4–g.3 Criticisms and Independent Review Rebutting  the  California  study  on  behalf  of  ATF,  Thompson  et  al.  (2002:15,  16)  argued  most  stridently  that  “all  of  [the  performance  test 

OCR for page 89
 CURRENT BALLISTIC IMAGING TECHNOLOGY results]  are  skewed  due  to  the  selection  of  Federal  Brand  ammunition.”  They argue that Federal is not the prescribed “ATF protocol ammunition  in any of the calibers of interest, due to the primer surface generally being  too  hard  in  comparison  to  the  ammunition  being  used  in  handguns.”  Instead, they suggested Remington-Peters ammunition as a more suitable  medium.  In  his  review,  De  Kinder  (2002b:9–11)  rejected  this  argument,  citing research by the Forensic Institute in The Netherlands on IBIS score  results using 18 common primers suggesting that Federal showed medium  performance in registering marks. (Unfortunately, the Dutch study did not  include  Remington  ammunition,  as  it  is  not  common  in  Europe.)  More  directly, De Kinder noted that hardness properties of primers are not well  known  but  hardness  measures  for  the  six  types  of  ammunition  used  in  California’s  Test  3  had  been  independently  conducted  by  the  Lawrence  Livermore  National  Laboratory.  The  Federal  brass  primers  were  directly  measured to be the least hard of the six (108 ± 5HV), including Remington- Peters’ nickel primers (157 ± 12HV).12  In  its  rebuttal  to  the  California  study,  FTI  (2002:5)  argued  that  “the  Ealuation has an overly pessimistic view of automated ballistics technol- ogy that discredits its conclusions.” In Test 1, FTI (2002:14–15), submits  that too great a focus on the 38 percent of possible matches missing from  the  top  15  ranks  unduly  discounts  the  48  percent  that  found  the  correct  match in the top rank on one of the marks and the 62 percent that matched  within the top 15 on either rank. “These results are sufficient to identify a  significant number of cartridge cases that merit manual study and would  have  produced  new  cold  hits.”  More  fundamentally,  FTI  (2002:13,  14)  holds  that  the  IBIS  system  was  held  to  an  unfair  standard  in  the  test.  A  firearms  examiner  manually  compared  the  cases  for  FTI  and  concluded  that he could not certify a match between eight of the Test 1 pairs and that  “approximately half had markings that were somewhat unfavorable.” As  a result, FTI suggested that at least the eight human-identified nonmatches  be excluded from the statistics, arguing: It  is  immediately  obvious  that  the  performance  of  an  automated  exami- nation  could  not,  and  should  not,  be  more  accurate  than  a  microscope  comparison by a firearms examiner. Thus, to the extent that the Ealua- tion  included  cartridge  cases  that  had  insufficient  marks  to  be  identified  by a firearms examiner, the results cannot support the hypothesis, and the  Ealuation must be without scientific value. 12  e Kinder also noted that the criticism of Federal ammunition was unusual, given that  D Federal had been chosen for a similar study by several of the same ATF authors (Thompson  et al., 1996).

OCR for page 89
 BALLISTIC IMAGING On these points, De Kinder (2002b:14) concluded that the FTI arguments  were an overreach. He countered that the passage quoted above “is the same  type of expression as saying at the beginning of the 1990[s] that automated  comparison  of  bullets  and  cartridge  casings  is  impossible.”  He  preferred  instead  the  revised  statement  that  “the  current  scientific  knowledge  and  state-of-the-art technology does not allow one to be more accurate than a  microscope comparison by a firearms examiner.” De Kinder held that drop- ping the believed-“unmatchable” exhibits from analysis is “unacceptable,”  particularly given that the study was oriented to studying the feasibility of  an  RBID.  “All  data  points  have  to  be  taken  into  consideration”  because  “the goal of [an RBID] is not restricted to those cartridge cases that can be  identified by a trained firearm examiner.” Generally,  De  Kinder  (2002b)  indicated  approval  of  the  conduct  and  interpretation  of  the  major  performance  tests  in  the  California  study.  He  suggested the need for further study in a variety of areas.