Strengthening Forensic Science in the United States: A Path Forward (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

6 Improving Methods, Practice, and Performance in Forensic Science In a presentation to the committee, Jennifer Mnookin, of the University of California, Los Angeles School of Law, cautioned against yielding to two extremes in developing expectations for the forensic science disciplines. The first is the risk of letting the “perfect” be the enemy of the “good.” That is, many forms of forensic investigation and analysis may work relatively well once appropriate tasks have been set for them. “The opposite danger is the risk of overconfidence about what we think we know—the risk of making unjustified inferences on the basis of limited information, or sometimes a resistance to gaining new information that would help us do it better.” Nonetheless, a number of the forensic science disciplines, as they are currently practiced, do not contribute as much to criminal justice as they could. This chapter discusses the improvements that are needed and makes four major recommendations. It does not evaluate the quality of evidence collection and management—steps that provide the inputs to forensic meth- ods—although, obviously, the quality of those steps is critical in maximizing the investigative and probative value of that evidence. INDEPENDENCE OF FORENSIC SCIENCE LABORATORIES The majority of forensic science laboratories are administered by law enforcement agencies, such as police departments, where the laboratory administrator reports to the head of the agency. This system leads to   J. Mnookin, Professor of Law, University of California, Los Angeles Law School. Presenta- tion to the committee. April 23, 2007. 183 

184 STRENGTHENING FORENSIC SCIENCE IN THE UNITED STATES significant concerns related to the independence of the laboratory and its budget. Ideally, public forensic science laboratories should be independent of or autonomous within law enforcement agencies. In these contexts, the director would have an equal voice with others in the justice system on matters involving the laboratory and other agencies. The laboratory also would be able to set its own priorities with respect to cases, expenditures, and other important issues. Cultural pressures caused by the different mis- sions of scientific laboratories vis-à-vis law enforcement agencies would be largely resolved. Finally, the forensic science laboratories would be able to set their own budget priorities and not have to compete with the parent law enforcement agencies. UNCERTAINTIES AND BIAS Few forensic science methods have developed adequate measures of the accuracy of inferences made by forensic scientists. All results for every fo- rensic science method should indicate the uncertainty in the measurements that are made, and studies must be conducted that enable the estimation of those values. For the identification sciences (e.g., friction ridge analysis, toolmark analysis, handwriting analysis), such studies would accumulate data about the intraindividual variability (e.g., how much one finger’s im- pressions vary from impression to impression, or how much one toolmark or signature varies from instance to instance) and the interindividual vari- ability (e.g., how much the impressions of many fingerprints vary across a population and in what ways). With that information, one could begin to attach confidence limits to individualization determinations and also begin to develop an understanding of how much similarity is needed in order to attain a given level of confidence that a match exists. Note that this necessary step would change the way the word “individualization” is commonly used. The concept of individualization is that an object found at a crime scene can be uniquely associated with one particular source. By acknowledging that there can be uncertainties in this process, the concept of “uniquely associated with” must be replaced with a probabilistic associa- tion, and other sources of the crime scene evidence cannot be completely discounted. The courts already have proven their ability to deal with some degree of uncertainty in individualizations, as demonstrated by the success- ful use of DNA analysis (with its small, but nonzero, error rate). Finally, as discussed in Chapter 4, the accuracy of forensic meth- ods resulting in classification or individualization conclusions needs to be evaluated in well-designed and rigorously conducted studies. The level of accuracy of an analysis is likely to be a key determinant of its ultimate probative value. Some initial and striking research has uncovered the effects of some 

IMPROVING METHODS, PRACTICE, AND PERFORMANCE 185 biases in forensic science procedures, but much more must be done to understand the sources of bias and to develop countermeasures. Some principles employed in other fields should be useful, although some (e.g., blinding) may not be feasible for some types of forensics work. The foren- sic science disciplines are just beginning to become aware of contextual bias and the dangers it poses. The traps created by such biases can be very subtle, and typically one is not aware that his or her judgment is being af- fected. An overview of the effect of bias in the forensic science disciplines can be found in Risinger et al., 2002. Decisions regarding what analyses need to be performed and in what order also can be influenced by bias and ultimately have the potential to skew results. Forensic scientists who sit administratively in law enforcement agencies or prosecutors’ offices, or who are hired by those units, are subject to a general risk of bias. Bias also is introduced through decisions made about evidence collection, which controls who is listed as a suspect. Evidence col- lection and crime scene investigation can require scientific knowledge and judgment, and these functions are normally outside the control of forensic scientists. REPORTING RESULTS There is a critical need in most fields of forensic science to raise the standards for reporting and testifying about the results of investigations. For example, many terms are used by forensic examiners in reports and in court testimony to describe findings, conclusions, and the degrees of association between evidentiary material (e.g., hairs, fingerprints, fibers) and particular people or objects. Such terms include but are not limited to “match,” “consistent with,” “identical,” “similar in all respects tested,” and “cannot be excluded as the source of.” The use of such terms can have a profound effect on how the trier of fact in a criminal or civil matter perceives and evaluates evidence. Yet the forensic science disciplines have not reached agreement or consensus on the precise meaning of any of these   E.g., I.E. Dror and D. Charlton. 2006. Why experts make errors. Journal of Forensic Identification 56 (4):600-616; I.E. Dror, D. Charlton, and A Peron. 2006. Contextual in- formation renders experts vulnerable to making erroneous identifications. Forensic Science International 156(1):74-78; D.E. Krane, S. Ford, J.R. Gilder, K. Inman, A. Jamieson, R. Koppl, I.L. Kornfield, D.M. Risinger, N. Rudin, M.S. Taylor, and W.C Thompson. 2008. Sequential unmasking: A means of minimizing observer effects in forensic DNA interpretation. Journal of Forensic Sciences 53(4):1006-1007; L.S. Miller. 1987. Procedural bias in forensic science examinations of human hairs. Law and Human Behavior 11(2):157-163.   See the discussion of biases provided in Chapter 4.   D.M. Risinger, M.J. Saks, W.C. Thompson, and R. Rosenthal. 2002. The Daubert/Kumho implications of observer effects in forensic science: Hidden problems of expectation and sug- gestion. California Law Review 90:1-56; Krane, et al., op. cit. 

186 STRENGTHENING FORENSIC SCIENCE IN THE UNITED STATES terms. Although some disciplines have developed vocabulary and scales to be used in reporting results, they have not become standard practice. This imprecision in vocabulary stems in part from the paucity of research in fo- rensic science and the corresponding limitations in interpreting the results of forensic analyses. Publications such as Evett et al.,  Aitken and Taroni, and Evett provide the essential building blocks for the proper assessment and communication of forensic findings. As a general matter, laboratory reports generated as the result of a sci- entific analysis should be complete and thorough. They should describe, at a minimum, methods and materials, procedures, results, and conclusions, and they should identify, as appropriate, the sources of uncertainty in the procedures and conclusions along with estimates of their scale (to indicate the level of confidence in the results). Although it is not appropriate and practicable to provide as much detail as might be expected in a research paper, sufficient content should be provided to allow the nonscientist reader to understand what has been done and permit informed, unbiased scrutiny of the conclusion. Some forensic laboratory reports meet this standard of reporting, but most do not. Some reports contain only identifying and agency information, a brief description of the evidence being submitted, a brief description of the types of analysis requested, and a short statement of the results (e.g., “The green, brown plant material in item #1 was identified as marijuana”). The norm is to have no description of the methods or procedures used, and most reports do not discuss measurement uncertainties or confidence limits. Many disciplines outside the forensic science disciplines have standards, templates, and protocols for data reporting. Although some of the Scientific Working Groups have a scoring system for reporting findings, they are not uniformly or consistently used. Forensic science reports, and any courtroom testimony stemming from them, must include clear characterizations of the limitations of the analyses, including associated probabilities where possible. Courtroom testimony should be given in lay terms so that all trial participants can understand how to weight and interpret the testimony. In order to enable this, research must be undertaken to evaluate the reliability of the steps of the various identification methods and the confidence intervals associated with the overall conclusions.   I.W. Evett, G. Jackson, J.A. Lambert, and S. McCrossan. 2000. The impact of the prin- ciples of evidence interpretation on the structure and content of statements. Science and Justice 40(4):233-239.   C.G.G. Aitken and F. Taroni. 2004. Statistics and the Evaluation of Evidence for Forensic Scientists. 2nd ed. V. Barnett, ed. Chichester, UK: John Wiley & Sons Ltd.   I.W. Evett. 1990. The theory of interpreting scientific transfer evidence. Forensic Science Progress 4:141-179. 

IMPROVING METHODS, PRACTICE, AND PERFORMANCE 187 THE NEED FOR RESEARCH Barry Fisher, Director of the Crime Laboratory of the Los Angeles County Sheriff’s Department, has said, “We run the risk of our science being questioned in the courts because there is so little research.” In 2001 Giannelli wrote, “In many areas [of forensic science] little system- atic research has been conducted to validate the field’s basic premises and techniques, and often there is no justification why such research would not be feasible.” As Smith et al. note, the United States has a renowned higher education system, and many basic research discoveries relating to the forensic science disciplines have been made in academia.10 However, the forensic science disciplines suffer from an inadequate research base: Few forensic scientists have the opportunity to conduct research, few academics are positioned to undertake such research, and, importantly, the funding for forensic research is insufficient. Others believe that the field suffers be- cause the research initiatives being funded and pursued lack an overarching strategic plan.11 There are several explanations for the relative lack of funding for ba- sic and applied research in the forensic science disciplines. First, forensic practice was started in, and has grown out of, the criminal justice and law enforcement systems. Many forensic science techniques were developed to aid in the investigatory phase of law enforcement and then were adapted to the role of aiding in prosecution by providing courtroom testimony. Thus, forensic practitioners who work in public crime laboratories often are seen as part of the prosecution team, not as part of the scientific enterprise. Second, some of the forensic science disciplines rely on an apprenticeship model for training, rather than on codifying their methods in a scientific framework. Third, federal agencies that fund scientific work, such as the National Science Foundation, the National Institutes of Health, and the Department of Defense, generally have not considered forensic science as part of the science base they need to support. It has been only in recent years that the National Institute of Justice has taken interest in funding fo- rensic science research, but the majority of these funds have been awarded to reduce case backlogs, especially for cases that involve the analysis of DNA (see Chapter 2).   K. Pyrek. 2007. Forensic Science Under Siege: The Challenges of Forensic Laboratories and the Medico-Legal Investigation System. Burlington, MA: Academic Press, p. 231.   P.C. Giannelli. 2001. Scientific evidence in civil and criminal cases. Arizona State Law Journal 103:112. 10  F.P. Smith, R.H. Liu, and C.A. Lindquist. 1988. Research experience and future criminal- ists. Journal of Forensic Sciences 33(4):1074-1080. 11  IAI Positions and Recommendations to the NAS Committee to Review the Forensic Sci- ences. September 19, 2007. See presentation by K.F. Martin, IAI President, to the committee. December 6, 2007. 

188 STRENGTHENING FORENSIC SCIENCE IN THE UNITED STATES The forensic science disciplines need to develop rigorous protocols for performing subjective interpretations, and they must pursue equally rigor- ous research and evaluation programs. The development of such research programs can benefit significantly from work in other areas, notably from the large body of research that is available on the evaluation of observer performance in diagnostic medicine and from the findings of cognitive psy- chology on the potential for bias and error in human observers. In evaluating the accuracy of a forensic analysis, it is crucial to clarify the type of question the analysis is called upon to address. Thus, although some techniques may be too imprecise to permit the accurate identification of a specific individual, they may still provide useful and accurate informa- tion about questions of classification. For example, microscopic hair analy- sis may provide reliable evidence on the subpopulation of the individual from which the specimen was derived, even if it cannot associate reliably the hair with a specific individual. However, the definition of the appropri- ate question is only a first step in evaluating the performance of a forensic technique. The research design should address the questions that arise in the specific context of forensics. A complete research agenda should include studies to establish the strengths and limitations of each procedure, sources of bias and varia- tion, quantification of uncertainties created by these sources, measures of performance, procedural steps in the process of analyzing the forensic evidence, and methods for continual monitoring and improving the steps in that process. CONCLUSIONS AND RECOMMENDATIONS Wide variability is found across forensic science disciplines not only with regard to techniques and methodologies (see Chapter 5), but also with regard to reliability, error rates, reporting, research foundations, general acceptability, and published material. Some of the disciplines are labora- tory based (e.g., nuclear and mitochondrial DNA analysis, toxicology and drug analysis, and analyses of fibers and fire debris); others are based on expert interpretation of observed patterns (e.g., of fingerprints, writing samples, toolmarks, bite marks, and hairs). The briefings and materials that informed this report illustrate that the level of scientific development and evaluation varies substantially among the forensic science disciplines. In most areas of forensic science, no well-defined system exists for determining error rates, and proficiency testing shows that some examin- ers perform poorly. In some disciplines, such as forensic odontology, the methods of evidence collection are relatively noncontroversial, but disputes arise over the value and reliability of the resulting interpretations. In most forensic science disciplines, no studies have been conducted 

IMPROVING METHODS, PRACTICE, AND PERFORMANCE 189 of large populations to establish the uniqueness of marks or features. Yet, despite the lack of a statistical foundation, examiners make probabilistic claims based on their experience. A statistical framework that allows quan- tification of these claims is greatly needed. These disciplines also critically need to standardize and clarify the terminology used in reporting and tes- tifying about the results and in providing more information. Little rigorous systematic research has been done to validate the basic premises and techniques in a number of forensic science disciplines. The committee sees no evident reason why conducting such research is not feasi- ble; in fact, some researchers have proposed research agendas to strengthen the foundations of specific forensic disciplines.12 Much more federal fund- ing is needed to support research in forensic science and forensic pathology in universities and in private laboratories committed to such work. The forensic science and medical examiner communities (see Chapter 9) will be improved by opportunities to collaborate with the broader science and engi- neering communities. In particular, collaborative efforts are urgently needed to: (1) develop new technical methods or provide in-depth grounding for advances developed in forensic science; (2) provide an interface between the forensic science and medical examiner communities and basic sciences; and (3) create fertile grounds for discourse among the communities. The proposed National Institute of Forensic Science (NIFS) should recommend, implement, and guide strategies for supporting such initiatives. Although a long-term research agenda will require a thorough assess- ment of each of the assumptions that underlie forensic science techniques, many concerns regarding the forensic science disciplines can be addressed immediately through studies in which forensic science practitioners are presented with a standardized set of realistic training materials that vary in complexity. Such studies will not explore the components of the decision process, but they will permit an assessment of the extent to which skilled forensic science practitioners will reach the same or similar conclusions when presented with the types of materials that lead to disagreements. Recommendation 2: The National Institute of Forensic Science (NIFS), after review- ing established standards such as ISO 17025, and in consultation with its advisory board, should establish standard terminology to be used in reporting on and testifying about the results of forensic science investigations. Similarly, it should establish model labora- tory reports for different forensic science disciplines and specify 12  See, e.g., L. Haber and R.N. Haber. 2008. Scientific validation of fingerprint evidence under Daubert. Law, Probability and Risk 7(2):87-109. 

190 STRENGTHENING FORENSIC SCIENCE IN THE UNITED STATES the minimum information that should be included. As part of the accreditation and certification processes, laboratories and forensic scientists should be required to utilize model laboratory reports when summarizing the results of their analyses. Recommendation 3: Research is needed to address issues of accuracy, reliability, and validity in the forensic science disciplines. The National Institute of Forensic Science (NIFS) should competitively fund peer-reviewed research in the following areas: (a)  Studies establishing the scientific bases demonstrating the validity of forensic methods. (b)  The development and establishment of quantifiable mea- sures of the reliability and accuracy of forensic analyses. Studies of the reliability and accuracy of forensic tech- niques should reflect actual practice on realisticcase sce- narios, averaged across a representative sample of forensic scientists and laboratories. Studies also should establish the limits of reliability and accuracy that analytic methods can be expected to achieve as the conditions of forensic evidence vary. The research by which measures of reliabil- ity and accuracy are determined should be peer reviewed and published in respected scientific journals. (c)  The development of quantifiable measures of uncertainty in the conclusions of forensic analyses. (d)  Automated techniques capable of enhancing forensic technologies. To answer questions regarding the reliability and accuracy of a foren- sic analysis, the research must distinguish between average performance (achieved across individual practitioners and laboratories) and individual performance (achieved by the specific practitioner and laboratory). Whether or not a forensic procedure is sufficient under the rules of evidence govern- ing criminal and civil litigation raises difficult legal issues that are outside the realm of scientific inquiry. Recommendation 4: To improve the scientific bases of forensic science examinations and to maximize independence from or autonomy within the law enforcement community, Congress should authorize and appropri- 

IMPROVING METHODS, PRACTICE, AND PERFORMANCE 191 ate incentive funds to the National Institute of Forensic Science (NIFS) for allocation to state and local jurisdictions for the purpose of removing all public forensic laboratories and facilities from the administrative control of law enforcement agencies or prosecutors’ offices. Recommendation 5: The National Institute of Forensic Science (NIFS) should encourage research programs on human observer bias and sources of human error in forensic examinations. Such programs might include stud- ies to determine the effects of contextual bias in forensic practice (e.g., studies to determine whether and to what extent the results of forensic analyses are influenced by knowledge regarding the background of the suspect and the investigator’s theory of the case). In addition, research on sources of human error should be closely linked with research conducted to quantify and characterize the amount of error. Based on the results of these studies, and in consultation with its advisory board, NIFS should develop stan- dard operating procedures (that will lay the foundation for model protocols) to minimize, to the greatest extent reasonably possible, potential bias and sources of human error in forensic practice. These standard operating procedures should apply to all forensic analyses that may be used in litigation.