SOPs must be created, and once an entire system is created, another validation must be performed. The SOPs should contain sufficient detail, including, where appropriate,

  1. All steps in the procedure,
  2. Proper controls,
  3. All reagents and preparations,
  4. Calibration,
  5. Criteria for analysis of results, and
  6. Interpretation of results.

Analysis and interpretation are two different things. Analysis could be “I have a positive result.” Interpretation would be, “It is ten times more likely to observe this result if X compared with if Y.” Budowle emphasized that there are feedback validation mechanisms in both approaches.

Validation is an ongoing process; it does not stop once the method is up and running. To illustrate its importance, he described an incident in which a lot of commercially available human DNA identification kits were sold to customers, and subsequently the manufacturer discovered that the deoxynucleotide triphosphates (dNTPs) could degrade while kits sat on the shelf. The company sent letters to crime laboratories apprising them of this possibility, but some laboratories continued using the kits. The signal of results was dropping (i.e., a loss of sensitivity of detection) and in actuality could have presented negative results. This outcome is a very serious problem when working with small amounts of DNA. The positive control supplied with the kit also was diminishing in signal intensity. Yet some users ignored it or were not cognizant of the signal loss. Instead, as long as they had a qualitative call that was consistent with the profile of the positive control, the users were satisfied that quality control was acceptable. Both qualitative and quantitative signals should have been monitored. Had the users done so, it would have been evident early on that the system was not performing appropriately and the assay could have been halted rather than consuming valuable evidence.

Error is an important consideration because error is ubiquitous. Different types of error are shown in Figure 6-2. There is, for example, systematic error, which includes measurement error. There also is bias, both in methods and in the individuals interpreting the data. It is human nature to want to accommodate data to scenarios that make sense to us, regardless of alternative hypotheses or explanations. This human nature affects how we perceive false positives and false negatives. There is no absolute point where the line can be drawn between the two false categories for acceptable performance. As the line shifts in one direction or the other, more false positives will be obtained at the expense of false nega-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement