Page 75
3
Ensuring High Standards of Laboratory Performance
If DNA from an evidence sample and DNA from a suspect or victim share a profile that has a low frequency in the population, this suggests that the two DNA samples came from the same person; the lower the frequency, the stronger the evidence. But the possibility remains that the match is only apparent—that an error has occurred and the true profile of one of the sources differs from that reported by the laboratory. We describe here ways that laboratory errors, particularly errors that might falsely incriminate a suspect, can arise, how their occurrence might be minimized, and how to take into account the fact that the error rate can never be reduced to zero.
Although this report focuses mainly on methods for computing the frequencies of profiles in various populations and the uncertainty in estimates of such quantities (Chapters 4 and 5), it is important to understand that those estimates will be of little value if there has been an error in determining that the two DNA profiles match. A reported match in DNA samples that is the result of error in the handling or analysis of the samples could lead to the conviction of an innocent person, and an erroneously reported exclusion could also have serious consequences. Although there are more ways for an error to lead to a false exclusion than a false match, the US system of justice is more concerned with the latter, since it regards false conviction as worse than false acquittal.
We recognize that some risk of error is inevitable, as in any human endeavor, whatever efforts a laboratory takes to eliminate mistakes. Nonetheless, safeguards can be built into the system to prevent both types of errors and to identify and correct them. It is important that forensic laboratories use strict quality-control standards to minimize the risk of error.
Page 76
Quality Control and Quality Assurance in the Laboratory
The maintenance of high laboratory standards rests on a foundation of sound quality control (QC) and quality assurance (QA). Quality control and quality assurance refer to related but distinct components of a laboratory's effort to deliver a quality product (ANSI/ASQC A3-1978). Quality control refers to measures that are taken to ensure that the product, in this case a DNA-typing result and its interpretation, meets a specified standard of quality. Quality assurance refers to measures that are taken by a laboratory to monitor, verify, and document its performance. Regular proficiency testing and regular auditing of laboratory operations are both essential components of QA programs. QA thus serves as a functional check on QC in a laboratory. Demonstration that a laboratory is meeting its QC objectives provides confidence in the quality of its product.
Current QC and QA Guidelines
The 1992 report (NRC 1992) outlined many features of desirable QC and QA as part of a proposed regulatory program (p 104-105):
· ''Individual analysts have education, training, and experience commensurate with the analysis performed and testimony provided.
· "Analysts have a thorough understanding of the principles, use, and limitations of methods and procedures applied to the tests performed.
· "Analysts successfully complete periodic proficiency tests and their equipment and procedures meet specified criteria.
· "Reagents and equipment are properly maintained and monitored.
· "Procedures used are generally accepted in the field and supported by published, reviewed data that were gathered and recorded in a scientific manner.
· "Appropriate controls are specified in procedures and are used.
· "New technical procedures are thoroughly tested to demonstrate their efficacy and reliability for examining evidence material before being implemented in casework.
· "Clearly written and well-understood procedures exist for handling and preserving the integrity of evidence, for laboratory safety, and for laboratory security.
· "Each laboratory participates in a program of external proficiency testing that periodically measures the capability of its analysts and the reliability of its analytic results.
· "Case records—such as notes, worksheets, autoradiographs, and population data banks—and other data or records that support examiners' conclusions are prepared, retained by the laboratory, and made available for inspection on court order after review of the reasonableness of a request."
Page 77
Although not QC or QA features, the following are listed as desirable aspects of a regulatory program (NRC 1992, p 105):
· "Redundancy of programs is avoided, so that unnecessary duplication of effort and costs can be eliminated.
· "The program is widely accepted by the forensic-science community.
· "The program is applicable to federal, state, local, and private laboratories.
· "The program is enforceable—i.e., . . . failure to meet its requirements will prevent a laboratory from continuing to perform DNA typing tests until compliance is demonstrated.
· "The program can be implemented within a relatively short time.
· "The program involves appropriate experts in forensic science, molecular biology, and population genetics."
This list substantially summarizes more-detailed and more-specific guidelines developed by the Technical Working Group on DNA Analysis Methods (TWGDAM), a group composed of forensic DNA analysts from government and private laboratories around the United States and Canada. TWGDAM meets several times a year to discuss problems, report on cooperative studies, and share procedures and experiences. It has published guidelines and reports that address various aspects of forensic DNA analysis and laboratory procedure (TWGDAM 1989, 1990a,b, 1991, 1994b,c, 1995). The most recent guidelines define current accepted standards of practice for forensic DNA laboratories in North America.
The crime laboratory accreditation program sponsored by the Laboratory Accreditation Board of the American Association of Crime Laboratory Directors (ASCLD-LAB) requires extensive documentation of all aspects of laboratory operations (including the education, training, and experience of personnel; the specification and calibration of equipment and reagents; the validation and description of analytic methods, the definition of appropriate standards and controls, the procedures for handling samples, and the guidelines for interpreting and reporting data), proficiency testing, internal and external audits of laboratory operations, and a plan to address deficiencies with corrective action and weigh their importance for laboratory competence. The TWGDAM QC and QA guidelines are specifically endorsed by ASCLD-LAB as part of the foundation for accreditation. Laboratories that seek accreditation must submit all their documentation to an accreditation review team and must undergo a week-long site inspection by that team. The site inspection includes a critical evaluation of randomly selected case files to verify that the QC standards as documented are being met. Accredited laboratories must annually certify to ASCLD-LAB that they continue to meet defined standards; they submit proficiency test results to ASCLD-LAB for review. The ASCLD-LAB accreditation program began in 1981; by the end of 1994, 128 forensic laboratories in the United States, one in Canada, and two in Australia had received accreditation. Forensic laboratories in Australia, New
Page 78
Zealand, Singapore, and Hong Kong were also preparing for ASCLD-LAB accreditation, as was the FBI laboratory in Washington, DC.
The College of American Pathologists (CAP) has recently established a program for laboratory accreditation in molecular pathology, which includes forensic identity-testing and parentage-testing. The program is similar to the ASCLD-LAB program in its requirements for documentation of procedures and of equipment and facilities, QC, QA, etc., and it requires proficiency-testing in the form of participation in an approved program for interlaboratory comparison. As with the ASCLD-LAB program, the accreditation process includes on-site inspection of laboratory operations and records.
The American Society of Crime Laboratory Directors (ASCLD) has published general guidelines for forensic-laboratory management (ASCLD 1987). (Despite the similarity in their names, ASCLD and ASCLD-LAB are separate entities with distinct governing bodies.) The guidelines cover all aspects of forensic analysis and affirm the key element of QA: the responsibility of laboratory managers for all aspects of laboratory operations and performance, including definition and documentation of standards for personnel training, procedures, equipment and facilities, and performance review.
The DNA Identification Act of 1994 establishes a federal framework for setting national standards on QA and proficiency-testing. It authorizes grant funding to be made available to state and local jurisdictions to improve the quality and availability of DNA analysis in forensic laboratories. To be eligible for funding, these jurisdictions must certify that a laboratory will satisfy or exceed QA standards published by the director of FBI; that DNA samples and analyses will be made available only to criminal-justice agencies, courts, and defendants; and that each DNA analyst will undergo external proficiency-testing at intervals not exceeding 180 days. The standards for QA and the standards for testing proficiency of forensic laboratories are to be developed by the DNA Advisory Board (See Chapter 1).
The Role of Proficiency-Testing and Audits
Proficiency-testing and audits are key assessment mechanisms in any program for critical self-evaluation of laboratory performance. Proficiency-testing entails the testing of specimens submitted to the laboratory in the same form as evidence samples. Audits are independent reviews of laboratory operations conducted to determine whether the laboratory is performing according to a defined standard. Both forms of assessment can be conducted internally or externally, that is, by people inside or outside the laboratory. Good QA programs have a mixture of regular internal and external assessment.
The most straightforward form of proficiency-testing is open, or declared. The analyst is presented with a set of samples, typically about five, in a mock case scenario and is asked to determine which samples could have a common
Page 79
source. The analyst is aware that the samples are being used in a proficiency test. Open proficiency-testing evaluates analytical methods and interpretation of results; it identifies systematic problems due to equipment, materials, the laboratory environment (such as contamination), and analyst misjudgment. A benefit of open proficiency-testing conducted by external entities is that many laboratories can test the same set of samples, thus allowing interlaboratory comparison of performance and statistical evaluation of collective results. At present, external proficiency-testing in forensic DNA analysis is offered by three vendors: Collaborative Testing Services, Cellmark Diagnostics (UK), and the College of American Pathologists. All provide summary reports on the results of each proficiency test.
Open proficiency-testing is required under TWGDAM guidelines and is a requirement both for laboratory accreditation by ASCLD-LAB and for board certification of analysts by the American Board of Criminalistics (ABC). TWGDAM specifies that each analyst take at least two proficiency tests per year; the results, including any corrective action for discrepancies, are to be documented. The ASCLD-LAB accreditation program follows TWGDAM in requiring at least two proficiency tests for analysts per year and requires in addition that one of the tests be external. Results are reported by the proficiency-test vendor to ASCLD-LAB as a condition of continuing accreditation. A committee of ASCLD-LAB reviews the discrepancies and may invoke sanctions up to and including suspension of accreditation. ABC similarly requires at least one external proficiency test per year, the results of which are to be reported to ABC.
A second form of proficiency-testing, full-blind proficiency-testing, goes a step beyond open proficiency-testing in that the analyst does not know that a proficiency test is being conducted. It has been argued that full-blind testing provides a truer test of functional proficiency because the analysts will not take extra care in analyzing samples. Whether or not that is so, this form of proficiency-testing evaluates a broader aspect of laboratory operation, from the receipt of the "evidence" at the front desk through analysis and interpretation to final reporting.
The logistics of full-blind proficiency-tests are formidable. The "evidence" samples have to be submitted through an investigative agency in the jurisdiction of the laboratory and have to arrive in the laboratory with case documentation and an identified contact investigator. Without such full cover, a case would likely be recognized as nonroutine, and a blind test suspected. The TWGDAM guidelines recommend one full-blind proficiency test per laboratory per year if such a program can be implemented. The DNA Identification Act of 1994 required that the director of the National Institute of Justice (NIJ) report to Congress on the feasibility of establishing a full-blind proficiency-testing program. The NIJ has reported that, although several of the large laboratory systems conduct blind testing in-house, there is no blind, external, DNA profiency-testing program generally available to public or private laboratories. The report mentioned some potentially serious issues with blind testing, including the cost of implementation, the risk that DNA data from an innocent donor to the test might end up in
Page 80
criminal DNA databanks, and the chance that the test would impose excessive costs and time demands on law-enforcement agencies. The NIJ has contracted a study to review current testing programs and to examine alternative ways of performing blind tests.
Regular audits of laboratory operations complement proficiency-testing in the monitoring of general laboratory performance. The objective of the audit is to compare a laboratory's performance with its professed quality policies and objectives. Audits cover all phases of laboratory operations related to performance and accordingly touch on matters not covered by proficiency-testing, such as equipment-calibration schedules and case-management records. The TWGDAM QA guidelines recommend audits every two years (TWGDAM 1995) by persons independent of the DNA laboratory operation, preferably including at least one from another organization (typically a laboratory from a jurisdiction in another state).
The objective of both proficiency-testing and auditing is to improve laboratory performance by identifying problems that need to be corrected. Neither is designed to measure error rates.
Safeguarding against Error
Every human activity is associated with some risk of error. There are potential sources of error at every stage in the processing of physical evidence, from collection in the field through laboratory analysis to interpretation of results of analysis. Not all lapses have deleterious consequences; many have no consequences. Many are readily identified and can be corrected. The lapses of most concern, however, are the ones that might lead to a false match. False exclusions are important but are unlikely to lead to false convictions. There is no single solution to the problem of error. To achieve accurate results, care and attention to detail and independent checks must be used at all stages of the analytical process. This section surveys potential sources of error, the consequences of errors, and safeguards to prevent them.
Sample Mishandling and Data-Recording Errors
Mixups or mislabelings of samples or results can occur at any point where evidence is handled or data recorded, that is, from the time of evidence collection in the field to the writing of the final report. The consequences of sample mishandling depend on which samples are mishandled. There are circumstances in which undetected mishandling can lead to false matches; the genetic types of the samples might be determined correctly but the inferred connections among the samples can be incorrect because of sample mixup. Sample mishandling and incorrect recording of data can happen with any kind of physical evidence and are of great concern in all fields of forensic science. The concern regarding mishandling is
Page 81
compounded by the reality that most forensic laboratories have little or no control over the handling of evidence elsewhere. Accordingly, it is desirable to have safeguards not only to protect against mixups in the laboratory but also to detect mixups that might have occurred anywhere in the process.
Safeguards against sample mishandling in the field include proper training of personnel involved in sample collection (such as crime-scene personnel) and submission of complete evidence items (rather than clippings or scrapings) to the laboratory. Mixups in the laboratory as samples are being removed from evidence items for analysis can be minimized by sample-handling policies that allow only one evidence item to be handled at a time. Sample mixup or mislabeling in the analysis stream (for example, transfer of a sample solution to the wrong tube, loading of a sample into the wrong lane on an electrophoresis gel, and misrecording of data) can be minimized by rigorous adherence to defined procedures for sample-handling and data entry.
Redundancy in testing provides a check on sample integrity. Testing of multiple items can serve as a check on consistency of results: inconsistencies among items believed to be of common origin can signal a mixup. For example, demonstration that bloodstains from different evidence items have the same DNA profile is less likely if a sample mixup occurred. Gender testing in cases in which both males and females are involved can also serve as a consistency check and has been used to verify suspected mislabeling. One benefit of the high discriminating power of DNA typing is the detection of sample-mishandling errors that might not have been recognized with classical blood-group and protein-marker testing.
Because an analyst might fail to notice an inconsistent result or a recording error, it is important to have analytical results reviewed by a second person, preferably one not familiar with the origin of the samples or issues in question. An independent reviewer can also catch flaws in analytical reasoning and interpretation. Independent "second reading" is common in forensic laboratories and is required by the guidelines (TWGDAM 1991, 1995).
The ultimate safeguard against error due to sample mixup is to provide an opportunity for retesting. In most cases, it is possible to retain portions of the original evidence items and portions of the samples from different stages of the testing. Sample retention is particularly easy when PCR-based typing methods are used for testing. If samples have been retained, questions of error due to mishandling can be resolved by retesting. Allegations of sample mishandling lose credibility if those making the allegation have rejected the opportunity for a retest. Sample retention whenever possible is recommended in the TWGDAM QA guidelines and is standard in many laboratories. As stated in the Guidelines (TWGDAM 1995), "testing of evidence and evidence samples should be conducted to provide the maximum information with the least consumption of the sample. Whenever possible, a portion of the original sample should be retained or returned to the submitting agency, as established by laboratory policy."
Page 82
Even the strongest evidence will be worthless—or worse, could lead to a false conviction—if the evidence sample did not originate in connection with the crime. Given the great individuating potential of DNA evidence and the relative ease with which it can be mishandled or manipulated by the careless or the unscrupulous, the integrity of the chain of custody is of paramount importance.
Faulty Reagents, Equipment, Controls, or Technique
Problems with reagents, equipment, controls, or technique usually lead to failed tests (no results) or to ambiguous test results. Situations in which such problems might lead to a false match or a false exclusion will be uncommon if testing is accompanied by appropriate controls. In any case, adherence to a standard QC program provides safeguards against these kinds of laboratory error. Regular monitoring of reagents and equipment is part of any standard QA program. Use of appropriate QC standards and of positive and negative controls is part of routine testing; failure of the standards and controls to behave as expected in a test signals a problem with the analytical system and might disqualify test results. Moreover, regular monitoring of test outcomes with standards and controls allows recognition of gradually emerging problems with reagents, equipment, controls, standards, and overall procedure that might otherwise be overlooked. For example, almost all North American forensic laboratories that perform VNTR analysis use DNA from the human cell line K562 as a positive typing control; correct sizing of restriction fragments from K562 DNA is prerequisite to accepting a typing result as reportable. Monitoring of K562 fragment-size measurements within a laboratory over time and comparison of measurements between laboratories allow identification of "drift" due to procedural modification, reagent variation, or equipment deterioration.
Inevitably, breakdowns in reagent quality, equipment, controls, or technique occur at times. For example, in the loading of an electrophoresis gel, a sample loaded in one lane might leak into an adjacent lane, which might then appear to contain a mixed sample. Confusion resulting from lane-leakage problems is typically avoided by leaving alternate lanes empty or by placing critical samples in nonadjacent lanes, and this should always be done. In this and other situations involving such lapses, a breakdown is usually readily apparent from the appearance of the results. Review of analytical results by a second analyst who is unfamiliar with the issues in the case protects against lapses of judgment on the part of the primary analyst.
Evidence Contamination
Contamination has been used as an umbrella term to cover any situation in which a foreign material is mixed with an evidence sample. Different kinds of contamination have different consequences for analysis. Contamination with
Page 83
nonbiological materials (gasoline, grit, etc.) or with nonhuman biological materials (microorganisms, plant materials, etc.) can result in test failures but not in genetic typing errors. Part of marker validation includes testing to determine whether the marker can be detected in nonhuman species and if so, whether its presence there might cause confusion in typing. It is generally found that the markers identified by the single locus probes used in forensic VNTR analysis and by PCR-based typing are detected in but a few nonprimate species; if such markers are used, that fact should obviously be taken into account. That is an advantage of DNA typing over enzyme and blood-group testing. Contamination with human material, however, is a possible source of concern for DNA tests.
Three kinds of sample contamination were described in the 1992 National Research Council report (p 65-67) and are briefly summarized here. For each, appropriate safeguards and controls can be built into the analytical system to protect against contamination and to detect it when it does occur.
· Inadvertent contamination can occur in the course of sample-handling by investigative or laboratory personnel or by others. The background environment from which the evidence is collected can also cause contamination. The concern about contamination is not peculiar to biological evidence; extraneous evidence (such as a detective's cigarette butt found at the scene) is always a concern. The important consequences of those sorts of contamination are that samples might appear to be mixtures of material from several persons and, in the worst case, that only the contaminating type might be detected. The concern is greater with PCR-based typing methods than with VNTR analysis because PCR can amplify very small amounts of DNA. A false match could occur if the genetic type of the contaminating materials by chance matched the genetic type of a principal (such as a suspect) in the case or, worse, if the contaminant itself came from a suspect in the case. The best safeguard against inadvertent contamination is to have rigorous procedures for sample-handling from field to laboratory. Particular attention should be given to keeping evidence samples separated from reference samples. In VNTR analysis, evidence and reference samples can be kept apart up to the time they are loaded onto the analytical gel. With PCR-based typing, evidence and reference samples can be analyzed separately as well. Contamination from sample-handling or from the background environment can be detected in several ways. Background control samples—samples collected from areas adjacent to bloodstains or other evidence sites—can be used to determine whether background contamination is present. Background control testing is not a new idea; it has long been used in forensic blood-grouping. Knowledge of the genetic types of people who might contribute contaminating material can be used to assess the possibility of contamination from those people. Testing for multiple loci increases the chance of differentiating between contaminant and true sources of a sample. Finally, redundancy in testing provides a consistency check; the chance that multiple samples would all be contaminated the same way is small.
Page 84
· Mixed samples are contaminated by their very nature. Postcoital vaginal swabs, for example, are expected to contain a mixture of semen and vaginal fluids, and shed blood from different persons might run together. Such samples are part of the territory of forensic science and must be dealt with whenever feasible. Sperm DNA can be separated from nonsperm DNA with differential DNA extraction. Detection of sample mixtures of other kinds is generally revealed with genetic typing. Mixtures show the composite of the individual types present; the proportions of the different types reflect the proportions of the contributors to the mixture. Testing samples collected from different areas of a mixed stain can sometimes allow the genetic types of the contributors to be more clearly distinguished.
· Carryover contamination is well recognized in PCR testing, although it is not an issue in VNTR analysis. This kind of contamination occurs when a PCR amplification product finds its way into a reaction mix before the target template DNA is added. The carryover product can then be amplified along with the DNA from an evidence sample, and the result can be that an incorrect genetic type is assigned to the evidence sample. A false match can occur if the genetic type of the contaminant matches by chance the genetic type of a principal in the case; in the worst case, the contaminant originates from another party in the case. Primary safeguards against carryover contamination include the use of different work areas for pre-PCR and post-PCR sample-handling, the use of biological safety hoods, the use of dedicated equipment (such as pipetters), and maintenance of a one-way flow of material from pre-PCR to post-PCR work areas so that PCR product cannot come into contact with sample materials. Those safeguards are outlined in the TWGDAM QC and QA guidelines (TWGDAM 1991, 1995). Sterile precautions similar to those used in handling infectious-disease agents in microbiology laboratories may also protect against carryover contamination; many of the contamination issues in PCR work and in infectious-disease microbiology are largely the same. Procedural safeguards can also be used. Genetic typing of evidence samples before the typing of reference samples protects against contamination of the former with the latter. Standard blank controls can be used to detect reagent and work area contamination. If there is any question regarding PCR carryover contamination, retained portions of the evidence item can be tested.
Analyst Bias
An analyst can be biased, consciously or unconsciously, in either direction. Genetic-typing results, however, are usually unambiguous; one cannot make one genetic type look like another simply by wishing it so. In VNTR analysis, patterns must meet empirically defined objective match criteria to be said to match. If enough loci are tested, it is extremely unlikely that two unrelated persons would have indistinguishable VNTR banding patterns.
Bias in forensic science usually leads to sins of omission rather than commis-
Page 85
sion. Possibly exculpating evidence might be ignored or rejected. Contradictory test results or evidence of sample mixture may be discounted. Such bias is relatively easy to detect if test results are reviewed critically. Both TWGDAM and ASCLD-LAB accreditation guidelines stipulate that case files be reviewed internally by a qualified second analyst before a report is released. That not only reveals bias but also reveals mistakes in recording and oversights. Independent review by a defense expert provides even stronger protection against the possibility that bias will lead to a false match. This is most effective if the defense expert is thoroughly familiar with the standard procedures of the testing laboratory so that exceptions from the standard can be noted.
It has been argued that when the analysis of a test result involves subjective judgment, expectations or other biases can influence an analyst's interpretation (Nisbett and Ross 1980). For example, it has been suggested that analysts examining VNTR autoradiographs sometimes interpret faint bands as real or artifactual so as to produce a match with a suspect's profile (Lander 1989; Thompson and Ford 1991, p 140-141; Thompson 1995). The protocols of the next paragraph should greatly reduce such bias, if it exists.
Laboratory procedures should be designed with safeguards to detect bias and to identify cases of true ambiguity. Potential ambiguities should be documented; in particular, any visual overrides of the computer-assisted imaging devices used for making measurements in VNTR analysis must be noted and explained. Internal review can detect cases of bias and true ambiguity as well as oversights and mistakes in recording.
Should an Error Rate Be Included in Calculations?
Some commentators have argued that the rate of profile matching due to laboratory error should be estimated and combined with the random-match probability (calculated with methods described in Chapter 4) to give only a single, summary statistic. But withholding the components of the summary statistic from the judge or jury would deprive the trier of fact of the opportunity to evaluate separately the possibility that the profiles match by coincidence as opposed to the possibility that they are reported to match by reason of laboratory or handling error. We discuss the legal arguments for and against such an exclusionary rule in Chapter 6. Here, we consider whether statistical analysis can provide a meaningful and accurate estimate of the probability of a laboratory or handling error that would produce a reported match between samples of nonmatching DNA.
· The question to be decided is not the general error rate for a laboratory or laboratories over time but rather whether the laboratory doing DNA testing in this particular case made a critical error. The risk of error in any particular case depends on many variables (such as number of samples, redundancy in
Page 86
testing, and analyst proficiency), and there is no simple equation to translate these variables into the probability that a reported match is spurious.
· To estimate accurately, from proficiency test results, the overall rate at which a laboratory declares nonmatching samples to match, as has been suggested, would require a laboratory to undergo an unrealistically large number of proficiency trials. Suppose that two laboratories each have under specific conditions a false-positive error rate of 0.10%—one match per 1,000 nonmatching proficiency trials. To establish that rate accurately, it would be necessary for each laboratory to undergo many thousands of trials. If one laboratory were to pass 1,000 proficiency tests without error, the 95% upper confidence limit for the error rate would be 0.30%. If the other laboratory had made one error, the limit would be 0.47%.1 Those results are not significantly different statistically. Both laboratories could have a true rate of 0.10%, but a court or jury might regard the laboratory that made no errors in the test as significantly better than the one that made a single error. To put the numbers in context, only the largest forensic laboratories could have performed DNA testing in as many as 1,000 cases; no laboratory performs more proficiency tests than case tests, and none should be expected to.
· The pooling of proficiency-test results across laboratories has been suggested as a means of estimating an ''industry-wide" error rate (Koehler et al. 1995). But that could penalize the better laboratories; multiple errors on a single test by one laboratory could substantially affect the overall estimated false-match error rate. Surveys of proficiency test results in the pre-DNA era show that the preponderance of errors originated in a small proportion of laboratories (Sensabaugh and Northey 1985; Sensabaugh 1987). Laboratories that made such errors today would have to document corrective action, which might include suspension of the analysts responsible for the errors (TWGDAM 1991).
· Estimating rates at which nonmatching samples are declared to match from historical performance on proficiency tests is almost certain to yield wrong values. When errors are discovered, they are investigated thoroughly so that corrections can be made. A laboratory is not likely to make the same error again, so the error probability is correspondingly reduced. There has been much publicity about proficiency-trial errors made by Cellmark in 1988 and 1989, the first years of its operation. Two matching errors were made in comparing 125 test samples, for an error rate of 1.6% in that batch. The causes of the two errors were discovered, and sample-handling procedures were modified to prevent their recurrence. There have been no errors in 450 additional tests through 1994. Clearly, an estimate of 0.35% (2/575) is inappropriate as a measure of the chance of error at Cellmark today.
1For the first case, with no errors, the upper 95% confidence limit, L, was calculated from the equation: (1 - L)N = 0.05, where N is the number of error-free tests. In the case where one error was made in N tests, the equation was (1 - L)N + NL(1 - L)N-1 = 0.05. The interpretation of a confidence limit is discussed in Chapter 5.
Page 87
For all those reasons, we believe that a calculation that combines error rates with match probabilities is inappropriate. The risk of error is properly considered case by case, taking into account the record of the laboratory performing the tests, the extent of redundancy, and the overall quality of the results. However, there is no need to debate differing estimates of false-match error rates when the question of a possible false match can be put to direct test, as discussed in the next section.
Retesting
A wrongly accused person's best insurance against the possibility of being falsely incriminated is the opportunity to have the testing repeated. Such an opportunity should be provided whenever possible. As we have previously noted, retesting provides an opportunity to identify and correct errors that might have been made during the course of analysis.
Whenever feasible, investigative agencies and testing laboratories should provide for repeat testing. Evidence items should be divided into two or more parts at the earliest possible time, and one or more parts retained for possible repeat testing. Ideally, the division should be made before DNA is extracted, and each part should be handled by different personnel. If division before DNA extraction is not feasible, the division should be made as soon as possible afterward and certainly before any analytical tests are initiated. Retained samples should be stored separately from analyzed samples under conditions that inhibit deteriorative loss, that is, at freezer temperatures and, for intact specimens, in the dry state. If retesting is called for, it should be done by an independent laboratory with different personnel. A defendant who believes that the match is spurious should welcome the opportunity for an independent repeat test. Legal aspects of retesting are discussed in Chapter 6.
Conclusions and Recommendations
Laboratory Errors
The occurrence of errors can be minimized by scrupulous care in evidence collecting, sample-handling, laboratory procedures, and case review. Detailed guidelines for QC and QA (quality control and quality assurance), which are updated regularly, are produced by several organizations, including TWGDAM. ASCLD-LAB is established as an accrediting agency. The 1992 NRC report recommended that a National Committee on Forensic DNA Typing (NCFDT) be formed to oversee the setting of DNA-analysis standards. The DNA Identification Act of 1994 gives this responsibility to a DNA Advisory Board appointed by the FBI. We recognize the need for guidelines and standards and for accreditation by appropriate organizations.
Page 88
Recommendation 3.1: Laboratories should adhere to high quality standards (such as those defined by TWGDAM and the DNA Advisory Board) and make every effort to be accredited for DNA work (by such organizations as ASCLD-LAB).
Proficiency Tests
Regular proficiency tests, both within a laboratory and by external examiners, are one of the best ways of ensuring high standards. To the extent that it is feasible, some of the tests should be blind.
Recommendation 3.2: Laboratories should participate regularly in proficiency tests, and the results should be available for court proceedings.
Duplicate Tests
We recognize that no amount of care and proficiency-testing can eliminate the possibility of error. However, duplicate tests, performed as independently as possible, can reduce the risk of error enormously. The best protection that an innocent suspect has against an error that could lead to a false conviction is the opportunity for an independent retest.
Recommendation 3.3: Whenever feasible, forensic samples should be divided into two or more parts at the earliest practicable stage and the unused parts retained to permit additional tests. The used and saved portions should be stored and handled separately. Any additional tests should be performed independently of the first by personnel not involved in the first test and preferably in a different laboratory.
