Strategies for Effective Improvements to the BioWatch System Proceedings of a Workshop (2018) / Chapter Skim
Currently Skimming:
Appendix F: Invited Paper: Current Quality Assurance (QA) Approach: DHS BioWatch Program
Appendix F: Invited Paper: Current Quality Assurance (QA) Approach: DHS BioWatch Program
Pages 126-143
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 126... ...
Appendix F Invited Paper: Current Quality Assurance (QA) Approach: DHS BioWatch Program Disclaimer: This white paper was prepared for the September 18-19, 2017, workshop on Strategies for Effective Biological Detection Systems hosted by the National Academies of Sciences, Engineering, and Medicine and does not necessarily represent the views of the National Academies, the Department of Homeland Security, or the U.S.
|
From page 127... ...
The methods and instruments used by each operational laboratory are determined by the BioWatch program and are specifically described in BioWatch program analytical standard operating procedures (SOPs)
|
From page 128... ...
in 2003 in support of fixed and mobile laboratories performing chemical, biological, radiological, nuclear, and explosives detection for DoD. Although the BioWatch QA program is fully tailored to the DHS BioWatch mission and constraints, DHS has maintained strong interagency communication and collaboration with JPEOCBD, the National Guard Bureau Weapons of Mass Destruction Civil Support Teams, and other DoD laboratories performing biothreat analyses in the area of QA.
|
From page 129... ...
PT samples are prepared on a half filter using a common background for the entire pro gram and laboratories are required to follow the protocols applied to operational samples, including the determination verification analyses to run based on the screening results, and the application of the decision algorithm to make a final call regarding the presence or absence of an agent. By focusing on decision-level results, PTs serve as a comple ment to the QA samples.
|
From page 130... ...
; traceability requirements; QA considerations and requirements associated with collection and analytical procedures; data review and reporting procedures; performance assessment (including quality control samples, external QA samples, proficiency tests, and metrics associate with performance assessment data) ; implementations of new methods, assays, and platforms; and QA reports to the DHS BioWatch program.
|
From page 131... ...
If a critical deficiency is identified, it is immediately communicated to the DHS BioWatch program and the affected laboratory. Corrective action is required to be implemented for all deficiencies identified during the audit.
|
From page 132... ...
If an operational laboratory is nonresponsive to the audit report after attempts by the DHS BioWatch program to resolve any issues, the DHS BioWatch program may temporarily suspend operational laboratory operations at that site and ship samples to an alternate operational laboratory for analysis, while working with the operational laboratory to bring them into compliance. In addition, for deficiencies determined to be critical, an immediate shutdown will be considered by the DHS BioWatch program until the laboratory has shown that the discrepancy has been
|
From page 133... ...
QA samples and PTs are planned, developed, and provided to the laboratories by the QA contractor and SSL, under the direction and guidance of the DHS BioWatch program. Details associated with QA and PT samples are provided below.
|
From page 134... ...
As shown in the chart, as the spike level increases, the number of false negatives due to the material concentration decreases, but the number of false negatives due to other contributors remains the same. In this example, levels 7 and 8 are considered optimal levels because they represent the level at which concentration-related false negatives do not occur so often that they mask other causes, making it difficult to discern what is contributing the errors, and the concentrations are not so high that the program loses the ability to detect issues that could affect operational samples (e.g., extraction efficiency and/or inhibition issues)
|
From page 135... ...
Sample matrices are quarter filters that contain environmental backgrounds specific to each laboratory and are obtained from the archive filter inventory maintained by the SSL. For laboratories that pool samples, QA sample sets are provided (i.e., for laboratories that pool operational samples 3:1, one spike and two blank QA samples are provided to the laboratory as a set to be pooled together for screening analysis)
|
From page 136... ...
Proficiency testing is also commonly recognized as a key element in a comprehensive QA program and helps to strengthen the overall defensibility of results from laboratories participating in the DHS BioWatch QA program. Each proficiency test currently consists of a panel of six sample sets per operational laboratory, with identical panels sent to each laboratory (but with a different sample ID order)
|
From page 137... ...
If an operational laboratory is nonresponsive or continues to observe proficiency test failures after attempts by the DHS BioWatch program to resolve any issues, the DHS BioWatch program may temporarily suspend operational laboratory operations at that site and ship samples to an alternate operational laboratory for analysis, while working with the operational laboratory to bring them into compliance. Metrics and Data Quality Expectations and Goals for QA and PT Samples The key laboratory analysis performance parameters, or metrics, include assay-level error rates and decision-level error rates.
|
From page 138... ...
Decisionlevel false-negative rates are computed as the proportion of test samples that have been spiked with an inactivated agent and/or other spiking material in which decision-level false negatives are observed. Evaluation of decision-level false negatives provides information about decision-level sensitivity.
|
From page 139... ...
The laboratory/assay false-positive rate is computed as the number of assay-level false positives for each individual assay at each individual laboratory, divided by the number of QA or PT samples analyzed by laboratory that were not spiked with materials that are expected to be reactive for the assay. Error Rates by Individual Assay The assay-specific program-wide assay-level false-negative rate is computed as the number of assay-level false negatives for each indi vidual assay across all laboratories, divided by the number of QA or PT samples analyzed by all laboratories that were spiked with materials (at a designated spike level)
|
From page 140... ...
.3 In order to meet the program expectation (false-positive or false-negative rate is not statistically significantly greater than 2 percent or 5 percent) , the 95 percent LCL must not exceed 2 percent or 5 percent, respective 3Together, a 95 percent LCL and a 95 percent UCL comprise a 90 percent confidence interval.
|
From page 141... ...
Rather, the focus of accuracy is on the closeness of the average Ct value for samples that have been spiked with a given agent and spike level to the expected Ct-value range for that agent and spike level based on historical responses observed at each individual laboratory. Accuracy is evaluated separately by spiking material or nucleic acid signature.
|
From page 142... ...
TABLE F-1 QA Reports to DHS BioWatch Program Report Description/Frequency Critical Elements Interpretation The QA contractor is Contributes to the responsible for reviewing evaluation of the external QA sample results performance and operational sample results capabilities and For each laboratory and (i.e., reactive rates) , as well as trends by assay: the SSL QC data, on an laboratory, Summary tables of ongoing basis.
|
From page 143... ...
Description of expected results and evaluation approach Each report includes Provides a routine Statistical summaries The QA contractor performs a summary of of performance comprehensive review of program laboratory capability and trends operational laboratory performance and or patterns in QA performance at least provides the basis results semiannually. This includes a for identifying and Trends Summary of PT review of external audits, addressing Analysis results summary reports from program-level Report Summary of laboratories (if received)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.