BioWatch is an air monitoring system deployed in jurisdictions around the country with the goal of detecting the presence of certain high-risk pathogenic microorganisms. It relies on a network of federal and nonfederal collaborative relationships to be successful, and is one part of a larger array of disease surveillance, intelligence-gathering, and biomonitoring activities in support of public safety and health.
The assays used in the BioWatch system to detect the presence of pathogens in collected samples rely on the technique of polymerase chain reaction (PCR) to sensitively and specifically amplify target nucleic acid sequences. The program and its users need an understanding of each assay’s performance characteristics in order to have confidence in the results and have the ability to appropriately interpret them. This confidence is partly provided through a performance standard, which gives information on the minimum requirements that must be met for the assay to be considered acceptable for its intended purpose and describes how testing to validate this performance is to be carried out.
The report discusses principles of performance standards, reviews information from several existing guidance documents and standards that might be applicable to BioWatch, and discusses assay testing efforts that have occurred or are ongoing. It provides recommendations on general principles and approaches for a performance standard and validation framework to meet BioWatch’s mission. It also provides a starting point for an approach to in silico and laboratory assay characterization for consideration by BioWatch and its stakeholders, with the aim of providing a reasonable amount of information on performance parameters such as an
assay’s limit of detection, sensitivity, and specificity. As tasked, the report focuses on the PCR reaction portion of a PCR assay; although it discusses key processes before and after the PCR, these are not addressed in detail.
One fundamental aspect of assay validation for a system such as BioWatch should be recognized and clearly communicated to program stakeholders—BioWatch is designed to detect rare but high-consequence events and all assays will have some associated false positive and false negative rates. It is not practical in time, effort, or money to conduct laboratory-based assay performance testing using large enough numbers of replicates under enough different conditions to validate the assay’s long-term false positive or false negative rate at the high levels of statistical confidence program users are likely to desire for operational decision making (e.g., to determine with statistical confidence that a false result will not occur more than once per x thousands or hundreds of thousands of samples). The assay design and laboratory performance validation that is the focus of the present report thus provides initial information that the assay is anticipated to perform reasonably well. It is critical for the system’s users to understand the testing that has been undertaken and the limitations of the data, and to combine this knowledge with ongoing analysis of data obtained from verification in jurisdictional laboratories and from operational deployment of the assay in order to achieve confidence in the results and to facilitate responses to positive BioWatch detections. The report emphasizes the importance of communication and, particularly, of discussing the details of performance data with jurisdictional laboratory experts and officials. The report also emphasizes the importance of obtaining a better understanding of the microorganism background that is present in BioWatch jurisdictions, information that remains very limited.
Finally, the report considers how developments in technology, particularly in multiplex PCR and next-generation sequencing (NGS), can contribute to the ability of the BioWatch program to meet current and future challenges. Sequencing currently can be used to follow up on unexpected assay results from the jurisdictions. In the nearer term, targeted approaches coupled with NGS may be useful as a replacement for the current real-time PCR secondary assays because of their ability to analyze many more genomic regions for identification and characterization. The applicability of techniques such as metagenomic NGS to the program would be a longer-term prospect. The report recommends that the Department of Homeland Security and the BioWatch program continue to monitor and evaluate technologies as they develop. The program also should plan to work with laboratory users in the BioWatch jurisdictions, along with technology experts, to ensure that new technology brought into the program not only incorporates the best technology but also functions smoothly for the stakeholder community.
