National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

Summary

BioWatch is an environmental surveillance system designed to detect the presence of pathogenic microorganisms in air samples in order to provide early warning to local, state, and federal stakeholders in the event that one of these potential biothreat agents is present. BioWatch exists in the context of a number of complementary efforts to provide infectious disease surveillance (Figure S-1). These efforts include monitoring disease trends and case reporting for human and animal illnesses, syndromic surveillance (e.g., monitoring for notable increases in the incidence of symptoms such as fevers), and intelligence-gathering activities. The BioWatch system aims to complement these efforts in order to provide public health, emergency management, and law enforcement with an alert to the presence of one of the tested biothreat agents prior to the onset of large numbers of clinical symptoms. The system is thus meant to provide decision makers with the opportunity to initiate further investigations or undertake initial responses to minimize risks to public safety and health.

BIOWATCH MUST FUNCTION EFFECTIVELY AS A TOOL FOR DECISION MAKING

Although BioWatch is a federally funded air monitoring program under the Department of Homeland Security (DHS), the program’s real impact lies in its ability to function as a decision-making tool for local and state jurisdictions that operate the program. The BioWatch system includes indoor and outdoor collectors deployed in jurisdictions around the coun-

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

images

FIGURE S-1 Multiple sources of information provide infectious disease surveillance. BioWatch is an example of an air monitoring system and forms part of this network to support public health and security. Circles represent targets of surveillance information that inform monitoring and detection activities. The boxes show the process from biosurveillance through response.
SOURCE: Adapted from IOM and NRC (2011).

try, primarily in large urban settings, and relies on laboratory analyses largely conducted within the country’s public health infrastructure. Each jurisdiction maintains a BioWatch Advisory Committee composed of law enforcement, public health, environmental health and protection, emergency management and medical services, National Guard Civil Support Teams, hazardous materials teams, and political officials that engages in close consultation with federal partners from a number of agencies. But ultimately, the actions taken or not taken based on results obtained from BioWatch aerosol collectors primarily impact the local level.

BioWatch relies on a series of process steps utilizing polymerase chain reaction (PCR) as its amplification and detection methodology to indicate whether or not nucleic acid from one of the tested pathogens is present on an air collection filter. Subsequent actions following a declaration of a BioWatch Actionable Result (BAR, indicating the presence of a pathogen’s DNA) may be time-consuming, potentially disruptive, or expensive. Similarly, if the result of the BioWatch PCR assays is negative, individuals with responsibility for public health and safety need to understand the limits

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

of the assays in determining that there is indeed no current threat to the public from one of the pathogens tested by the program.

It is critical that the BioWatch participants and the user community have confidence in the methodologies employed by BioWatch. An important element of this confidence is that the PCR assays used in the program have performance characteristics that have been validated and that provide proper and sufficient information to program stakeholders. This issue is the central topic of this report and is the focus of the committee’s statement of task (Box S-1).

BOX S-1
Statement of Task

An ad hoc committee will be convened to conduct a study and prepare a report that will evaluate and provide guidance on appropriate standards for the validation and verification of polymerase chain reaction (PCR) tests and assays in order to ensure that adequate performance data are available to public health and other key decision makers with a sufficient confidence level to facilitate the public health response to a BioWatch Actionable Response (BAR).

Specifically the ad hoc committee will:

  1. Determine PCR assay test and evaluation criteria that will provide a reasonable measure of confidence to federal, state, and local public health officials and key stakeholders.
  2. Identify and evaluate the Stakeholder Panel on Agent Detection Assays (SPADA), the Public Health Actionable Assays (PHAA), and any other existing and proposed standards applicable for use in defining the performance (validation and verification) of PCR assays for the BioWatch and other programs to ensure confidence as identified in Subtask 1, above. Standards are to be evaluated in terms of performance, cost, and public health applicability.
  3. Examine current PCR protocols used by the BioWatch program and other relevant biosurveillance programs and determine if the processes used to assess the performance of these protocols and assays are adequate to meet the standards identified in Subtask 2, above.
  4. Determine whether improvements could be made by adopting changes based on the evaluation in Subtasks 2 and 3, above.
  5. Determine if any existing standards approach is conducive, taking into consideration cost, schedule, and data requirements, to measuring performance of a PCR assay in multiplexed format.
  6. In the event that no approach currently in existence is judged to be appropriate in Subtask 5, above, provide recommendations for aspects that a standard must include to measure performance of multiplex PCR technology.

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

PERFORMANCE STANDARDS AND VALIDATION SUPPORT CONFIDENCE IN RESULTS

The existence of appropriate performance standards and a rigorous process of validation provide the basis for confidence that PCR assay results meet users’ needs. An assay performance standard describes minimum requirements that must be met for the assay to be considered acceptable for its intended purpose and describe how testing to validate this performance is to be carried out. To be applicable to BioWatch, a standard to characterize PCR assay performance and conduct assay validation would need to

  • Establish that the assay has sufficient sensitivity to detect the release of a tested biothreat agent at a program-relevant concentration above the baseline of environmental background (i.e., it is fit for purpose);
  • Establish that the assay has sufficient specificity to detect pathogenic strains of concern but not to cross-react with other strains and organisms, within acceptable program false positive and false negative rates;1 and
  • Determine that the assay is sufficiently robust for routine operational use by jurisdictional laboratories and is otherwise acceptable to program users.

From a practical standpoint, a decision maker would be most interested in knowing that the BioWatch system is sensitive enough to detect an amount of DNA reasonably expected to be present in a filter sample following a pathogen release of the scale for which BioWatch is designed. This question is complicated by the many factors involved in understanding overall system performance, including physical and biological characteristics of the threat agent particle, modeling analysis of environmental transport, the nature of the collection device and collection matrix, and others. The committee was not asked to examine the full spectrum of steps in the BioWatch system. Rather, the scope of the report is limited

_______________

1 A false positive is a positive detection result for a given target when the target is not actually present. A false negative, on the other hand, is a negative result when the target is present. BioWatch has experienced prior positive assay results from detections of microorganisms in the environment that were not the result of intentional releases of a biothreat agent. Opinions vary on whether these results are “false positives” at the assay level, which may have performed as designed but detected naturally occuring microbes or revealed cross-reactivity with an organism having sequence similarity in the regions targeted by the PCR amplification reactions). A prior report on BioWatch classified such events as “BAR false positives” because they resulted in declaration of a BAR when intentional release had not occurred (IOM and NRC 2011). False negative rates are difficult to determine in routine or standard operation.

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

to the process for validating assay performance in a laboratory setting, commonly starting from extracted nucleic acid. Understanding assay performance even in this limited setting provides useful information in establishing the maximum sensitivity and specificity of the system, comparing assays when making decisions on which to consider deploying, and comparing the performance of different instrument platforms on a given assay. While a standard to characterize the analytical performance of an assay is thus an important component, program users and stakeholders need to be aware of its limitations in the full operational context.

The report considers a variety of parameters addressed by PCR performance standards and analyzes the implications of different types of performance approaches. Dimensions discussed in the report include

  • The concentration of nucleic acid from a tested pathogen that will be detectable by the assay with a sufficiently high probability of detection (limit of detection [LOD]);
  • The ability of the assay to successfully detect an identified minimal amount of nucleic acid from the set of organisms that makes up the targeted inclusivity panel (sensitivity);
  • The ability of the assay to successfully not detect or cross-react with nucleic acid from the sets of organisms on the nontargeted exclusivity and environmental background panels (specificity);
  • The ability of assay protocols to amplify and detect nucleic acid from filters that contain anticipated environmental background substances, such as dust and pollen, which can interfere with the PCR reaction (environmental interference);
  • The ability of the assay to perform under real-world use, including robustness of the assay as it is performed by different operators and in different laboratories (robustness and reproducibility);
  • The genetic diversity present in microbial species and strains, the continuing acquisition of new microbial genetic knowledge, and the need to regularly reassess assay performance (quality control and databases); and
  • The controls, reference materials, and quality assurance processes associated with performance standards and validation (standards and quality assurance).

A BIOWATCH ASSAY STANDARD CAN BE INFORMED BY EXISITING GUIDANCE

The report reviews existing guidance on the process for conducting laboratory validation of the performance of PCR assays. A number of efforts have already been undertaken to provide a foundation of data

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

on the performance of PCR assays that are used or could be used by BioWatch. Testing of relevant assays has been undertaken or is ongoing by groups such as the Los Alamos National Laboratory (LANL) and the Centers for Disease Control and Prevention (CDC), while previous efforts such as BioNet compared the performance of detection assays then in use by the Department of Defense (DOD) and CDC. The BioWatch program and its partners should be recognized for the valuable efforts they have already undertaken, as well as for their efforts to increase communication among the many users who participate in BioWatch. In concert with third-party contractors, DHS and the BioWatch program have designed laboratory arrangements for undertaking assay and device testing, assembled and quality-controlled DNA from a number of applicable inclusivity and exclusivity panel organisms, conducted testing to obtain system operational data, and established a quality assurance program with the jurisdictional laboratories in the network. These approaches represent appropriate options for characterizing real-time PCR assays for detection of biothreat agents.

The report seeks to build on this base of information. It discusses principles addressed by PCR performance standards, explores how these principles could be applied to meet the needs of the BioWatch program, and draws comparisons between these needs and existing PCR performance approaches, such as the Standard Method Performance Requirements (SMPRs) developed by the Stakeholder Panel on Agent Detection Assays (SPADA) and assay performance standards captured by the Public Health Actionable Assays (PHAA) program and Federal Standards for Assay Performance and Equivalency (FSAPE).

The available documentation on these standards provides a wealth of detail on the characterization and validation processes they recommend, including the testing that must be conducted and the controls to be included. SPADA generally addresses the needs of first-responder and private-sector detection development communities in support of public safety actions, while the PHAA program addresses public health decision making, including deployment of medical countermeasures. FSAPE was developed to support assay performance comparisons across federally owned and operated surveillance programs. These standards approaches have many similarities but also significant differences. For example, the number of strains required for inclusivity, exclusivity, and environmental background panels under PHAA and FSAPE are generally more numerous than the strain panels for testing under SPADA. These differences have implications for the time and costs necessary to comply with these standards. The approaches of the different standards also differ in how they were developed. SPADA meeting information is posted online and the resulting performance requirements are published through the Journal

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

of AOAC International. PHAA and FSAPE are focused on federally owned programs; as such, their requirements are not published in the open literature.

The BioWatch program is a hybrid entity. Its results are used to support both public safety and public health actions. Although it is federally owned and relies on important federal relationships, it cannot succeed without close engagement with nonfederal partners. This fact makes it particularly important that any assay performance standard and associated validation program used by BioWatch be as transparent as possible to the range of the program’s users. BioWatch particularly relies on the expertise of public health laboratorians to conduct and interpret assay results within their jurisdictions, and these experts need to be aware of the available performance data and be engaged in discussions surrounding assay performance issues that arise.

BETTER UNDERSTANDING ENVIRONMENTAL BACKGROUND WOULD IMPROVE THE UNDERSTANDING OF ASSAY PERFORMANCE

The current BioWatch approach focuses on testing for the presence of a subset of known biothreat agents considered to be of particular priority. Assay performance is likewise defined by testing sets of known inclusivity and exclusivity organisms available in genomic databases and in various strain collections. A fundamental issue is that knowledge of the relatedness of strains used for assay testing to microorganisms actually present in BioWatch environments remains very limited. An assay can be tested against one hundred strains, but if these are not the most relevant ones it will mean very little in practice. A frustration among BioWatch jurisdictions is the occurrence of positive detections that appear not to indicate biothreat agent releases. Many pathogens of relevance to BioWatch can occur naturally in the environment—certain pathogens may predictably trigger detections at specific times of the year under particular conditions in some locations. Certain assays also may cross-react with an environmental organism that is not yet included on the strain panels against which the assays are tested. To help address these issues:

Recommendation 1. The Department of Homeland Security should undertake further characterization of environmental background organisms over time and at various locations. This sampling could build on the existing BioWatch network by using previously tested filters as samples and should also include judicious sampling of water and soil samples. Operational assay performance data in individual jurisdictions based on positive screening results might provide a source of invaluable data, as well.

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

A PERFORMANCE STANDARD FOR BIOWATCH SHOULD ENCOMPASS A FULLER RANGE OF STEPS THAN CURRENT GUIDANCE

In addition to PCR amplification, multiple steps influence PCR assay performance (Box S-2) and there are multiple ways in which the term “PCR assay” can be defined. In particular, steps in the overall process that feed into the PCR amplification are critically important to overall performance. Sample collection, extraction, and purification methods affect the type and quality of materials that are available as the input to the PCR. The Minimum Information for Publication of Quantitative Real-Time PCR Experiments guideline (Bustin et al. 2009), for example, considers information on the sample processing steps to be essential to a description of the assay and its results.

Guidance available through the SPADA SMPRs and in PHAA and FSAPE documents does not provide explicit advice on the sample extraction and preparation methodology to be used and one might also expect the most appropriate sample processing and preparation method to vary based on the specifics of a particular assay and its intended purpose. The BioWatch program makes protocols available to its jurisdictional laboratories in order to provide standardized methods for the daily operation of sample processing and DNA extraction from BioWatch filters. The program has indicated that it is considering opportunities to improve these processes, but that such efforts are currently decoupled from PCR assay test and validation needs that are the subject of the present study. As requested by DHS, the present report focuses on the core PCR reaction in the overall process from collection of a BioWatch air sample to a result, and the key processes before and after the PCR are not significantly addressed. However, the committee notes that evaluating performance of PCR amplification starting with purified strain nucleic acid represents a simplified and only partial view of a complex problem. The committee encourages the BioWatch program to consider the critical roles of the steps surrounding the PCR amplification and concludes that it would strengthen a BioWatch PCR assay performance standard to explicitly consider guidance on steps such as sample extraction and preparation as a standard is developed:

Recommendation 2. A PCR assay performance standard that meets program and user needs should include development and implementation of standards that encompass all of the individual steps in the assay from sample collection, extraction, amplification, detection, reagent quality control, and data analysis. There should be an integrated assay standard, as well, that includes all of the steps of a complete PCR assay as a unit.

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

BOX S-2
A PCR Assay Comprises Multiple Steps

PCR is a widely used nucleic acid amplification technique, and its inventor, Kary Mullis of Cetus Corporation (Emeryville, CA), was awarded the Nobel Prize in Chemistry in 1993. PCR involves polymerization of nucleotides in a reaction performed and mediated by enzymes and thermal conditions, resulting in a theoretical doubling of the amount of targeted nucleic acid in each cycle. In the BioWatch program, PCR is the basis of the targeted amplification of specific nucleic acid sequences in order to identify whether nucleic acids from biothreat agents tested by the program are present on collected filters.

The report focuses primarily on the PCR amplification in the overall PCR assay, including testing undertaken to determine the ability to detect or not detect relevant pathogen strains and environmental organisms, and to determine the limit of detection.

A complete PCR assay includes steps in addition to the amplification reaction, including input sample extraction and preparation, reaction conditions, method of detection and instrument platform, and data analysis. Setting standards for PCR performance in a realworld setting cannot be done in isolation from these other dimensions of the assay, which affect overall performance. Although the report mentions aspects of the steps that compose the full assay, it does not, for example, make recommendations on specific sample processing methods, detection methodology, or result algorithms that should be used by the BioWatch program.

images

BOX FIGURE. Several key steps precede the PCR amplification. When combined with a detection mode, the set of steps forms a complete PCR assay. The report focuses primarily on PCR amplification and subsequent steps.

STARTING POINT FOR A BIOWATCH ASSAY PERFORMANCE STANDARD

The approach currently used by BioWatch—sensitive real-time PCR to amplify target regions of DNA from a set of known pathogenic organisms judged to be of highest risk—represents the best system BioWatch has been able to deploy given current technology and assay options. The program needs a basis for understanding the performance of its assays, as tested in laboratory settings, in order to inform a more comprehensive

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

understanding of its system. The report highlights an example of a performance standards approach for conducting laboratory testing of assays that could provide reasonable statistical confidence for BioWatch users while seeking to keep numbers of tests and numbers of strains manageable for intended use. The specific details of the process ultimately used by the BioWatch program should be determined by the program and established in consultation with its stakeholder community.

In general, primary performance characterization and validation of an assay should be undertaken by the developer (or by a laboratory contracted by the developer). Further independent validation should be undertaken by DHS. Verification of assay and system performance should be undertaken by user laboratories independent from the developers. A smaller number of replicates can be used to verify assay performance (e.g., LOD, inclusivity, exclusivity, and environmental panel results) than was undertaken during full validation testing. The committee emphasizes that modeling and testing to understand performance needs in real operational contexts, monitoring sample flow through all stages of the system, and analyzing assay performance data in operational use must be conducted in addition to the initial laboratory assay validation. The committee recognizes that the BioWatch program has been undertaking and continues to undertake various types of system testing. In developing a performance standards approach to meet BioWatch’s mission:

Recommendation 3. BioWatch should strive to test its PCR assays against as many known strains of its target agents and relevant near neighbors as possible. Undertaking all such testing through laboratory analysis alone is likely to be cost, time, and feasibility prohibitive. Increased in silico testing should thus be incorporated into the program to support targeted laboratory testing.

Recommendation 4. To support the goal of Recommendation 3, the BioWatch program should work with other federal agencies and international partners to ensure that as many strains as possible are sequenced to at least a high-quality draft level and the data and associated reference materials made available to PCR assay and device developers through reference databases.

Recommendation 5. The panels of inclusivity and exclusivity strains specified for laboratory testing should seek to balance broad phylogenetic and geographic diversity within the realm of practical testing. Inclusivity panel strains for laboratory testing should sample the genetic diversity represented by available strain collections with an emphasis on prioritizing those strains that cause significant morbidity and mortality, have high

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

transmissibility, have a wide host range, or may be readily accessed by nefarious actors.

Recommendation 6. For laboratory testing of exclusivity panels (near neighbors and environmental organisms), a strategy of sample pooling is appropriate to increase testing efficiency. If a positive result is detected, each agent in the pool should be tested separately.

Recommendation 7. Assay performance in the presence of known environmental contamination (e.g., rail dust in a subway system) should be addressed by assessing the assay detection limit for a limited number of inclusivity and exclusivity panel agents spiked on jurisdictional background filters.

Recommendation 8. Federal agencies, including the Department of Homeland Security, the Department of Defense, the Department of Energy, the Department of Health and Human Services, and other partners, should collaborate to produce a reference document or database identifying which repositories contain strains or strain nucleic acids relevant for testing pathogen detection assays. Information on the requirements or procedures to obtain and use strains or strain materials (e.g., extracted nucleic acids) held by these repositories should be included, where possible. This information would provide a coherent source of information to both federal agency programs and to individuals or companies involved in assay and device development and validation.

Recommendation 9. The BioWatch program should institute a process to regularly review and update all assays in light of new genetic knowledge. The program should re-evaluate its assays through at least annual in silico screening to take advantage of new genomic information in databases, update signatures, and identify potential false positive or false negative issues. Similarly, the composition of inclusivity, exclusivity, and environmental panels used for testing should be reviewed regularly and laboratory re-testing conducted as needed. Annual in silico and periodic laboratory re-testing of assay performance should be part of any assay performance standard accepted for BioWatch.

The above recommendations provide strategies for a BioWatch PCR standard. Without duplicating the full level of detail contained in guidance documents such as the AOAC International validation guidelines, SPADA SMPRs, PHAA and FSAPE documents, and other sources, the committee also attempted to provide a specific starting point for methods to determine assay sensitivity and specificity that could be considered

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

by BioWatch in developing a standard to meet program needs. Further details, including the statistical confidence and limitations associated with this approach, are discussed in Chapters 3 and 4. In addition, the committee concluded that ongoing analysis of assay performance data from verification in user laboratories and from operational deployment of the assay in the program is critical. It will not be possible to undertake laboratory validation testing using large enough numbers of replicates on samples from the full spectrum of potential inclusivity, exclusivity, and environmental organisms in the presence of a wide range of different types of potentially interfering substances and different filter conditions to characterize the long-term false positive and false negative rates of an assay to a level of statistical confidence users may desire operationally. Nevertheless, some reasonable compromise must be achieved in order to undertake performance characterization and validation work on assays deployed or being considered for deployment in the program. Understanding and clearly communicating the strengths and limitations of the laboratory data obtained from the PCR assays is crucial to interpreting results and using them as a basis for taking action.

Recommendation 10. The following approach can serve as a starting point for a standard to provide confidence in PCR performance while seeking to undertake a reasonable amount of laboratory characterization and validation testing:

  • A reasonable approach to determining an assay’s analytical LOD in a laboratory setting is to conduct serial dilution at a range of concentrations bracketing the estimated LOD, using n = 60 replicates with acceptance criteria of at least 58/60 for a given concentration, followed by appropriate curve fitting. LOD testing should also be conducted in the presence of realistic background matrix, such as previously tested BioWatch filters, to gain an understanding of how the LOD may be affected by operational background.
  • The use of in silico screening presents a relatively fast and low-cost way to predict one part of assay performance using as many strains as are available as high-quality sequences in reference databases. In silico screening should be undertaken using the set of PHAA panels, if these strains’ sequences are available.
  • The strain panels recommended by SPADA represent a good starting point for laboratory validation testing of BioWatch assays and were developed with stakeholder participation. However, many were developed in 2010-2011 and should be reviewed and updated to account for new genetic knowledge.
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
  • The strategy used by LANL for testing detectability or nondetectability of inclusivity and exclusivity strains represents a reasonable model for BioWatch (testing n = 20 replicates, followed by testing 20 further replicates if an unexpected detect or nondetect result is obtained, with acceptance criteria of 20/20 or 39/40).

It is not possible to predict the precise costs of the committee’s proposed validation framework. Data available to the committee on costs of assembly and quality control of relevant strain materials and to undertake validation testing following SPADA and PHAA guidance are available in Table 3-3 for comparison. The costs of the committee’s approach appear to be comparable in scope.

It should also be noted that the BioWatch program currently employs an initial screening assay, provided through the DOD’s Critical Reagents Program, to detect a single target nucleic acid sequence per tested pathogen. If this screening assay yields a positive detection, a secondary assay panel, provided through the CDC Laboratory Response Network, is used to test for three or more additional pathogen sequences. A basic performance standard and validation framework can be appropriate for assays used for both screening and for secondary testing. However, the assessment of testing results in order to determine whether the assay meets program needs will be different depending on the way(s) the assay is to be used in the program. Because of the importance of specificity to the performance of a secondary assay, the program might conclude that it is useful, for example, to conduct laboratory validation using additional inclusivity and exclusivity strains than were used for an initial screening assay. An approach of serial testing or “tiers” of recommended testing makes sense from a practical perspective given limited resources. Thus, an assay meant to be used for secondary verification or confirmation might be validated against a larger number of inclusivity and exclusivity strains than an assay meant to be used as an initial screen.

ALTERNATIVES TO THE CURRENT SCREENING AND SECONDARY ASSAY APPROACH

The report also explores several options that BioWatch could consider as alternatives to its current model of initial screening and secondary PCR assays, which would have implications for the performance characterization and validation process required. These include the following:

  • Develop subsets of assays for each of the tested BioWatch agents rather than a single set of screening and secondary assays (e.g.,
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
  • incorporate additional types of rule-in and rule-out assays for certain pathogens, along with decision algorithms on when to use which tests);

  • Develop and deploy multiplex PCR assays; and
  • Incorporate next-generation sequencing (NGS) into the program as technology and software development makes this capability increasingly feasible.

Multiplex PCR

Each PCR assay currently used by BioWatch amplifies a single target sequence of pathogen nucleic acid (e.g., the assays are run as singleplex reactions). Assays also can be designed to detect multiple target sequences per reaction and technology in this area is moving fast, particularly in the development and use of multiplex assays in clinical settings. Multiplex PCR assays could be used by BioWatch and have the potential to reduce the labor incurred by jurisdictional laboratories as they conduct routine filter testing.

Appropriate standards and validation guidance recently have emerged to address the additional issues that arise when characterizing and validating multiplex assays. Because PCR remains the core detection technology, many of the same performance principles apply. Additional types of testing will need to be conducted for multiplex assays, for example, to determine that each of the target sequences can be sufficiently amplified in the reaction and that none of the components in the reaction interfere with each other. Other potential issues involved in developing successful multiplex assays include determining the concentrations of different reagents, primers, and probes (including the need to limit certain reagents) when aiming to amplify and detect two or more pathogen targets, and the potential for nonspecific amplification products to arise. A further complication arises when making subsequent changes to a previously validated multiplex assay, in terms of the extent of performance re-testing that is required.

Recent guidance from the Food and Drug Administration (FDA) provides a path forward in such circumstances, targeted to those submitting materials under FDA regulatory approval. For example, FDA guidance suggests that changes to already approved assays (such as inclusion of a new target) can be made using subsets of testing to confirm performance, rather than requiring the full range of validation testing to be repeated. The committee concluded that existing guidance on singleplex PCR performance for environmental biodetection assays in concert with FDA information should provide a good starting point for the performance testing and validation of multiplex assays by the BioWatch program .

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

Additional considerations will arise with multiplex assays as compared to singleplex assays, due to the need to validate mixtures of targets and to investigate potential cross-reactivities. However, the development of multiplex assays for use in BioWatch appears reasonable and could likely be achieved without an unacceptable loss of sensitivity and specificity for program users. More important, however, the use of multiplex assays in BioWatch would require agreement from program stakeholders, particularly the directors of the jurisdictional laboratories who conduct assay analyses and lead public health officials with responsibility for using the information obtained. Obtaining this agreement will require improved mechanisms for sharing and discussing detailed assay performance data. The additional issues that arise when designing, validating, and deploying multiplex assays will need to be considered when making the decision to switch to a multiplex format.

Next-Generation Sequencing

Moving forward, the committee urges the BioWatch program to consider the ways in which emerging technology, particularly next-generation sequencing, can help the program address some of the challenges inherent in its current approach, which is focused on detecting a subset of high-priority, previously known pathogens. Next-generation sequencing presents possibilities for the program to move to a system that can be significantly more flexible and broad-spectrum in identification. The program already can use sequencing as a tool to help resolve unusual assay results. In this way, sequencing can function as a tertiary confirmation and characterization tool. Several additional ways that NGS could be used by the program and current barriers to implementation are summarized in Table S-1.

In the relatively near term, targeted NGS approaches may become useful to the program in place of current secondary PCR assays because of the ability to analyze more regions for identification and characterization, thereby improving specificity of detection. Metagenomic sequencing, on the other hand, could assist in characterizing environmental background and might eventually enable the implementation of universal assay protocols that would minimize the effort and costs needed to continually update and revalidate PCR primers, probes, and assay conditions. Sequencing and associated bioinformatics are not yet at a stage at which they could replace PCR assays as the BioWatch program’s primary detection methodology because of cost, turnaround time, lack of standardized reference databases for comparison, and complexity of data analysis. Incorporating sequencing-based approaches into BioWatch will also require jurisdictional training, expertise, and infrastructure that do not

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

TABLE S-1 Applications of Next-Generation Sequencing for BioWatch

Key Challenges/Barriers to Implementation
Related Terms Purpose Advantages Turnaround Time Cost Complexity of Bioinformatics Analysis Challenges of Interpretation
Primary BioWatch assay Most inclusive X X X X
Environmental and detection filter background measurement Enable detection of "natural" positive controls (fungal spores, etc.) X X X
Metagenomic sequencing Shotgun sequencing, random sequencing, unbiased sequencing Better understand sources of false-positive signals
Improve design of primers and probe targets used for primary screening
Novel pathogen discovery Identification of novel or divergent pathogens that would go undetected by existing tests X X X
Microbial whole-genome sequencing Whole-genome resequencing, de novo genome assembly Optimize design of primers and probes Will identify new targets for PCR and improve primer/probe design X
Expand reference databases Will better capture target diversity and inform primer/probe design X X
Targeted sequencing Amplicon sequencing, biased sequencing, conserved region sequencing Investigate false-positive results Better understand sources of false-positive signals X X X
Primary BioWatch assay More informative than "yes-no" result from real-time PCR; more inclusive of diverse strains X X
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

currently exist in all laboratories. Nevertheless, exploring these options and incorporating them into the program through pilot testing will help BioWatch meet future challenges, such as the potential emergence of new or unknown pathogens or the possibilities for nefarious activities such as bioterrorism enabled by synthetic biology.

Recommendation 11. The Department of Homeland Security should monitor developments in technology, such as next-generation sequencing, as speed increases, cost decreases, and required informatics and databases improve. A program to evaluate technology and incorporate it into BioWatch would enable the program to improve its ability to address current challenges and enable it to meet those of the future. However, the program should plan to work with at least some early adopter laboratory users in the BioWatch jurisdictions, along with technology experts, to make sure that new technology brought into the program not only incorporates the best technology but also functions smoothly for the stakeholder community.

THE IMPORTANCE OF COMMUNICATION TO PROGRAM SUCCESS

Finally, one of the primary messages of the report is that the network of BioWatch stakeholders should cooperate and be integrally involved in the process of establishing standards that will meet the community’s needs and should have access to the relevant data to help them understand the performance and limitations of the assays in order to enable appropriate interpretation of the results. Building additional, specific mechanisms into the program to share assay and system performance results, discuss what these performance results reveal about the limits of the assay and the translation of assay results into actions, and share applicable detailed assay performance information across agencies such as DHS and CDC and with appropriate individuals within the network of jurisdictions appears to the committee to be a critical need. Even while recognizing the necessary security constraints, this goal should be achievable. Addressing this issue would go far in ensuring confidence in the system and its results in order to facilitate responses to a BAR.

Recommendation 12. A systematic process should be instituted to identify the root cause of positive assay results that do not appear to represent the targeted microorganism and to determine appropriate corrective actions to address the cause(s). This feedback and revision process should include the participation of laboratory experts in the jurisdictions and federal experts.

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×

Recommendation 13. Communication and information-sharing are necessary to establish confidence in a system. The BioWatch program, relevant federal agencies, and local and state jurisdictions should expand the communication and data-sharing that occurs among the network of federal and nonfederal partners involving both the screening and the secondary PCR assays. This effort would assist in establishing acceptable performance standards, enhance informed data interpretation and decision making, improve the ability to undertake root cause analysis of assay issues encountered by jurisdictions, and enable the collective identification and dissemination of actions as part of robust quality assurance.

The ability to detect pathogen nucleic acid from BioWatch collectors in such a way that the information is usable for decision making and is obtained on a sufficiently rapid timescale and at reasonable cost is fundamental to the acceptability and success of the program. The report aims to contribute to the discussions surrounding performance standards and validation approaches that can meet these BioWatch program needs. A valuable repository of data and performance guidance can be drawn on to support BioWatch’s mission and the program has a strong foundation from which it can develop performance and validation standards. Although it will require the participation of multiple groups, there is an opportunity to further strengthen the system by incorporating additional data-sharing and performance review features into the existing validation and quality assurance framework.

Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 3
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 4
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 5
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 6
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 7
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 8
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 9
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 10
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 11
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 12
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 13
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 14
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 15
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 16
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 17
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 18
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 19
Suggested Citation:"Summary." Institute of Medicine and National Research Council. 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/21658.
×
Page 20
Next: 1 Introduction »
BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version Get This Book
×
Buy Paperback | $52.00 Buy Ebook | $41.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

BioWatch is an air monitoring system deployed in jurisdictions around the country with the goal of detecting the presence of certain high risk pathogenic microorganisms. It relies on a network of federal and nonfederal collaborative relationships to be successful, and is one part of a larger array of disease surveillance, intelligence-gathering, and biomonitoring activities in support of public safety and health. The assays used in the BioWatch system to detect the presence of pathogens in collected samples rely on the technique of polymerase chain reaction (PCR) to sensitively and specifically amplify target nucleic acid sequences.

BioWatch PCR Assays evaluates and provides guidance on appropriate standards for the validation and verification of PCR tests and assays in order to ensure that adequate performance data are available to public health and other key decision makers with a sufficient confidence level to facilitate the public health response to a BioWatch Actionable Response. This report discusses principles of performance standards, reviews information from several existing guidance documents and standards that might be applicable to BioWatch, and discusses assay testing efforts that have occurred or are ongoing. BioWatch PCR Assays provides recommendations on general principles and approaches for a performance standard and validation framework to meet BioWatch's mission. The report also considers how developments in technology, particularly in multiplex PCR and next-generation sequencing, can contribute to the ability of the BioWatch program to meet current and future challenges.

This report has been determined to contain information exempt from disclosure under 5 U.S.C. 552(b). Section 15 of the Federal Advisory Committee Act provides that the National Academies shall make its final report available to the public unless the National Academies determines that the report would disclose matters described in one or more of the exemption provisions under the Freedom of Information Act (FOIA). In such case, the National Academies "shall make public an abbreviated version of the report that does not disclose those matters." This unrestricted, abbreviated version of the report represents, in so far as possible, the committee's findings, recommendations, and other substantive material without disclosing materials described in 5 U.S.C. 552(b).

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!