BioWatch PCR Assays: Building Confidence, Ensuring Reliability: Abbreviated Version (2015)

2

Performance Standards Provide Confidence in Results

A laboratory in a BioWatch jurisdiction obtains a positive result from the screening and secondary assays it runs to detect the presence of a biothreat agent. The collector from which the sample came is located in a busy station in the city’s mass transit system. A decision needs to be made regarding an actionable response. If officials evacuate the station, significant disruption will result. If they shut down the station for what turns out to be a non-event, they are likely to waste resources, face public anger, affect the local economy, undermine confidence in the city’s leadership, and face political consequences. On the other hand, if officials fail to take action for what turns out to be a release of a biological pathogen, the consequences could be far more severe. Therefore, the jurisdiction must know how to interpret the assay results it obtained and what risks are involved with that interpretation.

ESTABLISHING APPROPRIATE PERFORMANCE STANDARDS HELPS RESULTS BE ACTIONABLE

The BioWatch system is substantially more complex than described in the simple scenario above. Nevertheless, experts in the laboratory, along with public health and law enforcement decision makers, need a solid foundation from which to understand assay results and use them in the subsequent decision process. This foundation relies on defining the purpose for which the assay is used, establishing criteria the assay must fulfill to meet its intended purpose, and validating that the assay indeed performs

within requirements (or at least to define the assay’s limitations). These three pieces—defining purpose, establishing performance standards, and undertaking validation—provide crucial information on which to establish the significance of a result. This chapter and Chapter 3 focus on fundamental characteristics that form the basis for polymerase chain reaction (PCR) assay performance standards. Chapter 4 returns to the validation framework and explores how performance standards fit into the overall picture.

ASSUMPTIONS AND DEFINITIONS FOR “PCR ASSAY” AND RELEVANT STANDARDS

Many terms related to PCR performance characterization and validation have ambiguous definitions in common use. A description of the terms as they are used in the report is provided below.

PCR Assay

There are several ways in which the “PCR assay” step in the BioWatch system could be defined. For example, the assay could be defined to include the set of steps from sample extraction to readout (steps beyond simply thermal PCR amplification cycles and the associated chemistry). The Food and Drug Administration (FDA), for example, generally considers sample extraction in concert with amplification and detection in its device submission guidance. The Environmental Protection Agency (EPA) similarly includes sample collection and extraction in its guidance on PCR method validation because of the impact that these procedures have on the PCR amplification. The BioWatch program, on the other hand, appears to define the PCR assay as starting from an already-extracted sample of nucleic acid (it does not include the collection device or extraction processes), but does include the reagents and analytical platform used for amplification, detection, and data output. The BioWatch program makes protocols available to its jurisdictional laboratories in order to provide standardized methods for the daily operation of sample processing and DNA extraction from BioWatch filters. The program has indicated that it is considering opportunities to improve these protocols, but that such efforts are currently decoupled from the PCR assay test and validation needs that are the subject of the present study. For the purpose of the report, the committee likewise considers the PCR assay to begin with an extracted sample of nucleic acid. It includes the amplification and detection steps along with the required assay reagents and materials, but does not include the downstream decision making undertaken by BioWatch jurisdictions and other local, state, and federal partners following the generation of assay results. In Chapter 4, the committee returns to the impact of the system steps before and after the PCR assay.

Types of PCR Standards

There are multiple types of standards that could be applied to the development and use of PCR assays in BioWatch. These include

• PCR signature design and maintenance standards: PCR target sequences, or signatures, must be designed according to the best genomic data available at the time of assay creation. Furthermore, PCR signatures and the assays embodying them are an ongoing process rather than an end point; they must be scrutinized on a regular basis to see if new genomic information will predict false positive and false negative results. Standards guide the approaches used in signature design via in silico methods and through selected laboratory testing, and practices for signature maintenance.
• PCR assay performance standards for validation: ¹ These standards define broad guidelines for a PCR validation strategy, which of necessity must be tailored to some implementation details (e.g., what a “positive detection” means for particular detection chemistry). Independent or multilaboratory validation that the PCR assay meets performance standards is frequently undertaken as a complement to performance characterization work undertaken by the developer.
• PCR reference collection standards: PCR assays are tested against inclusivity and exclusivity strain panels (targets, near neighbors, and more distant nontarget organisms) and diverse backgrounds (soils, aerosols, pollens, dust, and other potentially problematic environmental backgrounds). A source of quality-controlled biological materials is needed. Reference standards collections may be provided by an ad hoc consortium or by individual researchers or maintained by a sponsor; access to the materials may be open to government, academia, and industry, or controlled by permission of the owner. In general, a collection of standard reference materials for an application such as BioWatch would include both extracted nucleic acids and live microbial cultures for use by qualified applicants.

___________________

¹ In this report, the term “validation” refers to the process of affirming that the test or methodology (in this case a BioWatch PCR assay) meets established performance requirements. Validation is also an overarching process or framework, rather than a single step, a concept discussed in further detail in Chapter 4. The term “verification,” on the other hand, is a process that affirms that a given user laboratory can obtain the anticipated results and assay performance from a validated PCR assay. These definitions are similar to the FDA’s definitions for the terms (FDA 2014c).

• PCR assay analysis and reporting standards: How assay results are generated and delivered is important to the BioWatch mission. These types of standards inform the use of a decision algorithm that considers the information provided by relevant controls and the measured result(s) of the PCR assay(s) in the test.
• End-to-end PCR system performance standards: Desired sample-in to result-out standards establish performance benchmarks for the overall system, such as the ability to detect a certain number of agents on a filter quadrant with defined overall success rates.
• PCR proficiency standards: Those using the BioWatch assays must be well trained in their use. BioWatch user laboratories verify that they can obtain the expected results from the assays they run. To achieve this, the laboratory will likely need to meet specific proficiency standards and undergo regular proficiency testing.

These different forms of PCR standards are interrelated. The committee’s task with regard to the BioWatch program focused primarily on the real-time PCR assays used by the program for biological agent detection. As a result, the committee focused particularly on PCR assay performance standards. However, the importance of regular signature maintenance, availability of standard reference collections, and the role of proficiency testing in the overall validation framework are mentioned in the report where appropriate.

Although not discussed in detail, the importance of performance characterization and validation applies equally to multiple processes in BioWatch operations that form part of understanding full system performance. When integrated into a system, several of the preliminary steps will directly impact the PCR assay step. For example, the composition of the collection matrix and airflow characteristics may influence how agents are deposited and recovered,² while the procedures used for extracting DNA from the filter directly affect the amount and quality of the material available for input to the PCR. Within an instrument, different methods for detecting and interpreting the signal from the PCR amplification can affect the assay results and performance for downstream decision-making algorithms. As Figure 2-1 emphasizes, these activities are interconnected within the BioWatch program and it is necessarily somewhat artificial to isolate the PCR assay step. Full system operational performance testing and the implementation of quality assurance standards for the program, discussed further in Chapter 4, do take into account additional steps from filter analysis.

___________________

² Currently, the BioWatch system makes use of aerosol deposition onto dry filters, but other types of collection approaches could be used, such as liquid-based collection.

FIGURE 2-1 BioWatch operations include multiple steps. PCR assay performance standards are associated with both the screening and secondary assays, and quality assurance standards are associated with the framework from sampling through output.

DEVELOPING AND USING PERFORMANCE STANDARDS

Guidance on the development and use of performance standards by the federal government is provided by the National Technology Transfer and Advancement Act of 1995 (NTTAA; P.L. 104-113) and Office of Management and Budget (OMB) Circular A-119 on Federal Participation in the Development and Use of Voluntary Consensus Standards and in Conformity Assessment Activities (OMB 1998³ ). These guidelines encourage agencies to make use of voluntary consensus standards and to participate in voluntary consensus standards efforts where they are relevant to the agency’s mission. The guidance reserves the right of agencies to use other types of standards, such as government-unique standards, where use of standards developed by a voluntary consensus body would be “inconsistent with applicable law or otherwise impractical” (NTTAA, § 12(d), 15 USC § 272).

___________________

³ The federal government currently is considering revisions to OMB A-119. See http://www.whitehouse.gov/sites/default/files/omb/inforeg/revisions-to-a-119-for-publiccomments.pdf.

Under the Homeland Security Act of 2002 (P.L. 107-296), the Department of Homeland Security (DHS) has been directed to conduct its standards activities in accordance with the NTTAA and OMB A-119.

In its analysis, the committee explored several assay performance approaches, some created by voluntary consensus bodies and published in the open literature and others considered to be government-unique and containing information restricted from public disclosure. The committee does not attempt to distinguish which are or are not performance standards under a formal definition of the term, but rather comments on the types of information they include and the potential tradeoffs and implications they present for BioWatch. For BioWatch to provide sufficient confidence to its users, however, some mechanism for engaging the participation of its relevant stakeholders is required, many of whom are outside of federal agencies. The report returns to this issue in Chapter 4.

PCR ASSAY PERFORMANCE AND VALIDATION APPROACHES THAT MAY BE APPLICABLE TO BIOWATCH

By 2005, several nonclinical surveillance programs were operational to detect selected biological pathogens. These included the BioWatch program, the Biohazard Detection System (BDS) used by the Postal Service to screen mail at major processing centers, and Department of Defense (DOD) programs such as the Guardian Installation Protection program to monitor specified U.S. and international military facilities. Two performance standards, in particular, also have since been developed that may be applicable for use or adaptation by the BioWatch program to validate the performance of the real-time PCR assays it employs. These are the performance requirements developed under the Public Health Actionable Assays (PHAA) program and the Standard Method Performance Requirements and Validation Guidelines established by the Stakeholder Panel on Agent Detection Assays (SPADA).

Public Health Actionable Assays

Under the leadership of the Science and Technology Directorate, DHS established a program to develop performance requirements for assays to detect specific microbial agents as Public Health Actionable Assays. The program was established in 2006 to support the goals of a national biomonitoring architecture called for in an interagency memorandum of understanding on monitoring for threat agents that had been signed in 2005. The original PHAA program subsequently split, with one piece continuing as PHAA and another as a Public Safety Actionable Assays (PSAA) program.

Under PHAA, DHS and the Centers for Disease Control and Prevention (CDC) consider such “public health actionable assays” to support high-confidence agent detection and characterization performed in a laboratory setting, and to support public health actions such as deployment of antibiotics. The PHAA program was built on and expanded the strategies used for testing assays in the Laboratory Response Network (LRN) (Morse 2014). The PHAA process involved participation of subject-matter experts from federal agencies and the academic community to develop recommended inclusivity and exclusivity panels and other aspects of the assay evaluation approach. In contrast to SPADA, PHAA was developed to be a government-unique standard with a focus on government-operated programs; details of the discussions and performance requirements are not disseminated in the open literature (DHS 2014; Pillai 2014).

The PHAA documents describe reference materials and testing approaches to be used for both clinical samples and environmental samples. Assays to detect environmental samples under PHAA, the subset that would be relevant to BioWatch, require validation testing with a different subset of exclusivity strain panels than clinical samples (e.g., they do not include the PHAA panel representing clinical background microorganisms). The CDC signed on to the PHAA standard in 2011 for use with its agent detection assays through the LRN (Merlin 2014).

Stakeholder Panel on Agent Detection Assays

SPADA operates under the auspices of AOAC International, an association of analytical communities, as a voluntary consensus standards body and is supported by funding from DHS. It engages the participation of over 100 representatives from federal, state, and local governments, the first-response and public health communities, academia, and industry. SPADA has established standards for real-time PCR methodologies to detect the pathogens Francisella tularensis, Yersinia pestis, Bacillus anthracis, Burkholderia pseudomallei, and Burkholderia mallei from aerosol (AOAC 2011a,b,c; AOAC in preparation-a,-b) and is currently working to develop standards for Variola virus (AOAC 2014). The standards are based on panels of recommended inclusivity and exclusivity strains, environmental substances, and validation test protocols. In addition to standards for PCR-based assays to detect aerosolized agents, SPADA has established standards for immuno-based handheld assays to screen suspicious powders for use by first responders (AOAC 2011d,e). The SPADA process for establishing these performance standards relies on 10- to 20-person working groups of subject-matter experts, who develop recommendations for discussion by the full membership of SPADA and voting by approximately 20-25 SPADA voting members. Once a standard has been

approved, it is publicly published in the Journal of AOAC International (Coates et al. 2011; Davenport 2014).

The performance requirements established by SPADA are considered by DHS and CDC to be Public Safety Actionable Assays, which are suitable for first responders in the field to use to support incident management, including building evacuation. Samples testing positive with a PSAA standard would subsequently be transported to the CDC LRN for further analysis.

Department of Defense Assay Transition Packages

The DOD also makes use of nucleic acid–based detection technology to support biosurveillance on military facilities, as well as other types of detection methods (e.g., protein based) and sensors. The approach taken by DOD is to standardize essential information that is captured and presented during assay development to sufficiently document performance. This process is defined in a Defense Technology Objective (DTO) CB.56 molecular assay transition package, a format that was developed by multiple DOD stakeholders and began to be used across the agency in 2005 (Goodwin 2014).⁴ According to DOD, “[a]n enabling technology DTO focuses on the development of knowledge to address a specific issue, and is a necessary intermediate step to achieve an operational capability” (DOD 2005). Thus, documentation such as the molecular assay transition packages would be part of the required process for transition into advanced development. The concept of the DTO CB.56 data transition package and the categories of information it includes have subsequently been recognized by the FDA as the basis for initial submission for pre–emergency use authorization (EUA) and as the framework for assay development and performance information by international DOD partners in Canada, the United Kingdom, and Australia (Emanuel 2014).⁵

For real-time PCR assays, the information contained in the DTO molecular assay transition packages is similar to guidance that has been described by the Minimum Information for Publication of Quantitative Real-Time PCR Experiments (Bustin et al. 2009), which were developed to harmonize descriptions of PCR assays reported in scientific journals. The DOD package generally includes such information as

___________________

⁴ Stakeholders involved in the discussion included the U.S. Army Medical Research Institute of Infectious Diseases, U.S. Army Dugway Proving Ground, U.S. Army Edgewood Chemical Biological Center, U.S. Navy, Joint Biological Agent Identification and Diagnostic System, regulators, and developers (Goodwin 2014).

⁵ For example, an EUA was recently issued by the FDA for in vitro diagnostic use of a real-time PCR Ebola virus detection assay developed within DOD and documented by a DTO CB.56 information package (FDA 2014b).

• Description of target with database accession numbers, gene location, and sequence;
• Database searches of target gene and sequences producing significant alignment (with accession numbers, names, and percent similarity);
• Software used to design primers and probe, primer and probe sequences and locations, characteristics such as length, melting temperature, and percent GC content, and information on database sequences producing significant alignment with primers and probe sequences;
• Assay reaction conditions including information on the specific reagents and instruments used and the thermal cycling parameters;
• Information on optimizing assay conditions (such as testing different primer or probe concentrations and the resulting threshold cycle [C_t] values and end point fluorescence);
• Calculations of assay linearity (and associated data such as equation, slope, and R² value);
• Determination of assay limit of detection; and
• Information on assay inclusivity and exclusivity revealed by testing against various strains.

The committee’s understanding is that over time, DOD assay developers have created a pool of potential pathogen detection assays, reagents, and the accompanying packages that characterize their performance, so that missions may select suitable assays as the need dictates. When an assay is selected and tested for a particular mission need, the committee assumes that the performance of the assay and system are validated in the context of the operational setting. The committee did not receive details on how DOD conducts performance and validation testing beyond dimensions described in the DTO CB.56 package when the assay is developed.

Within DOD, the Critical Reagents Program (CRP) serves as a consolidation point for quality control programs, standard operating procedures, conformance test plans, and assay reference materials. The CRP credits sustained funding through DOD 5-year budget cycles for successful long-term operation and maintenance of this resource (Goodwin 2014).

Other Federal Programs

The BDS is an autonomous air sampling and analysis system designed to detect the presence of Bacillus anthracis in postal facilities through PCR-based detection. If a positive sample is recorded, material is sent to the CDC’s LRN for confirmatory testing (O’Neill 2014). The EPA also pro-

vides guidance on validation for microbiological methods (EPA 2009). The guidance provided by the EPA is similar in nature to the performance guidance provided by SPADA, PHAA, DOD, and FDA. The EPA requires a written description of the method by the developing laboratory, summarizing the method; describing the equipment, reagents, and standards used; providing information on quality control measures, data, and statistical analysis; giving method performance results; and indicating other details. Within-laboratory performance parameters such as assay sensitivity, specificity, and precision (also referred to by EPA guidance as “primary validation”) are reported. The EPA takes a tiered approach to the nature and extent of multilaboratory validation required, based on the use of the assay (with a focus on conducting more extensive validation for methods that will, for example, be used nationally). The EPA guidance notes that different detection chemistries will influence PCR performance and that assays may perform differently on different types of instruments. As a result, validation is necessary in the context of the instrument and chemistry to be used, with appropriate documentation and additional validation likely to be necessary if these aspects change.

PRIOR FEDERAL EFFORTS ON PERFORMANCE STANDARDS AND ASSAY EQUIVALENCY

Given the fundamental role of performance standards and validation, the widespread use of PCR-based assays, and the increasing emphasis on biosurveillance since 2001, it is not surprising that federal agencies have a history of discussing these topics. Table 2-1 highlights several of the relevant events that have occurred.

One set of efforts has been directed at the issue of establishing federal assay performance equivalency. Multiple agencies and programs were using similar types of assays, and there was interest in having a common basis for comparing performance. In 2005, a Memorandum of Understanding (MOU) on Coordinated Monitoring of Biological Threat Agents was signed by DHS, the Department of Health and Human Services (HHS), DOD, the U.S. Postal Service (USPS), and the Department of Justice (DOJ) (Appendix A in DHS 2011). The agreement discussed development of a basis to demonstrate performance equivalency among the PCR assays used by federally owned biomonitoring activities to confirm the presence of biothreat agents. The text of the MOU, which expired on January 1, 2010, reads in part,

By August 2005, the Participants will use biological threat agent detection assays, protocols, and response algorithms for confirmation of a biological threat agent (presence of biothreat agent nucleic acid) that are

TABLE 2-1 Time Line of Selected Programs and Milestones

mutually acceptable to the Participants. . . . Using predetermined performance criteria, the Participants will evaluate rapid biological threat detection assays (e.g., PCR or other technologies), protocols, and response algorithms on select agents, similar substances, and other interferents to support a determination of equivalency. (Appendix A, lines 154-166, in DHS 2011)

The committee notes that the MOU refers to method equivalency for confirmation (emphasis added) of agent detection.

A “BioNet” study was undertaken between PCR assays deployed by DOD and those used by the CDC LRN. The multiyear, multi-milliondollar effort involved developing an agreed-upon test plan and comparing the performance of real-time PCR assays then in use by the CDC LRN, DOD Joint Program Executive Office for Chemical and Biological Defense, and DOD Naval Medical Research Center. Strain DNA was provided to Lawrence Livermore National Laboratory, which prepared 96 well plates of blinded samples that were sent to Battelle Memorial Institute for independent testing. Assays developed by CDC and DOD to detect three different pathogens of potential relevance to BioWatch were evaluated. Each assay’s limit of detection was characterized using a reference strain and assay specificity was characterized using panels of inclusivity, exclusivity, and background organisms, as well as background matrix provided by BioWatch and DOD Guardian air filters (Joan Gebhardt, Naval Medical Research Center [NMRC], personal communication, January 6, 2015).⁶ The strain panels used for testing were based on the strains obtainable; more limited strain typing data were available at the time, and thus it is not clear how well the strains available covered the genetic diversity of the tested pathogens (Segaran Pillai, DHS, personal communication, January 21, 2015).

To the best of the committee’s understanding, the report resulting from BioNet indicated that the CDC and DOD assays that were tested performed equivalently based on the collected data (Joan Gebhardt, NMRC, personal communication). On the other hand, the BioNet project appears not to have succeeded in a long-term, agreed-upon basis for establishing assay equivalency (Emanuel 2014). An ongoing effort under the leadership of the DHS Science and Technology Directorate (S&T) is now focused on establishing Federal Standards for Evaluating Assay Performance and Equivalency (FSAPE). The objective of this effort is to “expedite consequence management efforts responding to intention-

___________________

⁶ For each of the three tested pathogens, panels of approximately 30 inclusivity strains were assembled (tested at 10 pg of DNA) along with panels of varying numbers of near-neighbor exclusivity strains and a panel of background organisms (tested at 100 pg of DNA); panel composition was based on the strains that the effort was able to obtain.

ally released pathogens that have been detected by federally owned and operated biomonitoring programs. . . . This improved understanding will allow for interoperability, which is essential for coordinated consequence management efforts across agencies and to support public health actions and decisions (DHS 2011, p. 9). According to Segaran Pillai, Chief Medical and Science Advisor in DHS S&T, the development and implementation of FSAPE is being undertaken in collaboration with federal agencies such as DHS, DOD (Office of the Secretary of Defense), HHS (Office of the Assistant Secretary of Preparedness and Response, CDC, FDA), DOJ (Federal Bureau of Investigation), USPS, U.S. Secret Service, and state and local public health leaders and associations such as the Association for State and Territorial Health Officials, Association of Public Health Laboratories, and the National Association of County and City Health Officials (Pillai 2014).

The FSAPE document describes performance criteria and validation protocols that are to be used to document assays. These criteria and protocols are essentially identical in approach to the PHAA standard for validating assays from environmental (nonclinical) samples and so PHAA and FSAPE are discussed together where relevant in the remainder of the report. FSAPE appears to be an ongoing effort, and it is not clear to the committee whether all of the relevant federal agencies have formally signed on to this approach.

ASSAY PERFORMANCE CHARACTERIZATION CONDUCTED BY LOS ALAMOS NATIONAL LABORATORY

On behalf of the BioWatch program, Los Alamos National Laboratory (LANL) assembled and quality-controlled strain panel materials, developed a test plan to characterize the analytical performance of several of the assays being used by the program, and compared assay performance on vendor and reference systems to inform the BioWatch Gen-3 acquisition program on autonomous detection capabilities (LANL 2010, 2011a,b, 2012a,b). The LANL work partially followed SPADA guidelines, with some alterations. LANL used purified DNA for its analytical performance testing. Subsequent BioWatch testing that included extraction from a filter matrix was conducted at Dugway Proving Ground as part of full system testing. The discussion in the report is informed by approaches such as SPADA and PHAA/FSAPE, as well as by the procedures that LANL employed. DHS also has undertaken system testing beyond assay analytical performance. Table 2-2 summarizes recent activities.

Prior federal and stakeholder efforts on PCR assay performance standards provide a basis for discussing the tradeoffs of different approaches that may be applicable to BioWatch. A discussion of critical components of

TABLE 2-2 Recent BioWatch Performance Testing and Validation Activities ^a

^a A restricted version of this table may be found in Appendix A.

performance standards is presented in Chapter 3. The committee returns in Chapter 4 to the broader validation framework, the important role of monitoring and feedback, and the operational context of assay use. Although BioWatch is owned by a federal agency, it is managed and operated locally; it is the experience and confidence in results of the users in the public health laboratories and the jurisdictional decision makers that are critical to the success of BioWatch.

This page intentionally left blank.