BioWatch PCR Assays Building Confidence, Ensuring Reliability: Abbreviated Version (2015) / Chapter Skim
Currently Skimming:
3 Key Components of a Performance Standard for BioWatch
3 Key Components of a Performance Standard for BioWatch
Pages 53-100
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 53... ...
The committee determined that a PCR standard to characterize assay performance and conduct validation, applicable to BioWatch, would need to meet the following basic goals: • Establish that the assay has a sufficient limit of detection and sensitivity to detect the release of a tested biothreat agent at a pro gram-relevant concentration above the baseline of environmental background. The BioWatch system was not deployed with the goal of detecting a single airborne particle, but rather to monitor for a substantial release event.
|
From page 54... ...
If a nucleic acid from a tested pathogen is present then the assay should detect it and minimize the rate of obtaining a false negative result. A false positive result is less detrimental at this stage because all positive screening results will proceed to a subsequent secondary assay.
|
From page 55... ...
, Dynamic Range and Probability of Detection at Relevant Concentrations Test Ability to Detect Inclusivity Strains Test Ability to Not Detect Exclusivity Strains Test Against Environmental Strains and Substances Test Assay Performance and Robustness Under Operational Use Routinely Reevaluate and Make Adjustments as Needed FIGURE 3-1 Flowchart representing steps in assay analytical performance characterization. two-step assay process has improved their experiences with regard to preliminary positive results and downstream decision making.
|
From page 56... ...
Because of the importance of specificity to the performance of a secondary assay, the program might conclude that it is useful, for example, to conduct laboratory validation using additional inclusivity and exclusivity strains than were used for an initial screening assay. An assay meant to be used for a secondary level of verification or confirmation could thus be validated in the laboratory against a second tier containing additional inclusivity, exclusivity, and environmental strains in order to demonstrate the requirement that this type of assay have high specificity.
|
From page 57... ...
This model may be more challenging to implement in practice -- it is difficult to draw definitive links between the human infectious dose of a pathogen, the pathogen concentration in an air plume that would be required to cause illness, the factors that influence pathogen deposition onto an aerosol point collector like the BioWatch devices, and the effects of collection medium and sample processing prior to entry into the PCR reaction in order to set the appropriate AMDL. Other assay performance approaches essentially have taken an alternate approach at the assay evaluation stage by assuming that an assay will be designed to be as sensitive as possible and determining the achievable lower analytical detection limit.
|
From page 58... ...
58 BIOWATCH PCR ASSAYS Figure 2. Comparison of the POD model terminology to other obsolete terms.
|
From page 59... ...
As can be seen in Table 3-1, depending on the interval calculation method employed, testing 47-56 samples with zero failures, 79-85 samples with one failure, and 107-112 samples with two failures will achieve probabilities of detection of approximately 95 percent with 95 percent confidence. 2 For inclusivity strain testing, testing was conducted not at the concentration for which the assay yielded 95 percent probability of detection of the agent reference strain, but at the higher agent concentration for which the assay yielded a 99.5 percent probability of detection of the reference strain.
|
From page 60... ...
60 BIOWATCH PCR ASSAYS TABLE 3-1 Comparison of Binomial Confidence Intervals for Varying Numbers of Tests, Calculated Using Several Confidence Interval Methods x n Agresti-Coull Clopper-Pearson Wilson "successes" trials lower upper lower upper lower upper 3 3 0.470 ≤p≤ 1.000 0.368 ≤p≤ 1.000 0.526 ≤p≤ 1.000 3 5 0.271 ≤p≤ 0.859 0.189 ≤p≤ 0.924 0.272 ≤p≤ 0.857 4 5 0.422 ≤p≤ 0.968 0.343 ≤p≤ 0.990 0.435 ≤p≤ 0.954 5 5 0.599 ≤p≤ 1.000 0.549 ≤p≤ 1.000 0.649 ≤p≤ 1.000 9 9 0.731 ≤p≤ 1.000 0.717 ≤p≤ 1.000 0.769 ≤p≤ 1.000 8 10 0.533 ≤p≤ 0.939 0.493 ≤p≤ 0.963 0.541 ≤p≤ 0.931 9 10 0.636 ≤p≤ 0.994 0.606 ≤p≤ 0.995 0.652 ≤p≤ 0.977 10 10 0.751 ≤p≤ 1.000 0.741 ≤p≤ 1.000 0.787 ≤p≤ 1.000 18 20 0.730 ≤p≤ 0.975 0.717 ≤p≤ 0.982 0.738 ≤p≤ 0.966 19 20 0.791 ≤p≤ 1.000 0.784 ≤p≤ 0.997 0.804 ≤p≤ 0.989 20 20 0.859 ≤p≤ 1.000 0.861 ≤p≤ 1.000 0.881 ≤p≤ 1.000 38 40 0.854 ≤p≤ 0.989 0.851 ≤p≤ 0.991 0.860 ≤p≤ 0.983 39 40 0.887 ≤p≤ 1.000 0.887 ≤p≤ 0.999 0.895 ≤p≤ 0.994 40 40 0.924 ≤p≤ 1.000 0.928 ≤p≤ 1.000 0.937 ≤p≤ 1.000 58 60 0.900 ≤p≤ 0.993 0.899 ≤p≤ 0.994 0.904 ≤p≤ 0.989 59 60 0.923 ≤p≤ 1.000 0.923 ≤p≤ 0.999 0.929 ≤p≤ 0.996 60 60 0.948 ≤p≤ 1.000 0.951 ≤p≤ 1.000 0.957 ≤p≤ 1.000 94 96 0.936 ≤p≤ 0.996 0.936 ≤p≤ 0.996 0.939 ≤p≤ 0.993 95 96 0.951 ≤p≤ 1.000 0.952 ≤p≤ 0.999 0.955 ≤p≤ 0.998 96 96 0.967 ≤p≤ 1.000 0.969 ≤p≤ 1.000 0.973 ≤p≤ 1.000 118 120 0.948 ≤p≤ 0.997 0.948 ≤p≤ 0.997 0.951 ≤p≤ 0.994 119 120 0.960 ≤p≤ 1.000 0.961 ≤p≤ 1.000 0.964 ≤p≤ 0.998 120 120 0.973 ≤p≤ 1.000 0.975 ≤p≤ 1.000 0.978 ≤p≤ 1.000 47 47 0.935 ≤p≤ 1.000 0.938 ≤p≤ 1.000 0.946 ≤p≤ 1.000 53 53 0.942 ≤p≤ 1.000 0.945 ≤p≤ 1.000 0.951 ≤p≤ 1.000 56 56 0.945 ≤p≤ 1.000 0.948 ≤p≤ 1.000 0.954 ≤p≤ 1.000 57 57 0.946 ≤p≤ 1.000 0.949 ≤p≤ 1.000 0.955 ≤p≤ 1.000 78 79 0.941 ≤p≤ 1.000 0.941 ≤p≤ 0.999 0.945 ≤p≤ 0.997 84 85 0.945 ≤p≤ 1.000 0.945 ≤p≤ 0.999 0.949 ≤p≤ 0.997 85 86 0.945 ≤p≤ 1.000 0.946 ≤p≤ 0.999 0.950 ≤p≤ 0.997 105 107 0.942 ≤p≤ 0.996 0.942 ≤p≤ 0.997 0.945 ≤p≤ 0.994 110 112 0.945 ≤p≤ 0.997 0.945 ≤p≤ 0.997 0.947 ≤p≤ 0.994 Notes: Confidence intervals calculated using the package ‘bionom' version 1.1-1 (Dorai-Raj 2014) in the statistical computing software R, version 3.1.2 (R Project for Statistical Computing, available at http://www.r-project.org/)
|
From page 61... ...
Because of this, the committee notes that the two-step strategy employed by PHAA/FSAPE, which entails testing a limited number of replicates to initially estimate an approximate LOD, could be helpful in setting the appropriate concentration boundaries for the actual LOD calculation. It can also be noted that data obtained from lower concentrations of a serial dilution series help support confidence in results contained at higher concentrations (e.g., correctly detecting DNA in all tested samples at a lower concentration helps bolster confidence in the results from correctly detecting DNA in tested samples at a higher concentration)
|
From page 62... ...
when >1 nondetect = fail reference strain is present at AMDL -- goal is 0.95 If 95/96 samples, the 95% POD at AMDL with 95% lower confidence limit on confidence the POD is approx.
|
From page 63... ...
copy numbers/ >1 nondetect = fail If 9/9 samples rxn) In the absence are able to detect If 20/20 samples of nontarget the concentration, are able to detect nucleic acid or the 95% lower the concentration, environmental confidence limit on the 95% lower background the ALOD is 66% confidence limit is 86%; for 39/40 samples the 95% lower confidence limit is 89% continued
|
From page 64... ...
reference strain; study must produce 10 valid POD at least 0.95 with 95% data sets and report confidence limit reproducibility and standard deviation (total = at least 120) See below for testing in absence of agent
|
From page 65... ...
Approach Testing Details Approach Testing Details Phase II: n = 20 replicates by "Threshold 2012-2014: System Intermediate 3 operators (total of Probable characterization and precision/ = 60) Detection" field tests conducted repeatability measured at the Criteria: BioWatch system Test reference If 0 negative results level strain spiked = pass at the limit of detection For 60/60 samples, calculated in the 95% lower Phase I confidence limit of detection at the Tested in absence ALOD is 95% of nontarget/ background If ≥1 unexpected result, evaluate assay at next concentration above ALOD and/ or may modify the signatures or test (if so, assay must repeat process)
|
From page 66... ...
66 BIOWATCH PCR ASSAYS TABLE 3-2 Continued SPADA Characteristic Approach Testing Details Detection Limit, continued Not applicable Not applicable Not applicable Not applicable
|
From page 67... ...
confidence limit on the measurement is Under FSAPE: 66% Background of 20 "pooled processed BioWatch PCR-negative samples" Phase IV: n = 20 replicates by Not applicable Not applicable Operational 3 operators (total Intermediate = 60) precision/ repeatability Criteria: 0 false negatives expected Test reference strain spiked at If unexpected the operational results, evaluate at ALOD calculated next concentration in Phase III above the ALOD in presence of (see description for background Phase II)
|
From page 68... ...
95% Exclusivity Strain Panels Test exclusivity n=1 panel organisms at concentration of 10× 0 unexpected results = pass; AMDL If an unexpected result, re test with n = 96, 0 failures allowed For 95/96 samples, the 95% lower confidence limit of detection approx.
|
From page 69... ...
result, "assay does concentration of not definitively Step 3: Test reference strain Inclusivity panel meet performance using additional expected to give of strains are standard" and will SPADA strains 0.995 POD based each tested at be investigated obtained by on the target strain 1pg/rxn DNA or LANL probability of RNA (bacteria) For 3/3 samples, detection (Step 1)
|
From page 70... ...
in the absence of the lower confidence limit is target agent at AMDL 5%; for 95/96 samples, the 95% lower confidence limit is approx.
|
From page 71... ...
Approach Testing Details Approach Testing Details Not applicable Not applicable Not applicable Not applicable Phase V: n=3 Step 4: Test n = 20 replicates Evaluation of using SPADA Assay Exclusivity 0 unexpected results environmental If an unexpected = pass; panel organisms result, each strain Tested in the If 1-2 unexpected in pools from pool tested absence of target results, analyze individually 20 each component of Tested as times Up to 10 the pool (number individual organisms per of replicates organisms T1-11 Tested at 10× the pool at 100 pg/ unspecified) and 8 pools concentration of organism (total of reference strain 1000 pg/rxn)
|
From page 72... ...
To meet the SPADA requirement of 95/96 or 96/96 detections with greater than 43 percent probability, the true detection rate must be over 98 percent; if the true detection rate is 95 percent the test
|
From page 73... ...
. PCR assays are highly sensitive, and validating the minimum LOD of the assay could be useful information because it helps set the maximum achievable performance.
|
From page 74... ...
also should be included on defined environmental exclusivity panels. The composition of the panels defined in assay standards are established through consultation with subject-matter experts knowledgeable about the genetic diversity of the pathogen and its near neighbors.
|
From page 75... ...
The committee suggests that inclusivity panel strains for laboratory characterization and validation of assay performance should sample the genetic diversity represented by available strain collections with the emphasis on prioritizing those strains that cause significant morbidity or mortality, have high transmissibility, or have a wide host range. The strain panel contents are likely to evolve over time as novel diversity is encountered, and it is likely that each agent of interest to BioWatch will have its own unique rate of such evolution.
|
From page 76... ...
bThese strain panels continue to evolve as new strains have been identified. Numbers listed reflect strains in DHS (2014)
|
From page 77... ...
KEY COMPONENTS OF A PERFORMANCE STANDARD FOR BIOWATCH 77 Jackson/ FSAPE / PHAAb Morse Jackson/Morse Suggestionsc Genetically monomorphic; 48 15 SPADA panels sufficient for 61 (+ 7 on environmental panel) 22 inclusivity and exclusivity Genetically diverse; 40 35 for inclusivity and exclusivity, 42 19 supplement SPADA panels; provided suggestions for additional strains Genetically diverse; 46 40 for inclusivity, supplement SPADA 37 32 panel; provided suggestions for additional strains; for exclusivity, use FSAPE panel 31 31 Genetically diverse; 73 82 for inclusivity and exclusivity of D 91 98 and E, use FSAPE panels 45 Did not review this agent 26 Arthropod: 20 N/A Did not comment on these panels Bacterial: 112 Botanical: 10 Fungal: 14 Protists: 10 Vertebrates: 20 Viral: 9 Total = 195 N/A N/A Did not comment on these panels 664
|
From page 78... ...
. As noted above, testing assay performance against inclusivity and exclusivity strains should be conducted through a combination of in silico prediction and laboratory analysis.
|
From page 79... ...
. For n = 1/1, there is 95 percent confidence that the lower confidence limit on the measurement is only 5 percent (e.g., the long-term detection rate is at least 5 percent)
|
From page 80... ...
This yields a statistical lower confidence limit of 86 percent for 20/20 expected results and 89 percent for 39/40 samples. A potential false positive or false negative performance rate on the assay of 1 – 0.86 = 0.14 (14 percent)
|
From page 81... ...
This ongoing data analysis will build confidence in the performance of the assay or reveal issues that need to be addressed. As a result, the committee concluded that the approach and number of replicates used for inclusivity and exclusivity strain testing by LANL provides a reasonable level of confidence in assay performance for the laboratory validation stage.
|
From page 82... ...
Judicious sampling of microorganisms present in soil and water samples would also be informative. The significance of environmental background to confidence in assay results echoes a recommendation made in a previous NRC report (IOM and NRC 2011, pp.
|
From page 83... ...
As with selection of the inclusivity strains that should form part of the standard panels, employing a strategy to prioritize the most appropriate subset of environmental strains will be helpful in balancing the need to cover genetic diversity while reducing the size of strain panels and number of required tests. Using Environmental Background to Understand Predictive Value of the Assay Better understanding of environmental background would provide information on the expected probability of the tested organism or near neighbors in the specific jurisdictional environment.
|
From page 84... ...
The determination of PPV is affected by prevalence. For BioWatch, the true prevalence of tested pathogens in local environments is difficult to determine, because little is known about the actual environmental background of the agents and how they vary geographically and temporally, and because the prevalence of the tested microorganisms is likely to be generally very low.
|
From page 85... ...
It is thus critical that PCR assays targeting these viruses use primers derived from highly conserved regions of the genome to be able to capture the diversity of viruses within individuals as well as populations. In many cases, a single viral reference sequence may not be sufficient to detect all viral genomes within a population (Domingo et al.
|
From page 86... ...
B Effect this has on the selection of DNA signatures that can be used in designing PCR assays.
|
From page 87... ...
This option might be considered for use with certain pathogens in the context of certain collector sites or certain jurisdictions where it has been challenging to obtain PCR assays with sufficient specificity. This situation also suggests that a single set of initial and secondary assays sufficient for all BioWatch jurisdictions and all collection sites may not be feasible.
|
From page 88... ...
For analytical testing of a real-time PCR assay starting from known quantities of a purified nucleic acid, the accuracy of the assay result can at least be compared with the starting reference value.6 As the BioWatch program validates new assays or new detection technologies, it also can compare the performance of the new candidate method to the existing assay system. POSITIVE AND NEGATIVE PCR ASSAY CONTROLS Appropriate positive and negative controls are required when conducting all real-time PCR assays to ensure that the reagents and instruments are operating correctly, that there is no detectable contamination, and that PCR inhibitors are not present.
|
From page 89... ...
Ongoing monitoring and evaluation of assay and operational performance through a QA program is discussed further in Chapter 4. TESTING LABORATORY SETUP AND MANAGEMENT Assay performance and validation testing should be conducted using a laboratory setup designed to minimize the potential for contamination and with systems to record and manage the data obtained.
|
From page 90... ...
Even with select agent registration it can reportedly be difficult to access select agent materials and, in particular, federal agencies may not be willing to distribute select agents to nonfederal entities. Because of specific restrictions on the possession of, for example, Variola virus cell culture stocks and DNA, special considerations arise when undertaking assay performance testing and validation for orthopox detection assays.
|
From page 91... ...
The committee supports efforts to address the issue of which types of strain reference materials are necessary for biodetection assay testing and how they can be provided, and notes, as well, that the need for access to relevant reference materials does not imply that all such materials should be held in a single repository. Awareness of who holds nucleic acids from which microorganisms and how to obtain these materials is an important issue for the biodetection community and has the potential to limit assay development and validation efforts.
|
From page 92... ...
• Other Agency Collections: DHS and CDC have identified strains listed on the PHAA panels for agent detection assays as part of CDC LRN assay validation testing and documentation. • Other Resources: -- ndividual researchers may be sources for particular strains.
|
From page 93... ...
In parallel, the CRP undertook an effort to obtain stocks for additional microorganisms that it did not previously hold. As a result, the CRP now strives to hold as reference materials the strains listed on SPADA inclusivity and exclusivity panels that are biological weapons agents, and the strains held by CRP are in the process of being sequenced (Bruce Goodwin, CRP, personal communication, January 12, 2015)
|
From page 94... ...
. It might be feasible for panel materials eventually to be transferred to another entity to serve as the resource to provide them to DHS or others, as necessary for future assay performance and validation testing, but this option would need to be explored.
|
From page 95... ...
Similarly, MRI Global assembled SPADA strain panels (inclusivity targets, near-neighbor and background exclusivity panels, and potentially interfering environmental substances) for at least one biothreat agent of relevance to BioWatch.
|
From page 96... ...
• The greater the number of inclusivity, exclusivity, and environ mental materials against which assays should be tested, the more precisely assay specificity will be known but the higher the time and cost to conduct the assay testing. Similarly, requiring testing with a greater number of replicates will increase statistical confi dence in the assay performance results but increase time and cost.
|
From page 97... ...
Testing laboratory LANL for BioWatch $2.1M setup costs assay testing Assay performance BioWatch process 7 months $0.6M testing SPADA/PSAA 6 months $0.4M PHAA 2 or more years $1-2Mb Platform LANL for BioWatch 9 months, $4.4M performance vendor system setup to testing testing completion (comparison of two Gen-3 vendor systems with reference system) Ongoing assay and PCR assays and Available $909 per strain vial reference material reagents; most through CRP (~200 ng DNA/ provision strains on SPADA catalogue vial)
|
From page 98... ...
10 The RAZOR EX device was developed as a portable field detection system using pouches of reagents; it is not the same as the system of collectors and laboratory-based PCR assays used in the BioWatch program. However, it provides the only data point available to the committee on the time and cost associated with validating a biothreat detection device to the full AOAC/SPADA standard.
|
From page 99... ...
PHAA/FSAPE also uses a fairly similar overall testing approach to that suggested by the committee, although the committee has suggested conducting a larger number of exclusivity and environmental panel replicates than the n = 3 specified under PHAA/FSAPE. If CDC is willing to share its performance testing data with the BioWatch program, the committee does not mean to imply that the BioWatch program should duplicate all of the time, effort, and cost that will have gone into CDC's work on these assays.
|
From page 100... ...
The committee aims to recognize the valuable efforts that have been made to obtain assay performance data but does not wish the message to be that DHS, CDC, and others have con ducted all of the necessary testing and that jurisdictions should trust that fact without enabling data access and discussions. In this chapter, the committee explored key factors that will affect whether a PCR assay standard is suitable for performance testing and validation as part of the BioWatch program.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.