Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
APPENDIXES 69
A Evaluating Detector Signals The tester has access to challenge material, limited information about the test instrument such as inlet flow rate, power dissipation, etc. (in order to not restrict innovation and progress, the test instrument is treated as a âblack boxâ as much as possible), and the response of the test instrument to the challenge. The test protocols allow for the detector to produce only a binary response. If agent is detected, a positive response is recorded; if no agent is detected, a negative response is recorded. For instance, in a series of N aerosol challenge experiments, the instrument response R is recorded as shown in Table A.1. TABLE A.1 Experimental Protocol for Determining a Detectorâs Sensitivity Threshold Response Experiment R Comments 1 1 Positive â agent detected by the instrument 2 1 Positive ⦠i 1 Positive i+1 0 Negative â agent not detected by the instrument ⦠N 0 Negative Historically a the progression of test experiments began with a concentration, C, of target material that was then diluted until the instrument no longer detected the materialâusually done by comparison of a measurement to a threshold. In our example, this occurs at experiment i, and we designate Cthreshold = Ci the lowest target concentration to produce a positive detection. If the only material present in the challenge were viable agent target, a measurement in CFU, PFU, and animal toxicity testing (for bacteria, viruses, and toxins, respectively), for Cthreshold could be converted into BAULA units, and this would be the minimum detectable concentration of the instrument in BAULA. In the section, this simplified illustration of instrument response as a function of target concentration will be exploited to discuss aspects of performance testing for determining minimum detectable concentration in terms of BAULA units. We acknowledge here that in practice instrument response, R(c), will not have a sharp step-function in target concentration in which the detected quantity generates a robust positive signal down to some threshold value and then indicates a null detection below that value. More typically, R(c) will drop off over some extended range of concentration values with more of an âSâ shaped response function. For instruments that have a binary âtarget present/target absentâ response this means that the point of transition will fluctuate in a statistical sense over some range of target concentration. For example, if the trial illustrated in Table A.1. were repeated a number of times the transition point 71
72 represented by the ith+1 concentration would have some variance, meaning that on some tests the transition may occur at the ith or ith+2 absolute target concentrations. More importantly, for any given test, the transition may not be monotonic but could indicate a negative response on the ith concentration followed by a positive response on the ith+1 or lower concentration. The essence of this behavior is the concept of probability of detection. In general, for any fixed absolute target concentration, a large number of tests would yield a certain percentage of positives and a complementary number of negatives. In the limit of the number of tests becoming very large, we assign the percentage of positive responses for actual target concentrations as the probability of detection (PD). Therefore threshold concentration which is designated as the minimum detectable concentration must be selected on the basis of what PD is desired as an operational point. The two remaining fundamental metrics of a sensor system: (a) response time (integration time) and (b) probability of false positive are also interconnected to the determination of a minimum detectable threshold. In general, these four quantities are functionally related and it is not possible to specify one without also specifying the others. A complete outline of a test and evaluation procedure for determining the functional relationships of these four metrics is beyond the scope of this report. We continue this section under the assumptions that the aerosol challenge will be presented to the sensor being evaluated for a period of time consistent with the required response time, that the determination of the minimum concentration threshold is made in the absence of deliberate âconfounderâ or âinterferentâ materials, and that the detector response as a function of target concentration is sufficiently sharp to designate a reasonably repeatable and meaningful threshold concentration without having to define a precise PD. As was mentioned earlier, for instruments that are responsive to both viable and nonviable target agent (e.g., PCR, antibody capture assays, etc.), the estimate of test instrument detection threshold based on CFU, PFU, and animal toxicity testing in the referee system might be overly optimistic. To make this point more concretely, we provide an example where the referee system estimates both viable agent and previously viable agent in the challenge material. We then generalize to any number of parameters in the challenge that are under the control of the test. In both cases, we include the effect due to a background noise concentration and construct a mathematical approach to estimate the test instrumentâs minimum detectable concentration. Consider a procedure for determining a test instrumentâs detection threshold in BAULA consisting of: ⢠an idealized referee system that accurately measures active agent in a sample; ⢠a test instrument that uses a nucleic acid detection scheme that accurately senses the number of genome equivalent (GE) copies of the target agent to determine agent concentration with an inherent detection limit of 20 GE; and ⢠a target agent pathogenic enough that 1 active organism or molecule exposure or dosage is the LD50. As the goal of the test is to determine the test detectorâs sensitivity in BAULA, all data received from both the referee and test systems are converted to BAULA. (1 Referee unit converting to 1 BAULA; 1 GE of the test detector converting to 1 BAULA) In this scenario, on reaching the referee and test instrument, the challenge material has a concentration A of viable target agent and I of target agent that has degraded to become inactive. A series of dilution experiments are performed to test the detection threshold of the instrument. Assume a situation (Case 1) where there is twice as much active agent as inactive agent as in Table A.2. If the test instrument were
73 only sensitive to active agent, the limit of detection would have been determined in experiment dilution number 2, with a BAULA of 32 instead of the reported 48. Because the referee sees only BAULA, we would have empirically estimated the limit of detection to be 16 at dilution 3. Repeating the experiments will not remove this inherent measurement bias. TABLE A.2 Effect of detection of inactive sample on BAULA determination: Case 1, 1:2 Ratio of Inactive to Active Agent Concentration in Concentration in True Concentration BAULA BAULA Determined (per liter of air) of Determined by by Test Instrument Experiment Response of Agent at Location of Referee Measurements Number Test Detector Referee and Measurements (detects both active Instrument (detects only and inactive agent active agent) in GE) Inactive Active agent agent (A) (I) 0 128 64 128 192 Positive 1 64 32 64 96 Positive 2 32 16 32 48 Positive 3 16 8 16 24 Positive Negative (beyond 4 8 4 8 12 instrument sensitivity of 20 GE) 5 4 2 4 6 Negative 6 2 1 2 3 Negative Table A.3 describes test Case 2 where twice as much inactive agent as active agent is presented to the referee and the test instrument. In this experiment, when the referee is measuring 32 BAULA, the instrument is actually exploiting 96 GE-based BAULA. Assuming that the intrinsic GE copy detection capability is 20, then the sensor limit of detection would be empirically estimated to be 8 BAULA. In summary, as we decrease the ratio of inactive target agent to active agent from 1:2 in Case 1 to 2:1 in Case 2, the instrumentâs apparent minimum detectable concentration in BAULA, as experimentally estimated by a perfect referee system at the location of the test instrument, goes from 16 to 8.
74 TABLE A.3 Effect of detection of inactive agent on BAULA calculation: Case 2, 2:1 Ratio of Inactive to Active Agent Concentration in Concentration in True Concentration BAULA BAULA Determined (per liter of air) of Determined by by Test Instrument Experiment Response of Test Agent at Location of Referee Measurements Number Detector Referee and Measurements (detects both active Instrument (detects only and inactive agent active agent) in GE) Active Inactive agent (A) agent (I) 0 128 256 128 384 Positive 1 64 128 64 192 Positive 2 32 64 32 96 Positive 3 16 32 16 48 Positive 4 8 16 8 24 Positive Negative (beyond instrument 5 4 8 4 12 sensitivity of 20 GE) 6 2 4 2 6 Negative It is apparent from examining the tables that an estimate of GE copies (and conversion to BAULA) by the referee can be used to correct the bias. We introduce a matrix construct to help generalize to the different instruments that DOD currently and will potentially test. For a specific target agent define three variables: ⢠A as the concentration of active agent at the referee (and test instrument), using BAULA or an appropriate surrogate (e.g., infectious unit or toxic unit); ⢠I as the concentration of previously active agent (i.e., inactive agent) that can be used to infer what BAULA likely existed before inactivation; and ⢠N as the effective concentration of material that is noise to the instrument. This term captures noise as well as signal contributions from any components of the measurement technology in the test instrument that are not described by the other terms of the referee system. It is not related to parameters considered relevant to sensitivity of the test instrument or relevant to defining the target agent. In practice A is estimated using BAULA or surrogates for BAULA (e.g., CFU), I is estimated using genome copy (e.g., to estimate the number of active and inactive bacteria or viruses and remove the contribution from active agent using the estimate for A). The random variable N is not estimated by the referee system as defined. The response of the test instrument to changes in A , I, and N may be complex. Since we are seeking to find a minimum detectable concentration threshold, we make a simplifying assumption that near the minimum concentration value the three component variables contribute with linear (but unknown) weights. For a specific experiment âkâ where the transition from positive to negative detection occurs, aAk + iIk + nNk = Î, where Î is the threshold of the test instrument in BAULA-equivalent units. Recall that Cthreshold (see Table A.1) was selected by diluting concentration until the test instrument no
75 longer triggered. A second series of experiments can be conducted where the starting material is first diluted to a different ratio of active to inactive agent. Care must be taken to ensure that the ratios of active to inactive agent are randomized. For instance, dilution of the kth experiment by inactivating an aliquot fraction p of Ak would unfortunately result in a series of experiments given by a(1-p)Ak + i(pAk + Ik) + nNk = Î that are insufficient to uniquely determine a, i, and n. We can use control of the source material to vary the Ak / Ik ratio and the dilution protocol described in Tables A.2 and A.3 above to determine the instrument sensitivity with the referee estimates for active agent, Ak, and inactivated agent, Ik. By definition the referee does not estimate n or Nk. The collection of experiments can be represented in matrix form as: â¡ ... ⤠⡠a ⤠⡠... ⤠â¢A Ik 1 ⥠⢠i ⥠= â¢Î k ⥠⢠k â¥â¢ ⥠⢠⥠⢠⣠...⥠â¢nN k ⥠⢠... ⥠â¦â£ ⦠⣠â¦. When the challenge material is composed of viable agent only, Ik=0. The referee system estimates <Ak> and, therefore, the matrix reduces to one experiment: aAk + nNk = Îk As expected, the minimum concentration threshold is biased by a noise term nNk, and scaled by a, which can be interpreted as an instrument efficiency parameter. Since the noise term cannot be removed without a second experiment, it is necessary to change the Ak / Ik ratio and repeat the series of dilutions for the new Ak concentration of target agent and with nonzero inactive component Ik. We will now need to estimate the parameter i, in order to accurately estimate the detection threshold. We experimentally estimate Ak, Ik, and are given Îk by the test instrument as the estimated BAULA limit of detection. To solve for a, i, and nNk, we need three experiments that result in linearly independent rows in the matrix equation. A series of three experiments beginning with a range of Ak and Ik is sufficient to solve for a, i, and nNk uniquely. Using this series for an idealized noise-free example with the GE detector, our equations become â¡16 16 1⤠⡠a ⤠â¡32 ⤠⢠8 16 1⥠⢠i ⥠= â¢24⥠⢠â¥â¢ ⥠⢠⥠⢠0 32 1⥠â¢nNk ⥠â¢32 ⥠⣠â¦â£ ⦠⣠â¦. Because this is an example, the matrix was selected to have an inverse.
76 â¡1 â1 ⤠â¢8 0⥠8 â¢1 â1 1 ⥠⢠⥠⢠16 8 16 ⥠â¢â 2 4 â 1⥠⢠⣠⥠â¦. Solving, we determine a=1, i=1, and nNk=0. For data with a nonzero bias term or noise (i.e., where the three equations do not intersect at a single point), additional experiments can be conducted and a least squares approach used to solve â¡ a ⤠⡠... ⤠⢠⥠( ⢠i ⥠= MTM )â1 M â¢Îk ⥠T ⢠⥠â¢nNk ⥠⣠⦠⢠... ⥠⣠â¦, â¡ ... ⤠where M = k â¢A I 1 ⥠and MT and (MTM)-1 are the matrix transpose of M and left inverse of ⢠k ⥠⢠⣠...⥠⦠T M M, respectively. If desired, additional columns can be added to M for each signature considered acceptable or unacceptable to contribute to estimation of sensitivity of the instrument. Just as we added previously active agent and compensated for it, we could add columns that represent optical cross-section for a specific wavelength of light, antigen binding, etc. Basically any parameter that the referee estimates can be introduced into the matrix to organize calculation of the limit of detection. With the parameters estimated the tester can decide whether the components due to non-BAULA contributions are appropriate test surrogates or are appropriate contributors to positive detection. It should be emphasized that the test instrument response R(A, I, N) is most likely a complex function of the parameters that we have been discussing. Our representation of a linear combination of the parameters can be considered a low order approximation to R(A, I, N) or as a linearized estimate of the optimization search near the limit of detection. In either case it is an approximation. The measurement that is closest to the BAULA-defined limit of detection is the one that is often the most difficult or impossibleâ100% biologically active agent in the challenge with no material that interferes with the test instrument or the referee. As the test arrangement deviates from this challenge, the matrix formulation becomes a method to track contributions from BAULA and non-BAULA terms as well as an overall estimate of the noise. Selecting the most significant non-BAULA contributions is important to an accurate estimate of sensitivity in BAULA units. Consider Table A.4 where the test instrument uses GE to estimate BAULA, with the additional complexity that the test instrument is not as specific as the referee and it, therefore, senses the equivalent of 25 BAULA from a near neighbor organism that contaminated the challenge and is no longer viable. Repeating the experiment from Table A.3:
77 TABLE A.4 Effect of detection of inactive agent on BAULA calculation: Case 3, Test Sample Contamination with Near Neighbor Organism Concentration in Concentration in BAULA Determined True Concentration BAULA by Test Instrument (per liter of air) of Determined by Experiment Measurements Response of Test Agent at Location of Referee Number (detects both active Detector Referee and Measurements and inactive agent Instrument (detects only and contaminant in active agent) GE) Active Inactive agent (A) agent (I) 0 128 256 128 394 Positive 1 64 128 64 202 Positive 2 32 64 32 106 Positive 3 16 32 16 58 Positive 4 8 16 8 34 Positive 5 4 8 4 22 Positive Negative (beyond instrument 6 2 4 2 16 sensitivity of 20 GE) As before, a series of three experiments is sufficient to solve for a, i, and nNk uniquely. Using this series for the GE test instrument that is biased by a near neighbor contaminant (Î is increased by 10 in each experiment), our equations become â¡16 16 1⤠⡠a ⤠â¡42⤠⢠8 16 1⥠⢠i ⥠= â¢34 ⥠⢠â¥â¢ ⥠⢠⥠⢠0 32 1⥠â¢nNk ⥠â¢42⥠⣠â¦â£ ⦠⣠â¦. We use (MTM)-1 from before to solve (in a least squares sense) a=1, i=1, and nNk=10.