Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary (2005)

Prior to the present Final Report, the committee issued an Interim Report that addressed several of the committee’s tasks for this study (the committee’s statement of task can be found in the Appendix). A brief summary of the Interim Report is provided below.

The committee developed a technical framework that considers the decision process for test and evaluation, including the WSLAT option. The risk mitigating value of live agent simulants and component-level testing were evaluated in the context of biological point detection systems in general and JBPDS in particular. Three specific questions are integral to the decision framework.

1. Are simulants good surrogates for live agents?

2. Can whole system performance against live agents be predicted from whole system and component testing with simulants?

3. Are testing methodology and facilities adequate and complete?

The four most commonly used biothreat agent simulants under consideration, Bacillus subtilis var niger (BG), Erwinia herbicola (EH), Male Specific Coliphage (MS2), and Ovalbumin (OV) were evaluated for effectiveness as live agent surrogates. The consensus of the committee is that BG is the only reasonable surrogate for a full range of biothreat agent properties (e.g. physical, antigenic).

A single simulant is unlikely to meet the requirements for all live agents of concern; therefore, the committee believes that representative simulants may be selected for classes of similar or related live agents.

When available, killed related strains provide effective surrogates to biothreat agents, with the benefit of less stringent handling. The component tests can be conducted at lower biosafety levels than the corresponding live-agent tests, thereby greatly increasing the flexibility and variability of component testing (although the live-agent component testing is still required).

The committee supports the use of the current simulants in open-air whole-system tests as part of operational checkout, i.e., environmental tolerances, maintenance and logistical support.

The use of killed related strains may not obviate the need for a WSLAT facility. It remains to be determined whether whole system performance can be extrapolated from component testing.

Determination of the relationships between component and whole system measurements requires a high degree of systematic testing and analysis. Multiple performance models relating system response to simulants and live agents should be considered to address the different metrics, including sensitivity and false alarm rates, in component and whole system testing.

To address component-whole system correlation at an appropriate level of detail, there should be an integrated experimental plan, coordination of data analysis reports across component and system T&E, system modeling, and creation of an integrated evaluation team. Data will be needed for each simulant category from components and whole system testing, as well as data from component testing for selected agents of concern.

The committee considered the risks and benefits of several whole system testing approaches. The committee is of the opinion that whole system killed related agent testing (WSKRAT) may provide nearly the same level of confidence as WSLAT, while conferring low risk of failure because much of the testing can be done in existing facilities. This opinion assumes that killed related agents can be identified and demonstrated for all live biological agents of concern.

The committee has also examined the options for whole-system testing, including simulant only, killed agent, killed related strain, and live agent. Assuming that comparative tests are conducted and validate the use of killed related strain pathogens, they should be used when possible so that the opportunities for aerosol testing can be increased to include tunnel tests of the whole system.

Both the test methodology and the facilities appear to need additional resources to pursue WSLAT or any of the other options the committee investigated. A WSLAT facility would provide value to both military and counterterrorism programs. However, the committee has concerns that regulatory requirements and meeting all legal stipulations will delay construction and use of a WSLAT facility in time to impact JBPDS decisions. Even if the schedule were guaranteed, extensive testing will be required. Even more important from the point of view of facilities is the fact that there are pathogens that simply cannot be done with a WSLAT strategy.

To date there has been insufficient work to define “representative” backgrounds that could then be reproduced in controlled testing facilities.

A possible means for achieving this in the near term is to inject “qualitative” background aerosols into low containment whole system testing. In parallel, the DOD should undertake efforts to develop quantitatively defined, representative background aerosols that can be reproducibly injected into whole system tests.

To mitigate the risk of WSLAT the committee recommends an investigation of the predictive value of using killed related agents as simulants. A Whole System Killed Related Agent Test (WSKRAT) strategy significantly reduces schedule risks and possibly OT&E costs compared to WSLAT. In the committee’s opinion, an alternative approach to WSKRAT, using a Whole System Killed Agent (WSKAT) strategy, would not significantly reduce the costs or schedule risks compared with WSLAT. In addition to the challenges associated with biological agents, the T&E strategy should formally incorporate system performance variation associated with changing backgrounds and inhibitors that are representative of deployment environments, and should decrease technical risks.

The Whole System Live Agent Testing (WSLAT) proposal was not specifically evaluated in the Interim Report, but was generally considered as one option to assist in the JBPDS full-rate-production (FRP) decision process.¹ At the time of drafting its Interim Report the committee had little documentation to support its answers to the questions addressed in that report; subsequently the committee received several briefings, reports, data summaries, and communications with representatives of the Joint Program Executive Office for Chemical and Biological Defense (JPEO-CBD). Although it has not seen complete information, the committee believes that it is sufficiently aware of the types of data that have been generated to elaborate on and complete its recommendations. Thus, the purpose of this final report is to expand upon the recommendations provided in the interim report and to evaluate the specific WSLAT proposal under consideration.

The remainder of the report is structured into three chapters. Chapter 2 summarizes the committee’s high-level recommendations that result from its evaluation of data, briefings, and reports that became available after the release of the Interim Report and that address the need for development of a scientifically credible evaluation plan. Note that no information obtained subsequent to the release of the Interim Report has altered the primary recommendations that we made; rather, any new information has served primarily to strengthen our confidence in the credibility of these recommendations. Chapter 3 examines the WSLAT proposal under current consideration, addressing the questions;

1. Will construction of the WSLAT facility, as defined in the current proposal, provide a capability for JBPDS testing that currently does not exist? Yes, it would be a unique facility that will require an associated enhanced infrastructure.

2. Is this capability technically necessary and sufficient? The capability is necessary to provide a basis for a JBPDS FRP decision on a reasonable time scale, but not sufficient. Additional capabilities will be needed in new methods, modeling and analysis to extrapolate WSLAT data to detector performance in operational environments.

3. Can this facility be constructed within the schedule constraints imposed by the full-rate production decision? Uncertain, because the committee has identified areas for technical improvement in the specific proposal as well as in the overall T&E plan that increase schedule risks.

4. What is the technical risk to the JBPDS program of not fabricating the WSLAT facility as proposed? The technical risk to the JBPDS program is the risk of not having the methodology to properly use the results from WSLAT. Without good methodology there is also the risk of accepting (or rejecting) the JBPDS based on incomplete or faulty test data. If the technical evaluation cannot be completed, then the JBPDS FRP slips day for day until they can develop another strategy. Technical and schedule risk for WSLAT can be reduced by addressing the areas of concern the committee identifies in Chapter 3 and holding an external design review for the updated plan.

Chapter 4 provides an outline of recommended evaluation strategies that the committee believes are both technically defensible and that will more adequately support the Army’s needs for biological agent² detection system testing and evaluation than would the current WSLAT proposal alone.