National Academies Press: OpenBook
« Previous: 1 Introduction and Overview
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

2

Assessment of Current Performance

The next session featured presentations summarizing two invited papers on the performance of the current BioWatch system and a local perspective on what effect improvements might have on current operations. Molly Isbell, director of quality assurance and data science at Signature Science, discussed her paper on the program’s current approach to quality assurance, and David Brown, research scientist at Argonne National Laboratory (ANL), reviewed work using dispersion modeling tools to optimize system architecture. Jennifer Rakeman, assistant commissioner and director of the public health laboratory in the New York City Department of Health and Mental Hygiene, then spoke about how complicated BioWatch can be at the jurisdictional level and why jurisdictions rely on the BioWatch program’s work on quality assurance and dispersion modeling. An open discussion, moderated by John Clements, followed the three presentations.

OVERVIEW: THE CURRENT QUALITY ASSURANCE APPROACH

This section draws on a paper commissioned by the Planning Committee on Strategies for Effective Biological Detection Systems, “Current Quality Assurance (QA) Approach: DHS BioWatch Program,” by Molly Isbell (see Appendix F). Isbell, who in her position at Signature Science is responsible for coordinating the development of the quality assurance (QA) and quality control (QC) program for the BioWatch program, and continues to work with the Department of Homeland Security (DHS) to oversee and execute all QA activities, began her presentation by noting that development of the QA program for both laboratory and field activities began in 2010 and was fully implemented in 2011. Prior to implementing the formal QA program, BioWatch QA activities were left to the discretion of individual jurisdictions, she explained. Many of the laboratories and field teams, in fact, had developed and implemented their own QA and QC practices in close coordination with public health laboratories. While many of the public health laboratories that house BioWatch laboratories did have strong quality management system in place that the local jurisdictions took advantage of, there was no standardization across the BioWatch program and there were gaps in the system.

One of the first objectives of establishing the BioWatch QA program, said Isbell, was to implement systems for field collections and laboratory analyses that

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

would be pushed out to every field collection team and BioWatch laboratory to support consistently complete, accurate, and defensible results across the system and, at the same time, implement a monitoring system to verify that the field teams and laboratories were performing as expected. A second objective was to build data sets to provide insight into system performance, increase confidence in the day-to-day analytical results, and enable rapid identification and correction of assay performance issues before they affect the results from an operational sample. “I would argue this is one of the most important objectives of the quality assurance program,” said Isbell, explaining that it is much better to spot a problem or deficiency during an audit than when there is a BAR.

A third goal of the QA program, which Michael Walter and others had already spoken about, was the need to foster a strong, collaborative atmosphere among all of the program’s stakeholders, from local field and laboratory staff to federal partners. Isbell noted that she and her colleagues held many discussions at which a variety of opinions were voiced on how to implement a QA program to support the mission of BioWatch and at which many issues were addressed. “This was and remains a very important component of the QA program,” said Isbell.

Next, Isbell described the elements of the QA program, starting with the QA program plans for laboratory and field operations. These program plans detail the systems, procedures, and requirements to which the program needs to adhere to ensure the quality and defensibility of the results. The laboratory operations QA program plan, for example, is modeled after the ISO1705 standard,1 the internationally recognized standard for the competence of testing and calibration laboratories, as well as the ISO9001 standard2 on quality management systems, the Clinical Laboratory Improvement Amendments3 issued by the U.S. Centers for Medicare & Medicaid Services, and other standards for laboratory operations. One section of this program plan addresses the quality management system, including document control and records review, how to conduct audits and inspections, and how to take corrective action. Another section describes technical systems QA requirements, detailing training and qualification of staff, the type of facilities needed, decontamination procedures, workflow documentation, performance assessment, the procedure to follow when implementing new methods, and how to document QA performance in reports to DHS.

Isbell said the BioWatch program office reviews and updates the QA program plans annually in coordination with the laboratories, field teams, and various partners. She noted the plans have evolved over time to be more specific about how things must be done. A team of two auditors—at least one of whom is an American Society for Quality (ASQ) Certified Quality Auditor—audits every BioWatch laboratory and field team in accordance with ISO190114 auditing

___________________

1 See https://www.iso.org/standard/39883.html (accessed October 18, 2017).

2 See https://www.iso.org/iso-9001-quality-management.html (accessed October 18, 2017).

3 See https://www.cms.gov/Regulations-and-Guidance/Legislation/CLIA/index.html (accessed October 18, 2017).

4 See https://www.iso.org/standard/50675.html (accessed October 18, 2017).

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

standards at least every other year, with additional off-cycle or surprise audits conducted at DHS’s discretion. The audits focus on compliance with the QA program plans, BioWatch protocols, local standard operating procedures, and generally accepted good practices. The auditors review documents and records and try to watch every activity, from sample collection to reporting the results, multiple times to truly understand how the process is working and where there might be deficiencies.

Though a contractor conducts the audits, they are considered DHS audits. DHS reviews all of the audit reports and provides them to the laboratory or field teams along with appropriate feedback highlighting areas in which the laboratory or field teams need to focus. The reports also include best practices other jurisdictions may have developed and program-level observations. Isbell explained that the latter represent instances in which there might be some ambiguity in a requirement or a contradiction between documents that are not for the field teams or laboratories to address but fall under the responsibility of the program office to fix.

Any deficiencies identified require a formal corrective action report from the field or laboratory team. The auditors and DHS review that report and DHS provides feedback to the field or laboratory team. If DHS finds the response unacceptable or when deficiencies recur, the BioWatch program and DHS develop a tailored approach to assist the team. BioWatch maintains a database of all of the audit reports and responses to enable the program office to provide more informative feedback to the laboratories and field teams, said Isbell.

Another component of the QA program is to provide external QA samples, prepared by the Naval Surface Warfare Center, that the laboratories then use to monitor performance. The samples are nonreactive, archived quarter filters spiked with a cocktail and a QA tag to rule out QA sample cross-contamination that could lead to a BioWatch Actionable Result (BAR). The archived filters come from the jurisdictions so they represent the local environmental background. The spiking cocktail includes positive plasmid controls and Escherichia coli–transformed plasmid constructs, and the level at which the samples are spiked is the lowest level at which all assays can detect the constructs at least 95 percent of the time with 95 percent statistical confidence. “I want to be clear that we are not necessarily talking about decision-level false positives and false negatives,” said Isbell. “We are trying to understand at the very detailed level how well did these specific assays perform in the presence of environmental background on these operationally relevant matrices.”

Every laboratory runs the QA samples daily, to the extent possible, alongside operational samples, though not all the way through the full decision-making process, and laboratories enter the results into a DHS QA database. DHS reviews the results on an ongoing basis both from a statistical perspective and to identify likely causes of errors. The statistical review computes assay-level false-positive and false-negative rates and confidence intervals, and compares the results to program benchmarks. DHS reviews the data by laboratory and by assay, as well as by laboratories across assay and by assay across laboratories. Isbell noted there are automated routines in place for examining the incoming data to understand

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

what is going on in the field and laboratory and provide meaningful and timely feedback to BioWatch program and the local field teams and laboratories.

The QA program office issues weekly program-wide reports to DHS consisting of tables, graphical displays, and a short written summary focusing on assays and laboratories not meeting expectations, identifying issues that might be emerging, and including known or suspected causes of error. The QA program also provides weekly laboratory-specific summary reports to each laboratory and monthly laboratory-specific performance status reports to laboratory directors and laboratory leads. The status reports include a one-page summary of performance relative to benchmarks and other BioWatch laboratories, as well as a list of recent errors and causes.

In addition to the QA samples, every BioWatch laboratory participates three times a year in a program-wide proficiency test that an accredited proficiency test provider develops and oversees. Each laboratory is given a defined amount of time to analyze proficiency test samples as if they were operational samples, following the full concept of operations and decision algorithm to make a final call as to whether to declare a BAR. One difference between this assessment and the daily QA sample process is that the samples are prepared on a pooled rather than jurisdiction-specific environmental background. The laboratories all receive proficiency test results listing the expected result, what each laboratory obtained, and any issues encountered during the test. Each laboratory is also assigned a pass or fail score. Isbell said that, similar to audits, proficiency test failures and issues require formal corrective action responses. The program also encourages laboratories to respond to nonfailing errors and observations. Again, if a response is unacceptable to DHS or multiple failures occur, a tailored approach is applied to bring the team into compliance.

The final piece of the QA program is the system-wide performance monitoring report, which is a high-level, system-wide QA analysis report delivered twice a year to DHS. This report is a comprehensive, system-wide QA analysis that includes audit highlights, performance test summaries, and QA sample result trends and observations. Currently, said Isbell, DHS receives separate reports for laboratory and field performance.

OVERVIEW: DISPERSION MODELING TOOLS TO OPTIMIZE BIOLOGICAL DETECTION ARCHITECTURES

This section draws on a paper commissioned by the Planning Committee on Strategies for Effective Biological Detection Systems, “Use of Dispersion Modeling Tools in Optimizing Biological Detection Architectures,” by David Brown (see Appendix G). Within the BioWatch operating environment, modeling efforts focus on three critical domains: indoors, subway systems, and outdoors, said Brown. Indoor facilities include intermodal transit facilities, stadia and convention centers, airports, and high-value buildings. From a modeling perspective, indoor facilities are small scale—on the order of 100 to 1,000 meters—but complex.

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

Subway systems include heavy rail, such as the Washington, DC; New York City; and Chicago systems; light rail, such as in Seattle, Buffalo, and Boston’s Green Line; and regional rail, such as Amtrak in the Northeast corridor or the MARC system in Maryland and Washington, DC. Together, they represent a scale ranging from a few miles to 100-plus miles. The outdoor domain spans a scale of 10 to 50 miles. In the past, modelers have looked at each of these domains separately, but Brown said he and his colleagues are now starting to look at them as a combined problem.

Indoor modeling relies on a program called CONTAM, developed and maintained over the past 15 to 20 years by the National Institute of Standards and Technology (see Figure 2-1). CONTAM segments buildings into zones informed by heating, ventilation, and air conditioning (HVAC) systems, and it accounts for the ventilation characteristics of a building, openings in the building, and leakage that can be driven both by meteorology and external pressures, such as from an interconnected subway system. This model can account for size-dependent particle removal by filtration and deposition, and it uses “interzone mixing” to account for airflows related to people moving about indoors. Brown noted that he and his colleagues have coupled the CONTAM model to the ANL Below Ground Model (BGM) for subway systems to look at how contamination might be exchanged between these two domains if a release occurred in one venue or the other.

The Argonne BGM examines the transport dispersion of a biological or chemical agent throughout an entire subway system. It also accounts for emissions of materials to the outdoors via vents, station exits, and portals, and ingestion of an outdoor release into a subway system. Field studies in Washington, DC;

Image
FIGURE 2-1 Indoor modeling using CONTAM. SOURCE: Brown slide 4.
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

Boston; and New York City have validated BGM, said Brown, and it has modeled subway systems in Chicago, San Francisco, Los Angeles, and Seattle, among others. He noted that, when combined with an outdoor model, BGM can be a powerful tool for siting BioWatch collectors, and when combined with a passenger model it can look at dose effect on individuals and how passengers spread materials through the system.

For an outdoor model, Brown said he and his ANL colleagues are gravitating toward the Quick Urban and Industrial Complex (QUIC) dispersion model developed at Los Alamos National Laboratory. He explained that QUIC is “building aware,” which means it looks at airflow in and around every building in a defined grid and, thus, is good at handling street canyon effects (see Figure 2-2). Brown characterized QUIC as a fast-running model that includes algorithms for chemical, biological, and radiological releases. Several hundred groups in the United States and internationally have used QUIC, and it has been tested and evaluated in several major field studies. He noted that QUIC performs as well as more complex computational fluid dynamics models in a tiny fraction of the run time.

Image
FIGURE 2-2 The QUIC urban dispersion model handles airflow around buildings. SOURCE: Brown slide 6.
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

For the past 4 years, Brown and his colleagues have been working to link these models together to enable cross-domain modeling, thanks to funding from BioWatch and the DHS Science and Technology Directorate (S&T). Their current approach is to chain the output of one model into another; the output of the subway model, for example, serves as initialization data for the outdoor model. Brown hopes that, within a year, they will be able to run the models dynamically so that a single code addresses an entire city, at least to the extent that data are available to support that type of modeling.

Using these models to help with siting portable sampling units requires tens of thousands of individual model runs using a library of release scenarios and performing statistical analyses of the results. For example, a modeling study of the New York City subway system might include 600 release locations, each of which the model examines over a range of particle sizes, release amounts, ability of the agent to be airborne, how populations would be exposed to the agent, and other factors such as different HVAC operating conditions and train schedules.

For cross-domain collector siting, the key goal is to assess the ability of portable sampling units in one domain to detect releases in another. This approach is motivated by analyses and field data demonstrating that a robust subway collector network can detect a substantial fraction of outdoor releases in dense urban environments. “We are starting to explore that now to get a realistic sense of detection probabilities outdoors,” said Brown.

Two useful performance metrics for these models, he noted, are the fraction of population protected and the probability of detection. The fraction of the population protected is the preferred metric for optimizing collector deployments, Brown noted. Its key advantages are that it accounts implicitly for population by emphasizing more consequential events and deemphasizing release scenarios that are less consequential. This metric is also not overly sensitive to the ranges that bound the scenario library, such as the size of the release. The disadvantages of using this metric include that it can be a hard concept to explain to jurisdictional officials and policy makers, and it can be hard to define who would actually be exposed in an event in environments such as subway and transit systems with large transient populations. “Figuring out time of exposure can be tricky,” said Brown.

The probability of detection can be expressed in terms of the probability of detecting a release affecting 1,000 or more people or 10,000 or more people. The advantage of using it as a performance metric is that it is easy to define and communicate. Moreover, if placed in the proper context, it can provide quantifiable information on what fraction of the release scenarios are detected, said Brown. It is also independent of agent dose-response considerations. One main disadvantage of using probability of detection as a metric is that population exposure is not included, which means that all releases are of equal importance. This measure can be very sensitive to parameter ranges in the scenario library, particularly release size, and it can be of limited utility in considering indoor and subway facilities if the number of people affected is set too large—the probability of detecting a release affecting 10,000 people or more, for example—because it cannot

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

discern the benefit of additional collectors once the probability of detection approaches 100 percent.

In terms of modeling response strategies, Brown said that scenario libraries of urban biological agent releases generated for collector siting are ideal for constructing response frameworks. This approach uses statistical techniques that can rapidly interrogate the scenario library given a single BAR or set of BARs. Currently, he and his colleagues are working to marry the three domains to evaluate response scenarios for a realistic attack. One major challenge is to determine the location of the release absent other information. For example, a subway BAR could result from an outdoor release and vice versa, and distinguishing between the two would be impossible if collectors were only indoors or outdoors. A single-collector BAR provides little information with which to work, so he and his colleagues have looked at collector siting strategies that would minimize the probability of a single-collector BAR.

In summary, Brown said that detailed physical modeling is a cornerstone of collector siting and response in indoor, subway, and outdoor environments. Indoor, subway, and outdoor models are legacy tools, but they are under continual improvement in both their physics and their application. Toward that end, he noted that the ANL team has tested and evaluated computation models with real-world data, such as the data from the 2016 New York City Urban Transportation Restoration tracer experiments, linked computational models and siting analyses across domains, and evaluated new phenomena that can drive the problem, such as fomite5 transport. He and his colleagues have also used statistical modeling to try to reduce the probabilities of single-collector BARs, and they have begun developing techniques that can serve as response tools to assess source characteristics in the event of BARs across domains.

HOW IMPROVEMENTS MIGHT AFFECT CURRENT OPERATIONS AND THE QA PROCESS

Jennifer Rakeman began the panel discussion on how improvements to the BioWatch system could affect the existing QA systems and performance models. BioWatch’s QA program is incredibly important, said Rakeman, because in her position she needs to know that a positive result from a polymerase chain reaction (PCR) assay confirms the detection of DNA that matches the probes in the assay. Modeling and siting of the portable collection units is important when it comes to considering coverage and redundancy to reduce the chances of a single-collector BAR. “From where I sit, we need to pull together the laboratory results in the context of that robust quality assurance system, plus the modeling and model-based siting of the collectors throughout the jurisdiction, to appropriately interpret a BioWatch Actionable Result from the laboratory and allow us to answer the

___________________

5 A fomite is any object or substance capable of carrying infectious organisms, such as viruses or bacteria, and transferring them from one place to another.

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

question of whether a detection signals a real threat to public health,” said Rakeman.

New York City is complicated by the fact that its five boroughs are home to 8.55 million residents living at a density of 27,000 people per square mile, that 60 million tourists visit the city each year, and that on any given workday the population of Manhattan doubles because of commuters. In addition, 54 percent of households in New York City do not own a car and rely on public transportation. “A BioWatch result that shuts down the transportation system would be significant because people would not be able to get home,” said Rakeman.

Currently, New York City has more than 40 portable sampling units, many in indoor transit hubs, and there are three sample collections per day involving multiple shifts of staff, creating issues around training and competency. Each year, her laboratory tests more than 27,000 samples, or more than 10 percent of all U.S. BioWatch samples, and for any sample that tests positive, her laboratory needs to assess rapidly whether it is a false positive or it signals a public health threat. With that many samples, said Rakeman, there is a high probability for false positives.

New York City is also a transit hub for the Eastern seaboard, a site for many special events requiring additional BioWatch testing, and the nation’s financial hub, all of which means that shutting down the city in the event of a BAR would have effects across the globe. Moreover, there are many stakeholders, including public heath, environmental protection, law enforcement, transit officials, and others, including several federal partners. “Our stakeholder meetings are big events, with many people involved in decision making,” said Rakeman. She added that because New York City has been a terrorist target before, its response plans are forward leaning, including a mass prophylaxis plan that can deliver antibiotics to 8.5 million people even before determining if a BAR involves a viable organism and if that organism is susceptible to those antibiotics.

The current BioWatch system in New York City relies on a real-time PCR platform and a second verification panel for any potential positive result for any biological agent. “Our protocols are ahead of their time and are different than everyone else’s,” said Rakeman. In fact, she said, New York City’s work on a second verification panel led to nationwide implementation of a second verification panel for one particular agent. She noted that DHS’s transition to a critical reagents screening program led to a decrease in verification tests performed and this in conjunction with the city implementing a second verification panel led to a decrease in false-positive laboratory results.

There have been two real-life near-BARs in New York City. The first was on July 4, 2010, when an outdoor sampler on Manhattan’s west side had a positive result that passed the agent-specific algorithm after the verification panel. In this instance, the city’s public health laboratory had previously reported issues with the verification reagents to the CDC and DHS, and the decision was made to not declare a BAR and to unilaterally stop testing for this particular biological agent. Later that year, on the first day of the United Nations General Assembly, the

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

agent-specific algorithm was nearly met on multiple samplers over multiple times for a different biological agent, with similar results being reported up and down the East Coast. After a great deal of discussion, the decision was made that this was not a true attack and was likely the result of an endemic organism.

Given the potential consequences of declaring a BAR in New York City, an additional layer of discussion was created—the Department of Health Advisory Committee—that convenes between a first and second verification result to provide some time internally to talk about potential impacts of declaring a BAR. The mission of this advisory committee is to consider the available information and provide the health commissioner with recommendations on initial public health actions prior to reporting a BAR.

She noted that the New York City Police Department’s viewpoint is also important in the decision-making process since it manages the program from a law enforcement perspective. Declaring a BAR in the city, said Rakeman, could result in a rapid and complicated response that would surely be noticed given the population density in the city. “We cannot go in and do some Phase I sampling with nobody realizing it,” she explained. “We would shut down a subway station or be on a street in the middle of Manhattan.” In addition, the response would affect other jurisdictions, including those in New Jersey and other areas connected to the city by regional transit, as well as private facilities connected to the city transit system. “We need to work with those facilities to get on the same page as to how decision making happens,” said Rakeman.

In closing, Rakeman noted that because of the forward-leaning nature of the response, including its mass prophylaxis program, declaring a BAR in response to a false positive will not have zero impact, morbidity, and even mortality. As a result, the deliberations she and her colleagues engage in would involve a type of cost-benefit analysis that weighs the effect of not responding to a BAR if it is real versus the effect of responding to a BAR that does not pose a threat to public health. The bottom line, she said, is there is no room for a false positive in New York City.

DISCUSSION

As a prelude to the open discussion, Isbell reviewed the QA process. The elements of the QA program, she reminded the audience, are the QA program plans, laboratory and field audits, external QA samples, proficiency tests, and QA performance reviews and reports. Giving credit to the BioWatch program office, the laboratories, and the field operations, she said these elements create a strong foundation, and while some of the details might change, these elements are predicated on an already validated collection and analysis system. “So whenever there are improvements, it is going to be a matter of adjusting how we do QA and not having to completely build a new QA program,” she said.

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

Isbell also reiterated the goals of the QA program, which are to

  • Implement systems for field collections and laboratory analyses to support consistently complete, accurate, and defensible results;
  • Monitor the system to verify that methods are being implemented and are performing as intended;
  • Build data sets that provide insight into system performance and increase confidence in results;
  • Rapidly identify and correct system or assay performance issues; and
  • Foster collaboration among stakeholders.

Any improvement to the BioWatch system must undergo rigorous, statistically defensible validation, where validation is the process of demonstrating that the method is suitable for its intended purpose and that it satisfies QA program metrics, said Isbell. In that regard, validation of any improvement to the collection or analysis system should address sensitivity, specificity, accuracy and precision, the range of conditions in which the improvement will operate, uncertainty, and ruggedness in both indoor and outdoor environments. To a large extent, she expects BioWatch to remain a semiquantitative program.

With that, the open discussion began with an unidentified participant asking how Isbell saw QA changing if BioWatch increases the number of signatures it examines. Isbell replied that the biggest challenge she sees is creating materials that can be put in a matrix to assess the performance of analytes as the target list expands. She reminded the workshop that, prior to 2015, QA had the ability to spike filters with inactivated agents produced by Department of Defense (DoD) laboratories for that purpose. “We felt that was a good representation of what might be encountered operationally,” said Isbell. However, in 2015 there was an incident involving inadvertently shipping a live agent, so the program was discontinued. This is why QA now uses positive plasmid controls, which she said is an imperfect solution because it is not representative of what would be seen operationally. She added that while DoD is working on developing better surrogates for the specific agents on the BioWatch target analyte list, it may be necessary to “step back from the current luxury of being able to characterize every single target to the Nth degree and maybe use representative targets.”

John Clements, the session moderator, asked Isbell what the lead time would be to get a new QA and QC program operational if the decision was made to add another agent. She replied that if she had the appropriate spiking material, it would take no time at all. “We have the infrastructure and the system for doing it,” she said. The question is whether a suitable spiking material exists, she added. Rakeman commented that there are more steps involved to getting a new test online in a laboratory than just getting a QA program online. What she would need to do, she said, is develop, validate, and verify an appropriate standard operating procedure for the new agent and get the new assay online.

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

Responding to a question from an unidentified participant about how she would approach creating a representative background as the BioWatch program moves into a wider variety of environments, Isbell said that as long as there are filter remnants from those new environments that are verified to be nonreactive, those would be the most representative of the operational environment. If those remnants are not available as a result of a new technology, it would be necessary to collect a sample that is not an actual operational sample but is collected in the same manner. This would again provide a representative background sample. Rakeman noted that the matrices upon which background samples are based vary a great deal. In New York City, for example, some filters are “black and soot heavy,” some look clean, and others are between those two extremes. “It varies with the weather, the time of year, the time of collection, and the location of collection,” she said. The variability of the matrix, she added, creates challenges for validating an assay and looking at sensitivity, specificity, and other parameters.

Mark Buttner, associate laboratory director of the Nevada State Public Health Laboratory, University of Nevada, Las Vegas Branch, first complimented Isbell and her colleagues for establishing the well-run and rigorous QA program. He then noted that his laboratory spends more time on QA than it does running operational samples because it generates a great deal of information that he uses to inform his many stakeholders. He then asked Isbell how her team uses those data to inform the program and what changes she expects in the future with regard to verification markers. Isbell replied that the BioWatch program office makes decisions about verification markers based on the QA data she and her team provide. With respect to the QA data, she reiterated that her team provides weekly summaries to DHS that include notes about things that may not be performing as expected. The national program office then works on those specific issues.

Buttner also asked Isbell if the sensitivity of the assays could be used to estimate the lower limit of detection in terms of particles per cubic meter in the outdoor environment, for example. Isbell said the answer was yes to some extent with the current spiking materials. She added that if the program is allowed to start using inactivated surrogates again, the answer would be a more definitive yes.

An unidentified participant asked Brown if there are any modeling activities considering dispersion of contaminated people over time after exposure as a means of improving treatment and response—given the delays in identifying a positive sample from the time it is collected. Brown said this delay is a major concern because, while the initial attack on a subway system, for example, might be over relatively quickly, material will have already been transported to the outdoors, both through airflow and by people contaminated with the agent moving out of the subway system and dispersing through the city. “If you look at any major city, we can provide a fairly good estimate of a range of contamination on people,” said Brown, who added there are still uncertainties about the resuspension rate of particles coming off a contaminated person that he hopes to resolve over the next year. Nonetheless, said Brown, it is clear that human transport of contaminated material becomes a worldwide problem within 24 to 36 hours after

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

an attack because some of those who are contaminated will go to an airport and travel internationally. Identifying those people is a difficult challenge that he and his colleagues are only starting to try to understand.

John Vitko asked Brown if there is an adequate density of portable sampling units for a desired sensitivity, given the drive to use as few sampling units as possible because of economic concerns. Brown replied that, from a public health perspective, there are never enough detectors, particularly when trying to avoid the single-collector BAR. However, acknowledging that cost is a concern, the analyses the ANL team examines are at the marginal protection benefit to protect a specific area, population, or subway system. One confounding issue with subway systems is that the detection threshold in a dirty environment relative to a clean, outdoor environment is still something that needs to be determined. Brown did note that field experiments in a subway system demonstrated that the exact placement of a sampling unit in a subway station does not seem to make much difference with regard to detection because air in a subway station is well mixed. For an outdoor environment, exact siting could be more important, but often siting is determined by the location of electrical outlets and where the sampling unit does not interfere with the public or routine operations of the facility.

An unidentified participant asked Rakeman if she knew of the research DHS S&T performed several years ago that attempted to characterize resistance to antibiotics in the national stockpile. That research, the participant said, found widely varying single nucleotide polymorphisms (SNPs) associated with resistance, which would make creating a single rapid test difficult. She replied she was not aware of that research, but having such a test is on her wish list. Currently, the choice of antibiotics to dispense is based on what is known about the organism in general and that choice is balanced against possible adverse effects of those drugs, particularly when given to children and infants. To be useful, she would need information about antibiotic resistance in 5 to 6 hours, which suggests the test would need to be PCR based. She noted that clinical tests for resistance in tuberculosis require whole-genome sequencing to identify even some of the SNPs associated with antibiotic resistance. The problem is that naturally occurring antibiotic resistance is difficult to identify because of the multiple mechanisms and multiple mutations involved in developing resistance.

Clements then asked Rakeman if her laboratory attempts to culture the detected organism as soon as there is a positive result. “If we had a BioWatch Actionable Result on a filter we would make attempts to culture it off of the filter,” she said, adding that the organism on the filter may not even be alive, particularly if it is not an organism that forms spores. “We want to be able to collect samples in a way that we can culture them directly,” she said. One of the issues in declaring a BAR, Rakeman added, is deciding if a positive detection signal is a true public health threat without knowing if the detected organism is even viable and infectious.

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×

Sanjiv Shah, after complimenting Isbell’s team for developing such a robust QA program, asked if improvements to the BioWatch program will also include confidence-level improvements with regard to process validation. Isbell replied that validating the sensitivity, specificity, and sampling power of any new technologies or procedures will require working closely with the ANL group on dispersion modeling to know what levels the new process needs to be able to detect. She also said that she and her colleagues go through an extensive evaluation of any new technology to understand what sample size and how many data points are needed to set performance metrics and understand the statistical confidence that a positive result should trigger an action. Shah added there needs to be an educational effort to help decision makers and the public understand the challenges of triggering an event based on a positive test result.

Jennifer Heimberg from the National Academies of Sciences, Engineering, and Medicine asked Brown if there is information he would like to get from the BioWatch system that would benefit his modeling work. Brown replied that for sampling unit siting purposes, he would like a better idea of the statistics on detection itself. With regard to triggering a response, he would like more quantitative data about how many organisms are collected on a sample.

Regarding expanding the list or organisms BioWatch monitors, Clements asked Brown if modeling accounts for formulation and other characteristics that could differ from the Bacillus anthracis model organism. Brown replied there are two issues to consider. One is the viability of a given organism, and the other relates to how formulation affects fomite transport and resuspension of material. As far as he knows, there are no data in the open or classified literature on the latter, and this is important information to have when modeling dispersion. Brown did note that the models do account for biological decay in terms of how many people might be infected at a given dose. For example, outdoor models account for how quickly sunlight will denature specific organisms.

Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 18
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 19
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 20
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 21
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 22
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 23
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 24
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 25
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 26
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 27
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 28
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 29
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 30
Suggested Citation:"2 Assessment of Current Performance." National Academies of Sciences, Engineering, and Medicine. 2018. Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25014.
×
Page 31
Next: 3 Developing Effective Biological Detection Systems »
Strategies for Effective Improvements to the BioWatch System: Proceedings of a Workshop Get This Book
×
Buy Paperback | $65.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

BioWatch is the Department of Homeland Security’s (DHS’s) system for detecting an aerosolized biological attack using collectors that are positioned strategically across the country to continuously monitor the air for biological threats. As currently deployed, BioWatch collectors draw air through filters that field technicians collect daily and transport to laboratories, where professional technicians analyze the material collected on the filter for evidence of biological threats.

As part of the BioWatch program’s efforts to enhance its effectiveness and capabilities, particularly with regard to detecting biological threats in challenging indoor environments, DHS requested the National Academies of Sciences, Engineering, and Medicine hold a workshop to explore alternative and effective biodetection systems for aerosolized biological agents that would meet BioWatch’s technical and operational requirements, integrate into the existing system architecture and public health infrastructure, and be deployable by 2027. This publication summarizes the presentations and discussions from the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!