BioWatch
PCR Assays
Building Confidence,
Ensuring Reliability
Committee on PCR Standards for the BioWatch Program
Board on Life Sciences
Division on Earth and Life Studies
Board on Health Sciences Policy
Institute of Medicine
NATIONAL RESEARCH COUNCIL AND
INSTITUTE OF MEDICINE
OF THE NATIONAL ACADEMIES
Abbreviated Version
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract No. HSHQDC-11-D-00009/Order No. HSHQDC-13-J-00274 between the National Academy of Sciences and the U.S. Department of Homeland Security. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-36722-6
International Standard Book Number-10: 0-309-36722-0
Additional copies of this report are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2015 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America.
Suggested citation: NRC (National Research Council) and IOM (Institute of Medicine). 2015. BioWatch PCR Assays: Building Confidence, Ensuring Reliability. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
COMMITTEE ON PCR STANDARDS FOR
THE BIOWATCH PROGRAM
GEORGES C. BENJAMIN (Co-Chair),† Executive Director, American Public Health Association, Washington, DC
KENNETH I. BERNS (Co-Chair), Distinguished Professor of Molecular Genetics and Microbiology Emeritus, University of Florida College of Medicine, Gainesville
EDWIN P. PRZYBYLOWICZ (Co-Chair),‡ Senior Vice President and Director of Research (retired), Eastman Kodak Company, Webster, NY
BRUCE BUDOWLE, Executive Director, Institute of Applied Genetics, and Professor, Department of Molecular and Medical Genetics, University of North Texas Health Science Center, Fort Worth
CHARLES CHIU, Associate Professor, Laboratory Medicine and Medicine, Infectious Diseases; Director, UCSF-Abbott Viral Diagnostics and Discovery Center at China Basin; and Associate Director of the UCSF Clinical Microbiology Laboratory, University of California, San Francisco
JOHN M. HARDHAM, Associate Research Fellow and Chair, Emerging Infectious Disease Program, Zoetis, Inc., Kalamazoo, MI
GRACE KUBIN, Director, Laboratory Services Section, Texas Department of State Health Services, Austin
M. ALLEN NORTHRUP, Principal, Northrup Consulting Group, San Francisco, CA
TOM SLEZAK, Program Leader, Global Security Program, E.O. Lawrence Livermore National Laboratory, Livermore, CA
PETER M. VALLONE, Leader, Applied Genetics Group, Biomolecular Measurement Division, National Institute of Standards and Technology, Gaithersburg, MD
Study Staff
KATHERINE BOWMAN, Study Director, Board on Life Sciences
AUTUMN DOWNEY, Program Officer, Board on Health Sciences Policy
KATHRYN HUGHES, Senior Program Officer, Board on Chemical Sciences and Technology
JON Q. SANDERS, Program Coordinator, Board on the Health of Select Populations (until November 2014)
NIA D. JOHNSON, Senior Research Associate, Board on Army Science and Technology (from November 2014 to March 2015)
_________________
† Since June 2014.
‡ Until May 2014.
CLAIRE BALLWEG, Senior Program Assistant, Board on Environmental Studies and Toxicology (from May 2015)
BETHELHEM MEKASHA, Financial Associate, Board on Life Sciences
ANDREW POPE, Director, Board on Health Sciences Policy
FRANCES SHARPLES, Director, Board on Life Sciences
BOARD ON LIFE SCIENCES
JAMES P. COLLINS (Chair), Arizona State University, Tempe
ENRIQUETA C. BOND, Burroughs Wellcome Fund, Marshall, VA
ROGER D. CONE, Vanderbilt University Medical Center, Nashville, TN
JOSEPH R. ECKER, Salk Institute for Biological Studies, LaJolla, CA
SEAN EDDY, HHMI Janelia Farm Research Campus, Ashburn, VA
SARAH C. R. ELGIN, Washington University, St. Louis, MO
DAVID R. FRANZ, Consultant, Frederick, MD
STEPHEN FRIEND, Sage Bionetworks, Seattle, WA
ELIZABETH HEITMAN, Vanderbilt University Medical Center, Nashville, TN
RICHARD A. JOHNSON, Global Helix LLC, Washington, DC
JUDITH KIMBLE, University of Wisconsin, Madison
MARY E. MAXON, Science Philanthropy Alliance, Palo Alto, CA
KAREN E. NELSON, J. Craig Venter Institute, Rockville, MD
ROBERT M. NEREM, Georgia Institute of Technology, Atlanta
MARY E. POWER, University of California, Berkeley
MARGARET RILEY, University of Massachusetts, Amherst
LANA SKIRBOLL, Sanofi, Washington, DC
JANIS C. WEEKS, University of Oregon, Eugene
MARY WOOLLEY, Research!America, Alexandria, VA
Staff
FRANCES E. SHARPLES, Director
JO L. HUSBANDS, Scholar/Senior Project Director
JAY B. LABOV, Senior Scientist/Program Director for Biology Education
LIDA ANESTIDOU, Senior Program Officer, ILAR
KATHERINE W. BOWMAN, Senior Program Officer
MARILEE K. SHELTON-DAVENPORT, Senior Program Officer
KEEGAN SAWYER, Program Officer
AUDREY THEVENON, Associate Program Officer
VANESSA LESTER, Research Associate
BETHELHEM M. MEKASHA, Financial Associate
ANGELA KOLESNIKOVA, Administrative Assistant
JENNA OGILVIE, Senior Program Assistant
KANOKO MAEDA, Senior Program Assistant
BOARD ON HEALTH SCIENCES POLICY
JEFFREY KAHN (Chair), Johns Hopkins University, Baltimore, MD
ELI Y. ADASHI, Brown University, Providence, RI
HUDA AKIL, University of Michigan, Ann Arbor
WYLIE BURKE, University of Washington, Seattle
ROBERT M. CALIFF, Duke University Medical Center, Durham, NC
R. ALTA CHARO, University of Wisconsin, Madison
KATHLEEN A. DRACUP, University of California, San Francisco
MICHAEL EHLERS, Pfizer Inc., Cambridge, MA
NAOMI L. GERBER, George Mason University, Fairfax, VA
LEWIS R. GOLDFRANK, New York University School of Medicine, New York City
STEVEN E. HYMAN, The Broad Institute, Cambridge, MA
PATRICIA A. KING, Georgetown University Law Center, Washington, DC
HARRY T. ORR, University of Minnesota, Minneapolis
SHARON TERRY, Genetic Alliance, Washington, DC
REED V. TUCKSON, Tuckson Health Connections, LLC, Edina, MN
CLYDE YANCY, Northwestern University, Chicago, IL
Staff
ANDREW M. POPE, Director
BRUCE ALTEVOGT, Senior Program Officer
AUTUMN DOWNEY, Program Officer
Acknowledgments
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
Cynthia Bruckner-Lea, Pacific Northwest National Laboratory
Nancy D. Connell, Rutgers New Jersey Medical School
Christina Egan, New York State Department of Health
Sean Eddy, Howard Hughes Medical Institute Janelia Farm Research Campus
Joan Gebhardt, Naval Medical Research Center
Jonna A.K. Mazet, University of California, Davis
Suzet M. McKinney, Chicago Department of Public Health
R. Paul Schaudies, GenArraytion, Inc.
Victor Waddell, Arizona Department of Health Services
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report
before its release. The review of this report was overseen by Ronald M. Atlas, University of Louisville and David R. Walt, Tufts University. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
The committee thanks all those who provided information during the course of the study. The committee particularly thanks Kerby Shedden, University of Michigan, and Michael Cohen, National Research Council, for their assistance and thoughtful conversations on statistical confidence intervals and experimental design.
Preface
Knowing when a health threat enters a community is the cornerstone of biopreparedness. Historically, public health officials have had to rely on a clinical detection system that includes the astute clinician, the unanticipated lab result, or a cluster of disease cases to know that some disease threat was present. These methods of detection, while important, may put the health security system behind the curve in responding to a disease threat in the case of a bioterrorist attack. The BioWatch program is designed to get upstream of these clinical detection systems by providing early warning to local, state, and federal partners should an airborne release of certain types of biothreat agents occur. The program relies on collectors deployed in jurisdictions around the country, and on sample results provided through public health laboratories. It is a component of a much broader monitoring and surveillance effort to identify and respond to terrorist threats that ensures the health security of the nation.
Ensuring health security requires an early warning system with which its users have a high degree of confidence. The users need a clear understanding of the meaning and limitations of the data they receive and have trust that they can use the results to make subsequent, potentially high-consequence decisions. As the report discusses, standards and validation provide a necessary basis for confidence in the system.
Although standards may not seem glamorous, they are fundamental. Our own experiences studying infectious microorganisms and working to support a strong and effective public health system lent us motivation
for the task. The committee welcomed the opportunity to bring attention to the critical roles that standards play in supporting a program such as BioWatch, the efforts that have taken place to develop performance standards that could be applicable to program needs, and the base of information that has been gathered on BioWatch assay and system performance. Based on our statement of task, the report focuses on performance standards for the polymerase chain reaction (PCR) assays used in BioWatch, while emphasizing that all of the system’s components are necessarily interconnected. It is evident that the rapid development of next-generation sequencing offers the promise of a more definitive approach to the secondary verification or confirmation step required after the initial PCR assay in order to secure the trust required for responsible public health officials and other government authorities to initiate an appropriate response to a select agent detection.
We were joined in our task by fellow committee members with diverse areas of expertise in genetics and molecular biology, bioinformatics, technology development, and public health. We thank them for their commitment to the study topic and for giving so willingly of their time and energy. It has been a pleasure to work with each of them. The report was also informed by presentations and discussions with a number of experts in environmental biosurveillance, pathogen biology, biodetection, assay and standards development, and validation from within and outside the federal government, as well as from state and local BioWatch jurisdictions. Their contributions were critical in informing the committee and we thank them for their willingness to share their views with us.
Finally, on behalf of the committee, we would like to thank the staff of the National Research Council and the Institute of Medicine for their assistance and dedication in working alongside us. Their support was integral in bringing the project to fruition.
Georges C. Benjamin
Kenneth I. Berns
Co-Chairs
Appendix A to the Committee’s report has been determined by the sponsor to contain information described in one or more of the exemptions to the Freedom of Information Act, 5 U.S.C. § 552(b) (FOIA). Section 15 of the Federal Advisory Committee Act provides that the Academy shall make available to the public its final report unless the Academy determines that the report would disclose matters described in Section 552(b). In such case, the Academy “shall make public an abbreviated version of the report that does not disclose those matters.” This abbreviated version of the report includes all of the committee’s findings, recommendations, and other substantive material without disclosing materials described in 5 U.S.C. § 552(b).
This page intentionally left blank.
Contents
BioWatch Detects Airborne Releases of Specific Biological Agents
PCR Assay Standards Contribute to Program Success
2. PERFORMANCE STANDARDS PROVIDE CONFIDENCE IN RESULTS
Establishing Appropriate Performance Standards Helps Results Be Actionable
Assumptions and Definitions for “PCR Assay” and Relevant Standards
Developing and Using Performance Standards
PCR Assay Performance and Validation Approaches That May Be Applicable to BioWatch
Prior Federal Efforts on Performance Standards and Assay Equivalency
Assay Performance Characterization Conducted by Los Alamos National Laboratory
3. KEY COMPONENTS OF A PERFORMANCE STANDARD 53 FOR BIOWATCH
Tradeoffs in Performance and Assay Purpose
Assay Sensitivity and Limit of Detection
Assay Specificity: Strain and Substance Panels
Understanding Environmental Background
Assay Precision and Other Performance Parameters
Positive and Negative PCR Assay Controls
Testing Laboratory Setup and Management
Reference Material Availability
Use of Prior and Ongoing Performance Test Information
4. THE FRAMEWORK THAT SURROUNDS THE PCR REACTION AFFECTS PERFORMANCE
Sample Collection, Extraction, and Purification Affect PCR Assay Performance
Validation Is the Framework That Supports Quality
Continuous Monitoring and Assessment Is a Necessary Part of Validation
Applying Validation Principles to Assay Performance Characterization: Putting These into Practice
Constraints of the Current System and Operational Experience of BioWatch Jurisdictions
5. LOOKING TO THE FUTURE: MULTIPLEX PCR AND NEXT-GENERATION SEQUENCING
Multiplexing PCR Assays for Use in BioWatch
Taking Account of Step Changes in Technology
6. CONCLUSIONS AND RECOMMENDATIONS
Fundamental Principles of Performance Standards and Validation
Approaches to Assay Performance Characterization and Validation
Standards Approaches for Multiplexing Real-Time PCR Assays
Incorporating Advances in Sequencing Technology into the Program
This page intentionally left blank.
3-1 Flowchart representing steps in assay analytical performance characterization
3-4 Laboratory organization used by LANL in conducting assay testing
4-3 Steps in assay and system validation
TABLES
S-1 Applications of Next-Generation Sequencing for BioWatch
1-1 Selected Federal Strategy Documents Related to Environmental Biosurveillance
2-1 Time Line of Selected Programs and Milestones
2-2 Recent BioWatch Performance Testing and Validation Activities
3-2 Comparison of SPADA, PHAA/FSAPE, and BioWatch Acquisition Testing Approaches
This page intentionally left blank.
Acronyms and Abbreviations
APHL | Association of Public Health Laboratories |
BAR | BioWatch Actionable Result |
BDS | Post Office Biohazard Detection System |
BTRA | Biological Terrorism Risk Assessment |
CBRNE | chemical, biological, radiological, nuclear, and explosives |
CDC | Centers for Disease Control and Prevention |
CRP | Department of Defense Critical Reagents Program |
CRS | Congressional Research Service |
Ct | threshold cycle |
DHS | Department of Homeland Security |
DNA | deoxyribonucleic acid |
DOD | Department of Defense |
DOJ | Department of Justice |
DTO | Defense Technology Objective |
EPA | Environmental Protection Agency |
EUA | emergency use authorization |
FBI | Federal Bureau of Investigation |
FDA | Food and Drug Administration |
FSAPE | Federal Standards for Assay Performance and Equivalency |
GAO | Government Accountability Office |
HHS | Department of Health and Human Services |
IOM | Institute of Medicine |
JPEO-CBD | Joint Program Executive Office for Chemical and Biological Defense |
LANL | Los Alamos National Laboratory |
LLNL | Lawrence Livermore National Laboratory |
LOD | limit of detection |
LRN | Laboratory Response Network |
MFI | median fluorescence intensity |
MIQE | Minimum Information for Publication of Quantitative Real-Time PCR Experiments |
MOU | memorandum of understanding |
NAS | National Academy of Sciences |
NPV | negative predictive value |
NRC | National Research Council |
NTTAA | National Technology Transfer and Advancement Act |
OHA | DHS Office of Heath Affairs |
OIE | World Organisation for Animal Health |
OMB | Office of Management and Budget |
PCR | polymerase chain reaction |
PHAA | Public Health Actionable Assay |
POD | probability of detection |
PSAA | Public Safety Actionable Assay |
PPV | positive predictive value |
S&T | science and technology |
SMPR | Standard Method Performance Requirement |
SPADA | Stakeholder Panel on Agent Detection Assays |
USAMRIID | U.S. Army Medical Research Institute of Infectious Diseases |
USPS | U.S. Postal Service |