PERFORMANCE METRICS FOR THE
GLOBAL NUCLEAR
DETECTION ARCHITECTURE
Abbreviated Version
Committee on Evaluating the Performance Measures and Metrics
Development for the Global Nuclear Detection Architecture
Nuclear and Radiation Studies Board
Division on Earth and Life Studies
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract No. HSHQDC-11-D-00009 between the National Academy of Sciences and the Department of Homeland Security. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-29014-2
International Standard Book Number-10: 0-309-29014-7
Additional copies of this report are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2013 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
COMMITTEE ON EVALUATING THE PERFORMANCE
MEASURES AND METRICS DEVELOPMENT FOR THE
GLOBAL NUCLEAR DETECTION ARCHITECTURE
ARDEN BEMENT (Chair), Purdue University (retired), West Lafayette, Indiana
KELLEY COYNER, Northern Virginia Transportation Commission, Arlington, Virginia
MARTHA CRENSHAW, Stanford University, Stanford, California
JAMES S. DYER, University of Texas, Austin
ROGER L. HAGENGRUBER, University of New Mexico, Albuquerque
JOHN H. HOLMES, Port of Los Angeles, California
EDWARD H. KAPLAN, Yale School of Management, New Haven, Connecticut
JOHN MATTINGLY, North Carolina State University, Raleigh
GREGORY S. PARNELL, University of Arkansas, Fayetteville
DONALD PROSNITZ, Independent Consultant, Livermore, California
THOMAS C. SCHELLING, University of Maryland, College Park
DETLOF von WINTERFELDT, University of Southern California, Los Angeles
Staff
JENNIFER HEIMBERG, Study Director
ERIN WINGO, Senior Program Assistant
SHAUNTEÉ WHETSTONE, Senior Program Assistant (through July 8, 2013)
NUCLEAR AND RADIATION STUDIES BOARD
JAY C. DAVIS (Chair), Hertz Foundation, Livermore, California
BARBARA J. MCNEIL (Vice Chair), Harvard Medical School, Boston, Massachusetts
JOHN S. APPLEGATE, Indiana University School of Law, Bloomington
DAVID J. BRENNER, Columbia University, New York, New York
MARGARET S. Y. CHU, M.S. Chu & Associates, LLC, Albuquerque, New Mexico
MICHAEL L. CORRADINI, University of Wisconsin, Madison
PATRICIA J. CULLIGAN, Columbia University, New York, New York
ROBERT C. DYNES, University of California, San Diego
HEDVIG HRICAK, Memorial Sloan-Kettering Cancer Center, New York, New York
THOMAS H. ISAACS, Lawrence Livermore National Laboratory, Livermore, California
CAROL M. JANTZEN, Savannah River National Laboratory, Aiken, South Carolina
ANNIE B. KERSTING, Glenn T. Seaborg Institute, Lawrence Livermore National Laboratory, Livermore, California
MARTHA S. LINET, National Institutes of Health, Bethesda, Maryland
FRED A. METTLER, JR., New Mexico VA Health Care System, Albuquerque
BORIS F. MYASOEDOV, Russian Academy of Sciences, Moscow
LAWRENCE T. PAPAY, PQR, LLC, La Jolla, California
DANIEL O. STRAM, University of Southern California, Los Angeles
RICHARD J. VETTER, Mayo Clinic (retired), Rochester, Minnesota
Staff
KEVIN D. CROWLEY, Director
JENNIFER HEIMBERG, Senior Program Officer
OURANIA KOSTI, Senior Program Officer
TONI GREENLEAF, Administrative and Financial Associate
LAURA D. LLANOS, Administrative and Financial Associate
DARLENE GROS, Senior Program Assistant
ERIN WINGO, Senior Program Assistant
DANIEL POMEROY, Postdoctoral Fellow
SHAUNTEÉ WHETSTONE, Senior Program Assistant (through July 8, 2013)
Preface
The Domestic Nuclear Detection Office (DNDO) within the Department of Homeland Security (DHS) is responsible for developing and coordinating a cross-agency strategy, the Global Nuclear Detection Architecture (GNDA), to detect, analyze, and report on nuclear and radiological materials that are out of regulatory control. The GNDA is a global activity that involves programs, people, and technical systems in the United States and many other countries. It was mandated by presidential directive (in 2005) and public law (in 2006).
DNDO and its federal partners issued the Global Nuclear Detection Architecture Strategic Plan in 2010 which describes the high-level goals and federal agency responsibilities for implementing the GNDA. The U.S. government intends to undertake an annual review of the GNDA strategic plan to assess its effectiveness and identify new requirements arising from changes in technology and/or the threat environment.
DNDO has asked for advice from the National Research Council on developing quantitative approaches for assessing the effectiveness of the GNDA. This advice will be used to improve the GNDA strategic plan during future annual review cycles.
The committee approached this study by first gaining an understanding of what is meant by the “global nuclear detection architecture.” We reviewed its documentation (strategic plan, annual reviews, and the domestic implementation plan). We interviewed staff from DNDO and its many federal partners. We visited the Ports of Los Angeles (LA) and Long Beach and the LA Joint Regional Intelligence Center. The committee also invited other government agencies to tell us about their measures of effectiveness
and performance metrics (see Appendix A for full listing of the briefings received by the committee).
The GNDA is a worldwide network of detection and reporting capabilities controlled by many different entities and funding lines. It is meant to protect against a wide range of adaptive and committed adversaries. Developing metrics to measure the effectiveness of such a complex system of systems is a difficult problem.
DNDO and its partner agencies have developed documentation and an initial accounting of the many existing federal programs and activities that support nuclear detection and reporting objectives. The committee has developed metrics and an analysis framework that may help guide DNDO and its GNDA partners from this initial accounting to developing a capability to measure the effectiveness of the overall GNDA.
However, it became clear during the course of the study that the lack of a lead architect and a centralized GNDA budget (see Observation 1) make it difficult for the GNDA to function as a system rather than a collection of programs. The decision to address this concern (e.g., to assign clear leadership through organizational change) rests with the U.S. government.
I would like to extend special thanks to Captain John Holmes for organizing and the Ports of LA and Long Beach for hosting the committee during one of our information-gathering sessions. Our visit to the ports and the surrounding facilities and discussions with numerous stakeholders allowed the committee to see a unique example of federal, state, and local agencies truly working together toward a common goal of protecting the nation against threats.
Arden Bement, Chair
Acknowledgments
The committee wishes to acknowledge and thank a number of individuals and organizations for their valuable contributions to this study:
The Department of Homeland Security (DHS) Domestic Nuclear Detection Office (DNDO) for its sponsorship of this study, and especially DNDO staff members Brendan Plapp, John Zabko, Kimberly Koeppel, Greg Haugan, and Richard Passow.
Those who gave presentations at the committee’s information-gathering meetings provided insight and information that made this report possible (presentations are listed in Appendix A):
• Major General Julie A. Bentz, National Security Council
• Brendan Plapp, DNDO Architecture and Plans Directorate (currently with the National Defense University)
• John Zabko, DNDO Architecture and Plans Directorate
• Kevin Hart, DNDO Architecture and Plans Directorate
• Teri N. Leffer, Department of Energy, National Nuclear Security Administration, Second Line of Defense (NA-256)
• Steven Streetman, Data Architecture Solutions, Inc.
• Henry Willis, RAND Corporation
• David Kulp, Department of Defense
• Ernest Muenchau, DNDO Operation Support Directorate (OSD) (deceased)
• Colonel Robert Kolterman, Defense Threat Reduction Agency (DTRA)
• Brian Savage, DNDO OSD, Joint Analysis Center
• James Smith, Los Alamos National Laboratory
• Mark Oliphant, DNDO Red Team and Net Assessments Directorate
• J. J. Fisher, DNDO OSD
• Bernie Bogdan, Federal Bureau of Investigation (FBI)
• J. C. Wyss, State Department
• David Travers, U.S. Environmental Protection Agency
• Detective Jeff Shanaphy, Los Angeles Port Police
• Sgt. Peter Jackson, Los Angeles Sheriff’s Department
The Ports of Los Angeles and Long Beach graciously hosted one of our information gathering sessions and opened their doors to the committee for touring purposes, as did the National Marine Exchange in Los Angeles and the Joint Regional Intelligence Center (JRIC). The staff and committee are very grateful for the hospitality shown at these locations. The committee thanks the following agencies for providing briefings at that meeting:
• FBI
• Long Beach Police Department
• Los Angeles County Sheriff’s Department
• Los Angeles Fire Department
• Los Angeles Police Department
• U.S. Coast Guard
• U.S. Customs and Border Protection
• Los Angeles Port Police
The committee extends special thanks to the staff of the National Research Council for supporting this study. Staff members who have contributed to this effort are Jenny Heimberg (study director), Kevin Crowley (director of the Nuclear and Radiation Studies Board), Erin Wingo (senior program assistant), Shauntée Whetstone (senior program assistant), and Toni Greenleaf (financial and program associate).
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council. The purpose of this independent review is to provide candid and critical comments that will assist the National Research Council in making its published report as sound as possible and will ensure that this report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following individuals for their review of this report:
• Margaret Chu, M.S. Chu and Associates, LLC
• William Hagan, independent consultant
• Milton Levenson, independent consultant
• David Morton, University of Texas, Austin
• C. Paul Robinson, President Emeritus Sandia National Laboratories
• Tim Runyon, Illinois Emergency Management Agency (retired)
• Henry Willis, RAND Corporation
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Chris G. Whipple, ENVIRON, and John F. Ahearne, Sigma Xi. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were considered carefully. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
This page intentionally left blank.