PERFORMANCE METRICS FOR THE

GLOBAL NUCLEAR
DETECTION ARCHITECTURE

Abbreviated Version

Committee on Evaluating the Performance Measures and Metrics
Development for the Global Nuclear Detection Architecture

Nuclear and Radiation Studies Board
Division on Earth and Life Studies

NATIONAL RESEARCH COUNCIL
                     OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS

Washington, D.C.

www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Committee on Evaluating the Performance Measures and Metrics Development for the Global Nuclear Detection Architecture Nuclear and Radiation Studies Board Division on Earth and Life Studies

OCR for page R1
THE NATIONAL ACADEMIES PRESS  500 Fifth Street, NW  Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Govern- ing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineer- ing, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropri- ate balance. This study was supported by Contract No. HSHQDC-11-D-00009 between the National Academy of Sciences and the Department of Homeland Security. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organiza- tions or agencies that provided support for the project. International Standard Book Number-13:  978-0-309-29014-2 International Standard Book Number-10:  0-309-29014-7 Additional copies of this report are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360,Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu. Copyright 2013 by the National Academy of Sciences. All rights reserved. Printed in the United States of America

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Acad- emy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding en- gineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineer- ing programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is presi- dent of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Insti- tute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sci- ences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Coun- cil is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1

OCR for page R1
Committee on Evaluating the Performance Measures and Metrics Development for the Global Nuclear Detection Architecture ARDEN BEMENT (Chair), Purdue University (retired), West Lafayette, Indiana KELLEY COYNER, Northern Virginia Transportation Commission, Arlington, Virginia MARTHA CRENSHAW, Stanford University, Stanford, California JAMES S. DYER, University of Texas, Austin ROGER L. HAGENGRUBER, University of New Mexico, Albuquerque JOHN H. HOLMES, Port of Los Angeles, California EDWARD H. KAPLAN, Yale School of Management, New Haven, Connecticut JOHN MATTINGLY, North Carolina State University, Raleigh GREGORY S. PARNELL, University of Arkansas, Fayetteville DONALD PROSNITZ, Independent Consultant, Livermore, California THOMAS C. SCHELLING, University of Maryland, College Park DETLOF von WINTERFELDT, University of Southern California, Los Angeles Staff JENNIFER HEIMBERG, Study Director ERIN WINGO, Senior Program Assistant SHAUNTEÉ WHETSTONE, Senior Program Assistant (through July 8, 2013) v

OCR for page R1
Nuclear and Radiation Studies Board Jay c. Davis (Chair), Hertz Foundation, Livermore, California Barbara J. Mcneil (Vice Chair), Harvard Medical School, Boston, Massachusetts JOHN S. APPLEGATE, Indiana University School of Law, Bloomington DAVID J. BRENNER, Columbia University, New York, New York MARGARET S. Y. CHU, M.S. Chu & Associates, LLC, Albuquerque, New Mexico MICHAEL L. CORRADINI, University of Wisconsin, Madison PATRicia j. culligan, Columbia University, New York, New York ROBERT C. DYNES, University of California, San Diego HEDVIG HRICAK, Memorial Sloan-Kettering Cancer Center, New York, New York THOMAS H. ISAACS, Lawrence Livermore National Laboratory, Livermore, California CAROL M. JANTZEN, Savannah River National Laboratory, Aiken, South Carolina ANNIE B. KERSTING, Glenn T. Seaborg Institute, Lawrence Livermore National Laboratory, Livermore, California MARTHA S. LINET, National Institutes of Health, Bethesda, Maryland FRED A. METTLER, JR., New Mexico VA Health Care System, Albuquerque BORIS F. Myasoedov, Russian Academy of Sciences, Moscow LAWRENCE T. PAPAY, PQR, LLC, La Jolla, California DANIEL O. STRAM, University of Southern California, Los Angeles RICHARD J. VETTER, Mayo Clinic (retired), Rochester, Minnesota Staff KEVIN D. CROWLEY, Director JENNIFER HEIMBERG, Senior Program Officer OURANIA KOSTI, Senior Program Officer TONI GREENLEAF, Administrative and Financial Associate LAURA D. LLANOS, Administrative and Financial Associate DARLENE GROS, Senior Program Assistant ERIN WINGO, Senior Program Assistant DANIEL POMEROY, Postdoctoral Fellow SHAUNTEÉ WHETSTONE, Senior Program Assistant (through July 8, 2013) vi

OCR for page R1
Preface The Domestic Nuclear Detection Office (DNDO) within the Depart- ment of Homeland Security (DHS) is responsible for developing and coor- dinating a cross-agency strategy, the Global Nuclear Detection Architecture (GNDA), to detect, analyze, and report on nuclear and radiological materi- als that are out of regulatory control. The GNDA is a global activity that involves programs, people, and technical systems in the United States and many other countries. It was mandated by presidential directive (in 2005) and public law (in 2006). DNDO and its federal partners issued the Global Nuclear Detection Architecture Strategic Plan in 2010 which describes the high-level goals and federal agency responsibilities for implementing the GNDA. The U.S. government intends to undertake an annual review of the GNDA strategic plan to assess its effectiveness and identify new requirements arising from changes in technology and/or the threat environment. DNDO has asked for advice from the National Research Council on developing quantitative approaches for assessing the effectiveness of the GNDA. This advice will be used to improve the GNDA strategic plan dur- ing future annual review cycles. The committee approached this study by first gaining an understand- ing of what is meant by the “global nuclear detection architecture.” We reviewed its documentation (strategic plan, annual reviews, and the domes- tic implementation plan). We interviewed staff from DNDO and its many federal partners. We visited the Ports of Los Angeles (LA) and Long Beach and the LA Joint Regional Intelligence Center. The committee also invited other government agencies to tell us about their measures of effectiveness vii

OCR for page R1
viii PREFACE and performance metrics (see Appendix A for full listing of the briefings received by the committee). The GNDA is a worldwide network of detection and reporting capa- bilities controlled by many different entities and funding lines. It is meant to protect against a wide range of adaptive and committed adversaries. Developing metrics to measure the effectiveness of such a complex system of systems is a difficult problem. DNDO and its partner agencies have developed documentation and an initial accounting of the many existing federal programs and activities that support nuclear detection and reporting objectives. The committee has developed metrics and an analysis framework that may help guide DNDO and its GNDA partners from this initial accounting to developing a capabil- ity to measure the effectiveness of the overall GNDA. However, it became clear during the course of the study that the lack of a lead architect and a centralized GNDA budget (see Observation 1) make it difficult for the GNDA to function as a system rather than a collection of programs. The decision to address this concern (e.g., to assign clear leadership through organizational change) rests with the U.S. government. I would like to extend special thanks to Captain John Holmes for or- ganizing and the Ports of LA and Long Beach for hosting the committee during one of our information-gathering sessions. Our visit to the ports and the surrounding facilities and discussions with numerous stakeholders allowed the committee to see a unique example of federal, state, and local agencies truly working together toward a common goal of protecting the nation against threats. Arden Bement, Chair

OCR for page R1
Acknowledgments The committee wishes to acknowledge and thank a number of individu- als and organizations for their valuable contributions to this study: The Department of Homeland Security (DHS) Domestic Nuclear De- tection Office (DNDO) for its sponsorship of this study, and especially DNDO staff members Brendan Plapp, John Zabko, Kimberly Koeppel, Greg Haugan, and Richard Passow. Those who gave presentations at the committee’s information-gathering meetings provided insight and information that made this report possible (presentations are listed in Appendix A): • Major General Julie A. Bentz, National Security Council • Brendan Plapp, DNDO Architecture and Plans Directorate (cur- rently with the National Defense University) • John Zabko, DNDO Architecture and Plans Directorate • Kevin Hart, DNDO Architecture and Plans Directorate • Teri N. Leffer, Department of Energy, National Nuclear Security Administration, Second Line of Defense (NA-256) • Steven Streetman, Data Architecture Solutions, Inc. • Henry Willis, RAND Corporation • David Kulp, Department of Defense • Ernest Muenchau, DNDO Operation Support Directorate (OSD) (deceased) • Colonel Robert Kolterman, Defense Threat Reduction Agency (DTRA) ix

OCR for page R1
x ACKNOWLEDGMENTS • Brian Savage, DNDO OSD, Joint Analysis Center • James Smith, Los Alamos National Laboratory • Mark Oliphant, DNDO Red Team and Net Assessments Directorate • J. J. Fisher, DNDO OSD • Bernie Bogdan, Federal Bureau of Investigation (FBI) • J. C. Wyss, State Department • David Travers, U.S. Environmental Protection Agency • Detective Jeff Shanaphy, Los Angeles Port Police • Sgt. Peter Jackson, Los Angeles Sheriff’s Department The Ports of Los Angeles and Long Beach graciously hosted one of our information gathering sessions and opened their doors to the committee for touring purposes, as did the National Marine Exchange in Los Angeles and the Joint Regional Intelligence Center (JRIC). The staff and committee are very grateful for the hospitality shown at these locations. The committee thanks the following agencies for providing briefings at that meeting: • FBI • Long Beach Police Department • Los Angeles County Sheriff’s Department • Los Angeles Fire Department • Los Angeles Police Department • U.S. Coast Guard • U.S. Customs and Border Protection • Los Angeles Port Police The committee extends special thanks to the staff of the National Research Council for supporting this study. Staff members who have con- tributed to this effort are Jenny Heimberg (study director), Kevin Crowley (director of the Nuclear and Radiation Studies Board), Erin Wingo (senior program assistant), Shauntée Whetstone (senior program assistant), and Toni Greenleaf (financial and program associate). This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with proce- dures approved by the Report Review Committee of the National Research Council. The purpose of this independent review is to provide candid and critical comments that will assist the National Research Council in making its published report as sound as possible and will ensure that this report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following individuals for their review of this report:

OCR for page R1
ACKNOWLEDGMENTS xi • Margaret Chu, M.S. Chu and Associates, LLC • William Hagan, independent consultant • Milton Levenson, independent consultant • David Morton, University of Texas, Austin • C. Paul Robinson, President Emeritus Sandia National Laboratories • Tim Runyon, Illinois Emergency Management Agency (retired) • Henry Willis, RAND Corporation Although the reviewers listed above provided many constructive com- ments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Chris G. Whipple, ENVIRON, and John F. Ahearne, Sigma Xi. Appointed by the National Research Council, they were responsible for making certain that an in- dependent examination of this report was carried out in accordance with institutional procedures and that all review comments were considered carefully. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

OCR for page R1

OCR for page R1
Contents Summary 1 1 Introduction 9 2 GNDA Background 15 3 Strategic Planning and Metrics 27 4 A Notional Strategic Planning Example with Metrics 37 5 GNDA Analysis Framework 49 References 59 APPENDIXES A Presentations and Committee Information Gathering Meetings 63 B Statement of Task 65 C Biographical Sketches of the Committee 67 D Model-Based Approaches for the GNDA 75 E Acronyms 89 F Glossary 91 xiii

OCR for page R1