Summary

The Global Nuclear Detection Architecture (GNDA) is described as “a worldwide network of sensors, telecommunications, and personnel, with the supporting information exchanges, programs, and protocols that serve to detect, analyze, and report on nuclear and radiological materials that are out of regulatory control.”1 The Domestic Nuclear Detection Office (DNDO), an office within the Department of Homeland Security (DHS), coordinates the development of the GNDA with its federal partners. DNDO has asked the National Research Council (NRC) for advice on how to develop performance measures and quantitative metrics that can be used to evaluate the overall effectiveness and report on progress toward meeting the goals of the GNDA. The statement of task for this study can be found in Box S-1 (also Appendix B). The GNDA is a complex system of systems meant to deter and detect attempts to unlawfully transport radiological or nuclear (RN) material.2 It was established to enhance the U.S. government RN detection activities in response to the perceived increase of the risk of nuclear terrorism following the 9/11 attacks.3 Multiple federal, international, state, local, tribal, and industrial entities participate in activities that

_____________

1 See http://www.dhs.gov/architecture-directorate. Accessed August 1, 2013.“Out of regulatory control” describes materials that are being imported, possessed, stored, transported, developed, or used without authorization by the appropriate regulatory authority, either inadvertently or deliberately (DHS, 2011b, Vol. I, p. 4).

2 Radiological material is used in a radiological dispersion device (RDD); nuclear material is used in an improvised nuclear device (IND) or nuclear weapon. These are considered two distinct threats (e.g., http://www.dhs.gov/radiological-attack-what-it, http://www.dhs.gov/nuclear-attack-what-it). Accessed August 1, 2013.

3 Security and Accountability for Every Port Act of 2006 (P.L. 109-347).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary The Global Nuclear Detection Architecture (GNDA) is described as “a worldwide network of sensors, telecommunications, and personnel, with the supporting information exchanges, programs, and protocols that serve to detect, analyze, and report on nuclear and radiological materi- als that are out of regulatory control.”1 The Domestic Nuclear Detection Office (DNDO), an office within the Department of Homeland Security (DHS), coordinates the development of the GNDA with its federal partners. DNDO has asked the National Research Council (NRC) for advice on how to develop performance measures and quantitative metrics that can be used to evaluate the overall effectiveness and report on progress toward meeting the goals of the GNDA. The statement of task for this study can be found in Box S-1 (also Appendix B). The GNDA is a complex system of systems meant to deter and detect attempts to unlawfully transport radiological or nuclear (RN) material.2 It was established to enhance the U.S. government RN detection activities in response to the perceived increase of the risk of nuclear terrorism following the 9/11 attacks.3 Multiple federal, interna- tional, state, local, tribal, and industrial entities participate in activities that 1  See http://www.dhs.gov/architecture-directorate. Accessed August 1, 2013.“Out of regu- latory control” describes materials that are being imported, possessed, stored, transported, developed, or used without authorization by the appropriate regulatory authority, either inadvertently or deliberately (DHS, 2011b, Vol. I, p. 4). 2  Radiological material is used in a radiological dispersion device (RDD); nuclear mate- rial is used in an improvised nuclear device (IND) or nuclear weapon. These are considered two distinct threats (e.g., http://www.dhs.gov/radiological-attack-what-it, http://www.dhs.gov/ nuclear-attack-what-it). Accessed August 1, 2013. 3  Security and Accountability for Every Port Act of 2006 (P.L. 109-347). 1

OCR for page 1
2 GLOBAL NUCLEAR DETECTION BOX S-1 Statement of Task An ad hoc committee will conduct a study and prepare a report to the Do- mestic Nuclear Detection Office (DNDO) on quantitative approaches for evaluating the effectiveness of the Global Nuclear Detection Architecture (GNDA), specifi- cally in the context of the following two tasks: Task 1: Assess the feasibility of using performance measures and quantita- tive metrics for GNDA. The committee should assess the feasibility of using performance measures and quantitative metrics for evaluating progress in meeting these performance goals. This assessment should consider the following factors: •  efinition of performance measures for each of the performance goals in D the Strategic Plan. •  efinition of quantifiable performance metrics for each performance mea- D sure including, as appropriate, efficiency, output, and outcome-oriented performance measures. • Identification of data to be used to quantify these performance metrics. • dentification of methodologies to be used to collect and analyze these I data. •  pecification of performance target values for assessing the effectiveness S of each performance measure. If the use of performance measures and quantitative metrics is determined to be feasible, the committee should, to the extent practical, recommend specific performance measures, metrics, and the other supporting information described in the list above for consideration by the DNDO. If the use of performance measures and metrics is determined to be infea- sible, the committee should recommend alternative evaluation approaches. Task 2: Recommend approaches for evaluating the overall effectiveness of the GNDA. The committee should specifically recommend •  pproaches for developing an overall analysis framework to assess the A effectiveness of the GNDA in terms of its ability to detect, deny, confuse, and/or deter adversaries. •  pproaches for exercising this analysis framework using combinations of A modeling/simulation, red teaming, and/or related methods to assess the cost-effectiveness of and tradeoffs in GNDA components. In executing these tasks the committee should examine efforts by other or- ganizations to develop risk-informed metrics and analysis approaches for complex technological systems.

OCR for page 1
SUMMARY 3 can contribute to increasing the effectiveness of the global nuclear detection infrastructure. The challenge presented to the GNDA federal partners, who are re- sponsible through the Government Performance and Results Act (GPRA, P.L. 103-62) to report on the performance of the GNDA, is to develop meaningful metrics4 and gather the appropriate data to gauge its overall progress every year. This challenge has also been considered carefully by this committee (see Task 1 in Box S-1). The committee is directed to assess the feasibility of developing measures and metrics against existing perfor- mance goals of the strategic plan that can be used to measure the effective- ness of the GNDA. If infeasible, the committee is to recommend alternative approaches to evaluating GNDA effectiveness. There are significant challenges to developing metrics to gauge the overall effectiveness of the GNDA. The GNDA was created to address the threat of a high-consequence event that has never occurred. It must protect against a wide variety of adaptive and committed adversaries and threat materials. However, these types of challenges exist within other U.S. government agencies and are not unprecedented. The committee provides several examples and concludes that it is fundamentally possible to develop outcome-based metrics to gauge the effectiveness of the GNDA. However, the committee finds that it is not feasible to develop outcome-based metrics against the existing performance goals within the existing GNDA strategic plan. There are two reasons for this: the higher-level goals and objectives within the strategic plan are focused on process and activities (they are not primarily outcome-based) and many objectives are focused on individual GNDA layers or resources (not the full architecture); and the higher-level goals are disconnected from the objectives and lower-level performance goals. Furthermore, a new analysis framework is needed to evaluate the metrics and to prioritize the GNDA’s goals and objectives. This report presents a notional strategic plan (with notional outcome-based metrics) and new analysis framework. In addressing the statement of task, the committee identified several issues of concern that could limit the GNDA federal partners from imple- menting the findings and recommendations provided within this report. These concerns have been identified as observations. A summary of observations, findings, and recommendations is provided below. 4  Here and within the report, the committee uses the simplified “metric” to refer to “perfor- mance measures and quantitative metrics.” While acknowledging that “measures and metrics” exist separately within the scholarly literature and measurement theory, for the committee’s purposes in addressing the task statement, the simplified term “metric” is used.

OCR for page 1
4 GLOBAL NUCLEAR DETECTION OBSERVATIONS, FINDINGS, AND RECOMMENDATIONS The committee recommends an approach for developing an updated strategic plan, outcome-based metrics, and an analysis framework for evaluating the effectiveness of the overall GNDA. The GNDA, however, currently does not function as an integrated system but as a collection of programs. If it is decided that the GNDA should function as a system, then it will need to have clearly-defined lead authority and a centralized budget so that reallocations can be made across programs and agencies. Nonetheless, the committee’s recommendations for improved metrics and an analysis framework within the existing organizational structure could provide information to reallocate funds within agencies and to measure the overall cost of the GNDA. Observation 1: There is no clear lead architect or single entity to make final decisions about or to be held accountable for the design and operation of the GNDA. Furthermore, there is no centrally controlled GNDA bud- get; GNDA-related detection and reporting activities are intertwined with diverse mission activities across GNDA federal agencies and do not have specific lines of funding. Thus, there is no single congressional appropria- tion for the GNDA nor is there a single entity with budgetary control over GNDA activities across multiple agencies. The GNDA operates via a loosely confederated collection of federal, state/local and tribal programs and activities under what may be considered a “best-effort” budget. This is important to note, because it may not be possible to effectively utilize the results from an analysis framework and measures of effectiveness of the overall GNDA in a way that would change the contributions of participating agencies to the overall budget. This does not imply that developing improved metrics to guide resource decisions and establishing an analysis framework for the GNDA is without purpose. Establishing a capability to evaluate GNDA effectiveness can provide useful information to decision makers such as the gap between existing and opti- mal resource allocation and a measure of the cost of operating the GNDA. The issue of disconnected budgets’ impact on coordination of the GNDA has been highlighted previously.5 The committee provides an example of another federal program with similar challenges in Chapter 2. Observation 2: The GNDA operates within a larger nuclear counter- terrorism (NCT) mission. Its scope is limited to deterrence, detection, and 5 This issue has been identified through Senate hearings (U.S. Congress, Senate, 2010 Hearing 111-1096) but no actions have been taken. (http://www.gpo.gov/fdsys/pkg/CHRG- 111shrg58397/html/CHRG-111shrg58397.htm. Accessed on August 1, 2013).

OCR for page 1
SUMMARY 5 reporting. When considering how to address and define the GNDA strategy and goals, focusing solely on the detection and reporting mission may limit wider U.S. government actions that span multiple components of the NCT mission space. It is difficult to segregate actions and strategies focused on deterrence, detection, and reporting from other actions that support adjacent missions of federal agencies. The committee provides several examples of the impact of NCT federal mission boundaries on strategic planning and response options. In Chapters 3 and 4, the committee provides one finding, one recom- mendation and notional examples of a strategic plan and outcome-based metrics in response to Task 1. FINDING 1.1: It is fundamentally possible to create outcome-based metrics for the GNDA; however, it is not currently feasible to develop outcome- based metrics against the existing strategic plan’s goals, objectives, and performance goals because these components are primarily output- and process-based and are not linked directly to the GNDA’s mission. Two conditions must be met to use metrics to evaluate the effectiveness of the GNDA: 1.  new strategic plan with outcome-based goals and objectives must be A created and 2.  analysis framework must be developed to enable assessment of An outcome-based metrics. RECOMMENDATION 1.1: When DNDO and the GNDA partner agencies next update the GNDA Strategic Plan, the committee recommends that they take the following steps: 1. Generate a vision statement.  Without a clear, interagency-supported idea of the long-term goal of the GNDA, it is difficult to measure progress toward achieving it. 2. Simplify the plan.  Limit the strategic plan’s hierarchy to vision, mission, goals, and ob- jectives; the goals and objectives should be outcome-based and they should clearly describe the desired results and how they are directly related to the mission and vision of the GNDA. 3. Consider the broader nuclear counterterrorism problem before focusing on “detection.”

OCR for page 1
6 GLOBAL NUCLEAR DETECTION  strategic plan developed by solely focusing on deterrence, detection, A and reporting mission may not fully consider the activities that take place at the mission interfaces. Therefore, a broader perspective is needed to initially determine strategic goals and objectives before they are limited to those within the scope of the GNDA. 4. Determine the goals and objectives by focusing on the mission.  not limit the plan’s goals and objectives by focusing on what can be Do easily measured or by what data are readily available. Some important objectives may not lend themselves to direct measurement but they should not be excluded from the plan for that reason. 5. Use proxies when direct metrics are not available.  The metrics developed directly against outcome-based objectives will more readily be outcome-based and focused on measuring the full ar- chitecture. However, it is not always possible to develop metrics that meet these criteria. In those cases, proxies (i.e., indirect metrics that are frequently output- or process-based, such as the number of deployed detectors) can provide useful information as long as they can be directly linked to the objectives. Furthermore, in the absence of a GNDA design document the com- mittee suggests that the strategic plan clearly describes the GNDA’s design goals and how its existence enhances the otherwise disparate detection activities of GNDA partner agencies. In Chapter 5, the committee provides two findings, one recommenda- tion and an example of a new analysis framework in response to Task 2. FINDING 2.1: A new GNDA Analysis Framework is needed to assess the GNDA effective- ness as shown in Figure 5-1. The critical components of the framework are the following: 1.  GNDA Strategic Plan that contains outcome-oriented, broadly- A scoped goals, objectives and metrics and is directly connected to the components listed below; 2.  GNDA Architectural Definition that provides the conceptual and A physical descriptions of the GNDA, and that define needed input data for the models described below; 3.  suite of GNDA models that incorporate potential adversary objec- A tives, accurately represent existing and potential architecture capabili- ties, and calculate the metrics described below; 4. Metrics that can gauge overall GNDA effectiveness and assess potential GNDA resource decisions to increase GNDA effectiveness; and

OCR for page 1
SUMMARY 7 5.  Validation and Verification (V&V) program that evaluates the data A used in the GNDA architecture definition, models, and metrics. A ro- bust V&V program enhances the credibility of the analysis framework. FINDING 2.2: Current DNDO modeling, testing, red teaming, analysis, and training ca- pabilities provide a foundation for evaluating components of the GNDA, but these current capabilities are insufficient for validating and verifying the overall effectiveness of the GNDA. Evaluating the effectiveness of the overall GNDA requires an integrated and continuous model-based scenario testing, red teaming, analysis, peer review, and training supplemented with intelligence awareness. Recommendation 2.1: DNDO should develop a new GNDA Analysis Framework similar to the framework proposed by the committee. This framework defines an analytic process that clarifies the connections among strategic planning, architec- tural definition, models, metrics, and validation and verification efforts. Such an analysis framework can provide credible assessments of overall GNDA effectiveness.

OCR for page 1