The Global Nuclear Detection Architecture (GNDA) is described as “a worldwide network of sensors, telecommunications, and personnel, with the supporting information exchanges, programs, and protocols that serve to detect, analyze, and report on nuclear and radiological materials that are out of regulatory control.”1 The Domestic Nuclear Detection Office (DNDO), an office within the Department of Homeland Security (DHS), coordinates the development of the GNDA with its federal partners. DNDO has asked the National Research Council (NRC) for advice on how to develop performance measures and quantitative metrics that can be used to evaluate the overall effectiveness and report on progress toward meeting the goals of the GNDA. The statement of task for this study can be found in Box S-1 (also Appendix B). The GNDA is a complex system of systems meant to deter and detect attempts to unlawfully transport radiological or nuclear (RN) material.2 It was established to enhance the U.S. government RN detection activities in response to the perceived increase of the risk of nuclear terrorism following the 9/11 attacks.3 Multiple federal, international, state, local, tribal, and industrial entities participate in activities that
_____________
1 See http://www.dhs.gov/architecture-directorate. Accessed August 1, 2013.“Out of regulatory control” describes materials that are being imported, possessed, stored, transported, developed, or used without authorization by the appropriate regulatory authority, either inadvertently or deliberately (DHS, 2011b, Vol. I, p. 4).
2 Radiological material is used in a radiological dispersion device (RDD); nuclear material is used in an improvised nuclear device (IND) or nuclear weapon. These are considered two distinct threats (e.g., http://www.dhs.gov/radiological-attack-what-it, http://www.dhs.gov/nuclear-attack-what-it). Accessed August 1, 2013.
3 Security and Accountability for Every Port Act of 2006 (P.L. 109-347).
BOX S-1
Statement of Task
An ad hoc committee will conduct a study and prepare a report to the Domestic Nuclear Detection Office (DNDO) on quantitative approaches for evaluating the effectiveness of the Global Nuclear Detection Architecture (GNDA), specifically in the context of the following two tasks:
Task 1: Assess the feasibility of using performance measures and quantitative metrics for GNDA.
The committee should assess the feasibility of using performance measures and quantitative metrics for evaluating progress in meeting these performance goals. This assessment should consider the following factors:
• Definition of performance measures for each of the performance goals in the Strategic Plan.
• Definition of quantifiable performance metrics for each performance measure including, as appropriate, efficiency, output, and outcome-oriented performance measures.
• Identification of data to be used to quantify these performance metrics.
• Identification of methodologies to be used to collect and analyze these data.
• Specification of performance target values for assessing the effectiveness of each performance measure.
If the use of performance measures and quantitative metrics is determined to be feasible, the committee should, to the extent practical, recommend specific performance measures, metrics, and the other supporting information described in the list above for consideration by the DNDO.
If the use of performance measures and metrics is determined to be infeasible, the committee should recommend alternative evaluation approaches.
Task 2: Recommend approaches for evaluating the overall effectiveness of the GNDA.
The committee should specifically recommend
• Approaches for developing an overall analysis framework to assess the effectiveness of the GNDA in terms of its ability to detect, deny, confuse, and/or deter adversaries.
• Approaches for exercising this analysis framework using combinations of modeling/simulation, red teaming, and/or related methods to assess the cost-effectiveness of and tradeoffs in GNDA components.
In executing these tasks the committee should examine efforts by other organizations to develop risk-informed metrics and analysis approaches for complex technological systems.
can contribute to increasing the effectiveness of the global nuclear detection infrastructure.
The challenge presented to the GNDA federal partners, who are responsible through the Government Performance and Results Act (GPRA, P.L. 103-62) to report on the performance of the GNDA, is to develop meaningful metrics4 and gather the appropriate data to gauge its overall progress every year. This challenge has also been considered carefully by this committee (see Task 1 in Box S-1). The committee is directed to assess the feasibility of developing measures and metrics against existing performance goals of the strategic plan that can be used to measure the effectiveness of the GNDA. If infeasible, the committee is to recommend alternative approaches to evaluating GNDA effectiveness.
There are significant challenges to developing metrics to gauge the overall effectiveness of the GNDA. The GNDA was created to address the threat of a high-consequence event that has never occurred. It must protect against a wide variety of adaptive and committed adversaries and threat materials. However, these types of challenges exist within other U.S. government agencies and are not unprecedented. The committee provides several examples and concludes that it is fundamentally possible to develop outcome-based metrics to gauge the effectiveness of the GNDA. However, the committee finds that it is not feasible to develop outcome-based metrics against the existing performance goals within the existing GNDA strategic plan. There are two reasons for this: the higher-level goals and objectives within the strategic plan are focused on process and activities (they are not primarily outcome-based) and many objectives are focused on individual GNDA layers or resources (not the full architecture); and the higher-level goals are disconnected from the objectives and lower-level performance goals. Furthermore, a new analysis framework is needed to evaluate the metrics and to prioritize the GNDA’s goals and objectives. This report presents a notional strategic plan (with notional outcome-based metrics) and new analysis framework.
In addressing the statement of task, the committee identified several issues of concern that could limit the GNDA federal partners from implementing the findings and recommendations provided within this report. These concerns have been identified as observations.
A summary of observations, findings, and recommendations is provided below.
_____________
4 Here and within the report, the committee uses the simplified “metric” to refer to “performance measures and quantitative metrics.” While acknowledging that “measures and metrics” exist separately within the scholarly literature and measurement theory, for the committee’s purposes in addressing the task statement, the simplified term “metric” is used.
OBSERVATIONS, FINDINGS, AND RECOMMENDATIONS
The committee recommends an approach for developing an updated strategic plan, outcome-based metrics, and an analysis framework for evaluating the effectiveness of the overall GNDA. The GNDA, however, currently does not function as an integrated system but as a collection of programs. If it is decided that the GNDA should function as a system, then it will need to have clearly-defined lead authority and a centralized budget so that reallocations can be made across programs and agencies. Nonetheless, the committee’s recommendations for improved metrics and an analysis framework within the existing organizational structure could provide information to reallocate funds within agencies and to measure the overall cost of the GNDA.
OBSERVATION 1: There is no clear lead architect or single entity to make final decisions about or to be held accountable for the design and operation of the GNDA. Furthermore, there is no centrally controlled GNDA budget; GNDA-related detection and reporting activities are intertwined with diverse mission activities across GNDA federal agencies and do not have specific lines of funding. Thus, there is no single congressional appropriation for the GNDA nor is there a single entity with budgetary control over GNDA activities across multiple agencies.
The GNDA operates via a loosely confederated collection of federal, state/local and tribal programs and activities under what may be considered a “best-effort” budget. This is important to note, because it may not be possible to effectively utilize the results from an analysis framework and measures of effectiveness of the overall GNDA in a way that would change the contributions of participating agencies to the overall budget. This does not imply that developing improved metrics to guide resource decisions and establishing an analysis framework for the GNDA is without purpose. Establishing a capability to evaluate GNDA effectiveness can provide useful information to decision makers such as the gap between existing and optimal resource allocation and a measure of the cost of operating the GNDA. The issue of disconnected budgets’ impact on coordination of the GNDA has been highlighted previously.5 The committee provides an example of another federal program with similar challenges in Chapter 2.
OBSERVATION 2: The GNDA operates within a larger nuclear counterterrorism (NCT) mission. Its scope is limited to deterrence, detection, and
_____________
5 This issue has been identified through Senate hearings (U.S. Congress, Senate, 2010 Hearing 111-1096) but no actions have been taken. (http://www.gpo.gov/fdsys/pkg/CHRG-111shrg58397/html/CHRG-111shrg58397.htm. Accessed on August 1, 2013).
reporting. When considering how to address and define the GNDA strategy and goals, focusing solely on the detection and reporting mission may limit wider U.S. government actions that span multiple components of the NCT mission space.
It is difficult to segregate actions and strategies focused on deterrence, detection, and reporting from other actions that support adjacent missions of federal agencies. The committee provides several examples of the impact of NCT federal mission boundaries on strategic planning and response options.
In Chapters 3 and 4, the committee provides one finding, one recommendation and notional examples of a strategic plan and outcome-based metrics in response to Task 1.
FINDING 1.1: It is fundamentally possible to create outcome-based metrics for the GNDA; however, it is not currently feasible to develop outcome-based metrics against the existing strategic plan’s goals, objectives, and performance goals because these components are primarily output- and process-based and are not linked directly to the GNDA’s mission.
Two conditions must be met to use metrics to evaluate the effectiveness of the GNDA:
1. A new strategic plan with outcome-based goals and objectives must be created and
2. An analysis framework must be developed to enable assessment of outcome-based metrics.
RECOMMENDATION 1.1:
When DNDO and the GNDA partner agencies next update the GNDA Strategic Plan, the committee recommends that they take the following steps:
1. Generate a vision statement.
Without a clear, interagency-supported idea of the long-term goal of the GNDA, it is difficult to measure progress toward achieving it.
2. Simplify the plan.
Limit the strategic plan’s hierarchy to vision, mission, goals, and objectives; the goals and objectives should be outcome-based and they should clearly describe the desired results and how they are directly related to the mission and vision of the GNDA.
3. Consider the broader nuclear counterterrorism problem before focusing on “detection.”
A strategic plan developed by solely focusing on deterrence, detection, and reporting mission may not fully consider the activities that take place at the mission interfaces. Therefore, a broader perspective is needed to initially determine strategic goals and objectives before they are limited to those within the scope of the GNDA.
4. Determine the goals and objectives by focusing on the mission.
Do not limit the plan’s goals and objectives by focusing on what can be easily measured or by what data are readily available. Some important objectives may not lend themselves to direct measurement but they should not be excluded from the plan for that reason.
5. Use proxies when direct metrics are not available.
The metrics developed directly against outcome-based objectives will more readily be outcome-based and focused on measuring the full architecture. However, it is not always possible to develop metrics that meet these criteria. In those cases, proxies (i.e., indirect metrics that are frequently output-or process-based, such as the number of deployed detectors) can provide useful information as long as they can be directly linked to the objectives.
Furthermore, in the absence of a GNDA design document the committee suggests that the strategic plan clearly describes the GNDA’s design goals and how its existence enhances the otherwise disparate detection activities of GNDA partner agencies.
In Chapter 5, the committee provides two findings, one recommendation and an example of a new analysis framework in response to Task 2.
FINDING 2.1:
A new GNDA Analysis Framework is needed to assess the GNDA effectiveness as shown in Figure 5-1. The critical components of the framework are the following:
1. A GNDA Strategic Plan that contains outcome-oriented, broadlyscoped goals, objectives and metrics and is directly connected to the components listed below;
2. A GNDA Architectural Definition that provides the conceptual and physical descriptions of the GNDA, and that define needed input data for the models described below;
3. A suite of GNDA models that incorporate potential adversary objectives, accurately represent existing and potential architecture capabilities, and calculate the metrics described below;
4. Metrics that can gauge overall GNDA effectiveness and assess potential GNDA resource decisions to increase GNDA effectiveness; and
5. A Validation and Verification (V&V) program that evaluates the data used in the GNDA architecture definition, models, and metrics. A robust V&V program enhances the credibility of the analysis framework.
FINDING 2.2:
Current DNDO modeling, testing, red teaming, analysis, and training capabilities provide a foundation for evaluating components of the GNDA, but these current capabilities are insufficient for validating and verifying the overall effectiveness of the GNDA. Evaluating the effectiveness of the overall GNDA requires an integrated and continuous model-based scenario testing, red teaming, analysis, peer review, and training supplemented with intelligence awareness.
RECOMENDATION 2.1:
DNDO should develop a new GNDA Analysis Framework similar to the framework proposed by the committee. This framework defines an analytic process that clarifies the connections among strategic planning, architectural definition, models, metrics, and validation and verification efforts. Such an analysis framework can provide credible assessments of overall GNDA effectiveness.
