Performance Metrics for the Global Nuclear Detection Architecture: Abbreviated Version (2013)

4

A Notional Strategic Planning Example with Metrics

The committee finds in the preceding chapter that it is fundamentally possible to measure the effectiveness of the Global Nuclear Detection Architecture (GNDA) as long as a new strategic plan and analysis framework are created. To demonstrate feasibility, a notional strategic planning example with outcome-based metrics is presented in this chapter and a new analysis framework is presented in Chapter 5. The committee found the development of the notional example to be difficult. This chapter will highlight the difficulties that the committee encountered (such as the development of deterrence metrics) and to provide its recommendations for the GNDA partner agencies if they decide to revise the GNDA strategic plan following the committee’s recommendations. These recommendations are based both on the committee’s expertise and the lessons learned from developing this notional example.

This chapter consists of three sections. In the first section, a notional example to address Task 1 is introduced. It shows how to structure a strategic plan to enable the development of outcome-based metrics. The second section presents a discussion on developing metrics against deterrence goals. This is intended to address the Domestic Nuclear Detection Office’s (DNDO’s) interest in deterrence and more broadly to highlight the challenges of developing metrics against preventive goals. The final section analyzes the notional example using criteria and tools introduced in Chapter 3. The chapter ends with recommendations for strategic planning and metrics development based on lessons learned from this exercise.

4.1 NOTIONAL EXAMPLE

There are significant challenges to developing metrics to demonstrate progress toward achieving the GNDA mission. OMB notes that the development of performance measures for “programs that relate to deterrence or prevention of specific behaviors” are particularly challenging (OMB, 2003, p. 10).

The following notional example of an outcome-based strategic plan with associated outcome-based metrics illustrates an approach (and an existence proof) for measuring GNDA effectiveness. This example shows how a strategic plan can be structured to allow for the development of outcome-based metrics.

A strategic plan lays out the vision, mission, and high-level goals and objectives of an organization or program. Additional details on how the objectives and goals can be implemented are frequently provided in an implementation plan. The vision statement describes the long-term goal(s) of the organization or program whereas the mission, coupled with the goals and objectives, describes how to achieve the vision through shorter-term goals. A vision statement need not be readily achievable or define a clear end state; for example, the Environmental Protection Agency’s vision is “a cleaner, greener, more sustainable environment.” Some say that a vision statement should grab the heart whereas the mission statement should speak to the mind.

Goals and their underlying objectives derive from the mission and are not limited to those that can be measured easily. Often, strategic planning is guided by available data rather than by determining a logical set of goals and objectives to address the mission.

In developing the notional example, the committee considered the broad challenge of protecting the nation from radiological and nuclear terrorist attacks and outlining a vision, a mission, and a set of goals and objectives that would directly support that broader challenge. By approaching the problem from this broad perspective, strategic planning is not affected by the limited, federally defined scope of “detection” (see Figure 2-1 and the committee’s Observation 2).

The committee’s notional example is illustrated schematically in Figure 4-1 and described in Box 4-1. It consists of the following elements:

•   Vision: “For U.S. citizens to live free from the fear of nuclear or radiological terrorism.”

•   Mission: “Protect the nation from terrorist attacks that use radiological or nuclear (RN) materials.”

•   Several examples of outcome-based goals, objectives and metrics

Table 4-1 provides a list of the notional metrics. These metrics are meant to be a sampling of those that would be developed against a full strategic plan. In practice, each objective would have at least one associated metric. Some of these notional metrics were selected to highlight specific challenges. For example, the metric “Effectiveness of deterrence by denial” which directly supports the objective to “Deter terrorists’ RN attacks by demonstrating high likelihood of failure,” is discussed in detail in the Section 4.2.

Cost-effectiveness can be calculated using a combination of the metrics listed above but it is not identified as a separate metric. Within the proposed notional hierarchy in Figure 4-1, cost-effectiveness would be calculated by aggregating and comparing several different metrics. For example, a cost-effectiveness study might assess objectives related to threat, vulnerabilities, and consequences using the associated metrics and would weigh those results against other metrics (e.g., increased cost and side effects). Multiattribute utility analysis could be used to assign values to the different metrics (for more details on evaluating cost-effectiveness, see Appendix D).

Some notional objectives could be evaluated using proxy metrics. For example, the metric “Improve the probability of detecting an attempt to bring RN materials into the United States” could have proxy measures such as “Percent of POEs [Ports of Entry] with RN portals” or “number of interdictions of attempts to transport RN materials.” To be useful for this purpose, however, proxies must have a direct relationship to the goals and objectives.

In the final section of this chapter, the committee evaluates the notional example metrics against the characteristics of “useful metrics” (see Box 3-1) and the “scope-versus-type” matrices. First, the committee considers the challenging case of developing metrics to measure deterrence.

4.2 METRICS TO MEASURE DETERRENCE

Although not the main focus of the committee’s tasking, DNDO asked the committee to consider approaches for quantifying deterrence. The study of deterrence as it relates to the GNDA requires significantly more resources and time than was available for this study. However, the committee outlines several challenges associated with developing metrics to measure the effects of deterrence on the effectiveness of the GNDA. This discussion is not meant to overemphasize deterrence as a mission component to the GNDA—which is to both deter and detect.

Deterrence is a reduction in the likelihood that terrorists will attempt an attack using RN materials that are outside of regulatory control. This can be achieved by denial, increasing the cost and difficulty of an attack, providing a reward for not attacking, or retribution in case of a successful

FIGURE 4-1 Notional example of a GNDA strategic plan outlined in Box 4-1. A vision, mission statement and a set of goals and objectives are presented. Text of the goals and objectives has been truncated to fit them into the diagram. Not all

of the objectives are within the scope of the GNDA. Cells that have been filled in are within the scope of the GNDA; those not filled in are outside the scope of the GNDA.

attack. Deterrence by denial is achieved by demonstrating to the adversary that attacks would likely be detected, for example by hardening borders and restricting access to targets (NRC, 2002, p. 10). Efforts to increase the global effectiveness of detecting and reporting on RN material out of regulatory control contribute to deterrence by denial (e.g., increasing the cost to the terrorists by limiting the pathways available to move material without risk of detection), but they do not contribute to deterrence in other ways such as the threat of retribution. DNDO and GNDA partners are considering how to weigh efforts toward deterrence against the resulting threat reduction, which requires quantification.

The quantification and measurement of the impact of deterrence on terrorists has been discussed by several authors (Drake et al., 2003; Morral and Jackson, 2009; Willis et al., 2010; Haphuriwat et al., 2011). To assess the effectiveness of a deterrent, the following is needed at a minimum:

•   Information about the objectives and values of the adversary—conceptually, the adversary’s utility function for types of attack,

•   Information about the adversary’s perceptions of the likelihood of success of attacks,

•   Information about the adversary’s aversion to risk,

•   Information about the adversary’s decision rules when selecting attacks, and

•   An accounting of the potential of shifting of risk from one target to another (displaced risk).

The challenges associated with gathering, quantifying, and measuring this information are discussed in the following paragraphs.

Measuring the impact of deterrence has been considered in other fields, for example, criminology (Anthony, 2004). Deterrence and the Death Pen-

TABLE 4-1 Set of Notional Metrics for the GNDA

NOTE: This set was developed to illustrate how outcome-based metrics might be developed against an outcome-oriented strategic plan. It is not a full set of metrics for the notional plan; in practice, metrics would be developed against the full set of objectives.

alty (NRC, 2012a) reviews the past 30 years of research on the impact of capital punishment on murder rates. The committee that authored this report notes the importance of understanding the perception of potential penalties by would-be murderers: “It is not possible to interpret empirical evidence of the relationship of homicide rates to sanctions without understanding how potential murderers perceive sanction regimes” (NRC, 2012a, p. 105). The challenge is that perception of risk—although a critical component of deterrence—is subjective and difficult to measure. One of the conclusions of the NRC report is that studies on perception need to take place. The report also notes the challenges of identifying the small subset of potential murderers within the broader population and, once identified, extracting truthful responses from them.

This concern is echoed in another study, Discouraging Terrorism: Some Implications of 9/11 (NRC, 2002). In introducing the problem of measuring terrorists’ perceptions of risk, the report notes that the impact of deterrence is difficult to measure when state actors are involved; it becomes significantly more difficult with terrorists. The challenges are identified as follows:

(a)   difficulties in getting unambiguous and credible threats across to terrorists,

(b)   the unwillingness of terrorists to communicate except indirectly and on their own terms,

(c)   exceptionally high levels of mutual distrust,

(d)   uncertainty about how to affect what terrorists value, and

(e)   uncertainty about the targets to which threats should be directed. (NRC, 2002, p. 1)

Measuring the impact of deterrence requires understanding and characterizing the decision-making process of would-be terrorists and what terrorists consider a successful attack (e.g., any RN device detonated at a target versus detonated upon interdiction). The report Understanding and Managing Risk in Security Systems for the DOE Nuclear Weapons Complex (NRC, 2011) concludes that there is not currently a comprehensive analytical basis to support assessment of the probability of adaptive adversaries’ attacks.

Researchers have used game theory to model deterrence (Haphuriwat et al., 2011). Appendix D provides a simple model as an example to highlight the potential effectiveness of randomization strategies on adversaries’ decisions. Lacking firm data on terrorists’ perceptions, simple models can provide insight into potential strategies for deterrence. As with any modeling effort, validation through real events should be sought. Risk perceptions of adversaries will likely be collected and analyzed by intelligence agencies. Therefore, information from the intelligence community would be needed to address deterrence by denial through detection and reporting capabilities (e.g., What detection capabilities are known by potential terrorists? Does the physical location of detectors influence the attack strategies or plans of potential terrorists?).

The other aspect of measuring deterrence by denial is accounting for displaced risk (Morral and Jackson, 2009). The successful deterrence of a nuclear or radiological attack against a specific target needs to be weighed against the possibility that terrorists will then make alternative attack plans that could cause greater harm. Displacement can include shifts to other targets or other attack vectors (e.g., from nuclear to biological).

Deterrence is an important component of the GNDA mission even though it is difficult to characterize and measure. The committee did not find a credible or peer-reviewed deterrence model in its review. An indirect “deterrence effectiveness metric” could potentially be developed based on the assumption that the costlier an attack plan is for a terrorist organization, the less likely it is to happen. Other metrics identified in the notional example above could be combined and linked directly to increased costs of planning an attack and, as such, provide an indirect measure of the effectiveness of deterrence.

4.3 ANALYSIS OF NOTIONAL EXAMPLE

The committee provides an analysis of the notional example’s strategic plan and metrics using the approach described in Chapter 3.

4.3.1 Customer and Application of Metrics

The application for the notional example was primarily performance based (for U.S. security). The customers would be GNDA partner agencies, Congress, and the White House. None of the notional metrics were developed purely for management; however, one such metric would be “the percent of existing nuclear detection capabilities currently integrated into a model.”

4.3.2 Clear Connection to Consequences and Options

As discussed in Section 3.1, the most important criteria for metrics are that they be outcome-based and broadly focused. Figure 4-2 is the “scope-versus-type” matrix (introduced in Chapter 3) and is used here to analyze the notional metrics from Table 4-1.

This set of metrics is grouped toward the upper-right corner of the matrix which is the desirable region in this exercise. However, it is noteworthy that not all of the notional metrics are in the far upper-right corner box. In developing metrics, they will not always meet both criteria. Similar analyses can be conducted on the goals and objectives to illustrate that they are predominantly outcome-based and broadly-scoped. Consideration of the desirable characteristics of a metric (transparent, quantitative, reproducible, and verifiable) would show that not all of the notional metrics meet all of the characteristics. The matrices and the list of characteristics are tools that

FIGURE 4-2 A scope-versus-type matrix for the notional metrics from Table 4-1. The notional metrics have been categorized by type (input, process, output, and outcome) and scope (architecture, layer, resources/assets/capabilities).

may be used to guide future metric development—not all metrics will meet all of the criteria but striving to meet them will produce metrics that are more likely to be useful for assessing GNDA effectiveness.

4.4 RECOMMENDATION

RECOMMENDATION 1.1: When DNDO and the GNDA partner agencies next update the GNDA Strategic Plan, the committee recommends that they take the following steps:

1.   Generate a vision statement.
Without a clear, interagency-supported idea of the long-term goal of the GNDA, it is difficult to measure progress toward achieving it.

2.   Simplify the plan.
Limit the strategic plan’s hierarchy to vision, mission, goals, and objectives; the goals and objectives should be outcome-based and they should clearly describe the desired results and how they are directly related to the mission and vision of the GNDA.

3.   Consider the broader nuclear counterterrorism problem before focusing on “detection.”
A strategic plan developed by solely focusing on deterrence, detection, and reporting mission may not fully consider the activities that take place at the mission interfaces. Therefore, a broader perspective is needed to initially determine strategic goals and objectives before they are limited to those within the scope of the GNDA.

4.   Determine the goals and objectives by focusing on the mission.
Do not limit the plan’s goals and objectives by focusing on what can be easily measured or by what data are by readily available. Some important objectives may not lend themselves to direct measurement but they should not be excluded from the plan for that reason.

5.   Use proxies when direct metrics are not available.
The metrics developed directly against outcome-based objectives will more readily be outcome-based and focused on measuring the full architecture. However, it is not always possible to develop metrics that meet these criteria. In those cases, proxies (i.e., indirect metrics which are frequently output-or process-based, such as the number of deployed detectors) can provide useful information as long as they can be directly linked to the objectives.

Furthermore, in the absence of a GNDA design document, the committee suggests that the strategic plan clearly describes the GNDA’s design goals and how it enhances the otherwise disparate detection activities of GNDA partner agencies.

The findings and recommendations related to improved strategic planning efforts are general. They do not require or exclude specific types of planning (e.g., a capabilities-based planning). Therefore, the advice provided should be applicable regardless of the specific planning approach that the GNDA partners may decide to adopt should they proceed with implementation of the recommendations within this report.

Finally, the committee recognizes that significant organizational challenges exist that may impact the implementation of Recommendation 1.1—many of which are beyond DNDO’s control. The committee did not investigate this topic further than the general observations made in Section 2.3.