Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 11
1 Introduction In the National Defense Authorization Act for Fiscal Year 2010 (P.L. 111-84), Congress directed the Secretary of Defense (DoD) to develop and implement metrics to “measure the impact and effectiveness of activities of the Cooperative Threat Reduction Program,” (CTR) and directed DoD to request the National Academy of Sciences (NAS) to conduct “an assessment to review the metrics developed and implemented under subsection (a) and identify possible additional or alternative metrics, if necessary.” (See Appendix A.) The Secretary completed a report describing DoD’s metrics for the CTR Program (DoD, 2010; here called the DoD Metrics Report) in September 2010.9 As implied in the language cited above, a metric is an evidence-based tool that measures impact and effectiveness or performance of a program or a project toward its objectives. Metrics alone cannot ensure that the best options have been identified or are being implemented, but they can be helpful to program managers and overseers as they work to ensure that efforts are focused in productive directions. Impact and effectiveness in some projects is easily measured quantitatively, particularly where the objectives yield tangible products, such as constructing secure storage facilities, consolidating materials into a secure facility, or destroying a stock of a chemical agent. The impact and effectiveness of projects that have less tangible objectives (e.g., strengthening both a nation’s ability to detect and respond to disease and the relationships between U.S. experts and those in the partner nation) can be more difficult to measure. Indeed, what to measure is an important but difficult question for such projects, and this was a central motivation for Congress’s mandate for DoD to develop and NAS to review metrics for the CTR Program, which are described in the next section. The committee’s goal in this report is to assess the metrics DoD developed, and to provide recommendations on how to improve them or develop more effective alternative metrics. No metric will be perfect. Metrics are tools for evaluation, not substitutes for evaluation. Even in the missions with easy-to-understand, tangible actions, the obvious metrics (e.g., facilities completed, stocks of nuclear material consolidated) may not by themselves reflect the effectiveness that the United States cares about most (e.g., are the stocks secure?). With this limitation in mind, the committee provides background on the CTR Program, other studies of CTR, and the scope of this report before assessing the DoD Metrics Report. The committee wrote this report with two main audiences in mind: Those who are mostly concerned with the overall assessment and advice, and those readers directly involved in the CTR Program, who need the details of the DoD report assessment and of how to implement the approach that the committee recommends. For the first category of reader, the committee has included a summary of the DoD CTR metrics in Table 2-1. For readers who desire more detail on the DoD CTR Metrics Report, the entire report can be found in Appendix B. 9 See Appendix B for the DoD Metrics Report; Figure 1 of the Report shows the CTR Program within the DoD organizational structure. 11

OCR for page 11
12 IMPROVING METRICS FOR THE DOD CTR PROGRAM BACKGROUND The CTR Program, created in 1991, was based on an enlightened approach to national security conceived at a time of rapidly changing events surrounding the end of the Cold War: The United States and the Soviet Union (and its successor states) had a shared security interest in addressing the threat of nuclear weapons proliferation from the Soviet nuclear weapons complex at a time when the former Soviet Union (FSU) was experiencing profound political and economic crises. Because of the security risks precipitated by the unstable and evolving situation, the United States provided assistance to Russia and other states of the FSU to meet treaty obligations by dismantling weapon delivery systems and to address the challenges of improving protection, control, and accounting for nuclear weapons and materials; to work to prevent smuggling of nuclear weapons and their components; and to prevent transfer of actual weapons, components, and weapons-related knowledge to other countries or terrorist groups. How did the United States reach the point where it would provide such assistance to a set of new states that emerged from its former Cold War rival? In 1990, the Soviet Union possessed tens of thousands of nuclear weapons, more than all of the other countries on Earth combined. The Soviet Union was also beginning to break apart. Several of its constituent republics declared independence over a 21-month period, a group of hard-line Communist Party leaders staged a failed coup attempt in August 1991, and in December 1991, the Soviet Union officially ceased to exist. Throughout the process of the Union’s dissolution, nuclear security experts in the United States became increasingly concerned about the control of Soviet nuclear weapons in Russia and especially in other, newly independent states. Recognizing the danger of leaving nuclear weapons on the territories of newly devolved states facing profound economic and political difficulties, Soviet President Mikhail Gorbachev asked U.S. President George H.W. Bush for assistance in dismantling some Soviet nuclear weapons and began removing Soviet weapons from military bases outside of Russia. When three heads of the former Soviet republics signed the Minsk Agreement on December 8, 1991, followed by eight others on December 21, formally dissolving the Union of Soviet Socialist Republics and forming the Commonwealth of Independent States, approximately 4,000 nuclear weapons remained in Belarus, Kazakhstan, and Ukraine. These agreements and the subsequent Lisbon Protocol signed with the United States on May 23, 1992, helped to ensure strategic stability by defining roles and future responsibilities under existing arms control treaties and establishing commitments for consolidation of Soviet nuclear weapons in Russia. Further steps were also needed to implement these commitments and to deal with parts of the former Soviet nuclear arsenal itself. On December 12, 1991, President Bush signed into law The Soviet Nuclear Threat Reduction Act of 1991 (P.L. 102-228), commonly known as the “Nunn-Lugar” legislation, so named because Senators Sam Nunn and Richard Lugar were the sponsors of the bill. This legislation articulated two objectives for the CTR Program: “A) to facilitate on a priority basis the transportation, storage, safeguarding, and destruction of nuclear and other weapons in the Soviet Union, its republics, and any successor states; and B) to assist in the prevention of weapons proliferation.” During the 1990s, the CTR Program expanded to address threats from the discontinued clandestine biological weapons programs and the stalled chemical weapons destruction program of the Soviet Union. These tasks have been substantial. The production scale of these Soviet programs was enormous: devastating chemical agents were stockpiled in

OCR for page 11
INTRODUCTION 13 thousands of tons; biological agents were produced on a massive scale. The CTR Program consolidated declared inventories of these agents, increased security at storage sites, and helped destroy and dismantle declared production facilities. How has progress on cooperative threat reduction been measured? The “Nunn-Lugar scorecard,” was developed to illustrate the impact and effectiveness using simple, direct measures.10 It tracks the numbers of warheads and delivery systems dismantled and destroyed. As of January 2011, the scorecard reads as follows: TABLE 1-1 Nunn-Lugar Scorecard, January 2011 7,599 strategic nuclear warheads deactivated 791 intercontinental ballistic missiles (ICBMs) destroyed 498 ICBM silos eliminated 180 ICBM mobile launchers destroyed 659 submarine launched ballistic missiles (SLBMs) eliminated 492 SLBM launchers eliminated 32 nuclear submarines capable of launching ballistic missiles destroyed 155 bombers eliminated 906 nuclear air-to-surface missiles destroyed 194 nuclear test tunnels eliminated 503 nuclear weapons transport train shipments secured upgraded security at 24 nuclear weapons storage sites built and equipped 20 biological monitoring stations neutralized 1680.4 metric tons of Russian and Albanian chemical weapons agent While this table is succinct and readable, text supporting the scorecard captures a greater accomplishment of the CTR Program: “Perhaps most importantly, Ukraine, Kazakhstan, and Belarus are nuclear weapons free as a result of cooperative efforts under the Nunn-Lugar Program. Those countries were the third, fourth, and eighth largest nuclear weapons powers in 10 See http://lugar.senate.gov/nunnlugar/scorecard.html; accessed January 11, 2012.

OCR for page 11
14 IMPROVING METRICS FOR THE DOD CTR PROGRAM the world.”11 Since 1991, much has been done; both accomplishments that are easy to present with a number or score and ones that are not have been achieved. DoD Organization for CTR Figure 1-1 illustrates how DoD has organized responsibilities for the CTR Program within the department, giving responsibility for overall policy decision making (what countries to partner with and what high-level objectives to pursue) to the Office of the Under Secretary of Defense for Policy, decisions about how to pursue the objectives to the Deputy Assistant to the Secretary of Defense for Treaties and Threat Reduction, and implementation responsibility to the Defense Threat Reduction Agency (DTRA), both in the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. Different parts of the DoD Metrics Report that the committee reviewed may have been developed by different parts of DoD, but the committee has deliberately disregarded this. Congress tasked the Secretary of Defense to develop and implement metrics for the CTR Program, so the report is a DoD report, and the committee directs its comments and advice to DoD. Secretary of Defense (OSD) Deputy Secretary of Defense (OSD) Under Secretary of Defense Under Secretary of Defense (Policy) (Acquisition, Technology, and Logistics) (USD/P) (USD/AT&L) Assistant to the Secretary of Defense for Assistant Secretary of Defense Nuclear and Chemical and Biological Defense Programs Global Strategic Affairs (ATSD (NCB)) (ASD/GSA) Policy Implementation Guidance Guidance Deputy Assistant to the Defense Threat Reduction & Deputy Assistant Secretary of Defense for Secretary of Defense for Agency Acquisition Countering Weapons of Mass Destruction Treaties and Threat Oversight (DASD(CWMD)) Reduction (DATSD (T&TR)) Associate Director, Operations Cooperative Threat Reduction Policy Enterprise (CTR Policy) (ADOP) Cooperative Threat Reduction Directorate (OP-CT) FIGURE 1-1 The CTR Program within the DoD organizational structure (DoD 2010). 11 See http://lugar.senate.gov/nunnlugar/; accessed January 11, 2012.

OCR for page 11
INTRODUCTION 15 PREVIOUS CTR REPORTS Several studies have examined the CTR programs and given advice on how to make them more effective. Some of those studies are described in Appendix C. This section describes recent reports and developments. The original legal agreement between Russia and the United States that provided the basis for the CTR Program is due to expire in June 2013, and there have been many efforts to define what, if anything, should follow that agreement. Moreover, given that the nature of security threats is different today than in the 1990s, some have considered how best to address these 21st century challenges through an expanded CTR Program. In the National Defense Authorization Act of 2008 (P.L. 110-181), Congress directed DoD to ask the NAS to recommend ways to strengthen and expand the DoD CTR Program. The report from that study concluded that Expanding the nation’s cooperative threat reduction programs beyond the former Soviet Union, as proposed by Congress, would enhance U.S. national security and global stability. … The committee recommends that the DoD CTR Program should be expanded geographically, updated in form and function….Forging broad new partnerships to implement sustainable programs that employ hard and soft capabilities and are tailored to specific countries or regions will energize and strengthen global security efforts and result in tangible and intangible benefits to national security. It is essential to develop meaningful program metrics that highlight program impact, acknowledge the value to national security of intangible program results, incorporate partner metrics into the overall evaluation of programs, and link metrics to program selection criteria. (NRC 2010) In this sense, an expanded CTR Program is envisioned to expand not only beyond its core weapons of mass destruction (WMD) portfolio, and beyond the geographic regions in which it has thus far focused, but also beyond those measures of programmatic impact that have been utilized in the first decades of the Program. Concurrent with that study, the White House developed its own strategy for countering biological threats (NSC 2009), which the federal agencies have used to guide their efforts (see, e.g., Weber, 2010). Perceptions in Congress of the evolution of the Program and the challenges faced in the Program led to a congressional mandate for this study. The desire for new metrics for CTR is motivated by at least two concerns about the Program’s growing work on capacity-building efforts aimed at improving partner nations’ ability to deter, detect, and respond to emerging threats, particularly in the Cooperative Biological Engagement Program (CBEP). First, some national security constituencies question whether CBEP, which resides in a gray mission space that sometimes overlaps with public health, serves their national security goals and whether it should be carried out by DoD. These skeptics have seen the Biological Threat Reduction Program shift from efforts to redirect weapons scientists and Biopreparat (the Soviet biowarfare agency) to legitimate commercial ventures, to efforts to engage nonweapons bioscientists in Central Asia, Africa, and even India,12 and ask DoD to define the security value of today’s capacity building health and biology related programs today.13 Second, impact and effectiveness 12 See, for example, Weber (2010) and Royal (2011). 13 The committee heard skeptics of CBEP say that unless the partner country’s scientists or their programs are weapons programs, the engagement is not a defense mission and CBEP does not belong in DoD. DoD argues that biological threats have changed greatly in the last 20 years. Proliferation of ability to create dangerous biological agents has already happened globally, threats are asymmetric, and only a small amount of material is needed to

OCR for page 11
16 IMPROVING METRICS FOR THE DOD CTR PROGRAM success are difficult to measure in such efforts, and appropriators and program managers alike want to ensure that the implementers have accurate, timely feedback on progress to guide the Program, and that funds spent on the Program are expended effectively. BRIEF BACKGROUND ON DOD CTR METRICS REPORT DoD issued a report on metrics for the CTR programs titled “Report on Metrics for the Cooperative Threat Reduction Program, September 2010.” (See Appendix B).14 The DoD Metrics Report consists of a brief introduction followed by descriptions of the metrics for all but one of the CTR programs: CBEP Program, the Chemical Weapons Elimination Program, the Nuclear Weapons Safety and Security Program, and the WMD Proliferation Prevention Program. The Strategic Offensive Arms Elimination Program was not included because that program uses the “Nunn-Lugar Scorecard” metrics, and will continue to do so until that mission is completed. SCOPE OF THIS ACADEMY REPORT This report provides the study committee’s assessment of the metrics in the DoD Metrics Report, including the feasibility of using the metrics (can these measurements be made?), their alignment with the objectives of the program (do these metrics reveal progress toward the stated objectives of the program?), and their usefulness to people with responsibility for implementing, directing, overseeing, or otherwise supporting the program. To the extent that there are shortcomings in the proposed DoD metrics, the committee tries to provide useful suggestions for how to improve the metrics and the use of metrics overall. The committee does not recommend specific alternative metrics, but does recommend what it considers a more effective approach for DoD to use in developing metrics. DoD has to develop its own metrics with its partners. In the report, the committee provides an example of its recommended approach applied to CBEP. create a biological weapon that could cause billions of dollars of damage to our economy (one expert said that amount could even be “pocket-sized”). With the capability already spread widely, intent is more important now, and to know and affect intent the program must communicate with capable and knowledgeable individuals with whom we have built some degree of understanding and, in some cases, even trust. 14 The DoD Metrics Report is refreshingly brief, but unusual in that it has no indication of who wrote the report, and has no trappings of an official document (e.g., the agency name or seal, document number).