National Academies Press: OpenBook
« Previous: Summary
Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

1

Introduction

In the National Defense Authorization Act for Fiscal Year 2010 (P.L. 111-84), Congress directed the Secretary of Defense (DoD) to develop and implement metrics to “measure the impact and effectiveness of activities of the Cooperative Threat Reduction Program,” (CTR) and directed DoD to request the National Academy of Sciences (NAS) to conduct “an assessment to review the metrics developed and implemented under subsection (a) and identify possible additional or alternative metrics, if necessary.” (see Appendix A.) The Secretary completed a report describing DoD’s metrics for the CTR Program (DoD, 2010; here called the DoD Metrics Report) in September 2010.9

As implied in the language cited above, a metric is an evidence-based tool that measures impact and effectiveness or performance of a program or a project toward its objectives. Metrics alone cannot ensure that the best options have been identified or are being implemented, but they can be helpful to program managers and overseers as they work to ensure that efforts are focused in productive directions. Impact and effectiveness in some projects is easily measured quantitatively, particularly where the objectives yield tangible products, such as constructing secure storage facilities, consolidating materials into a secure facility, or destroying a stock of a chemical agent. The impact and effectiveness of projects that have less tangible objectives (e.g., strengthening both a nation’s ability to detect and respond to disease and the relationships between U.S. experts and those in the partner nation) can be more difficult to measure. Indeed, what to measure is an important but difficult question for such projects, and this was a central motivation for Congress’s mandate for DoD to develop and NAS to review metrics for the CTR Program, which are described in the next section.

The committee’s goal in this report is to assess the metrics DoD developed, and to provide recommendations on how to improve them or develop more effective alternative metrics. No metric will be perfect. Metrics are tools for evaluation, not substitutes for evaluation. Even in the missions with easy-to-understand, tangible actions, the obvious metrics (e.g., facilities completed, stocks of nuclear material consolidated) may not by themselves reflect the effectiveness that the United States cares about most (e.g., are the stocks secure?). With this limitation in mind, the committee provides background on the CTR Program, other studies of CTR, and the scope of this report before assessing the DoD Metrics Report.

The committee wrote this report with two main audiences in mind: Those who are mostly concerned with the overall assessment and advice, and those readers directly involved in the CTR Program, who need the details of the DoD report assessment and of how to implement the approach that the committee recommends. For the first category of reader, the committee has included a summary of the DoD CTR metrics in Table 2-1. For readers who desire more detail on the DoD CTR Metrics Report, the entire report can be found in Appendix B.

image

9 See Appendix B for the DoD Metrics Report; Figure 1 of the Report shows the CTR Program within the DoD organizational structure.

Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

BACKGROUND

The CTR Program, created in 1991, was based on an enlightened approach to national security conceived at a time of rapidly changing events surrounding the end of the Cold War: The United States and the Soviet Union (and its successor states) had a shared security interest in addressing the threat of nuclear weapons proliferation from the Soviet nuclear weapons complex at a time when the former Soviet Union (FSU) was experiencing profound political and economic crises. Because of the security risks precipitated by the unstable and evolving situation, the United States provided assistance to Russia and other states of the FSU to meet treaty obligations by dismantling weapon delivery systems and to address the challenges of improving protection, control, and accounting for nuclear weapons and materials; to work to prevent smuggling of nuclear weapons and their components; and to prevent transfer of actual weapons, components, and weapons-related knowledge to other countries or terrorist groups. How did the United States reach the point where it would provide such assistance to a set of new states that emerged from its former Cold War rival?

In 1990, the Soviet Union possessed tens of thousands of nuclear weapons, more than all of the other countries on Earth combined. The Soviet Union was also beginning to break apart. Several of its constituent republics declared independence over a 21-month period, a group of hard-line Communist Party leaders staged a failed coup attempt in August 1991, and in December 1991, the Soviet Union officially ceased to exist. Throughout the process of the Union’s dissolution, nuclear security experts in the United States became increasingly concerned about the control of Soviet nuclear weapons in Russia and especially in other, newly independent states.

Recognizing the danger of leaving nuclear weapons on the territories of newly devolved states facing profound economic and political difficulties, Soviet President Mikhail Gorbachev asked U.S. President George H.W. Bush for assistance in dismantling some Soviet nuclear weapons and began removing Soviet weapons from military bases outside of Russia. When three heads of the former Soviet republics signed the Minsk Agreement on December 8, 1991, followed by eight others on December 21, formally dissolving the Union of Soviet Socialist Republics and forming the Commonwealth of Independent States, approximately 4,000 nuclear weapons remained in Belarus, Kazakhstan, and Ukraine. These agreements and the subsequent Lisbon Protocol signed with the United States on May 23, 1992, helped to ensure strategic stability by defining roles and future responsibilities under existing arms control treaties and establishing commitments for consolidation of Soviet nuclear weapons in Russia. Further steps were also needed to implement these commitments and to deal with parts of the former Soviet nuclear arsenal itself.

On December 12, 1991, President Bush signed into law The Soviet Nuclear Threat Reduction Act of 1991 (P.L. 102-228), commonly known as the “Nunn-Lugar” legislation, so named because Senators Sam Nunn and Richard Lugar were the sponsors of the bill. This legislation articulated two objectives for the CTR Program: “A) to facilitate on a priority basis the transportation, storage, safeguarding, and destruction of nuclear and other weapons in the Soviet Union, its republics, and any successor states; and B) to assist in the prevention of weapons proliferation.” During the 1990s, the CTR Program expanded to address threats from the discontinued clandestine biological weapons programs and the stalled chemical weapons destruction program of the Soviet Union. These tasks have been substantial. The production scale of these Soviet programs was enormous: devastating chemical agents were stockpiled in

Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

thousands of tons; biological agents were produced on a massive scale. The CTR Program consolidated declared inventories of these agents, increased security at storage sites, and helped destroy and dismantle declared production facilities.

How has progress on cooperative threat reduction been measured? The “Nunn-Lugar scorecard,” was developed to illustrate the impact and effectiveness using simple, direct measures.10 It tracks the numbers of warheads and delivery systems dismantled and destroyed. As of January 2011, the scorecard reads as follows:

TABLE 1-1 Nunn-Lugar Scorecard, January 2011

7,599 strategic nuclear warheads deactivated

791 intercontinental ballistic missiles (ICBMs) destroyed

498 ICBM silos eliminated

180 ICBM mobile launchers destroyed

659 submarine launched ballistic missiles (SLBMs) eliminated

492 SLBM launchers eliminated

32 nuclear submarines capable of launching ballistic missiles destroyed

155 bombers eliminated

906 nuclear air-to-surface missiles destroyed

194 nuclear test tunnels eliminated

503 nuclear weapons transport train shipments secured

upgraded security at 24 nuclear weapons storage sites

built and equipped 20 biological monitoring stations

neutralized 1680.4 metric tons of Russian and Albanian chemical weapons agent

While this table is succinct and readable, text supporting the scorecard captures a greater accomplishment of the CTR Program: “Perhaps most importantly, Ukraine, Kazakhstan, and Belarus are nuclear weapons free as a result of cooperative efforts under the Nunn-Lugar Program. Those countries were the third, fourth, and eighth largest nuclear weapons powers in

image

10 See http://lugar.senate.gov/nunnlugar/scorecard.html; accessed January 11, 2012.

Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

the world.”11 Since 1991, much has been done; both accomplishments that are easy to present with a number or score and ones that are not have been achieved.

DoD Organization for CTR

Figure 1-1 illustrates how DoD has organized responsibilities for the CTR Program within the department, giving responsibility for overall policy decision making (what countries to partner with and what high-level objectives to pursue) to the Office of the Under Secretary of Defense for Policy, decisions about how to pursue the objectives to the Deputy Assistant to the Secretary of Defense for Treaties and Threat Reduction, and implementation responsibility to the Defense Threat Reduction Agency (DTRA), both in the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. Different parts of the DoD Metrics Report that the committee reviewed may have been developed by different parts of DoD, but the committee has deliberately disregarded this. Congress tasked the Secretary of Defense to develop and implement metrics for the CTR Program, so the report is a DoD report, and the committee directs its comments and advice to DoD.

image

FIGURE 1-1 The CTR Program within the DoD organizational structure (DoD 2010).

image

11 See http://lugar.senate.gov/nunnlugar/; accessed January 11, 2012.

Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

PREVIOUS CTR REPORTS

Several studies have examined the CTR programs and given advice on how to make them more effective. Some of those studies are described in Appendix C. This section describes recent reports and developments.

The original legal agreement between Russia and the United States that provided the basis for the CTR Program is due to expire in June 2013, and there have been many efforts to define what, if anything, should follow that agreement. Moreover, given that the nature of security threats is different today than in the 1990s, some have considered how best to address these 21st century challenges through an expanded CTR Program. In the National Defense Authorization Act of 2008 (P.L. 110-181), Congress directed DoD to ask the NAS to recommend ways to strengthen and expand the DoD CTR Program. The report from that study concluded that

Expanding the nation’s cooperative threat reduction programs beyond the former Soviet Union, as proposed by Congress, would enhance U.S. national security and global stability. … The committee recommends that the DoD CTR Program should be expanded geographically, updated in form and function….Forging broad new partnerships to implement sustainable programs that employ hard and soft capabilities and are tailored to specific countries or regions will energize and strengthen global security efforts and result in tangible and intangible benefits to national security. It is essential to develop meaningful program metrics that highlight program impact, acknowledge the value to national security of intangible program results, incorporate partner metrics into the overall evaluation of programs, and link metrics to program selection criteria. (NRC 2010)

In this sense, an expanded CTR Program is envisioned to expand not only beyond its core weapons of mass destruction (WMD) portfolio, and beyond the geographic regions in which it has thus far focused, but also beyond those measures of programmatic impact that have been utilized in the first decades of the Program.

Concurrent with that study, the White House developed its own strategy for countering biological threats (NSC 2009), which the federal agencies have used to guide their efforts (see, e.g., Weber, 2010).

Perceptions in Congress of the evolution of the Program and the challenges faced in the Program led to a congressional mandate for this study. The desire for new metrics for CTR is motivated by at least two concerns about the Program’s growing work on capacity-building efforts aimed at improving partner nations’ ability to deter, detect, and respond to emerging threats, particularly in the Cooperative Biological Engagement Program (CBEP). First, some national security constituencies question whether CBEP, which resides in a gray mission space that sometimes overlaps with public health, serves their national security goals and whether it should be carried out by DoD. These skeptics have seen the Biological Threat Reduction Program shift from efforts to redirect weapons scientists and Biopreparat (the Soviet biowarfare agency) to legitimate commercial ventures, to efforts to engage nonweapons bioscientists in Central Asia, Africa, and even India,12 and ask DoD to define the security value of today’s capacity building health and biology related programs today.13 Second, impact and effectiveness

image

12 See, for example, Weber (2010) and Royal (2011).

13 The committee heard skeptics of CBEP say that unless the partner country’s scientists or their programs are weapons programs, the engagement is not a defense mission and CBEP does not belong in DoD. DoD argues that biological threats have changed greatly in the last 20 years. Proliferation of ability to create dangerous biological agents has already happened globally, threats are asymmetric, and only a small amount of material is needed to

Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

success are difficult to measure in such efforts, and appropriators and program managers alike want to ensure that the implementers have accurate, timely feedback on progress to guide the Program, and that funds spent on the Program are expended effectively.

BRIEF BACKGROUND ON DOD CTR METRICS REPORT

DoD issued a report on metrics for the CTR programs titled “Report on Metrics for the Cooperative Threat Reduction Program, September 2010.” (see Appendix B).14 The DoD Metrics Report consists of a brief introduction followed by descriptions of the metrics for all but one of the CTR programs: CBEP Program, the Chemical Weapons Elimination Program, the Nuclear Weapons Safety and Security Program, and the WMD Proliferation Prevention Program. The Strategic Offensive Arms Elimination Program was not included because that program uses the “Nunn-Lugar Scorecard” metrics, and will continue to do so until that mission is completed.

SCOPE OF THIS ACADEMY REPORT

This report provides the study committee’s assessment of the metrics in the DoD Metrics Report, including the feasibility of using the metrics (can these measurements be made?), their alignment with the objectives of the program (do these metrics reveal progress toward the stated objectives of the program?), and their usefulness to people with responsibility for implementing, directing, overseeing, or otherwise supporting the program. To the extent that there are shortcomings in the proposed DoD metrics, the committee tries to provide useful suggestions for how to improve the metrics and the use of metrics overall. The committee does not recommend specific alternative metrics, but does recommend what it considers a more effective approach for DoD to use in developing metrics. DoD has to develop its own metrics with its partners. In the report, the committee provides an example of its recommended approach applied to CBEP.

image

create a biological weapon that could cause billions of dollars of damage to our economy (one expert said that amount could even be “pocket-sized”). With the capability already spread widely, intent is more important now, and to know and affect intent the program must communicate with capable and knowledgeable individuals with whom we have built some degree of understanding and, in some cases, even trust.

14 The DoD Metrics Report is refreshingly brief, but unusual in that it has no indication of who wrote the report, and has no trappings of an official document (e.g., the agency name or seal, document number).

Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 11
Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 12
Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 13
Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 14
Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 15
Suggested Citation:"1 Introduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 16
Next: 2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report »
Improving Metrics for the Department of Defense Cooperative Threat Reduction Program Get This Book
×
Buy Paperback | $38.00 Buy Ebook | $30.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Cooperative Threat Reduction (CTR) Program was created in 1991 as a set of support activities assisting the Former Soviet Union states in securing and eliminating strategic nuclear weapons and the materials used to create them. The Program evolved as needs and opportunities changed: Efforts to address biological and chemical threats were added, as was a program aimed at preventing cross-border smuggling of weapons of mass destruction. CTR has traveled through uncharted territory since its inception, and both the United States and its partners have taken bold steps resulting in progress unimagined in initial years. Over the years, much of the debate about CTR on Capitol Hill has concerned the effective use of funds, when the partners would take full responsibility for the efforts, and how progress, impact, and effectiveness should be measured.

Directed by Congress, the Secretary of Defense completed a report describing DoD's metrics for the CTR Program (here called the DoD Metrics Report) in September 2010 and, as required in the same law, contracted with the National Academy of Sciences to review the metrics DoD developed and identify possible additional or alternative metrics, if necessary. Improving Metrics for the DoD Cooperative Threat Reduction Program provides that review and advice.

Improving Metrics for the DoD Cooperative Threat Reduction Program identifies shortcomings in the DoD Metrics Report and provides recommendations to enhance DoD's development and use of metrics for the CTR Program. The committee wrote this report with two main audiences in mind: Those who are mostly concerned with the overall assessment and advice, and those readers directly involved in the CTR Program, who need the details of the DoD report assessment and of how to implement the approach that the committee recommends.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!