ASSESSMENT OF THE DOD METRICS REPORT
The committee evaluated the DoD CTR metrics described in the DoD Metrics Report based on whether the metrics provide decision makers the essential information to manage the effectiveness and impact of CTR programs. Most of the CTR programs try to develop a sustaining capability of some sort through a set of projects implemented with partner countries on their territory. The DoD Metrics Report contains reasonable metrics for the CTR programs that consolidate and eliminate weapons and weapons materials,2 and contains a solid starting point for developing metrics for the newer capacity-building programs. For meaningful evaluation, the committee assessed that DoD must (1) state the objectives of the program and the projects (i.e., the goals of the actual activities); (2) identify the capabilities it is trying to develop or maintain; (3) link those capabilities to metrics; (4) ensure that the metrics reflect program effectiveness and impact; and (5) plan for and measure sustainment.3 It is generally good practice for the program to establish minimum performance levels (performance that must be achieved for the project to not fail) and aspirational goals (the desired performance above the minimum) for each metric.4
• Overall, the DoD Metrics Report describes CTR’s highest-level objectives and difficulties in developing metrics clearly and succinctly in the introductory section. However, the report does not connect these objectives to threats for the capacity- building programs. This is not to say that there is no connection, nor that the explanation does not appear in other documents,5 just that the DoD Metrics Report does not describe the connection. For example, describing the connection would clarify how building the capacity to better track respiratory disease in East Africa reduces threats to U.S. national security. The lack of a concise statement of the objectives of each program and how the actions planned under the program are intended to reduce threat or risk is a deficiency that makes the Report less effective for communicating with people outside of the program and internally makes development and refinement of metrics more difficult.
• CTR programs are meant to be partnerships and work best when they are carried out as partnerships with other countries (hence, cooperative threat reduction). This includes joint development of the objectives with the partner countries. The DoD Metrics Report does not make clear whether and how partner countries participate in determining objectives and metrics, or the partner country’s role in measurement. The report reads as if metrics are U.S. measurements of the partner’s progress toward U.S. goals rather than the impact and effectiveness of the program measured with jointly agreed metrics.
2 The committee did not address the Strategic Offensive Arms Elimination Program, which is not discussed in the DoD Metrics Report because DoD plans to use the long-standing “Nunn-Lugar Scorecard” metrics for that program.
3 In its report, DoD seems to use the term sustainability to refer to both the ability to sustain the program, security state, or other improvements of the CTR programs, and to the actual result or act of sustaining them. The committee refers to the former as sustainability and the latter as sustainment.
4 In capabilities based planning, these are referred to as threshold values and objective values. Because the term “objective” is used in so many different ways, the committee has not adopted this usage.
5 See, for example, Nacht (2009).