2

Committee Assessment of the Department of Defense Cooperative
Threat Reduction Metrics Report

The first step in developing metrics is to establish what questions the metrics are meant to answer, and how the metrics will be used. Throughout this study, in conversations with the Department of Defense (DoD), other agencies, staff on Capitol Hill, and in the Executive Office of the President, and with officials in partner countries, the committee asked how the metrics will be used. The answers ranged based on the respondent, from those who sought measures of progress toward agreed goals, to cost effectiveness, to others who sought measures of how well the efforts align with the national defense mission. Given that different audiences want metrics for different purposes, it makes sense to begin by describing the committee’s perspective on the purposes of metrics.

Metrics are tools for evaluating the impact and effectiveness of programs and projects against strategic goals and for management within the program. The former are program metrics. The latter are project metrics. Different metrics may be appropriate for different stages of program and project development, but they all need to be tied together by the strategic objectives of the program as they fit with broader DoD Cooperative Threat Reduction objectives.

The committee evaluated the DoD CTR metrics described in the (CTR) DoD Metrics Report based on whether the metrics provide decision makers with the essential information to manage the effectiveness and impact of CTR programs. The committee drew on several frameworks for program planning, performance, and evaluation, but especially on capability based planning, to identify general steps needed for meaningful evaluation.15 Other frameworks are also valid and will either explicitly or implicitly include these elements. DoD must

 

•   state the objectives of the program and the project (i.e., the goals of the actual activities)

•   identify the capabilities it is trying to develop or maintain

•   define objectives for each capability and link those capabilities to metrics

•   ensure that the metrics reflect program effectiveness and impact

•   plan for and measure sustainment (see footnote 3 in the Summary)

It is generally good practice for the program to establish minimum performance levels and aspirational goals for each metric (see footnote 4 in the Summary). In Table 2-1, the committee

image

15 A much more elaborate metrics framework is emerging from analysts focused on the international development community. This highly structured approach treats the portfolio of projects as an experimental trial, with control projects or communities and strict adherence to a protocol established at the outset of the program or project. The committee chose a program evaluation framework more like the program planning, performance, and evaluation approaches already used in DoD because the latter are more familiar to DoD and, in the committee’s view, better suited to the CTR Program. The CTR Program engages in too few projects for meaningful control groups, and the political conditions, physical and social circumstances, budgets, and objectives for the program change, all of which make a trial-based approach incompatible with CTR today.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
2 Committee Assessment of the Department of Defense Cooperative Threat Reduction Metrics Report The first step in developing metrics is to establish what questions the metrics are meant to answer, and how the metrics will be used. Throughout this study, in conversations with the Department of Defense (DoD), other agencies, staff on Capitol Hill, and in the Executive Office of the President, and with officials in partner countries, the committee asked how the metrics will be used. The answers ranged based on the respondent, from those who sought measures of progress toward agreed goals, to cost effectiveness, to others who sought measures of how well the efforts align with the national defense mission. Given that different audiences want metrics for different purposes, it makes sense to begin by describing the committee’s perspective on the purposes of metrics. Metrics are tools for evaluating the impact and effectiveness of programs and projects against strategic goals and for management within the program. The former are program metrics. The latter are project metrics. Different metrics may be appropriate for different stages of program and project development, but they all need to be tied together by the strategic objectives of the program as they fit with broader DoD Cooperative Threat Reduction objectives. The committee evaluated the DoD CTR metrics described in the (CTR) DoD Metrics Report based on whether the metrics provide decision makers with the essential information to manage the effectiveness and impact of CTR programs. The committee drew on several frameworks for program planning, performance, and evaluation, but especially on capability based planning, to identify general steps needed for meaningful evaluation.15 Other frameworks are also valid and will either explicitly or implicitly include these elements. DoD must • state the objectives of the program and the project (i.e., the goals of the actual activities) • identify the capabilities it is trying to develop or maintain • define objectives for each capability and link those capabilities to metrics • ensure that the metrics reflect program effectiveness and impact • plan for and measure sustainment (see footnote 3 in the Summary) It is generally good practice for the program to establish minimum performance levels and aspirational goals for each metric (see footnote 4 in the Summary). In Table 2-1, the committee 15 A much more elaborate metrics framework is emerging from analysts focused on the international development community. This highly structured approach treats the portfolio of projects as an experimental trial, with control projects or communities and strict adherence to a protocol established at the outset of the program or project. The committee chose a program evaluation framework more like the program planning, performance, and evaluation approaches already used in DoD because the latter are more familiar to DoD and, in the committee’s view, better suited to the CTR Program. The CTR Program engages in too few projects for meaningful control groups, and the political conditions, physical and social circumstances, budgets, and objectives for the program change, all of which make a trial-based approach incompatible with CTR today. 17

OCR for page 17
18 IMPROVING METRICS FOR THE DOD CTR PROGRAM TABLE 2-1 Summary of metrics from the DoD Metrics Report CTR Program What metrics are intended to measure Metrics Examples of measures of effectivenessa 1. Secure and consolidate collections of especially dangerous pathogens (EDP) and their associated research at a minimum Partner country EDP collections and number of secure health and agricultural laboratories or associated research are consolidated into related facilities. a minimum number of locations. Partner country EDPs and associated 2. Enhance partner country/region’s capability to prevent the research are secured in a manner sale, theft, diversion, or accidental release of biological consistent with standards. weapons (BW)-related materials, technology, and expertise by Partner country has BS&S laws and improving biological safety and security (BS&S) standards regulations governing work with EDPs. and procedures. Cooperative Biological Partner country disease detection and 3. Enhance partner country/region’s capability to detect, Engagement Program diagnosis capability meets U.S. and/or diagnose, and report endemic and epidemic, man-made or (CBEP) international guidelines for biosafety. natural EDPs, bio-terror attacks, and potential pandemics. Partner country has preparedness and 4. Ensure the developed capabilities are designed to be response plans. sustainable within each partner country/region’s current Partner country disease surveillance system operating budget. is capable of detecting and reporting 5. Facilitate the engagement of partner country’s/regional suspect EDP cases to those responsible scientific and technical personnel in research areas of interest for human and animal health. to both the partner country/region and the United States. 6. Eliminate any BW-related infrastructure and technologies encountered in a partner/country region. Quantity of CW agent destroyed Chemical Weapons Destruction of agent through the Russian Federation’s safe, Number of munitions destroyed Elimination Program efficient operation of the destruction facility Project Metrics (CWE) Scheduled facility downtime Unscheduled facility downtime Facility achieved availability 1. Performance/capability assurance Examples of metrics Number of nuclear weapons storage 2. Configuration management Nuclear Weapons Safety sites upgraded 3. Procedures and process and Security (NWSS) Number of personnel trained 4. Training supports Nuclear Weapons Establishment of regional technical 5. Organization and personnel Transport Security Program centers 6. Life-cycle management Note: Some metrics still under 7. Maintenance development 8. Logistics 1. Enhance partner country capability to perform effective risk Examples of metrics management Miles of green (land) / blue (maritime) border with added security 2. Enhance partner country capability to perform border Number of ports of entry enhanced security command, control, communications and computers Number of border facilities (land and 3. Enhance partner country capability to perform border maritime) provided increased security surveillance capability to detect WMD 4. Enhance partner country capability to perform WMD Number of personnel trained and detection equipped to perform border security 5. Enhance partner country capability to perform border interdiction security interdiction Initial and refresher training system 6. Assist partner country with the development of a Weapons of Mass exists sustainment budget for all systems delivered under this Destruction Proliferation Number of personnel exchanges program Prevention Program (training, professional development) 7. Enhance partner country capability to support and maintain (WMD-PPP) Mean time to negotiate agreements delivered equipment and/or systems Number of regional relationships 8. Enhance partner country capability to sustain delivered facilitated training 9. Enhance a partner country capability to capture and disseminate information including that concerning WMD incidents 10. Increase the awareness of partner countries as to their critical role in WMD non-proliferation (partner country buy- in) 11. Develop interagency, bilateral, regional, and multilateral cooperation a The terms “goal,” “objective,” “indicator,” “attribute,” and “metric” are used inconsistently among the different program descriptions in the DoD Metrics Report. CBEP uses the term “measure of effectiveness” to describe what is being measured and the term “indicator” to denote how it is being measured (i.e., what is being counted).

OCR for page 17
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 19 attempts to summarize the disparate metrics from the DoD Metrics Report and Table 2-2 summarizes the committee’s assessments of DoD’s CTR Metrics Report with respect to the elements described above. Because of differences among the programs, inconsistencies in their terminologies, and the complexity of their metrics, the program metrics and indeed the assessments are difficult to represent in a uniform tabular structure, so the committee commends readers seeking deeper familiarity with the metrics to the DoD Metrics Report, which is reproduced in full in Appendix B. The committee did not address the Strategic Offensive Arms Elimination Program, which is not discussed in the DoD Metrics Report. DoD indicated that it plans to use long-standing “Nunn-Lugar Scorecard” metrics for that program. METRICS AND OBJECTIVES In its introduction, the DoD Metrics Report states that “We can measure the amount of equipment provided and the number of training events conducted or scientists engaged; however, we need better measures to show that these efforts actually result in changed practices or additional effectiveness.” The committee agrees and notes that the example given conflates inputs and project metrics (those that measure progress on the means to meet objectives) and outputs and programmatic or strategic metrics (those that measure progress relative to/toward the objectives themselves). Effectiveness and efficiency are important criteria for the evaluation of government programs. A government program is effective if it achieves the objectives set forth by the partners engaged in the program.16 A government program is efficient if it minimizes the resources required to achieve its goals. A common approach is to manage and assess programs based on their cost-effectiveness, which is an efficiency measure. Although measures of effectiveness and impact are the most challenging and should continue to receive the most attention, measures of efficiency and cost-effectiveness are also necessary and should be included in future iterations of the CTR Program metrics.17 OVERALL ASSESSMENT The DoD Metrics Report contains reasonable metrics for the CTR programs aimed at consolidation and elimination of weapons and weapons materials, and contains a reasonable starting point for developing metrics for the newer capacity-building programs. 16 It may not be possible to directly measure the higher-level outcomes from some CTR programs or projects, even with the best metrics. For example, DoD may never know how many illegal shipments were not interdicted at a border crossing where a CTR program provides assistance, or how many patients sick with an illness of interest did not go to the hospital. Recognizing this fact at the outset will help to avoid wasted time and effort and avoid false expectations. 17 While cost-effectiveness is important, particularly as a part of prioritization, it is essential that efficiency (which is relatively easy to measure) not drive out effectiveness (which is more difficult to measure). In other words, programs that are extremely cost-efficient may be pursuing the wrong result, or be solving the wrong problem. Balancing cost-effectiveness against other measures is a necessary part of developing prioritized metrics.

OCR for page 17
20 IMPROVING METRICS FOR THE DOD CTR PROGRAM Individual Program Objectives The DoD Metrics Report describes both CTR’s highest-level objectives and DoD’s difficulties in developing metrics, and it does so clearly and succinctly in the introductory section of its report. However, the individual program objectives (those of the CWE, NWSS, CBEP and WMD-PPP) and their connections to threats to U.S. national security are not well articulated or not addressed. Each program has objectives, and they may be described well in other DoD documents (see, e.g., Nacht, 2009), but the DoD Metrics Report itself needs to articulate the objectives in a way that makes apparent the connection between the actions taken under the Program and the higher level objectives. Not making those connections poses a problem for the Program because proper metrics indicate progress toward objectives, or more specifically, progress in developing the capabilities to fulfill the objectives. Those connections are the logical starting point for the presentation and for the actual development and analysis of metrics. Again, this is not to say that there is no connection, but that the DoD Metrics Report does not describe the connection. For example, describing the connection would clarify how building the capacity to better track respiratory disease in East Africa reduces threats to U.S. national security or why Ukraine’s interdiction of smuggled frozen chicken legs indicates a positive impact from a CTR program. A strong case can be made for each of these, but not without stating the Program objectives and connect them to program objectives. For the programs that do not already have such objectives stated in the DoD Metrics Report, a concise statement of the objectives of each program and how the actions planned under the programs are intended to reduce threat or risk would provide a connection between the objectives and the metrics. Partnership CTR programs are intended to be and work best when they are carried out as partnerships (cooperative) with other countries. This includes joint development of the objectives with the partner countries. Most of the metrics in the DoD Metrics Report do not reflect such joint development. The report reads as if metrics are U.S. measurements of our partner’s progress toward U.S. goals. This can create misunderstandings between the United States and our partners, which can undermine CTR efforts. Prioritization DoD did not use a consistent framework for developing and articulating objectives, priorities, and metrics, and DoD did not prioritize among its metrics. As a consequence, the DoD Metrics Report mixes project management measures with higher-level program performance metrics for some of the CTR programs, and weights equally metrics that are critically important and others are not. Cooperation DoD plans to leverage other United States Government agencies’ experience, capabilities, and assets as CTR expands to new countries and as it continues existing programs.

OCR for page 17
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 21 DoD also needs to communicate, coordinate, and cooperate with relevant agencies. Examination of other U.S. Government documents suggests that CBEP is the most consistently planned and coordinated across the other agencies.18 The committee notes that previous Academy reports and government agencies themselves have stated the importance of such coordination for program success, so a metric for it seems reasonable and valuable in theory. To the committee’s knowledge, there is no metric to reflect interagency coordination or a “whole of government” approach to working with a partner country, rather than a piecemeal approach. The committee does not have a solution to this problem (see Chapter 3 for further discussion). In addition to working with these other agencies, DoD CTR can learn something from them. They, too, engage in capacity building programs, work on similar missions, and measure performance. U.S. Customs and Border Protection conducts a mission similar to the PPP along many thousands of kilometers of border and has developed metrics for its mission and operations. The U.S. Department of Agriculture’s Animal and Plant Health Inspection Service is a leader in an international surveillance network that has many parallels to the global network DoD leadership envisions for reducing biothreats. The U.S. Agency for International Development operates programs with partners across the world to foster democratic institutions. As a further example, a partner’s compliance with the International Health Regulations would reflect interests and activities of DoD, Health and Human Services, and Department of State, and the extent of their coordination and cooperation in achieving that outcome is another metric. Even other organizations within DoD could provide different models: The Defense Security Cooperation Agency builds security capacities with partner nations. DoD’s CTR Program can learn from these other agencies successes and challenges (and even its own experience) if it incorporates a lessons-learned mechanism for the Program. Furthermore, the CTR Program will be working with these other agencies in a whole-of-government effort, and the others might already have mechanisms in place for measuring impact and effectiveness that would be useful to CTR. Change The DoD Metrics Report deliberately does not consider future missions or changes in objectives, but it is difficult to see how CTR metrics can be designed to respond to change if this is not discussed in DoD’s report. Some factors are under the Program’s control. Far more are outside of the Program’s control. Every responsible business, military operation, and government program builds in resilience that is the ability to deal with exogenous change. The CTR programs are susceptible to external change ranging from budget reductions to shifts in the host country’s political or economic environment. The practical consequences of some of the shortcomings listed above, such as not articulating the connection to threat or risk, might not be large for projects in progress under longstanding agreements. But they are important for new projects and especially for new partnerships. A more detailed examination of the programs can be found in the next section. 18 See Weber, A. 2010.

OCR for page 17
22 IMPROVING METRICS FOR THE DOD CTR PROGRAM PROGRAM-BY-PROGRAM ASSESSMENT The committee has summarized several important aspects of its assessment of the DoD Metrics Report in Table 2-2. The column headings represent a logical chain from objectives through metrics to sustainability and the minimum performance required, read from left to right. Chemical Weapons Elimination (CWE) The treatment of the CWE program in the DoD Metrics Report may be adequate for the current project under the program. The stated program goal looks very much like a project goal: “[T]o destroy nerve agent-filled munitions located in the Planovy [chemical weapons] storage facility in a safe, secure, and environmentally sound manner.” The United States and the Group of Eight Global Partnership against the Spread of Weapons and Materials of Mass Destruction (G8 Global Partnership) supported the design and construction of the chemical weapons destruction facility at Shchuch’ye, Russia, which began operating in 2009. At Russia’s request, the CWE program is providing technical support for the commissioning and operation of the facility. So the desired capability to support the program and project objectives is defined as adequate operation (by the Russians with U.S. technical support) of the Shchuch’ye destruction facility. In its Metrics Report, DoD refers to the quantities of chemical weapons agent destroyed (neutralized and immobilized) as the program metrics. DoD’s only project level metrics are scheduled facility downtime and unscheduled facility downtime. These metrics align well with the “destroy nerve agent” aspect of the stated program objectives, but the metrics only implicitly address the “safe, secure, and environmentally sound manner” aspect of the objectives. Worker safety and environmental emissions are also important indicators of proper operation of the facility, but are not mentioned in the DoD Metrics Report. In a facility that has strong regulatory controls and/or strong internal management controls on safety and environmental performance, problems with safety or environmental releases affect the operation of the facility, i.e., the facility will shut down some or all of its operations to correct problems. If the facility does not have strong controls in place, safety and environmental performance might not affect continued operation of the facility, so tracking facility downtime might not factor in safety and environmental performance. DoD is better positioned than the committee to assess this aspect of the Shchuch’ye facility and can justify its decision either to include such metrics or to exclude them. Likewise, the neutralization process is supposed to convert 99.99 percent of the agent to slightly hazardous components and then immobilize them for disposal. DoD does not explain the acceptable range of destruction percentage (is 90 percent destruction acceptable?) and whether it is important to track that parameter. Finally, the committee notes that the Planovy Chemical Weapons Depot near Shchuch’ye contains approximately one seventh of the Russian stockpile of chemical weapons that awaits destruction. If DoD hopes that the support provided at Shchuch’ye will not only enable the Russian Federation to operate Shchuch’ye independently, but also affect progress at other depots, then DoD should include a metric for that objective too.

OCR for page 17
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 23 TABLE 2-2. Program-by-program summary of objectives, capabilities, and metrics in the DoD Metrics Report. The committee’s assessment (shaded) is followed by a summary of the committee’s advice Objectives Desired Linkage of Do metrics Do metrics Are minimum (program/ capabilities capabilities to measure address performance & aspirational project) are identified? metrics impacts and sustainability? specified? effectiveness? goals specified? Not in DoD Report;a Adequately Yes Yes – for the None Yes Chemical Some confusion operating operation of Weapons between program and facility Shchuch’ye Elimination project objectives.a Shchuch’ye (CWE) Assessment N/A Consider This linkage Worker safety & This aspect could N/A CWE whether needs to be plant emissions be developed. Advice development established. may also be at other appropriate & facilities is important needed. metrics. Yesa Yes/Yes Yes, for Yes No, only Yes Nuclear sustainmentb measure input. Weapons Safety & Security (NWSS) Assessment Clearly link overall N/A This linkage Although there N/A N/A NWSS objectives to threat should be made are constraints Advice reduction. more explicit, on measurement, particularly as improved metrics the program should have expands to more focus on other countries. impact & effectiveness. Yes/Yes Yes Yes Some, but not Yes Yes Cooperative prioritized. Biological Engagement Program (CBEP) Assessment Need to be more N/A Would benefit Need to be N/A N/A CBEP clearly linked to from linking prioritized, Advice threat reductiona capabilities reduced in metrics with number, and threat reduction. direct metrics, where possible. Yes/Yes Yes, but Program Program: no. Yes No Weapons of vague. metrics not Project: Some of Mass linked to them do, but not Destruction capabilities. prioritized, not Proliferation Project, yes clearly linked to Prevention program level. Program (WMD-PPP) Assessment Recognize links to Need to be Once capabilities More thought Sustainability The minimum WMD-PPP other related/similar clarified. are better should be given metrics also need performance and Advice programs; address articulated, they to prioritization to be prioritized aspirational integrated assistance need to be better and impact and and focused on goals need to efforts across the linked to the effectiveness. capabilities. be specified. USG and other aid program and providers. project metrics. a Not available in the DoD Metrics Report but available in other documents. b NWSS has completed upgrades at agreed facilities, so the current program is focused on providing sustainment.

OCR for page 17
24 IMPROVING METRICS FOR THE DOD CTR PROGRAM Cooperative Biological Engagement Program (CBEP) For the biosecurity mission, there are two main thrusts:19 (1) secure and consolidate especially dangerous pathogens, and any work with those pathogens, to a safe, secure facility or facilities; and (2) engage bioscience institutions20 and authorities in the partner country to establish a culture of responsible practice, detection, and international reporting of emergent pathogens and transfers of the pathogens. In principle, the first thrust is relatively easy to measure, if the work with those pathogens is known to the governments. The second thrust, the aim of which is in part to develop relationships of trust and practices that are trustworthy, is more difficult for the United States to measure (see Measuring Trust, Confidence, and Goodwill in Chapter 3). The metrics DoD developed for CBEP are numerous and mostly indirect with respect to trust, but relatively concrete and measureable. Some additional steps are needed to make these metrics effective tools for evaluating impact and effectiveness and managing the programs. The DoD metrics for CBEP have nearly all of the elements that the committee thinks are needed for development of useful metrics. The CBEP objectives and desired capabilities are clearly stated, although their connection to threat reduction is not stated in the DoD Metrics Report. The metrics are linked to the desired partner country capabilities, and some of them measure impacts and effectiveness, although others do not.21 DoD also factors sustainability into its rating of the program’s performance. The greatest shortcoming of this otherwise rather complete metrics structure is its lack of prioritization. DoD lists 49 metrics for CBEP and in its first application of the metrics, DoD weighted all of them equally. It is difficult to draw meaningful conclusions about overall progress from 49 metrics that are all weighted as equally important. Some of the metrics are far more important to DoD and to threat reduction than are others (see, e.g., Footnote 7 in this chapter), and not all 49 are needed, even after duplicated metrics are eliminated. The committee discusses metrics for the CBEP more extensively in Chapter 3, where the committee illustrates how an improved method for developing, prioritizing, and using metrics should be applied to CTR programs. 19 This is a summary of the four objectives DoD describes in its Metrics Report: 1. Secure and consolidate collections of especially dangerous pathogens (EDPs) and their associated research at a minimum number of secure health and agricultural laboratories or related facilities; 2. Enhance partner country/region’s capability to prevent the sale, theft, diversion, or accidental release of biological weapons related materials, technology, and expertise by improving biological safety and security standards and procedures; 3. Enhance partner country/region’s capability to detect, diagnose, and report endemic and epidemic, man-made or natural EDPs, bio-terror attacks, and potential pandemics; and 4. Ensure the developed capabilities are designed to be sustainable within each partner country/region’s current operating budget. 20 It seems reasonable to enhance this priority to include engaging bioscientists, not just bioscience institutions. 21 For example, one of the measures of effectiveness concerns the partner country laws and regulations for biological safety and security, and their conformance to U.S. standards. While the U.S. implementers may believe that having such laws and regulations is better than not having them, the actual practices are more important than the details of the laws. The practices may be poor despite good laws and regulations or good despite poor laws and regulations. As a result, a partner that otherwise has shown significant improvements with respect to biological safety and security might get a poor rating based on a regulatory system that is different from that in the United States.

OCR for page 17
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 25 Nuclear Weapons Safety and Security (NWSS) The NWSS section of the DoD Metrics Report gives candid assessments of the metrics it has developed (e.g., “Depending on the level of cooperation, DoD may not be able to track this metric independently.”), and overall gives the impression of a metrics effort in development. This reflects the current state of the NWSS program, but it also gives an incomplete picture of DoD’s thinking about measuring the impact and effectiveness of this program. NWSS has completed upgrades at all of the facilities covered by the agreement between the United States and the Russian Federation, and DoD plans to continue to use the number of nuclear weapons storage sites upgraded as a metric of the impact and effectiveness of the program in Russia. Given that the upgrades are done, the current program is focused on (1) providing sustainment in the form of training, and on (2) the Nuclear Weapons Transportation Security Program. The DoD Metrics Report also states that “with the potential global expansion of the NWSS program to other partner countries, Defense Threat Reduction Agency DTRA has been working to develop enhanced metrics that better reflect how our efforts are contributing to overall threat reduction.” The current NWSS program has severe constraints on measurements. The CTR program trains people who train the operators of the nuclear weapons storage sites, but the CTR training takes place at facilities away from the nuclear weapons storage sites, and CTR personnel do not have access to any of the sites. CTR supports supply of upgraded secure railcars for the transportation mission, but CTR personnel do not see them in operation. In both cases, the lack of access makes it difficult or impossible to measure the outcomes directly linked to the program’s objectives. Recognizing these constraints, DoD has identified the objectives of the program and the capabilities desired to achieve those objectives. The DoD Metrics Report describes what DoD would like to measure and what might actually be possible. For example, DoD will track the number of railcar shipments supported and the number of secure railcars provided. DoD can measure the latter itself; it cannot measure the former. DoD included some metrics in its report because they were negotiated among DoD, Department of Energy, and the partner, Russia’s Ministry of Defense. Such mutually agreed metrics are important and notable, as discussed elsewhere in this report. DoD presented additional information and ideas to the committee in November 2010.22 These ideas include a systematic method for weighting or prioritizing among the lower level factors that contribute to the metrics. Without endorsing the specific details presented to the committee, the committee agrees that DoD would benefit from further development of ideas like those in the November presentation. Weapons of Mass Destruction Proliferation Prevention Program (WMD-PPP) The WMD-PPP focuses on land and maritime border security, also known as green and blue borders respectively, in partner nations including Azerbaijan and Ukraine. Three program metrics are defined: • Miles of green (land) border provided sustainable security enhancements (in progress/complete) 22 See Appendix E.

OCR for page 17
26 IMPROVING METRICS FOR THE DOD CTR PROGRAM • Miles of blue (maritime) border provided sustainable security enhancements (in progress/complete) • Number of ports of entry provided sustainable capability enhancements (in progress/complete) In introducing these metrics, the DoD Metrics Report notes: “The ability to measure simply and objectively the impact that WMD-PPP assistance has had on threat reduction is challenging due to the nature of the program: we are providing a capability to our partners that gives them an ability to deter proliferation.” DoD articulates both program and project objectives for the WMD-PPP. The difficulty for WMD-PPP program managers and decision makers results from the vagueness of the capabilities desired to achieve the goals or objectives. DoD clearly found it difficult to translate the desired capabilities into metrics. Miles of green and blue border with additional security enhancements, and the number of ports of entry where capabilities have been enhanced are ill defined. Are all enhancements equivalent? Is every kilometer of border equivalent in terms of threat reduction? Securing a border is complicated. The first approximation of progress—the number of kilometers of green border deemed secure—might not accurately represent risk reduction for several reasons. First, not all kilometers of border represent equal risks. Second, adversaries can adapt to exploit the weakest link. Third, what constitutes a secure kilometer of border? Areas where security enhancements have been provided might still have known smuggling routes that regularly defeat security enhancements may be improperly counted in success measures. These challenges are well known to DTRA, and in implementing WMD-PPP, DTRA has not treated all borders as if they were equally important. For instance, some stretches of border may be sufficiently impassable due to the natural topography of the area as to render any additional security enhancement unnecessary while others may include commonly used smuggling routes. Metrics should instead focus on the risk posed and the likelihood of evading detection. In Ukraine, both DTRA and the Ukrainian State Border Guard Service conducted a simple risk ranking and agreed jointly on the highest risk stretch of border. They focused their prototype and demonstration facility efforts on that stretch of border. Thus the implementation is better than the metrics would suggest. Interdiction events involving materials useful for WMD are rare and experts are cautiously optimistic that this is because smuggling of WMD material is rare, not just difficult to observe. Border protection services do not now have, and endeavor to never have, enough WMD incidents to make tracking them a useful statistical tool, so they use statistics from other border violations as proxies to indicate the effectiveness of their border controls. The definition of positive outcome must be sufficiently broad to prevent unintentionally causing a reduction of inspections or alarms that might prevent the detection of an illicit item. For instance, if the risk management device is using rules based targeting efforts or non-intrusive inspection equipment, identifying a shipment that comes from a legitimately high-risk source or known bad actor might be a positive outcome, even if the resulting inspection did not uncover illicit material. Similarly, the physical inspection of a container that uncovers a significant quantity of lead or similarly dense material may be a positive outcome to imaging analysis of a container x-ray that shows a density anomaly. Noting those inspections as negative outcomes may result in the overly narrowing of risk factors, thereby increasing the risk of missing an actual smuggling event.

OCR for page 17
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 27 Progress in securing a border can be measured statistically (with some degree of accuracy, Chapter 3) if the country has sufficient control or awareness to even estimate the violations. Border protection services can track radiation alarms and smuggling incidents involving drugs, firearms, people, stolen goods, and goods avoiding tariffs. It may be that partner countries do not need the ability to perform maritime interdictions themselves. Instead, one might measure the response time to conduct a maritime interdiction, understanding that another country’s team or a regional/multi-national team may be best situated and equipped to respond. Measuring the time for interdiction might more accurately measure the objective. For instance, the coastline of Georgia may be such that it needs fewer boarding teams than required to respond adequately to the much larger coastline of Ukraine. Another option might be to take the numerator/denominator approach, measuring the current capability against an assessed need. At a project level, the metrics in the DoD Metrics Report are better linked to the capabilities desired. Some of the project metrics DoD developed for the WMD-PPP measure impacts and effectiveness, such as alerts resulting in a positive outcome, and others do not, such as the number of maintenance personnel trained. There are also a number of other programs of the United States and other countries that provide border control and law enforcement assistance that affect border control and trafficking that may provide useful insights for DoD. METRICS AND EVALUATION Metrics are inputs to evaluation, not evaluations themselves or substitutes for evaluations. The committee cautions those who use program metrics to use them judiciously. Some people will always trust quantitative or numerical metrics over qualitative metrics in a belief that they are somehow more rigorous or objective. But numerical metrics are not necessarily objective and especially when taken by themselves can be misleading because different countries will have different baselines and different objectives. Likewise, it would be unfortunate if otherwise appropriate attention to metrics results in funds being taken out of activities that are useful and shifted into activities that are measurable. The WMD-PPP illustrates some of the caveats that must be kept in mind with quantitative metrics. In addition to the shortcomings to the “miles of border secured” metric described above, using a more nuanced quantitative evaluation of border security based on interdiction statistics requires interpretation beyond the numbers. A country that has poor control of its borders or poor ability to interdict illegal trafficking through its border crossings (either because of inability to detect illegal trafficking or because of corruption) may have statistics tracking its progress in securing its borders, but those statistics will require interpretation. Does a rise in interdictions indicate higher risk (more trafficking) or lower risk (more traffickers getting caught)? See Chapter 3 for a more general description of what metrics cannot do.

OCR for page 17