Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 17
2
Committee Assessment of the Department of Defense Cooperative
Threat Reduction Metrics Report
The first step in developing metrics is to establish what questions the metrics are meant to
answer, and how the metrics will be used. Throughout this study, in conversations with the
Department of Defense (DoD), other agencies, staff on Capitol Hill, and in the Executive Office
of the President, and with officials in partner countries, the committee asked how the metrics will
be used. The answers ranged based on the respondent, from those who sought measures of
progress toward agreed goals, to cost effectiveness, to others who sought measures of how well
the efforts align with the national defense mission. Given that different audiences want metrics
for different purposes, it makes sense to begin by describing the committee’s perspective on the
purposes of metrics.
Metrics are tools for evaluating the impact and effectiveness of programs and projects
against strategic goals and for management within the program. The former are program metrics.
The latter are project metrics. Different metrics may be appropriate for different stages of
program and project development, but they all need to be tied together by the strategic objectives
of the program as they fit with broader DoD Cooperative Threat Reduction objectives.
The committee evaluated the DoD CTR metrics described in the (CTR) DoD Metrics
Report based on whether the metrics provide decision makers with the essential information to
manage the effectiveness and impact of CTR programs. The committee drew on several
frameworks for program planning, performance, and evaluation, but especially on capability
based planning, to identify general steps needed for meaningful evaluation.15 Other frameworks
are also valid and will either explicitly or implicitly include these elements. DoD must
• state the objectives of the program and the project (i.e., the goals of the actual activities)
• identify the capabilities it is trying to develop or maintain
• define objectives for each capability and link those capabilities to metrics
• ensure that the metrics reflect program effectiveness and impact
• plan for and measure sustainment (see footnote 3 in the Summary)
It is generally good practice for the program to establish minimum performance levels and
aspirational goals for each metric (see footnote 4 in the Summary). In Table 2-1, the committee
15
A much more elaborate metrics framework is emerging from analysts focused on the international development
community. This highly structured approach treats the portfolio of projects as an experimental trial, with control
projects or communities and strict adherence to a protocol established at the outset of the program or project. The
committee chose a program evaluation framework more like the program planning, performance, and evaluation
approaches already used in DoD because the latter are more familiar to DoD and, in the committee’s view, better
suited to the CTR Program. The CTR Program engages in too few projects for meaningful control groups, and the
political conditions, physical and social circumstances, budgets, and objectives for the program change, all of which
make a trial-based approach incompatible with CTR today.
17
OCR for page 18
18 IMPROVING METRICS FOR THE DOD CTR PROGRAM
TABLE 2-1 Summary of metrics from the DoD Metrics Report
CTR Program What metrics are intended to measure Metrics
Examples of measures of effectivenessa
1. Secure and consolidate collections of especially dangerous
pathogens (EDP) and their associated research at a minimum Partner country EDP collections and
number of secure health and agricultural laboratories or associated research are consolidated into
related facilities. a minimum number of locations.
Partner country EDPs and associated
2. Enhance partner country/region’s capability to prevent the
research are secured in a manner
sale, theft, diversion, or accidental release of biological
consistent with standards.
weapons (BW)-related materials, technology, and expertise by
Partner country has BS&S laws and
improving biological safety and security (BS&S) standards
regulations governing work with EDPs.
and procedures.
Cooperative Biological
Partner country disease detection and
3. Enhance partner country/region’s capability to detect,
Engagement Program
diagnosis capability meets U.S. and/or
diagnose, and report endemic and epidemic, man-made or
(CBEP)
international guidelines for biosafety.
natural EDPs, bio-terror attacks, and potential pandemics.
Partner country has preparedness and
4. Ensure the developed capabilities are designed to be
response plans.
sustainable within each partner country/region’s current
Partner country disease surveillance system
operating budget.
is capable of detecting and reporting
5. Facilitate the engagement of partner country’s/regional
suspect EDP cases to those responsible
scientific and technical personnel in research areas of interest
for human and animal health.
to both the partner country/region and the United States.
6. Eliminate any BW-related infrastructure and technologies
encountered in a partner/country region.
Quantity of CW agent destroyed
Chemical Weapons Destruction of agent through the Russian Federation’s safe, Number of munitions destroyed
Elimination Program efficient operation of the destruction facility Project Metrics
(CWE) Scheduled facility downtime
Unscheduled facility downtime
Facility achieved availability
1. Performance/capability assurance Examples of metrics
Number of nuclear weapons storage
2. Configuration management
Nuclear Weapons Safety sites upgraded
3. Procedures and process
and Security (NWSS) Number of personnel trained
4. Training
supports Nuclear Weapons Establishment of regional technical
5. Organization and personnel
Transport Security Program centers
6. Life-cycle management
Note: Some metrics still under
7. Maintenance development
8. Logistics
1. Enhance partner country capability to perform effective risk Examples of metrics
management Miles of green (land) / blue (maritime)
border with added security
2. Enhance partner country capability to perform border
Number of ports of entry enhanced
security command, control, communications and computers
Number of border facilities (land and
3. Enhance partner country capability to perform border
maritime) provided increased
security surveillance
capability to detect WMD
4. Enhance partner country capability to perform WMD
Number of personnel trained and
detection
equipped to perform border security
5. Enhance partner country capability to perform border
interdiction
security interdiction
Initial and refresher training system
6. Assist partner country with the development of a
Weapons of Mass exists
sustainment budget for all systems delivered under this
Destruction Proliferation Number of personnel exchanges
program
Prevention Program (training, professional development)
7. Enhance partner country capability to support and maintain
(WMD-PPP) Mean time to negotiate agreements
delivered equipment and/or systems
Number of regional relationships
8. Enhance partner country capability to sustain delivered
facilitated
training
9. Enhance a partner country capability to capture and
disseminate information including that concerning WMD
incidents
10. Increase the awareness of partner countries as to their
critical role in WMD non-proliferation (partner country buy-
in)
11. Develop interagency, bilateral, regional, and multilateral
cooperation
a
The terms “goal,” “objective,” “indicator,” “attribute,” and “metric” are used inconsistently among the different program
descriptions in the DoD Metrics Report. CBEP uses the term “measure of effectiveness” to describe what is being measured and
the term “indicator” to denote how it is being measured (i.e., what is being counted).
OCR for page 19
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 19
attempts to summarize the disparate metrics from the DoD Metrics Report and Table 2-2
summarizes the committee’s assessments of DoD’s CTR Metrics Report with respect to the
elements described above. Because of differences among the programs, inconsistencies in their
terminologies, and the complexity of their metrics, the program metrics and indeed the
assessments are difficult to represent in a uniform tabular structure, so the committee commends
readers seeking deeper familiarity with the metrics to the DoD Metrics Report, which is
reproduced in full in Appendix B. The committee did not address the Strategic Offensive Arms
Elimination Program, which is not discussed in the DoD Metrics Report. DoD indicated that it
plans to use long-standing “Nunn-Lugar Scorecard” metrics for that program.
METRICS AND OBJECTIVES
In its introduction, the DoD Metrics Report states that “We can measure the amount of
equipment provided and the number of training events conducted or scientists engaged; however,
we need better measures to show that these efforts actually result in changed practices or
additional effectiveness.” The committee agrees and notes that the example given conflates
inputs and project metrics (those that measure progress on the means to meet objectives) and
outputs and programmatic or strategic metrics (those that measure progress relative to/toward the
objectives themselves).
Effectiveness and efficiency are important criteria for the evaluation of government
programs. A government program is effective if it achieves the objectives set forth by the
partners engaged in the program.16 A government program is efficient if it minimizes the
resources required to achieve its goals. A common approach is to manage and assess programs
based on their cost-effectiveness, which is an efficiency measure. Although measures of
effectiveness and impact are the most challenging and should continue to receive the most
attention, measures of efficiency and cost-effectiveness are also necessary and should be
included in future iterations of the CTR Program metrics.17
OVERALL ASSESSMENT
The DoD Metrics Report contains reasonable metrics for the CTR programs aimed at
consolidation and elimination of weapons and weapons materials, and contains a reasonable
starting point for developing metrics for the newer capacity-building programs.
16
It may not be possible to directly measure the higher-level outcomes from some CTR programs or projects, even
with the best metrics. For example, DoD may never know how many illegal shipments were not interdicted at a
border crossing where a CTR program provides assistance, or how many patients sick with an illness of interest did
not go to the hospital. Recognizing this fact at the outset will help to avoid wasted time and effort and avoid false
expectations.
17
While cost-effectiveness is important, particularly as a part of prioritization, it is essential that efficiency (which is
relatively easy to measure) not drive out effectiveness (which is more difficult to measure). In other words,
programs that are extremely cost-efficient may be pursuing the wrong result, or be solving the wrong problem.
Balancing cost-effectiveness against other measures is a necessary part of developing prioritized metrics.
OCR for page 20
20 IMPROVING METRICS FOR THE DOD CTR PROGRAM
Individual Program Objectives
The DoD Metrics Report describes both CTR’s highest-level objectives and DoD’s
difficulties in developing metrics, and it does so clearly and succinctly in the introductory section
of its report. However, the individual program objectives (those of the CWE, NWSS, CBEP and
WMD-PPP) and their connections to threats to U.S. national security are not well articulated or
not addressed. Each program has objectives, and they may be described well in other DoD
documents (see, e.g., Nacht, 2009), but the DoD Metrics Report itself needs to articulate the
objectives in a way that makes apparent the connection between the actions taken under the
Program and the higher level objectives. Not making those connections poses a problem for the
Program because proper metrics indicate progress toward objectives, or more specifically,
progress in developing the capabilities to fulfill the objectives. Those connections are the logical
starting point for the presentation and for the actual development and analysis of metrics. Again,
this is not to say that there is no connection, but that the DoD Metrics Report does not describe
the connection.
For example, describing the connection would clarify how building the capacity to better
track respiratory disease in East Africa reduces threats to U.S. national security or why Ukraine’s
interdiction of smuggled frozen chicken legs indicates a positive impact from a CTR program. A
strong case can be made for each of these, but not without stating the Program objectives and
connect them to program objectives.
For the programs that do not already have such objectives stated in the DoD Metrics
Report, a concise statement of the objectives of each program and how the actions planned under
the programs are intended to reduce threat or risk would provide a connection between the
objectives and the metrics.
Partnership
CTR programs are intended to be and work best when they are carried out as partnerships
(cooperative) with other countries. This includes joint development of the objectives with the
partner countries. Most of the metrics in the DoD Metrics Report do not reflect such joint
development. The report reads as if metrics are U.S. measurements of our partner’s progress
toward U.S. goals. This can create misunderstandings between the United States and our
partners, which can undermine CTR efforts.
Prioritization
DoD did not use a consistent framework for developing and articulating objectives,
priorities, and metrics, and DoD did not prioritize among its metrics. As a consequence, the
DoD Metrics Report mixes project management measures with higher-level program
performance metrics for some of the CTR programs, and weights equally metrics that are
critically important and others are not.
Cooperation
DoD plans to leverage other United States Government agencies’ experience,
capabilities, and assets as CTR expands to new countries and as it continues existing programs.
OCR for page 21
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 21
DoD also needs to communicate, coordinate, and cooperate with relevant agencies. Examination
of other U.S. Government documents suggests that CBEP is the most consistently planned and
coordinated across the other agencies.18 The committee notes that previous Academy reports and
government agencies themselves have stated the importance of such coordination for program
success, so a metric for it seems reasonable and valuable in theory. To the committee’s
knowledge, there is no metric to reflect interagency coordination or a “whole of government”
approach to working with a partner country, rather than a piecemeal approach. The committee
does not have a solution to this problem (see Chapter 3 for further discussion).
In addition to working with these other agencies, DoD CTR can learn something from
them. They, too, engage in capacity building programs, work on similar missions, and measure
performance. U.S. Customs and Border Protection conducts a mission similar to the PPP along
many thousands of kilometers of border and has developed metrics for its mission and
operations. The U.S. Department of Agriculture’s Animal and Plant Health Inspection Service is
a leader in an international surveillance network that has many parallels to the global network
DoD leadership envisions for reducing biothreats. The U.S. Agency for International
Development operates programs with partners across the world to foster democratic institutions.
As a further example, a partner’s compliance with the International Health Regulations would
reflect interests and activities of DoD, Health and Human Services, and Department of State,
and the extent of their coordination and cooperation in achieving that outcome is another metric.
Even other organizations within DoD could provide different models: The Defense
Security Cooperation Agency builds security capacities with partner nations. DoD’s CTR
Program can learn from these other agencies successes and challenges (and even its own
experience) if it incorporates a lessons-learned mechanism for the Program. Furthermore, the
CTR Program will be working with these other agencies in a whole-of-government effort, and
the others might already have mechanisms in place for measuring impact and effectiveness that
would be useful to CTR.
Change
The DoD Metrics Report deliberately does not consider future missions or changes in
objectives, but it is difficult to see how CTR metrics can be designed to respond to change if this
is not discussed in DoD’s report. Some factors are under the Program’s control. Far more are
outside of the Program’s control. Every responsible business, military operation, and government
program builds in resilience that is the ability to deal with exogenous change. The CTR programs
are susceptible to external change ranging from budget reductions to shifts in the host country’s
political or economic environment.
The practical consequences of some of the shortcomings listed above, such as not
articulating the connection to threat or risk, might not be large for projects in progress under
longstanding agreements. But they are important for new projects and especially for new
partnerships. A more detailed examination of the programs can be found in the next section.
18
See Weber, A. 2010.
OCR for page 22
22 IMPROVING METRICS FOR THE DOD CTR PROGRAM
PROGRAM-BY-PROGRAM ASSESSMENT
The committee has summarized several important aspects of its assessment of the DoD
Metrics Report in Table 2-2. The column headings represent a logical chain from objectives
through metrics to sustainability and the minimum performance required, read from left to right.
Chemical Weapons Elimination (CWE)
The treatment of the CWE program in the DoD Metrics Report may be adequate for the
current project under the program. The stated program goal looks very much like a project goal:
“[T]o destroy nerve agent-filled munitions located in the Planovy [chemical weapons] storage
facility in a safe, secure, and environmentally sound manner.” The United States and the Group
of Eight Global Partnership against the Spread of Weapons and Materials of Mass Destruction
(G8 Global Partnership) supported the design and construction of the chemical weapons
destruction facility at Shchuch’ye, Russia, which began operating in 2009. At Russia’s request,
the CWE program is providing technical support for the commissioning and operation of the
facility. So the desired capability to support the program and project objectives is defined as
adequate operation (by the Russians with U.S. technical support) of the Shchuch’ye destruction
facility.
In its Metrics Report, DoD refers to the quantities of chemical weapons agent destroyed
(neutralized and immobilized) as the program metrics. DoD’s only project level metrics are
scheduled facility downtime and unscheduled facility downtime.
These metrics align well with the “destroy nerve agent” aspect of the stated program
objectives, but the metrics only implicitly address the “safe, secure, and environmentally sound
manner” aspect of the objectives. Worker safety and environmental emissions are also important
indicators of proper operation of the facility, but are not mentioned in the DoD Metrics Report.
In a facility that has strong regulatory controls and/or strong internal management controls on
safety and environmental performance, problems with safety or environmental releases affect the
operation of the facility, i.e., the facility will shut down some or all of its operations to correct
problems. If the facility does not have strong controls in place, safety and environmental
performance might not affect continued operation of the facility, so tracking facility downtime
might not factor in safety and environmental performance. DoD is better positioned than the
committee to assess this aspect of the Shchuch’ye facility and can justify its decision either to
include such metrics or to exclude them. Likewise, the neutralization process is supposed to
convert 99.99 percent of the agent to slightly hazardous components and then immobilize them
for disposal. DoD does not explain the acceptable range of destruction percentage (is 90 percent
destruction acceptable?) and whether it is important to track that parameter.
Finally, the committee notes that the Planovy Chemical Weapons Depot near Shchuch’ye
contains approximately one seventh of the Russian stockpile of chemical weapons that awaits
destruction. If DoD hopes that the support provided at Shchuch’ye will not only enable the
Russian Federation to operate Shchuch’ye independently, but also affect progress at other depots,
then DoD should include a metric for that objective too.
OCR for page 23
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 23
TABLE 2-2. Program-by-program summary of objectives, capabilities, and metrics in the DoD Metrics Report. The
committee’s assessment (shaded) is followed by a summary of the committee’s advice
Objectives Desired Linkage of Do metrics Do metrics Are minimum
(program/ capabilities capabilities to measure address performance &
aspirational
project) are identified? metrics impacts and sustainability?
specified? effectiveness? goals specified?
Not in DoD Report;a Adequately Yes Yes – for the None Yes
Chemical
Some confusion operating operation of
Weapons
between program and facility Shchuch’ye
Elimination
project objectives.a Shchuch’ye
(CWE)
Assessment
N/A Consider This linkage Worker safety & This aspect could N/A
CWE
whether needs to be plant emissions be developed.
Advice
development established. may also be
at other appropriate &
facilities is important
needed. metrics.
Yesa
Yes/Yes Yes, for Yes No, only Yes
Nuclear
sustainmentb measure input.
Weapons
Safety &
Security
(NWSS)
Assessment
Clearly link overall N/A This linkage Although there N/A N/A
NWSS
objectives to threat should be made are constraints
Advice
reduction. more explicit, on measurement,
particularly as improved metrics
the program should have
expands to more focus on
other countries. impact &
effectiveness.
Yes/Yes Yes Yes Some, but not Yes Yes
Cooperative
prioritized.
Biological
Engagement
Program
(CBEP)
Assessment
Need to be more N/A Would benefit Need to be N/A N/A
CBEP
clearly linked to from linking prioritized,
Advice
threat reductiona capabilities reduced in
metrics with number, and
threat reduction. direct metrics,
where possible.
Yes/Yes Yes, but Program Program: no. Yes No
Weapons of
vague. metrics not Project: Some of
Mass
linked to them do, but not
Destruction
capabilities. prioritized, not
Proliferation
Project, yes clearly linked to
Prevention
program level.
Program
(WMD-PPP)
Assessment
Recognize links to Need to be Once capabilities More thought Sustainability The minimum
WMD-PPP
other related/similar clarified. are better should be given metrics also need performance and
Advice
programs; address articulated, they to prioritization to be prioritized aspirational
integrated assistance need to be better and impact and and focused on goals need to
efforts across the linked to the effectiveness. capabilities. be specified.
USG and other aid program and
providers. project metrics.
a
Not available in the DoD Metrics Report but available in other documents.
b
NWSS has completed upgrades at agreed facilities, so the current program is focused on providing sustainment.
OCR for page 24
24 IMPROVING METRICS FOR THE DOD CTR PROGRAM
Cooperative Biological Engagement Program (CBEP)
For the biosecurity mission, there are two main thrusts:19 (1) secure and consolidate
especially dangerous pathogens, and any work with those pathogens, to a safe, secure facility or
facilities; and (2) engage bioscience institutions20 and authorities in the partner country to
establish a culture of responsible practice, detection, and international reporting of emergent
pathogens and transfers of the pathogens. In principle, the first thrust is relatively easy to
measure, if the work with those pathogens is known to the governments. The second thrust, the
aim of which is in part to develop relationships of trust and practices that are trustworthy, is more
difficult for the United States to measure (see Measuring Trust, Confidence, and Goodwill in
Chapter 3). The metrics DoD developed for CBEP are numerous and mostly indirect with respect
to trust, but relatively concrete and measureable. Some additional steps are needed to make these
metrics effective tools for evaluating impact and effectiveness and managing the programs.
The DoD metrics for CBEP have nearly all of the elements that the committee thinks are
needed for development of useful metrics. The CBEP objectives and desired capabilities are
clearly stated, although their connection to threat reduction is not stated in the DoD Metrics
Report. The metrics are linked to the desired partner country capabilities, and some of them
measure impacts and effectiveness, although others do not.21 DoD also factors sustainability into
its rating of the program’s performance.
The greatest shortcoming of this otherwise rather complete metrics structure is its lack of
prioritization. DoD lists 49 metrics for CBEP and in its first application of the metrics, DoD
weighted all of them equally. It is difficult to draw meaningful conclusions about overall
progress from 49 metrics that are all weighted as equally important. Some of the metrics are far
more important to DoD and to threat reduction than are others (see, e.g., Footnote 7 in this
chapter), and not all 49 are needed, even after duplicated metrics are eliminated. The committee
discusses metrics for the CBEP more extensively in Chapter 3, where the committee illustrates
how an improved method for developing, prioritizing, and using metrics should be applied to
CTR programs.
19
This is a summary of the four objectives DoD describes in its Metrics Report:
1. Secure and consolidate collections of especially dangerous pathogens (EDPs) and their
associated research at a minimum number of secure health and agricultural laboratories or
related facilities;
2. Enhance partner country/region’s capability to prevent the sale, theft, diversion, or accidental
release of biological weapons related materials, technology, and expertise by improving
biological safety and security standards and procedures;
3. Enhance partner country/region’s capability to detect, diagnose, and report endemic and
epidemic, man-made or natural EDPs, bio-terror attacks, and potential pandemics; and
4. Ensure the developed capabilities are designed to be sustainable within each partner
country/region’s current operating budget.
20
It seems reasonable to enhance this priority to include engaging bioscientists, not just bioscience institutions.
21
For example, one of the measures of effectiveness concerns the partner country laws and regulations for biological
safety and security, and their conformance to U.S. standards. While the U.S. implementers may believe that having
such laws and regulations is better than not having them, the actual practices are more important than the details of
the laws. The practices may be poor despite good laws and regulations or good despite poor laws and regulations. As
a result, a partner that otherwise has shown significant improvements with respect to biological safety and security
might get a poor rating based on a regulatory system that is different from that in the United States.
OCR for page 25
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 25
Nuclear Weapons Safety and Security (NWSS)
The NWSS section of the DoD Metrics Report gives candid assessments of the metrics it
has developed (e.g., “Depending on the level of cooperation, DoD may not be able to track this
metric independently.”), and overall gives the impression of a metrics effort in development.
This reflects the current state of the NWSS program, but it also gives an incomplete picture of
DoD’s thinking about measuring the impact and effectiveness of this program.
NWSS has completed upgrades at all of the facilities covered by the agreement between
the United States and the Russian Federation, and DoD plans to continue to use the number of
nuclear weapons storage sites upgraded as a metric of the impact and effectiveness of the
program in Russia. Given that the upgrades are done, the current program is focused on (1)
providing sustainment in the form of training, and on (2) the Nuclear Weapons Transportation
Security Program. The DoD Metrics Report also states that “with the potential global expansion
of the NWSS program to other partner countries, Defense Threat Reduction Agency DTRA has
been working to develop enhanced metrics that better reflect how our efforts are contributing to
overall threat reduction.”
The current NWSS program has severe constraints on measurements. The CTR program
trains people who train the operators of the nuclear weapons storage sites, but the CTR training
takes place at facilities away from the nuclear weapons storage sites, and CTR personnel do not
have access to any of the sites. CTR supports supply of upgraded secure railcars for the
transportation mission, but CTR personnel do not see them in operation. In both cases, the lack
of access makes it difficult or impossible to measure the outcomes directly linked to the
program’s objectives.
Recognizing these constraints, DoD has identified the objectives of the program and the
capabilities desired to achieve those objectives. The DoD Metrics Report describes what DoD
would like to measure and what might actually be possible. For example, DoD will track the
number of railcar shipments supported and the number of secure railcars provided. DoD can
measure the latter itself; it cannot measure the former. DoD included some metrics in its report
because they were negotiated among DoD, Department of Energy, and the partner, Russia’s
Ministry of Defense. Such mutually agreed metrics are important and notable, as discussed
elsewhere in this report.
DoD presented additional information and ideas to the committee in November 2010.22
These ideas include a systematic method for weighting or prioritizing among the lower level
factors that contribute to the metrics. Without endorsing the specific details presented to the
committee, the committee agrees that DoD would benefit from further development of ideas like
those in the November presentation.
Weapons of Mass Destruction Proliferation Prevention Program (WMD-PPP)
The WMD-PPP focuses on land and maritime border security, also known as green and
blue borders respectively, in partner nations including Azerbaijan and Ukraine. Three program
metrics are defined:
• Miles of green (land) border provided sustainable security enhancements (in
progress/complete)
22
See Appendix E.
OCR for page 26
26 IMPROVING METRICS FOR THE DOD CTR PROGRAM
• Miles of blue (maritime) border provided sustainable security enhancements (in
progress/complete)
• Number of ports of entry provided sustainable capability enhancements (in
progress/complete)
In introducing these metrics, the DoD Metrics Report notes: “The ability to measure simply and
objectively the impact that WMD-PPP assistance has had on threat reduction is challenging due
to the nature of the program: we are providing a capability to our partners that gives them an
ability to deter proliferation.”
DoD articulates both program and project objectives for the WMD-PPP. The difficulty
for WMD-PPP program managers and decision makers results from the vagueness of the
capabilities desired to achieve the goals or objectives. DoD clearly found it difficult to translate
the desired capabilities into metrics. Miles of green and blue border with additional security
enhancements, and the number of ports of entry where capabilities have been enhanced are ill
defined. Are all enhancements equivalent? Is every kilometer of border equivalent in terms of
threat reduction?
Securing a border is complicated. The first approximation of progress—the number of
kilometers of green border deemed secure—might not accurately represent risk reduction for
several reasons. First, not all kilometers of border represent equal risks. Second, adversaries can
adapt to exploit the weakest link. Third, what constitutes a secure kilometer of border? Areas
where security enhancements have been provided might still have known smuggling routes that
regularly defeat security enhancements may be improperly counted in success measures. These
challenges are well known to DTRA, and in implementing WMD-PPP, DTRA has not treated all
borders as if they were equally important. For instance, some stretches of border may be
sufficiently impassable due to the natural topography of the area as to render any additional
security enhancement unnecessary while others may include commonly used smuggling routes.
Metrics should instead focus on the risk posed and the likelihood of evading detection. In
Ukraine, both DTRA and the Ukrainian State Border Guard Service conducted a simple risk
ranking and agreed jointly on the highest risk stretch of border. They focused their prototype and
demonstration facility efforts on that stretch of border. Thus the implementation is better than the
metrics would suggest.
Interdiction events involving materials useful for WMD are rare and experts are
cautiously optimistic that this is because smuggling of WMD material is rare, not just difficult to
observe. Border protection services do not now have, and endeavor to never have, enough WMD
incidents to make tracking them a useful statistical tool, so they use statistics from other border
violations as proxies to indicate the effectiveness of their border controls. The definition of
positive outcome must be sufficiently broad to prevent unintentionally causing a reduction of
inspections or alarms that might prevent the detection of an illicit item. For instance, if the risk
management device is using rules based targeting efforts or non-intrusive inspection equipment,
identifying a shipment that comes from a legitimately high-risk source or known bad actor might
be a positive outcome, even if the resulting inspection did not uncover illicit material. Similarly,
the physical inspection of a container that uncovers a significant quantity of lead or similarly
dense material may be a positive outcome to imaging analysis of a container x-ray that shows a
density anomaly. Noting those inspections as negative outcomes may result in the overly
narrowing of risk factors, thereby increasing the risk of missing an actual smuggling event.
OCR for page 27
COMMITTEE ASSESSMENT OF THE DOD CTR METRICS REPORT 27
Progress in securing a border can be measured statistically (with some degree of
accuracy, Chapter 3) if the country has sufficient control or awareness to even estimate the
violations. Border protection services can track radiation alarms and smuggling incidents
involving drugs, firearms, people, stolen goods, and goods avoiding tariffs.
It may be that partner countries do not need the ability to perform maritime interdictions
themselves. Instead, one might measure the response time to conduct a maritime interdiction,
understanding that another country’s team or a regional/multi-national team may be best situated
and equipped to respond. Measuring the time for interdiction might more accurately measure the
objective. For instance, the coastline of Georgia may be such that it needs fewer boarding teams
than required to respond adequately to the much larger coastline of Ukraine. Another option
might be to take the numerator/denominator approach, measuring the current capability against
an assessed need.
At a project level, the metrics in the DoD Metrics Report are better linked to the
capabilities desired. Some of the project metrics DoD developed for the WMD-PPP measure
impacts and effectiveness, such as alerts resulting in a positive outcome, and others do not, such
as the number of maintenance personnel trained. There are also a number of other programs of
the United States and other countries that provide border control and law enforcement assistance
that affect border control and trafficking that may provide useful insights for DoD.
METRICS AND EVALUATION
Metrics are inputs to evaluation, not evaluations themselves or substitutes for evaluations.
The committee cautions those who use program metrics to use them judiciously. Some people
will always trust quantitative or numerical metrics over qualitative metrics in a belief that they
are somehow more rigorous or objective. But numerical metrics are not necessarily objective and
especially when taken by themselves can be misleading because different countries will have
different baselines and different objectives. Likewise, it would be unfortunate if otherwise
appropriate attention to metrics results in funds being taken out of activities that are useful and
shifted into activities that are measurable.
The WMD-PPP illustrates some of the caveats that must be kept in mind with
quantitative metrics. In addition to the shortcomings to the “miles of border secured” metric
described above, using a more nuanced quantitative evaluation of border security based on
interdiction statistics requires interpretation beyond the numbers. A country that has poor control
of its borders or poor ability to interdict illegal trafficking through its border crossings (either
because of inability to detect illegal trafficking or because of corruption) may have statistics
tracking its progress in securing its borders, but those statistics will require interpretation. Does a
rise in interdictions indicate higher risk (more trafficking) or lower risk (more traffickers getting
caught)? See Chapter 3 for a more general description of what metrics cannot do.
OCR for page 28
