National Academies Press: OpenBook

Improving Metrics for the Department of Defense Cooperative Threat Reduction Program (2012)

Chapter: Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report

« Previous: Appendix A: Congressional Mandate
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

APPENDIX B

REPRODUCTION OF THE DEPARTMENT OF DEFENSE COOPERATIVE THREAT REDUCTION METRICS REPORT

Delivered by Department of Defense to Congress

September 2010

Introduction

This report is submitted in accordance with Section 1304 of the National Defense Authorization Act for Fiscal Year 2010, Title XIII of Public Law 111-84. Section 1304 provides that the Secretary of Defense shall develop and implement metrics to measure the impact and effectiveness of activities of the Department of Defense (DoD) Nunn-Lugar Cooperative Threat Reduction (CTR) Program. The metrics to be developed would measure the effectiveness of projects and activities undertaken by DoD to prevent the proliferation of chemical, nuclear, and biological weapons and weapons of mass destruction (WMD)-related materials, technologies, and expertise.

Background

In the National Academy of Sciences’ (NAS) 2009 report, Global Security Engagement: A New Model for Cooperative Threat Reduction, NAS affirmed the usefulness of the traditional metrics of the DoD CTR Program for evaluating effectiveness in traditional CTR WMD elimination or site security program areas, in which concrete actions such as chemical munitions, warhead delivery systems, and launchers destroyed or chemical or nuclear warheads and weapons materials secured clearly lead to threat reduction. Such activities break down into discrete events with clear end points, and thus metrics that can be incrementally reported, as on the CTR Scorecard (Appendix 1), continue to be appropriate for those types of programs. However, the NAS report accurately noted that CTR program responses to new, often less quantifiable threats in this century will require new methods to measure success. The challenge is to find measureable performance indicators that capture the true value of program activities that may not lend themselves readily to raw quantitative measurement, thus making it more difficult to document the relative success of the activity.

In recent years, the CTR program has increasingly become involved in capacity-building efforts, notably in the Cooperative Biological Engagement Program (CBEP), formerly the Biological Threat Reduction Program (BTRP), and the WMD Proliferation Prevention Program (WMD-PPP). In these programs, threat reduction is accomplished through building the capacity of the partner state to conduct surveillance to deter or detect and report or respond to emerging biological and WMD trafficking threats. Similarly, the CTR Program is building the capacity of the Russian Federation to sustain upgraded physical protection systems at nuclear weapons

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

storage sites. Although increased partner state capacity makes an important contribution to a reduction in the WMD threat to the United States, this is more difficult to quantify than numbers of missiles or other forms of strategic offensive arms destroyed as in the traditional programs.

CTR’s prevention mission is difficult to quantify. Although a number of output measures can be counted, they are proxies for the outcome of deterring terrorists and proliferators from accessing WMD and related materials and expertise. We can measure the amount of equipment provided and the number of training events conducted or scientists engaged; however, we need better measures to show that these efforts actually result in changed practices or additional effectiveness.

As an example that makes evident the findings in the NAS report and reflects a matter DoD has recognized in connection with the CTR Program, a metric based on the number of Zonal Diagnostic Laboratories built under the BTRP may not accurately reflect the partner country’s enhanced capability for pathogen detection and reporting. Indeed, such a metric could create unintended incentives to continue building additional laboratories that do not make a positive contribution to threat reduction, when what is really needed might be additional training to make better use of the existing laboratories’ capabilities for pathogen detection. In WMD-PPP, the raw number of radars, sensors, and patrol boats provided for border security, although useful, may not be as important as the effectiveness of the information sharing among those systems, which is a function of system integration, training and effective decision making rather than of the number of systems provided.

In light of these issues, Congress has mandated the development and implementation of metrics to measure the impact and effectiveness of projects and activities of the CTR program to address threats arising from the proliferation of chemical, nuclear, and biological weapons and WMD-related materials, technologies, and expertise. Additionally, Section 1304 requires submission, no later than 270 days after the enactment of the Act, of a report describing the metrics developed and implemented. This report addresses these Section 1304 requirements.

Discussion

The metrics described in this report are designed to measure appropriately the impact of each CTR program area, as the CTR program is currently constituted for Fiscal Year 2010, and does not attempt to speculate on what metrics might be appropriate for future areas of programmatic or geographic expansion. Neither do the metrics attempt to determine whether the activities of the CTR program are the “right” activities. The metrics described in this report are intended to best measure the effectiveness of the CTR Program in conducting these established programs. These metrics are not intended to revise the method for establishing these objectives.

Figure 1 illustrates the organization of the Department of Defense as it relates to the CTR program. CTR executes programs as directed in program guidance from the Deputy Assistant to the Secretary of Defense (Treaties and Threat Reduction) (DATSD(T&TR)). This program guidance provides instruction on how to implements policy guidance issued by the Office of the Under Secretary of Defense for Policy (USD/P), which has been coordinated with all relevant agencies within the Executive Branch.

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Those responsible for each CTR program area, in concert with applicable stakeholders, reviewed their specific program objectives and either established new program metrics or confirmed the validity of existing program metrics. Although not culled out specifically in this report, except as noted in the CTR Scorecard (Appendix 1), the metrics currently in use for the Strategic Offensive Arms Elimination (SOAE) program will not change. The results of this review and analysis process are provided in this report. Each report section provides a brief program description, methodology for metrics development, and a description of metrics that includes reporting and re-evaluation activities.

Conclusion

CTR program metrics serve as a means to evaluate how well program objectives have been accomplished. Several traditional CTR program metrics are still very applicable and useful measures of CTR Program threat reduction achievements. For those programs involved in capacity-building efforts, notably the CBEP, basic metrics need to be specifically tailored and evaluated individually at a project level. A number of program benefits that would be useful and meaningful in the larger USG threat reduction effort and are routinely considered in establishing execution plans, such as partner country contributions and commitments, or ability to leverage multiple sources of expertise or funding, were considered but not recommended since these benefits ultimately do not trace back directly to the CTR program objectives. As the CTR program continues to evolve, we will continue to assess its metrics to ensure they best represent achievement of new objectives.

Figure 1. The CTR program within the DoD organizational structure

image

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Cooperative Biological Engagement Program

Program Description

The Cooperative Biological Engagement Program (CBEP) has four main objectives that are identified in the current policy and implementation guidance issued to the Defense Threat Reduction Agency (DTRA) for implementation:

1.   Secure and consolidate collections of especially dangerous pathogens (EDPs) and their associated research at a minimum number of secure health and agricultural laboratories or related facilities;

2.   Enhance partner country/region’s capability to prevent the sale, theft, diversion, or accidental release of biological weapons (BW)-related materials, technology, and expertise by improving biological safety and security (BS&S) standards and procedures;

3.   Enhance partner country/region’s capability to detect, diagnose, and report endemic and epidemic, man-made or natural EDPs, bio-terror attacks, and potential pandemics; and

4.   Ensure the developed capabilities are designed to be sustainable within each partner country/region’s current operating budget.

The fourth objective identified above, sustainability, is a cross-cutting objective to ensure that the capabilities provided by the CBEP are adapted and owned by the partner country. The ultimate aim for CBEP implementation is that the partner country take ownership of a sustainable capacity to conduct biosurveillance, meet BS&S standards, and conduct research effectively on its own. Promoting partner country ownership is one of the critical program objectives and one of the most significant CBEP challenges.

The CBEP’s supplemental objectives – objectives noted as supplemental in the guidance received – include facilitating the engagement of partner country’s/regional scientific and technical personnel in research areas of interest to both the partner country/region and the United States and eliminating any BW infrastructure and technologies encountered in a partner country/region. CBEP assistance also supports host country objectives in meeting the World Health Organization (WHO) International Health Regulations (IHR).

CBEP implementation to meet these objectives includes collaborative biological research (CBR), biosurveillance, and enhancement of BS&S. CBR efforts bring together U.S. and international scientists in cooperative research of mutual interest in support of threat reduction priorities. Biosurveillance projects include providing training to animal and human health experts on disease recognition, diagnosis, and reporting; support for laboratory diagnostic equipment and reagents; and implementation of the Electronic Integrated Disease Surveillance System (EIDSS). BS&S enhancements include bioethics training, training on safe clinical and laboratory practices; installation of security systems at laboratories; implementation of the Pathogen Asset Control System (PACS) to track inventory; and support for development of biorisk management processes and procedures.

Methodology for Development of New Metrics

Developing metrics for the CBEP presented several challenges. First, the program seeks to achieve concrete improvements in disease detection, biological safety, and research systems in an environment not fully under its control, with the result that accountability and analysis of

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

program impact are complex, difficult matters. Also, the metrics for the CBEP need to establish a solid, traceable link from CBEP-funded efforts to concrete threat reduction achievements.

To address these challenges and establish actionable metrics, the CBEP adapted a Joint Capabilities Integration and Development System-like process to develop mission and performance measures. This effort took place over the course of one year of collaborative development. The CBEP metrics team first developed an initial capabilities document that lays out the key capabilities required in the program’s strategic guidance and the attributes supporting these capabilities. From these attributes, the team then developed measures of effectiveness (MOEs) to assess mission accomplishment. These MOEs are directly traceable to the program’s policy and implementation guidance. Finally, the team developed measurement values to serve as indicators that the critical MOEs have been achieved. The objective and minimum thresholds for these measurement values indicate the desired end state and the minimally acceptable level of achievement respectively.

Throughout the process, the CBEP engaged program stakeholders from across disciplines as diverse as clinical practice, epidemiology, laboratory, biosafety, veterinary, biosecurity, systems engineering, and training. The use of small working groups by discipline and progressive reviews by outside parties and program leadership enhanced the accuracy, comprehensiveness, and usefulness of the final product.

Description of New Metrics

The table in Appendix 2 depicts the linkages and associations across 6 program objectives, 5 program capabilities, 10 attributes, 20 MOEs, measurement values, and minimum/objective values.

The CBEP will utilize these revised metrics throughout planning, implementation, and final assessment of country/region engagements.

Using a systems engineering approach, all project officers will use metrics to plan and execute work, including designing a concept of operations to support MOEs in a partner country or region. The project implementation team will focus on achieving a capability in a partner country or region as defined by the MOEs.

Interim performance against the metrics will be assessed through a seamless, collaborative test and evaluation process that will include tabletop and field training exercises, as well as informal assessments by CBEP collaborators. In addition, ongoing proficiency and competency testing will assess capability levels, target areas for improvement, and build capacity on the micro level. Results will also be fed back to the implementation teams, allowing them to make adjustments to their execution plans. Feedback from CBEP collaborators is a part of each engagement activity in each country or region, and the program holds an integrated country review at least every quarter of the year to identify issues, necessary changes in approach, action items, and those responsible for taking action. This iterative, ongoing assessment approach helps to ensure that enhancements are taking place and that the implementation approach is delivering results.

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

In conducting formal, large-scale assessments of program achievements relative to the metrics, the CBEP will leverage international authorities such as the WHO, the Food and Agriculture Organization of the United Nations (FAO), or the World Organization for Animal Health (OIE) to seek third-party validation of program accomplishments. These formal reviews will assess the cumulative results of CBEP engagement and identify areas for continuing cooperation.

All of the above assessment efforts will document performance relative to the 20 MOEs identified in Appendix 2. The use of multiple metrics allows for a thorough, broad evaluation of program performance; however, it makes providing a succinct, top-level report of performance a challenge. The CBEP plans to report the cumulative performance of each country against the comprehensive metrics on an annual basis. This summary report would offer an overview of each country’s “proficiency” relative to the CBEP objectives. Proficiency could be indicated in some manner on a summary graphic; however, the complexity of the program does not lend itself to simplification into a single scorecard metric.

Chemical Weapons Elimination

Program description

The original program task for the Chemical Weapons Elimination program was to assist the Russian Federation in establishing a chemical weapons destruction facility (CWDF) near Shchuch’ye, located in the Kurgan District, to destroy nerve agent-filled munitions located in the Planovy CW storage facility in a safe, secure, and environmentally sound manner. The assistance provided included the following:

•   Site clearing, water drainage, and preparation of the land for construction;

•   Evaluating, optimizing, and scaling-up of the Russian-developed nerve agent destruction process;

•   Developing, designing, fabricating, and testing of the munitions processing equipment;

•   Integrating and systemizing the processing and associated support equipment and facilities; and

•   Commissioning the facilities as an acceptable industrial complex under Russian Federation standards and laws.

The United States and other members of the Group of Eight Global Partnership against the Spread of Weapons and Materials of Mass Destruction assisted with completion of this facility.

As construction of the first destruction building and other parts of the complex neared completion, the Russian Federation requested continued U.S. technical assistance to monitor and evaluate the agent destruction system and support equipment maintenance through the remaining agent elimination efforts as a means of assuring continued full functionality of the systems and equipment. The Russian Federation initiated destruction in the first building in March 2009. The second destruction building is still under construction and is scheduled to be complete and in operation in 2011.

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Program-Level Metrics

Methodology for Development of New Program-Level Metrics

Technical assistance will primarily take the form of engineering advice to help resolve process and equipment failures, identify potential future failures, and ensure replacement spares are on hand. Quantifying how this advice/assistance provided by CTR contributes to the overall availability of the facility to perform its agent destruction mission is the challenge. Two quantifiable factors that can be identified and tracked from a strategic perspective are the amount of funding invested in providing technical assistance and the amount of agent and munitions destroyed by the facility.

Although it can be quantified, tracking funds expended on technical assistance provides no clear measure of whether the technical assistance actually is contributing to the facility’s ability to destroy nerve agent. Costs are incurred whether or not any deficiency is successfully identified, isolated, and corrected. However, the number of munitions processed and the tonnage of nerve agent destroyed does provide a valuable and useful link to determining how successful the provided technical assistance has been toward keeping the facility functioning at capacity. Progress in the destruction of agents and processing of munitions is clearly indicative of continued facility functionality and operation. Therefore, we determined that using the amount of agent destroyed and tracking the number of munitions processed would be the best metrics to measure the success of the technical support presently being provided.

Description of New Program-Level Metrics

As nerve agent-filled munitions are brought to the facility for processing, the weight of nerve agent extracted and neutralized is determined and the number of individual munitions bodies processed is captured by the plant’s automated process control system. The Russian Federation’s operations staff consolidates and reports the two quantities, and the U.S. on-site technical staff obtains the data from the Russian Federation and maintains a cumulative total of both data points. The on-site staff then provides the information weekly to the U.S. Project Management Office.

Table 1 below provides an example of the metrics provided in the weekly report:

Table 1. CWE Metrics

Report Date Total Rounds Total Metric Tons
August 3, 2010 532,806 1461.8

However, since the number of munitions (total rounds) correlates closely to the amount of nerve agent destroyed, for the purposes of reporting externally in the “CTR scorecard” only the metric tons of agent destroyed will be reported. (Appendix 1),

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Project-Level Metrics

Additional metrics will be maintained at a project level to assess how effectively the destruction facility is being used in relation to performance of maintenance and repairs that help ensure the safety of the destruction process.

Methodology for Development of New Project-Level Metrics

Reliability, Availability, and Maintainability (RAM) Analysis

When fully constructed, the Shchuch’ye CWDF will consist of two main munitions destruction buildings, with two destruction process lines (DPLs) operating in each building. A RAM analysis to establish the baseline threshold and objective capacities of the facility in this configuration was completed in August 2005. That analysis determined that the design capacity threshold for the completed facility is 1,000 metric tons of agent destruction per year, with a design capacity objective of 1,700 metric tons of agent destruction per year (capacity for two main destruction buildings with two DPLs operating in each).

Description of new project-level metrics:

Scheduled Facility Downtime

Scheduled Facility Downtime (SFD) tracks any days when it is planned that the facility will be idle and not processing munitions. This can be due to scheduled maintenance, campaign changeovers, or any administrative downtime where the idling of the process was planned in advance. In the RAM analysis, the SFD was estimated to average 25 percent. For a facility of this complexity, with its associated preventive maintenance requirements both for the destruction process and all of the support systems, such an estimate is reasonable.

Minimizing down days, whether scheduled or unscheduled, is extremely important. Using scheduled downtime efficiently and effectively by scheduling periodically required maintenance actions for those periods can maintain overall production capabilities, as well as the safety of the facility. U.S. technical support efforts help the Russian Federation to plan and optimize scheduled maintenance outages and helps ensure that the scheduled outage does not extend beyond the originally scheduled duration by having the right parts, equipment and vendor support pre-staged to take full advantage of the outage.

Since the start of munitions destruction, with only one of the two destruction buildings in operation, the cumulative SFD has been 19.25%. A rolling 12-month average helps identify potential trends that might get masked over the long term. The Shchuch’ye CWDF’s current rolling 12-month SFD average is 26.85%. This higher percentage is attributable to the recent scheduled changeover from one type of munition to another.

In terms of the objectives and thresholds for agent tons per year, an objective SFD average of 20 percent per year would support meeting the 1,700 metric tons of agent per year average with two

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

main production facilities operational. Similarly, an SFD average of 35 percent would result in meeting the threshold average of 1,000 metric tons per year.

Unscheduled Facility Downtime

A second key area for assessing overall throughput is Unscheduled Facility Downtime (UFD). Efficient and effective technical support helps minimize the number of UFD days. This support encompasses proactive failure root-cause analysis, analysis of spare parts consumption, provisioning of the right amount and types of spare parts, and the selective replacement of items before failure during scheduled maintenance outages.

Objective and threshold limits for this metric are being developed.

The Shchuch’ye CWDF’s current overall cumulative UFD percentage is 6.68%, with a 12-month rolling average of 4.66%. The UFD for the Shchuch’ye CWDF is significantly lower than would be expected for a complex CWDF, based upon experience with similar facilities in the U.S. program, especially during the initial year of operation.

Facility Achieved Availability

The Achieved Availability metric encompasses the overall throughput of the facility achieved versus the actual design capacity of the facility. During the current phase of operations, it is based upon the one completed main destruction building (overall capacity based upon two DPLs). The achieved availability with one main destruction building is a good indicator of what could be expected with two main destruction buildings in terms of meeting the overall objective of 1,700 metric tons per year for a facility operating with two main destruction facilities.

Based upon the RAM analysis, the estimated Achieved Availability was 35.4 percent, which would result in an overall annual throughput rate of 1,382.5 metric tons for the facility with two main destruction buildings in operation with two operational DPLs in each. In order to achieve the objective of 1,700 metric tons per year, an overall Achieved Availability of approximately 44 percent would be required. Similarly, it is estimated that the 1,000 metric tons per year threshold would be met with an Achieved Availability of 26 percent. To date, Building 101A has an Achieved Availability of 54 percent, much higher than originally anticipated.

The Achieved Availability calculation takes into account Scheduled and Unscheduled Facility Downtimes. Based on operations to date, these two components account for almost 26 percent of the nearly 45 percent of lost production time for Building 101A associated with an Achieved Availability of 54 percent. The remaining 19 percent of downtime is primarily due to unplanned and planned maintenance of one or the other of the two DPLs while one remains in operation.

Summary

We are using three project-level metrics to track and evaluate the trend of the effectiveness of the facility and the provided technical support. Based upon an analysis of available RAM reports,

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

the following are the project-level metrics, with associated objective and threshold values, as of June 30, 2010:

Scheduled Facility Downtime (SFD):

Objective: 20 percent (RAM analysis rate of 1,700 metric tons/year)

Threshold: 35 percent (RAM analysis rate of 1,000 metric tons/year)

Cumulative SFD: 19.25 percent

Last 12 Months SFD: 26.85 percent

Unscheduled Facility Downtime (UFD):

Cumulative UFD: 6.7 percent

Last 12 Months UFD: 4.7 percent

Facility Achieved Availability (FAA):

Objective: 44 percent

Threshold: 26 percent

Cumulative FAA: 53.97 percent

Last 12 Months FAA: 52.51 percent

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Nuclear Weapons Safety and Security

Program Description

This program supports proliferation prevention by enhancing the security systems of nuclear weapons storage sites using DoD nuclear security standards as a basis for design. It also trains cadres of security systems operators, administrators, and maintenance and repair technicians and provides capabilities, such as the personnel reliability program and guard force training systems. In response to a request from Russia and former President Bush’s commitment at Bratislava, Slovakia in February 2005, DoD and the Department of Energy (DOE) enhanced security systems at requested locations that permanently or temporarily store strategic and non-strategic (tactical) nuclear weapons. DoD upgraded a total of 24 sites and, additionally, supports proliferation prevention by enhancing the security and safety of nuclear weapons during shipment through the Nuclear Weapons Transportation Security (NWTS) Program.

Methodology for Development of New Metrics

Nuclear Weapons Safety and Security (NWS&S) projects have been focused on efforts exclusively in Russia, and the metric used to measure program success reported the number of nuclear weapons storage sites upgraded. This remains an appropriate measure of the impact and effectiveness of program efforts in Russia and, therefore, this information will continue to be reported on the CTR Scorecard (Appendix 1) in its present format.

Recently, however, with the potential global expansion of the Nuclear Weapons Storage Security (NWSS) program to other partner countries, DTRA has been working to develop enhanced metrics that better reflect how our efforts are contributing to overall threat reduction. These new metrics are aimed at measuring the capability of Russia (and in the future, other countries) to sustain the ability to store and transport nuclear materials safely and securely. The development of these metrics presents a substantial challenge, as we need to develop measures that accurately and correctly reflect an intangible capability. DoD and DOE have been working together on this issue and have jointly developed eight principles to evaluate sustainment capabilities. Although these principles are presently used as internal measures of effectiveness and are still being refined, the NWSS program has already begun to apply these principles to the sustainment of the site security projects in Russia.

Description of New Metrics

DoD and DOE jointly developed eight sustainment and logistics principles for the Russia NWSS projects, and proposed metrics would follow these standards. These metrics also meet other international standards, including those used by the International Atomic Energy Agency (IAEA) for logistics and maintenance management of nuclear facilities. The metrics include:

•   Performance/Capability Assurance,

•   Configuration Management,

•   Procedures and Processes,

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

•   Training,

•   Organization and Personnel,

•   Life Cycle Management,

•   Maintenance, and

•   Logistics.

These principles/goals are further defined in Table 3 below with the associated objectives and the metrics being tracked. Further development of the methodology is planned jointly with DOE to clarify roles and responsibilities more completely. These metrics would be provided as an appendix to the CTR Scorecard

Note: NWTS program metrics will remain the same, as these continue to track the threat reduction impact of this project. The number of railcar shipments will continue to be included in the scorecard and will measure the number of nuclear weapon transport shipments supported annually. An additional metric will be added to the CTR Scorecard to reflect the number of new nuclear weapons guard and cargo railcars with enhanced security systems and off-train communications provided.

TABLE 3.1 NWSS Goals, Objectives, and Metrics

Goals Objectives Metrics

1.0 Performance/Capability Assurance

1.1 Partner Country institutes an assessment program, including vulnerability assessments and system readiness tests, to ensure site security effectiveness is sustained.

•   Metrics to fulfill this goal are still being determined based on current sustainment activities. This goal is challenging, as it relates to a host country’s ability to self-assess its performance in the below areas. Depending on the level of cooperation, DoD may not be able to track this metric independently. The metric is included, however, as it has been shared with and agreed to by the host country to assist it in taking over sustainment activities once U.S. assistance stops.

2.0 Configuration Management

2.1 Partner Country institutes a program to maintain configuration control for all site security elements.

•   Functional technical library exists.

•   Equipment lists and design specifications by site are maintained.

3.0 Procedures and Process

3.1 Partner Country institutes systems, processes and procedures to ensure site security effectiveness is sustained.

•   Metrics to fulfill this goal are still being determined based on current sustainment activities. This goal is challenging, as it relates to a host country’s internal procedures and processes. Depending on the level of cooperation, DoD may not be able to track this metric independently. The metric is included, however, as it has been shared with and agreed to by the host country to assist it in taking over sustainment activities once U.S. assistance stops.

4.0 Training

4.1 Partner Country institutes a training program and delivers the majority of training necessary through an organic capability to operate and maintain security elements.

•   Number of personnel trained

o   Operators

o   Administrators

o   Maintainers

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Goals Objectives Metrics

•   Establishment or enhancement of regional technical and training centers

•   Functional technical library exists.

5.0 Organization and Personnel

5.1 Partner Country staffs sustainment centers and sites with personnel qualified to sustain operations and maintenance of security elements.

•   Quantity of trained Physical Protection System Operators, Administrators and Maintainers/Repair Technicians

6.0 Life Cycle Management

6.1 Partner Country regularly assesses system reliability, availability and maintainability of security elements, using usage and failure data to plan and implement improvements and upgrades to sustain security effectiveness.

•   Failure data is tracked in order to determine trends in equipment life cycle.

•   Partner country has spares and consumables available and a preventive management system in place.

7.0 Maintenance

7.1 Partner Country ensures that an organic capability exists to perform the majority of maintenance and repair tasks to sustain security elements.

•   Failure data reports generated and provided to DoD indicating occurrence of maintenance activity, failures, and successful repair/replacement of failed parts/systems.

•   Functional technical library exists

8.0 Logistics

8.1 Partner Country performs all necessary logistics tasks to sustain security elements.

•   Necessary equipment has been procured and delivered, is available and is operational. Overall security systems are functioning effectively.

•   Reports received on the amounts of support equipment/spares provided, annual usage rate, and fill rate.

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Weapons of Mass Destruction Proliferation Prevention Program

Program Description

The continued threat from possible trafficking of WMD and related materials demands an expansion of and a new approach to proliferation prevention efforts. The WMD Proliferation Prevention Program (WMD-PPP) is focused on enhancing partner country border security capabilities to prevent proliferation and will assist those countries in the development of effective and sustainable capabilities to prevent the proliferation of WMD, related materials, and technologies threatening U.S. national security and global stability.

WMD-PPP has evolved from an “initiative” into an established program and will apply the “lessons learned” as the program expands globally. This expansion requires a new vision, expanded mission, and associated strategic goals with a global focus on stopping the movement of WMD. With its unique expertise in enhancing deterrence, detection, and interdiction capabilities on land or at sea, the WMD-PPP is capable of expanding its reach beyond individual countries to include regional and global partnerships. This program is able to provide a system of materiel and non-materiel solutions tailored to the needs of the partner state to counter the threat of WMD smuggling. Furthermore, by ensuring that the assistance provided can be effectively sustained by our partners, we are able to be confident that our investment in proliferation prevention solutions will be long lasting.

Methodology for Development of WMD-PPP Metrics

To determine measures of success, metrics have been developed to address three strategic program goals. Specific objectives for each goal are listed under the program goal. They are as follows:

1.   Strengthen the capability of partner countries to secure their borders against illicit movement of WMD, related materials, and technologies.

a.   Enhance partner country capability to perform effective risk management.

b.   Enhance partner country capability to perform border security command, control, communications, and computers (C4).

c.   Enhance partner country capability to perform border security surveillance.

d.   Enhance partner country capability to perform WMD detection.

e.   Enhance partner country capability to interdict border violations

2.   Assist partner countries to achieve long-term sustainment of border security capabilities.

a.   Assist partner country with the development of a sustainment budget for all systems delivered under this program.

b.   Enhance partner country capability to support and maintain delivered equipment and/or systems.

c.   Enhance partner country capability to sustain delivered training.

3.   Promote cooperative relationships to further U.S. national security and global stability.

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

a.   Enhance partner country capability to capture and disseminate information concerning WMD incidents to applicable USG agencies.

b.   Increase the awareness of partner countries as to their critical role in WMD proliferation prevention.

c.   Develop bilateral, regional, and multilateral cooperation.

Program Metrics

The ability to measure simply and objectively the impact that WMD-PPP assistance has had on threat reduction is challenging due to the nature of the program: we are providing a capability to our partners that gives them an ability to deter proliferation. Thus, as we are unable to show quantitatively that proliferation has, in fact, been prevented, we must measure the efficiency and effectiveness of the capacity provided. It is possible to quantify the amount of work the WMD-PPP is accomplishing in efforts to provide this capability indirectly by measuring and reporting the following metrics:

•   Miles of green and blue border with additional security enhancements

•   Number of ports of entry where capabilities have been enhanced

On an annual basis, the WMD-PPP will provide updated totals for the above two metrics, which henceforth will appear on the CTR Scorecard.

Objective quantification of the impact of these enhancements is more difficult to establish. Although data can be gathered on the number of detected and interdicted attempts to cross the border, it is impossible to know what number of attempts have not been detected. Furthermore, enhancing deterrence can reduce the number of detections and interdictions that would otherwise have been made. In order to overcome this challenge, one way to measure the effectiveness of enhancements objectively is through testing; that is, standardized exercises that can be conducted before and after enhancements are made to measure the impact of those enhancements.

Project-Level Metrics

The table below provides a description of new project-level metrics. These lower-level metrics provide greater detail as to the impact of each individual project and can be provided as an addendum to the simpler Scorecard metrics.

TABLE 4. WMD-PPP Goals, Objectives, and Metrics

Goals Objectives Metrics

1.0 Strengthen the capability of partner countries to secure their borders against illicit movement of WMD, related materials, and technology.

1.1 Enhance partner country capability to perform effective risk management (RM)

•   Percentage of alerts from RM system that result in a positive outcome

•   Increase in efficiency of RM system

•   Increase in effectiveness of RM system (percentage of alerts that result in a positive outcome)

•   Number of alerts provided to operational units

•   Number of independent sources integrated into RM system

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Goals Objectives Metrics

   

1.2 Enhance partner country capability to perform border security command, control, communications and computers (C4)

•   Number of border security facilities provided with enhanced C4

•   Number of border stations and ports of entry provided with enhanced communications

•   Provided partner country capability to “manage” or “control ”X number of maritime ports or X miles of border

•   Number of command centers connected to lower level units

•   Number of sensor and information types integrated into command & control (C2) system

1.3 Enhance partner country capability to perform border security surveillance

•   Provided partner country capability to surveil/monitor X number of miles/km of maritime or land border

•   Nautical miles of shipping lanes surveilled

1.4 Enhance partner country capability to perform WMD detection

•   Number of border facilities (land and maritime) provided increased capability to detect WMD

•   Number of personnel trained and equipped to perform WMD detection

1.5 Enhance partner country capability to perform border security interdiction

•   Number of personnel trained and equipped to perform border security interdiction

•   Number of border units (border stations and vessels) provided improved capability to perform interdiction

•   Enhanced X number of interdiction platforms

•   Number of boarding teams trained and equipped to perform maritime interdiction

2.0 Assist partner countries to achieve long-term sustainment of border security capabilities

2.1 Assist partner country with the development of a sustainment budget for all systems delivered under this program

•   Percentage of sustainment cost budgeted by partner country for all systems delivered under this program

•   Percentage of sustainment cost funded by partner country

2.2 Enhance partner country capability to support and maintain delivered equipment and/or systems

•   Number of maintenance personnel trained

•   Number of support equipment sets provided

•   Operational availability of delivered equipment and/or systems

•   Does partner country have spares and consumables available and existing preventive maintenance system?

•   Functional technical library

2.3 Enhance partner country capability to sustain delivered training

•   Initial and refresher training system exists

•   Current operators/maintainers trained

•   Establish training centers

•   Number of mobile training teams

•   Established training records

•   Training materials (programs of instruction, computer based training, manuals, etc)

•   Training and certifications maintained

•   Incorporation of training in existing doctrine and academies

3.0 Promote cooperative relationships to further

3.1 Enhance a partner country capability to

•   Number of countries in which we have established reporting mechanisms for WMD

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Goals Objectives Metrics

      U.S. national security and global stability

      capture and disseminate information including that concerning WMD incidents

     incidents (means, procedure, agreement to share)

3.2 Increase the awareness of partner countries as to their critical role in WMD non-proliferation (partner country buy-in)

•   Number of personnel exchanges (training, professional development)

•   Increase in partner country’s commitment to adhere to UNSCR 1540 and other nonproliferation regimes

3.3 Develop interagency, bilateral, regional, and multilateral cooperation

•   Mean time to negotiate agreements

•   Number of regional relationships facilitated

•   Number of regional/multilateral exercises

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

image

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Appendix 2: CBEP Objective, Capability and MOE Traceability

CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
CBEP Objective 1: Secure and consolidate collections of especially dangerous pathogens (EDP) and their associated research at a minimum number of secure health and agricultural laboratories or related facilities. Capability 1: Secure and consolidate collections of EDPs and their associated research. Consolidation MOE 1: Partner country EDP collectionsi and associated research are consolidated into a minimum numberii of locations.iii 1.1: Number of EDP collections. Min: No increase in the number of EDP collections over baselineiv.
Obj: No more than 1 EDP collection.
Security MOE 2: Partner country EDPs and associated research are secured in a manner consistent with standards. 2.1: Biosecurity compliance. Min: Consistent with the European Committee for Standardization (CEN) Workshop Agreement (CWA) Biorisk Management requirements.
Obj: Consistent with the requirements of DoDI 5210.89 outlined in CTB-09-405 Memorandum “Biological Threat Reduction Program (BTRP) Biosafety and Security Standards (BS&S),” dated November 20, 2009 (CTB-09-405 Memorandum).
CBEP Objective 2: Enhance partner country/region’s capability to prevent the sale, theft, diversion, or accidental release of biological weapons (BW)-related materials, technology, and expertise by improving biological safety and security (BS&S) standards and procedures. Capability 1: Secure and consolidate collections of EDP and their associated research. Consolidation MOE 1: Same as Objective 1, Capability 1 MOE 1 1.1: Same as Objective 1, Capability 1 MOE 1. Min: Same as Objective 1, Capability 1 MOE 1.
Obj: Same as Objective 1, Capability 1 MOE 1.
Security MOE 2: Same as Objective 1 Capability 1 MOE 2 2.1: Same as Objective 1, Capability 1 MOE 2 Min: Same as Objective 1, Capability 1 MOE 2.
Obj: Same as Objective 1, Capability 1 MOE 2.
Capability 2: Enhance partner country BS&S standards and practices. Enforceability MOE 3: Partner country has BS&S laws and regulations governing work with EDPs. 3.1: Established legal frameworkv. Min: Partner country laws, guidelines, and regulations demonstrate commitment and progress towards compliance with international laws, regulations, and treaties (BWC, WHO, OIE, UN, etc.).
Obj: Partner country laws, guidelines, and regulations are compliant with DODI 5210.89 to the maximum extent possible and CDC/NIH “Biosafety in Microbiological and Biomedical Laboratories, 5th edition, 2007” (BMBL 5) and subsequent versions.
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
3.2: Level of Regulation. Min: Compliance at the national level.
Obi: Compliance at the local, regional, and national levels.
Safety MOE 4: Partner country guidelines for work with EDPs meet U.S. or international guidelines for biosafety. 4.1: Biosafety guidelines. Min: Consistent with the minimum requirements of WHO IHR.
Obi: Consistent with the requirements of BMBL 5 and subsequent editions for safety in accordance with CTB-09-405 Memorandum.
4.2: Facility specific biosafety plans exist. Min/Obi: Yes.
Security MOE 5: Partner country standards for work with EDPs are consistent with standards for biosecurity. 5.1: Biosecurity standards. Min: Consistent with the CWA Biorisk Management requirements.
Obi: Consistent with the requirements of DoDI 5210.89 outlined in CTB-09-405 Memorandum.
5.2: Facility specific biosecurity plans exist. Min/Obi: Yes.
Transparency MOE 6: Partner country BS&S standards and practices are transparent. 6.1: BS&S standards are available. Min: Available to independent third party.
Obi: Available to U.S. Government.
6.2: Biosecurity Event Notification. Min/Obi: Immediate notification of the theft or loss of an EDP to the appropriate Ministry and law enforcement/security organizations, as well as in accordance with international obligations and treaties.
6.3: Biosafety Event Notification. Min/Obi: Immediate notification of the release of an EDP causing occupational exposure or release of an EDP outside of the primary barriers of the biocontaimnentvi area to the appropriate Ministry and law enforcement/security organizations, as well as in accordance with international obligations and treaties.
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
CBEP Objective 3: Enhance partner country/region’s capability to detect, diagnose, and report endemic and epidemicvii, man- made or natural EDPs, bio-terror attacks, and potential pandemics. Capability 1: Enhance partner country disease detection, diagnosis viii, and reporting capabilities. Safety MOE 1: Partner country disease detection and diagnosis capability meets U.S. and/or international guidelines for biosafety. 1.1: Biosafety guidelines Min: Consistent with the minimum requirements of WHO IHR.
Obi: Consistent with the requirements of BMBL 5 and subsequent editions for safety in accordance with CTB-09-405 Memorandum.
Security MOE 2: Partner countiy disease detectionix and diagnostic infrastructure complies with standards for biosecurity. 2.1: Biosecurity standards Min: Consistent with the CWA Biorisk Management requirements.
Obi: Consistent with the requirements of DoDI 5210.89 outlined in CTB-09-405 Memorandum.
Comprehensive MOE 3: Partner country has preparedness and response plans. 3.1: National Pandemic Influenza Preparedness and Response planx. Min: National Pandemic Influenza Preparedness and Response Plan has been developed.
Obi: Plan conforms to WHO guidance document for Pandemic Influenza Preparedness and Response.
3.2: Bioterrorism preparedness and response plan. Min: Bioterrorism preparedness and response are covered in a developed Public Health Emergency/Bioterrorism Response Plan.
Obi: Plan conforms to HHS/CDC templatexi.
3.3: Multi-Hazard xii National Public Health Emergency Preparedness and Response Plan. Min: Multi-hazard National Public Health Emergency Preparedness and Response Plan for Biological Hazards has been developed.
Obi: Plan conforms to core biological capacity requirements IAW WHO/IHRxiii Monitoring Framework.
3.4: National Emergency Preparedness and Response Plan for animal diseases Min: National Emergency Preparedness and Response Plan for animal diseases lias been developed.
Obi: Plan conforms to FAO/OIE manualxiv.
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
Comprehensive MOE 4: Partner country understands its EDP disease and pathogen baselines. 4.1: National disease surveillance plan. Min: National disease surveillance plan exists.
Obi: National active and passive surveillancexv is established and executed IAW WHO Recommended Surveillance Standards, Second Edition.
Comprehensive MOE 5: Partner country disease surveillance system is capable of detecting and reporting suspect EDP cases to those responsible for human and animal health. 5.1: Disease surveillance system is capable of detecting EDP cases. Min: Disease surveillance system is capable of detecting endemic EDP suspect cases.
Obi: Disease surveillance system is capable of detecting all suspect EDP cases.
5.2: Epidemiological data from EDP case investigations are shared with those responsible for human and animal health. Min/Obi: Epidemiological EDP data from case investigations are shared with those responsible for human and animal health within partner country’s own government structure.
5.3: Laboratory test results are provided to those responsible for human and animal health. Min/Obi: Laboratory test reports from EDP cases are shared with those responsible for human and animal health within partner country’s own government structure.
5.4: All reported human epidemiological events constituting a Public Health Emergency of International Concern (PHEIC) are reported to the WHOxvi Min/Obi: All events constituting a PHEIC are reported to WHO IAW IHR regulations.
5.5: All animal reportable diseases are reported to OIExvii Min/Obi: All events constituting OIE animal reportable diseases are reported to OIE IAW OIE guidelines.
5.6: Case data is shared via appropriate reporting systems. Min/Obj: Reporting systems meet WHOxviii and OIExix standards for early detection and timely reporting.
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
Timeliness MOE 6: Partner country disease surveillance system is capable of early detection and timely reporting of EDP cases to those responsible for human and animal health. 6.1: Epidemiological data from EDP case investigations are promptly reported to those responsible for human and animal health. Min/Obi: EDP case investigation reports are shared within 24 hoursxx of investigation.
6.2: Laboratory test results are promptly provided to those responsible for human and animal health. Min/Obi: Laboratory test results are shared within 24-hours.
6.3: All human epidemiological events constituting a Public Health Emergency of International Concern (PHEIC) are promptly reported to the WHO.xxi Min/Obi: All PHEIC reports are shared IAW WHO IHR timelines.
6.4: All animal reportable diseases are promptly reported to the OIE. xxii Min/Obi: All reportable animal disease reports are shared with OIE IAW OIE timelines.
Comprehensive MOE 7: Partner country is capable of investigating and diagnosing EDP cases. 7.1: Suspect EDP cases are investigated and documented by those responsible for human or animal health. Min: Standardized case investigation form is completed and complies with WHO/OIE standard.
Obi: Standardized case investigation form is completed in an electronic system and complies with WHO/OIE standard.xxiii
7.2: Appropriate samples are collected and promptly transported under optimum conditionsxxiv for Min/Obi: Samples are collected and transported IAW CBEP procedures.
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
laboratory confirmation of the diagnosis.
7.3: Partner country is able to diagnose endemic EDPs. Min: Diagnosis is made IAW CBEP procedures and demonstrated through proficiency testing program.
Obi: Diagnosis is made IAW CBEP procedures and demonstrated in country during real time case investigation and diagnosis.
7.4: Partner country is able to utilize international reference laboratories when there is no country diagnostic capability. Min: Capable of shipping samples IAW IATA (International Airline Transport Association) regulations and requirement agreements of the international reference lab.
Obi: Established agreement exists within international reference labs to which samples will be routinely shipped.
Timeliness MOE 8: Suspect EDP cases are promptly investigated and diagnosed. 8.1: Suspect cases are promptly investigated and assessed by those responsible for human or animal health. Min/Obi: Within 48-hours of suspect case being reported to those responsible for human and animal health IAW WHO IHR/OIE guidelines.xxv
8.2: Appropriate samples are collected and promptly transported under optimum conditions for laboratory confirmation of the diagnosis. Min: Initiate sample transport within 24- hours of case investigation.
Obi: Initiate sample transport immediately following case investigation.
8.3: Partner country promptly initiates diagnostic testing of endemic EDPs. Min: Initiate appropriate diagnostic testing protocols/SOPs within 24-hours of receipt of samples.
Obi: Initiate appropriate diagnostic testing protocols/SOPs immediately upon receipt of samples.
8.4: Partner country is able to rapidly Min/Obi: Shin samples to international reference laboratory within 24-hours of
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
utilize international reference laboratories when there is no in- country diagnostic capability. acquiring samples.
CBEP Objective 4: Ensure the developed capabilities are designed to be sustainable within each partner country/region’s current operating budget. Capability 1: Secure and consolidate collections of EDP and their associated research. Sustainability MOE 1: Partner country EDP collection methods and associated research are sustainable. 1.1: Sustainability plan to physically maintain collection. Min/Obi: Plan identifies human, resource and operational needs to maintain collection of viruses and bacteria over time to support pathogen baseline and facilitate U.S. efforts to mitigate biological risks.
Capability 2: Enhance partner country BS&S standards and practices. Sustainability MOE 2: Partner countiy BS&S practices and regulations concerning EDPs are absorbable with respect to cost. 2.1: Sustaimnent cost of application and enforcement of BS&S guidelines and regulations. Min/Obi: Cost is within minimally budgeted resources of the partner country and in partnership with others.
Capability 3: Enhance partner country disease detection, diagnosis, and reporting capabilities Sustainability MOE 3: Partner country disease detection, diagnosis, and reporting capabilities are absorbable with respect to cost.xxvi 3.1: Sustainment Cost. Min/Obi: Cost is within minimally budgeted resources of the partner country and in partnership with others.
Capability 4: Enhance partner country ability to train/retain personnel trained to desired biosafety/ biosecurity standards. Sustainability MOE 4: Partner country demonstrates an ability to retain and/or train qualified personnel to desired standards. 4.1 Trainee test results. Min/Obj: 70% of trained biosafetv/biosecuritv officers can demonstrate continued proficiency (bv means of either written or hands-on testing) at 6, 12, and 18 months following completion of training.
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
CBEP Supplemental Objective 5: Facilitate the engagement of partner country’s/regional scientific and technical personnel in research areas of interest to both the partner country/region and the United States. Capability 1: Engage partner country life science personnel in areas of interest to partner country/ region and the United States. Legitimacy MOE 1: Research objectives of partner country scientists and institutes are aligned with national and international EDP priorities. 1.1: Credible, appropriate and executable EDP research projects. Min: CBEP sponsored research projects are technically reviewed and approved by CBEP SMEs.
Obi: Country projects on EDPs are competitively peer reviewed and funded by international review board.
Transparency MOE 2: Partner countiy scientists and institutes openly share research information and copies of EDP strains. 2.1: U.S. access to research data funded by CBEP. Min/Obi: U.S. lias access to all data generated by joint research projects.
2.2: Copies of EDP strains transferred to US. Min: Copies of all requested EDP strains from joint research projects are transferred to the US.
Obi: Copies of all requested EDP strains are transferred to the U.S.
2.3: Contribution to international scientific community. Min: Partner country scientists present BTRP research results at relevant international conferences, and contribute to research articles/presentations published/presented within the international community.
Obi: Partner country scientists are invited to present their research results at relevant international conferences, and have lead role in publishing research articles and/or conducting presentations within the international community.
Safety MOE 3: Work of partner country scientists and institutes with EDPs meets U.S. or international guidelines for biosafety. 3.1: Biosafety guidelines. Min: Consistent with the minimum requirements of WHO IHR.
Obi: Consistent with the requirements of BMBL 5 and subsequent editions for safety in accordance with CTB-09-405 Memorandum.
Safety Security MOE 4: Partner country scientists and institutes work with EDPs is consistent with standards for biosecurity. 4.1: Biosecurity standards. Min: Consistent with the CWA Biorisk Management requirements.
Obi: Consistent with the requirements of DoDI 5210.89 outlined in CTB-09-405 Memorandum.
CBEP Supplemental Objective 6: Capability 1: Eliminate bioweapons Comprehensive MOE 1: Disclosed BW-related infrastructure, materials, and technologies are eliminated. 1.1: Extent of elimination Min: Scale and scope consistent with peaceful purposes and BWC treaty.
Obi : 100% of disclosed BW-related
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
CBEP Objective CBEP Capability Attribute Measure of Effectiveness
(MOE)
Indicator Evaluation Criteria
Eliminate any BW- related infrastructure and technologies encountered in a partner/country region. related infrastructure and technologies. infrastructure is eliminated.

i Collection is defined as any collections of U.S. Select agents in the same location/facility.

ii Number of locations must effectively mitigate identified risks based on a threat/risk assessment.

iii Confirmation of EDP diagnosis does not require EDP storage except at central repository.

iv CBEP refers to baseline as no increase in number of EDP collections during pre-engagement.

v An established legal framework should span all levels (local, regional, national, and international).

vi Biocontainment is the containment of extremely pathogenic organisms (as viruses) usually by isolation in secure facilities to prevent their accidental release especially during scientific research.
ttp://www.merriam-webster.com/dictionary/biocontainment

vii Epidemic is the occurrence of more cases of disease than expected in a given area or among a specific group of people over a particular period of time.
http://www.cdc.gov/excite/classroom/outbreak/objectives.htm

viii BTRP refers to diagnosis as the identification and analysis of the nature and underlying cause of an illness or infection. Diagnosis involves laboratory confirmation of the presence or absence of infection and is made by a chemical, microscopic, microbiologic, immunologic, or pathologic study of samples collected from suspect case of infection. For BTRP, diagnosis does not refer to medical diagnosis in which a medical condition or disease is identified/recognized by its outward signs and symptoms.

ix Deleted reporting capability since not applicable to security.

x Metric specifically references Pandemic Influenza Preparedness plans because these are the only pandemic plans being created globally, http://www.who.int/csr/disease/influenza/pipguidance2009/en/index.html

xi World Health Organization (WHO) References CDC Emergency Preparedness and Response for templates on Bioterrorism Response Planning, http://www.bt.cdc.gov/planning/responseguide.asp

xii Multi-hazard IHR hazards of relevance to BTRP include biological (infectious, zoonotic, and foodborne human pathogens). Source: International Health Regulations (IHR) 2005 Monitoring Framework for monitoring Progress in the implementation of IHR Core Capacities in State Parties. Annual Data Collection Tool (2010). http://www.who.int/ihr/checklist/en/index.html

xiii “At the national level, state parties are required to assess all reports of urgent events within their territories within 48 hours by applying algorithm contained in Annex II of the IHR.” “When a State Party identifies an event as notifiable, it must be notified to

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

WHO immediately*, i.e. within 24 hrs after having carried out the assessment of public health information related to the event. Such notification will include details of any health measure employed in response to the event as well as accurate and sufficiently detailed public health information available, including case definitions, laboratory results, and number of cases and deaths.” “In addition to notification and consultation, state parties are required to inform WHO within 24 hrs of receipt of evidence of public health risks occurring outside their territory that may cause international disease spread.” IHR (2005) defines immediately as: within 24 hrs. http://www.who.int/ihr/about/10things/en/index.html

xiv FAO Animal Health Manual on the preparation of national animal disease emergency preparedness plans. http://www.fao.org/DOCREP/004/X2096E/X2096E00.htm

xv Active surveillance includes activities in which those responsible for human and animal health identify new cases of disease (case finding) and/or involve interviewing patients, reviewing medical records to detect cases or after index case has been reported.

Passive surveillance includes activities in which available data on reportable diseases is used or reporting is mandated or requested by those responsible for human and animal health. (Leon Gordis, Epidemiology Second Edition). WHO Recommended Surveillance Standards, Second edition, http://www.who.int/csr/resources/publications/surveillance/whocdscsrisr992.pdf

xvi Events constituting a PHEIC are assessed by country using the decision instrument in Annex 2 of IHR. http://www.who.int/ihr/9789241596664/en/index.html

xvii OIE reportable diseases http://www.oie.int/eng/maladies/en_classification2010.htm?eld7

xviii WHO IHR Monitoring Framework for monitoring Progress in the implementation of IHR Core Capacities in State Parties. Annual Data Collection Tool (2010). http://www.who.int/ihr/checklist/en/index.html

xix OIE World Animal Health Information System (WAHIS). http://www.oie.int/ENG/info/en_info.htm

xx Immediate reporting of EDPs as required by CSTE and referenced by CDC for guidelines regarding nationally notifiable diseases. http://www.cste.org/dnn/LinkClick.aspx?fileticket=7CCxM20JUGg%3dtabid=36∣=1496 http://www.who.int/ihr/about/10things/en/index.html

xxi Events constituting a PHEIC are assessed by country using the decision instrument in Annex 2 of the IHR. http://www.who.int/ihr/9789241596664/en/index.html

xxii OIE reportable diseases http://www.oie.int/eng/maladies/en_classification2010.htm?eld7

xxiii WHO/OIE standards http://www.who.int/ihr/IHR_Monitoring_Framework_Checklist_and_lndicators.pdf

xxiv Optimum conditions include adherence to appropriate BS&S regulations, cold-chain maintenance, and universal precautions.

xxv http://www.who.int/ihr/IHR_Monitoring_Framework_Checklist_and_lndicators.pdf

xxvi Cost is the combined metric of multiple sustainment topics, including but not limited to: operations, maintenance, technology, infrastructure, knowledge, skills, abilities, educational, logistics, regulatory, cultural) etc. Operations cost is for biosecurity and collection facility. “Maintenance costs” is for biosecurity and collection facility.

Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 65
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 66
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 67
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 68
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 69
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 70
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 71
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 72
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 73
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 74
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 75
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 76
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 77
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 78
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 79
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 80
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 81
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 82
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 83
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 84
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 85
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 86
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 87
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 88
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 89
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 90
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 91
Suggested Citation:"Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 92
Next: Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction »
Improving Metrics for the Department of Defense Cooperative Threat Reduction Program Get This Book
×
Buy Paperback | $38.00 Buy Ebook | $30.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Cooperative Threat Reduction (CTR) Program was created in 1991 as a set of support activities assisting the Former Soviet Union states in securing and eliminating strategic nuclear weapons and the materials used to create them. The Program evolved as needs and opportunities changed: Efforts to address biological and chemical threats were added, as was a program aimed at preventing cross-border smuggling of weapons of mass destruction. CTR has traveled through uncharted territory since its inception, and both the United States and its partners have taken bold steps resulting in progress unimagined in initial years. Over the years, much of the debate about CTR on Capitol Hill has concerned the effective use of funds, when the partners would take full responsibility for the efforts, and how progress, impact, and effectiveness should be measured.

Directed by Congress, the Secretary of Defense completed a report describing DoD's metrics for the CTR Program (here called the DoD Metrics Report) in September 2010 and, as required in the same law, contracted with the National Academy of Sciences to review the metrics DoD developed and identify possible additional or alternative metrics, if necessary. Improving Metrics for the DoD Cooperative Threat Reduction Program provides that review and advice.

Improving Metrics for the DoD Cooperative Threat Reduction Program identifies shortcomings in the DoD Metrics Report and provides recommendations to enhance DoD's development and use of metrics for the CTR Program. The committee wrote this report with two main audiences in mind: Those who are mostly concerned with the overall assessment and advice, and those readers directly involved in the CTR Program, who need the details of the DoD report assessment and of how to implement the approach that the committee recommends.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!