APPENDIX B

REPRODUCTION OF THE DEPARTMENT OF DEFENSE COOPERATIVE THREAT REDUCTION METRICS REPORT

Delivered by Department of Defense to Congress

September 2010

Introduction

This report is submitted in accordance with Section 1304 of the National Defense Authorization Act for Fiscal Year 2010, Title XIII of Public Law 111-84. Section 1304 provides that the Secretary of Defense shall develop and implement metrics to measure the impact and effectiveness of activities of the Department of Defense (DoD) Nunn-Lugar Cooperative Threat Reduction (CTR) Program. The metrics to be developed would measure the effectiveness of projects and activities undertaken by DoD to prevent the proliferation of chemical, nuclear, and biological weapons and weapons of mass destruction (WMD)-related materials, technologies, and expertise.

Background

In the National Academy of Sciences’ (NAS) 2009 report, Global Security Engagement: A New Model for Cooperative Threat Reduction, NAS affirmed the usefulness of the traditional metrics of the DoD CTR Program for evaluating effectiveness in traditional CTR WMD elimination or site security program areas, in which concrete actions such as chemical munitions, warhead delivery systems, and launchers destroyed or chemical or nuclear warheads and weapons materials secured clearly lead to threat reduction. Such activities break down into discrete events with clear end points, and thus metrics that can be incrementally reported, as on the CTR Scorecard (Appendix 1), continue to be appropriate for those types of programs. However, the NAS report accurately noted that CTR program responses to new, often less quantifiable threats in this century will require new methods to measure success. The challenge is to find measureable performance indicators that capture the true value of program activities that may not lend themselves readily to raw quantitative measurement, thus making it more difficult to document the relative success of the activity.

In recent years, the CTR program has increasingly become involved in capacity-building efforts, notably in the Cooperative Biological Engagement Program (CBEP), formerly the Biological Threat Reduction Program (BTRP), and the WMD Proliferation Prevention Program (WMD-PPP). In these programs, threat reduction is accomplished through building the capacity of the partner state to conduct surveillance to deter or detect and report or respond to emerging biological and WMD trafficking threats. Similarly, the CTR Program is building the capacity of the Russian Federation to sustain upgraded physical protection systems at nuclear weapons



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 65
APPENDIX B REPRODUCTION OF THE DEPARTMENT OF DEFENSE COOPERATIVE THREAT REDUCTION METRICS REPORT Delivered by Department of Defense to Congress September 2010 Introduction This report is submitted in accordance with Section 1304 of the National Defense Authorization Act for Fiscal Year 2010, Title XIII of Public Law 111-84. Section 1304 provides that the Secretary of Defense shall develop and implement metrics to measure the impact and effectiveness of activities of the Department of Defense (DoD) Nunn-Lugar Cooperative Threat Reduction (CTR) Program. The metrics to be developed would measure the effectiveness of projects and activities undertaken by DoD to prevent the proliferation of chemical, nuclear, and biological weapons and weapons of mass destruction (WMD)-related materials, technologies, and expertise. Background In the National Academy of Sciences’ (NAS) 2009 report, Global Security Engagement: A New Model for Cooperative Threat Reduction, NAS affirmed the usefulness of the traditional metrics of the DoD CTR Program for evaluating effectiveness in traditional CTR WMD elimination or site security program areas, in which concrete actions such as chemical munitions, warhead delivery systems, and launchers destroyed or chemical or nuclear warheads and weapons materials secured clearly lead to threat reduction. Such activities break down into discrete events with clear end points, and thus metrics that can be incrementally reported, as on the CTR Scorecard (Appendix 1), continue to be appropriate for those types of programs. However, the NAS report accurately noted that CTR program responses to new, often less quantifiable threats in this century will require new methods to measure success. The challenge is to find measureable performance indicators that capture the true value of program activities that may not lend themselves readily to raw quantitative measurement, thus making it more difficult to document the relative success of the activity. In recent years, the CTR program has increasingly become involved in capacity-building efforts, notably in the Cooperative Biological Engagement Program (CBEP), formerly the Biological Threat Reduction Program (BTRP), and the WMD Proliferation Prevention Program (WMD- PPP). In these programs, threat reduction is accomplished through building the capacity of the partner state to conduct surveillance to deter or detect and report or respond to emerging biological and WMD trafficking threats. Similarly, the CTR Program is building the capacity of the Russian Federation to sustain upgraded physical protection systems at nuclear weapons 65

OCR for page 65
66 IMPROVING METRICS FOR THE DOD CTR PROGRAM storage sites. Although increased partner state capacity makes an important contribution to a reduction in the WMD threat to the United States, this is more difficult to quantify than numbers of missiles or other forms of strategic offensive arms destroyed as in the traditional programs. CTR’s prevention mission is difficult to quantify. Although a number of output measures can be counted, they are proxies for the outcome of deterring terrorists and proliferators from accessing WMD and related materials and expertise. We can measure the amount of equipment provided and the number of training events conducted or scientists engaged; however, we need better measures to show that these efforts actually result in changed practices or additional effectiveness. As an example that makes evident the findings in the NAS report and reflects a matter DoD has recognized in connection with the CTR Program, a metric based on the number of Zonal Diagnostic Laboratories built under the BTRP may not accurately reflect the partner country’s enhanced capability for pathogen detection and reporting. Indeed, such a metric could create unintended incentives to continue building additional laboratories that do not make a positive contribution to threat reduction, when what is really needed might be additional training to make better use of the existing laboratories’ capabilities for pathogen detection. In WMD-PPP, the raw number of radars, sensors, and patrol boats provided for border security, although useful, may not be as important as the effectiveness of the information sharing among those systems, which is a function of system integration, training and effective decision making rather than of the number of systems provided. In light of these issues, Congress has mandated the development and implementation of metrics to measure the impact and effectiveness of projects and activities of the CTR program to address threats arising from the proliferation of chemical, nuclear, and biological weapons and WMD- related materials, technologies, and expertise. Additionally, Section 1304 requires submission, no later than 270 days after the enactment of the Act, of a report describing the metrics developed and implemented. This report addresses these Section 1304 requirements. Discussion The metrics described in this report are designed to measure appropriately the impact of each CTR program area, as the CTR program is currently constituted for Fiscal Year 2010, and does not attempt to speculate on what metrics might be appropriate for future areas of programmatic or geographic expansion. Neither do the metrics attempt to determine whether the activities of the CTR program are the “right” activities. The metrics described in this report are intended to best measure the effectiveness of the CTR Program in conducting these established programs. These metrics are not intended to revise the method for establishing these objectives. Figure 1 illustrates the organization of the Department of Defense as it relates to the CTR program. CTR executes programs as directed in program guidance from the Deputy Assistant to the Secretary of Defense (Treaties and Threat Reduction) (DATSD(T&TR)). This program guidance provides instruction on how to implements policy guidance issued by the Office of the Under Secretary of Defense for Policy (USD/P), which has been coordinated with all relevant agencies within the Executive Branch.

OCR for page 65
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 67 Those responsible for each CTR program area, in concert with applicable stakeholders, reviewed their specific program objectives and either established new program metrics or confirmed the validity of existing program metrics. Although not culled out specifically in this report, except as noted in the CTR Scorecard (Appendix 1), the metrics currently in use for the Strategic Offensive Arms Elimination (SOAE) program will not change. The results of this review and analysis process are provided in this report. Each report section provides a brief program description, methodology for metrics development, and a description of metrics that includes reporting and re-evaluation activities. Conclusion CTR program metrics serve as a means to evaluate how well program objectives have been accomplished. Several traditional CTR program metrics are still very applicable and useful measures of CTR Program threat reduction achievements. For those programs involved in capacity-building efforts, notably the CBEP, basic metrics need to be specifically tailored and evaluated individually at a project level. A number of program benefits that would be useful and meaningful in the larger USG threat reduction effort and are routinely considered in establishing execution plans, such as partner country contributions and commitments, or ability to leverage multiple sources of expertise or funding, were considered but not recommended since these benefits ultimately do not trace back directly to the CTR program objectives. As the CTR program continues to evolve, we will continue to assess its metrics to ensure they best represent achievement of new objectives. Figure 1. The CTR program within the DoD organizational structure Secretary of Defense (OSD) Deputy Secretary of Defense (OSD) Under Secretary of Defense Under Secretary of Defense (Policy) (Acquisition, Technology, and Logistics) (USD/P) (USD/AT&L) Assistant to the Secretary of Defense for Assistant Secretary of Defense Nuclear and Chemical and Biological Defense Programs Global Strategic Affairs (ATSD(NCB) ) (ASD/GSA) Policy Implementation Guidance Guidance Deputy Assistant to the Defense Threat Reduction Agency & Deputy Assistant Secretary of Defense for Secretary of Defense for (DTRA) Acquisition Countering Weapons of Mass Destruction Treaties and Threat Oversight (DASD(CWMD)) Reduction (DATSD(T&TR)) Associate Director, Operations Cooperative Threat Reduction Policy Enterprise (CTR Policy) (ADOP) Cooperative Threat Reduction Directorate (OP-CT)

OCR for page 65
68 IMPROVING METRICS FOR THE DOD CTR PROGRAM Cooperative Biological Engagement Program Program Description The Cooperative Biological Engagement Program (CBEP) has four main objectives that are identified in the current policy and implementation guidance issued to the Defense Threat Reduction Agency (DTRA) for implementation: 1. Secure and consolidate collections of especially dangerous pathogens (EDPs) and their associated research at a minimum number of secure health and agricultural laboratories or related facilities; 2. Enhance partner country/region’s capability to prevent the sale, theft, diversion, or accidental release of biological weapons (BW)-related materials, technology, and expertise by improving biological safety and security (BS&S) standards and procedures; 3. Enhance partner country/region’s capability to detect, diagnose, and report endemic and epidemic, man-made or natural EDPs, bio-terror attacks, and potential pandemics; and 4. Ensure the developed capabilities are designed to be sustainable within each partner country/region’s current operating budget. The fourth objective identified above, sustainability, is a cross-cutting objective to ensure that the capabilities provided by the CBEP are adapted and owned by the partner country. The ultimate aim for CBEP implementation is that the partner country take ownership of a sustainable capacity to conduct biosurveillance, meet BS&S standards, and conduct research effectively on its own. Promoting partner country ownership is one of the critical program objectives and one of the most significant CBEP challenges. The CBEP’s supplemental objectives – objectives noted as supplemental in the guidance received – include facilitating the engagement of partner country’s/regional scientific and technical personnel in research areas of interest to both the partner country/region and the United States and eliminating any BW infrastructure and technologies encountered in a partner country/region. CBEP assistance also supports host country objectives in meeting the World Health Organization (WHO) International Health Regulations (IHR). CBEP implementation to meet these objectives includes collaborative biological research (CBR), biosurveillance, and enhancement of BS&S. CBR efforts bring together U.S. and international scientists in cooperative research of mutual interest in support of threat reduction priorities. Biosurveillance projects include providing training to animal and human health experts on disease recognition, diagnosis, and reporting; support for laboratory diagnostic equipment and reagents; and implementation of the Electronic Integrated Disease Surveillance System (EIDSS). BS&S enhancements include bioethics training, training on safe clinical and laboratory practices; installation of security systems at laboratories; implementation of the Pathogen Asset Control System (PACS) to track inventory; and support for development of biorisk management processes and procedures. Methodology for Development of New Metrics Developing metrics for the CBEP presented several challenges. First, the program seeks to achieve concrete improvements in disease detection, biological safety, and research systems in an environment not fully under its control, with the result that accountability and analysis of

OCR for page 65
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 69 program impact are complex, difficult matters. Also, the metrics for the CBEP need to establish a solid, traceable link from CBEP-funded efforts to concrete threat reduction achievements. To address these challenges and establish actionable metrics, the CBEP adapted a Joint Capabilities Integration and Development System-like process to develop mission and performance measures. This effort took place over the course of one year of collaborative development. The CBEP metrics team first developed an initial capabilities document that lays out the key capabilities required in the program’s strategic guidance and the attributes supporting these capabilities. From these attributes, the team then developed measures of effectiveness (MOEs) to assess mission accomplishment. These MOEs are directly traceable to the program’s policy and implementation guidance. Finally, the team developed measurement values to serve as indicators that the critical MOEs have been achieved. The objective and minimum thresholds for these measurement values indicate the desired end state and the minimally acceptable level of achievement respectively. Throughout the process, the CBEP engaged program stakeholders from across disciplines as diverse as clinical practice, epidemiology, laboratory, biosafety, veterinary, biosecurity, systems engineering, and training. The use of small working groups by discipline and progressive reviews by outside parties and program leadership enhanced the accuracy, comprehensiveness, and usefulness of the final product. Description of New Metrics The table in Appendix 2 depicts the linkages and associations across 6 program objectives, 5 program capabilities, 10 attributes, 20 MOEs, measurement values, and minimum/objective values. The CBEP will utilize these revised metrics throughout planning, implementation, and final assessment of country/region engagements. Using a systems engineering approach, all project officers will use metrics to plan and execute work, including designing a concept of operations to support MOEs in a partner country or region. The project implementation team will focus on achieving a capability in a partner country or region as defined by the MOEs. Interim performance against the metrics will be assessed through a seamless, collaborative test and evaluation process that will include tabletop and field training exercises, as well as informal assessments by CBEP collaborators. In addition, ongoing proficiency and competency testing will assess capability levels, target areas for improvement, and build capacity on the micro level. Results will also be fed back to the implementation teams, allowing them to make adjustments to their execution plans. Feedback from CBEP collaborators is a part of each engagement activity in each country or region, and the program holds an integrated country review at least every quarter of the year to identify issues, necessary changes in approach, action items, and those responsible for taking action. This iterative, ongoing assessment approach helps to ensure that enhancements are taking place and that the implementation approach is delivering results.

OCR for page 65
70 IMPROVING METRICS FOR THE DOD CTR PROGRAM In conducting formal, large-scale assessments of program achievements relative to the metrics, the CBEP will leverage international authorities such as the WHO, the Food and Agriculture Organization of the United Nations (FAO), or the World Organization for Animal Health (OIE) to seek third-party validation of program accomplishments. These formal reviews will assess the cumulative results of CBEP engagement and identify areas for continuing cooperation. All of the above assessment efforts will document performance relative to the 20 MOEs identified in Appendix 2. The use of multiple metrics allows for a thorough, broad evaluation of program performance; however, it makes providing a succinct, top-level report of performance a challenge. The CBEP plans to report the cumulative performance of each country against the comprehensive metrics on an annual basis. This summary report would offer an overview of each country’s “proficiency” relative to the CBEP objectives. Proficiency could be indicated in some manner on a summary graphic; however, the complexity of the program does not lend itself to simplification into a single scorecard metric. Chemical Weapons Elimination Program description The original program task for the Chemical Weapons Elimination program was to assist the Russian Federation in establishing a chemical weapons destruction facility (CWDF) near Shchuch’ye, located in the Kurgan District, to destroy nerve agent-filled munitions located in the Planovy CW storage facility in a safe, secure, and environmentally sound manner. The assistance provided included the following: • Site clearing, water drainage, and preparation of the land for construction; • Evaluating, optimizing, and scaling-up of the Russian-developed nerve agent destruction process; • Developing, designing, fabricating, and testing of the munitions processing equipment; • Integrating and systemizing the processing and associated support equipment and facilities; and • Commissioning the facilities as an acceptable industrial complex under Russian Federation standards and laws. The United States and other members of the Group of Eight Global Partnership against the Spread of Weapons and Materials of Mass Destruction assisted with completion of this facility. As construction of the first destruction building and other parts of the complex neared completion, the Russian Federation requested continued U.S. technical assistance to monitor and evaluate the agent destruction system and support equipment maintenance through the remaining agent elimination efforts as a means of assuring continued full functionality of the systems and equipment. The Russian Federation initiated destruction in the first building in March 2009. The second destruction building is still under construction and is scheduled to be complete and in operation in 2011.

OCR for page 65
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 71 Program-Level Metrics Methodology for Development of New Program-Level Metrics Technical assistance will primarily take the form of engineering advice to help resolve process and equipment failures, identify potential future failures, and ensure replacement spares are on hand. Quantifying how this advice/assistance provided by CTR contributes to the overall availability of the facility to perform its agent destruction mission is the challenge. Two quantifiable factors that can be identified and tracked from a strategic perspective are the amount of funding invested in providing technical assistance and the amount of agent and munitions destroyed by the facility. Although it can be quantified, tracking funds expended on technical assistance provides no clear measure of whether the technical assistance actually is contributing to the facility’s ability to destroy nerve agent. Costs are incurred whether or not any deficiency is successfully identified, isolated, and corrected. However, the number of munitions processed and the tonnage of nerve agent destroyed does provide a valuable and useful link to determining how successful the provided technical assistance has been toward keeping the facility functioning at capacity. Progress in the destruction of agents and processing of munitions is clearly indicative of continued facility functionality and operation. Therefore, we determined that using the amount of agent destroyed and tracking the number of munitions processed would be the best metrics to measure the success of the technical support presently being provided. Description of New Program-Level Metrics As nerve agent-filled munitions are brought to the facility for processing, the weight of nerve agent extracted and neutralized is determined and the number of individual munitions bodies processed is captured by the plant’s automated process control system. The Russian Federation’s operations staff consolidates and reports the two quantities, and the U.S. on-site technical staff obtains the data from the Russian Federation and maintains a cumulative total of both data points. The on-site staff then provides the information weekly to the U.S. Project Management Office. Table 1 below provides an example of the metrics provided in the weekly report: Table 1. CWE Metrics Report Date Total Rounds Total Metric Tons August 3, 2010 532,806 1461.8 However, since the number of munitions (total rounds) correlates closely to the amount of nerve agent destroyed, for the purposes of reporting externally in the “CTR scorecard” only the metric tons of agent destroyed will be reported. (Appendix 1),

OCR for page 65
72 IMPROVING METRICS FOR THE DOD CTR PROGRAM Project-Level Metrics Additional metrics will be maintained at a project level to assess how effectively the destruction facility is being used in relation to performance of maintenance and repairs that help ensure the safety of the destruction process. Methodology for Development of New Project-Level Metrics Reliability, Availability, and Maintainability (RAM) Analysis When fully constructed, the Shchuch’ye CWDF will consist of two main munitions destruction buildings, with two destruction process lines (DPLs) operating in each building. A RAM analysis to establish the baseline threshold and objective capacities of the facility in this configuration was completed in August 2005. That analysis determined that the design capacity threshold for the completed facility is 1,000 metric tons of agent destruction per year, with a design capacity objective of 1,700 metric tons of agent destruction per year (capacity for two main destruction buildings with two DPLs operating in each). Description of new project-level metrics: Scheduled Facility Downtime Scheduled Facility Downtime (SFD) tracks any days when it is planned that the facility will be idle and not processing munitions. This can be due to scheduled maintenance, campaign changeovers, or any administrative downtime where the idling of the process was planned in advance. In the RAM analysis, the SFD was estimated to average 25 percent. For a facility of this complexity, with its associated preventive maintenance requirements both for the destruction process and all of the support systems, such an estimate is reasonable. Minimizing down days, whether scheduled or unscheduled, is extremely important. Using scheduled downtime efficiently and effectively by scheduling periodically required maintenance actions for those periods can maintain overall production capabilities, as well as the safety of the facility. U.S. technical support efforts help the Russian Federation to plan and optimize scheduled maintenance outages and helps ensure that the scheduled outage does not extend beyond the originally scheduled duration by having the right parts, equipment and vendor support pre-staged to take full advantage of the outage. Since the start of munitions destruction, with only one of the two destruction buildings in operation, the cumulative SFD has been 19.25%. A rolling 12-month average helps identify potential trends that might get masked over the long term. The Shchuch’ye CWDF’s current rolling 12-month SFD average is 26.85%. This higher percentage is attributable to the recent scheduled changeover from one type of munition to another. In terms of the objectives and thresholds for agent tons per year, an objective SFD average of 20 percent per year would support meeting the 1,700 metric tons of agent per year average with two

OCR for page 65
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 73 main production facilities operational. Similarly, an SFD average of 35 percent would result in meeting the threshold average of 1,000 metric tons per year. Unscheduled Facility Downtime A second key area for assessing overall throughput is Unscheduled Facility Downtime (UFD). Efficient and effective technical support helps minimize the number of UFD days. This support encompasses proactive failure root-cause analysis, analysis of spare parts consumption, provisioning of the right amount and types of spare parts, and the selective replacement of items before failure during scheduled maintenance outages. Objective and threshold limits for this metric are being developed. The Shchuch’ye CWDF’s current overall cumulative UFD percentage is 6.68%, with a 12-month rolling average of 4.66%. The UFD for the Shchuch’ye CWDF is significantly lower than would be expected for a complex CWDF, based upon experience with similar facilities in the U.S. program, especially during the initial year of operation. Facility Achieved Availability The Achieved Availability metric encompasses the overall throughput of the facility achieved versus the actual design capacity of the facility. During the current phase of operations, it is based upon the one completed main destruction building (overall capacity based upon two DPLs). The achieved availability with one main destruction building is a good indicator of what could be expected with two main destruction buildings in terms of meeting the overall objective of 1,700 metric tons per year for a facility operating with two main destruction facilities. Based upon the RAM analysis, the estimated Achieved Availability was 35.4 percent, which would result in an overall annual throughput rate of 1,382.5 metric tons for the facility with two main destruction buildings in operation with two operational DPLs in each. In order to achieve the objective of 1,700 metric tons per year, an overall Achieved Availability of approximately 44 percent would be required. Similarly, it is estimated that the 1,000 metric tons per year threshold would be met with an Achieved Availability of 26 percent. To date, Building 101A has an Achieved Availability of 54 percent, much higher than originally anticipated. The Achieved Availability calculation takes into account Scheduled and Unscheduled Facility Downtimes. Based on operations to date, these two components account for almost 26 percent of the nearly 45 percent of lost production time for Building 101A associated with an Achieved Availability of 54 percent. The remaining 19 percent of downtime is primarily due to unplanned and planned maintenance of one or the other of the two DPLs while one remains in operation. Summary We are using three project-level metrics to track and evaluate the trend of the effectiveness of the facility and the provided technical support. Based upon an analysis of available RAM reports,

OCR for page 65
74 IMPROVING METRICS FOR THE DOD CTR PROGRAM the following are the project-level metrics, with associated objective and threshold values, as of June 30, 2010: Scheduled Facility Downtime (SFD): Objective: 20 percent (RAM analysis rate of 1,700 metric tons/year) Threshold: 35 percent (RAM analysis rate of 1,000 metric tons/year) Cumulative SFD: 19.25 percent Last 12 Months SFD: 26.85 percent Unscheduled Facility Downtime (UFD): Cumulative UFD: 6.7 percent Last 12 Months UFD: 4.7 percent Facility Achieved Availability (FAA): Objective: 44 percent Threshold: 26 percent Cumulative FAA: 53.97 percent Last 12 Months FAA: 52.51 percent

OCR for page 65
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 75 Nuclear Weapons Safety and Security Program Description This program supports proliferation prevention by enhancing the security systems of nuclear weapons storage sites using DoD nuclear security standards as a basis for design. It also trains cadres of security systems operators, administrators, and maintenance and repair technicians and provides capabilities, such as the personnel reliability program and guard force training systems. In response to a request from Russia and former President Bush’s commitment at Bratislava, Slovakia in February 2005, DoD and the Department of Energy (DOE) enhanced security systems at requested locations that permanently or temporarily store strategic and non-strategic (tactical) nuclear weapons. DoD upgraded a total of 24 sites and, additionally, supports proliferation prevention by enhancing the security and safety of nuclear weapons during shipment through the Nuclear Weapons Transportation Security (NWTS) Program. Methodology for Development of New Metrics Nuclear Weapons Safety and Security (NWS&S) projects have been focused on efforts exclusively in Russia, and the metric used to measure program success reported the number of nuclear weapons storage sites upgraded. This remains an appropriate measure of the impact and effectiveness of program efforts in Russia and, therefore, this information will continue to be reported on the CTR Scorecard (Appendix 1) in its present format. Recently, however, with the potential global expansion of the Nuclear Weapons Storage Security (NWSS) program to other partner countries, DTRA has been working to develop enhanced metrics that better reflect how our efforts are contributing to overall threat reduction. These new metrics are aimed at measuring the capability of Russia (and in the future, other countries) to sustain the ability to store and transport nuclear materials safely and securely. The development of these metrics presents a substantial challenge, as we need to develop measures that accurately and correctly reflect an intangible capability. DoD and DOE have been working together on this issue and have jointly developed eight principles to evaluate sustainment capabilities. Although these principles are presently used as internal measures of effectiveness and are still being refined, the NWSS program has already begun to apply these principles to the sustainment of the site security projects in Russia. Description of New Metrics DoD and DOE jointly developed eight sustainment and logistics principles for the Russia NWSS projects, and proposed metrics would follow these standards. These metrics also meet other international standards, including those used by the International Atomic Energy Agency (IAEA) for logistics and maintenance management of nuclear facilities. The metrics include: • Performance/Capability Assurance, • Configuration Management, • Procedures and Processes,

OCR for page 65
82 IMPROVING METRICS FOR THE DOD CTR PROGRAM

OCR for page 65
Appendix 2: CBEP Objective, Capability and MOE Traceability CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria Capability (MOE) Consolidation MOE 1: Partner country EDP 1.1: Number of EDP Min: No increase in the number of EDP CBEP Objective 1: Capability 1: i iv Secure and Secure and collections. collections and associated collections over baseline . consolidate consolidate research are consolidated into Obj: No more than 1 EDP collection. ii collections of collections of a minimum number of especially dangerous EDPs and their iii locations. pathogens (EDP) associated and their associated research. research at a Security MOE 2: Partner country 2.1: Biosecurity Min: Consistent with the European minimum number of EDPs and associated research compliance. Committee for Standardization (CEN) secure health and are secured in a manner Workshop Agreement (CWA) Biorisk agricultural consistent with standards. Management requirements. laboratories or Obj: Consistent with the requirements of related facilities. DoDI 5210.89 outlined in CTB-09-405 Memorandum "Biological Threat Reduction Program (BTRP) Biosafety and Security Standards (BS&S) ,” dated November 20, 2009 (CTB-09-405 Memorandum). Consolidation MOE 1: Same as Objective 1, 1.1: Same as Min: Same as Objective 1, Capability 1 CBEP Objective 2: Capability 1: Enhance partner Secure and Capability 1 MOE 1 Objective 1, MOE 1. country/region’s consolidate Capability 1 MOE 1. Obj: Same as Objective 1, Capability 1 capability to prevent collections of MOE 1. the sale, theft, EDP and their Security MOE 2: Same as Objective 1 2.1: Same as Min: Same as Objective 1, Capability 1 diversion, or associated Capability 1 MOE 2 Objective 1, MOE 2. accidental release of research. Capability 1 MOE 2 Obj: Same as Objective 1, Capability 1 APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT biological weapons MOE 2. (BW)-related Enforceability MOE 3: Partner country has 3.1: Established legal Min: Partner country laws, guidelines, and Capability 2: materials, Enhance partner BS&S laws and regulations frameworkv. regulations demonstrate commitment and technology, and country BS&S governing work with EDPs. progress towards compliance with expertise by standards and international laws, regulations, and treaties improving biological practices. (BWC, WHO, OIE, UN, etc.). safety and security Obj: Partner country laws, guidelines, and (BS&S) standards regulations are compliant with DODI and procedures. 5210.89 to the maximum extent possible and CDC/NIH “Biosafety in Microbiological and Biomedical Laboratories, 5th edition, 2007” (BMBL 5) and subsequent versions. 83

OCR for page 65
84 CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria Capability (MOE) 3.2: Level of Min: Compliance at the national level. Regulation. Obj: Compliance at the local, regional, and national levels. Safety MOE 4: Partner country 4.1: Biosafety Min: Consistent with the minimum guidelines for work with EDPs guidelines. requirements of WHO IHR. meet U.S. or international Obj: Consistent with the requirements of guidelines for biosafety. BMBL 5 and subsequent editions for safety in accordance with CTB-09-405 Memorandum. 4.2: Facility specific Min/Obj: Yes. biosafety plans exist. Security MOE 5: Partner country 5.1: Biosecurity Min: Consistent with the CWA Biorisk standards for work with EDPs standards. Management requirements. are consistent with standards Obj: Consistent with the requirements of for biosecurity. DoDI 5210.89 outlined in CTB-09-405 Memorandum. 5.2: Facility specific Min/Obj: Yes. biosecurity plans exist. Transparency MOE 6: Partner country 6.1: BS&S standards Min: Available to independent third party. BS&S standards and practices are available. Obj: Available to U.S. Government. are transparent. 6.2: Biosecurity Min/Obj: Immediate notification of the Event Notification. theft or loss of an EDP to the appropriate Ministry and law enforcement/security organizations, as well as in accordance with international obligations and treaties. 6.3: Biosafety Event Min/Obj: Immediate notification of the Notification. release of an EDP causing occupational exposure or release of an EDP outside of the vi primary barriers of the biocontainment area to the appropriate Ministry and law enforcement/security organizations, as well as in accordance with international obligations and treaties. IMPROVING METRICS FOR THE DOD CTR PROGRAM

OCR for page 65
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria Capability (MOE) MOE 1: Partner country 1.1: Biosafety Min: Consistent with the minimum CBEP Objective 3: Capability 1: Enhance partner Enhance partner Safety disease detection and diagnosis guidelines requirements of WHO IHR. country/region’s country disease capability meets U.S. and/or Obj: Consistent with the requirements of capability to detect, detection, international guidelines for BMBL 5 and subsequent editions for safety viii diagnose, and report biosafety. in accordance with CTB-09-405 diagnosis , and endemic and Memorandum. reporting vii epidemic , man- capabilities. Security MOE 2: Partner country 2.1: Biosecurity Min: Consistent with the CWA Biorisk made or natural ix standards Management requirements. EDPs, bio-terror disease detection and Obj: Consistent with the requirements of attacks, and potential diagnostic infrastructure DoDI 5210.89 outlined in CTB-09-405 pandemics. complies with standards for Memorandum. biosecurity. Comprehensive MOE 3: Partner country has 3.1: National Min: National Pandemic Influenza preparedness and response Pandemic Influenza Preparedness and Response Plan has been plans. Preparedness and developed. x Obj: Plan conforms to WHO guidance Response plan . document for Pandemic Influenza Preparedness and Response. 3.2: Bioterrorism Min: Bioterrorism preparedness and preparedness and response are covered in a developed Public response plan. Health Emergency/Bioterrorism Response Plan. Obj: Plan conforms to HHS/CDC xi APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT template . xii Min: Multi-hazard National Public Health 3.3: Multi-Hazard Emergency Preparedness and Response Plan National Public Health for Biological Hazards has been developed. Emergency Obj: Plan conforms to core biological Preparedness and xiii capacity requirements IAW WHO/IHR Response Plan. Monitoring Framework. 3.4: National Min: National Emergency Preparedness and Emergency Response Plan for animal diseases has been Preparedness and developed. xiv Response Plan for Obj: Plan conforms to FAO/OIE manual . animal diseases. 85

OCR for page 65
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria 86 Capability (MOE) Comprehensive MOE 4: Partner country 4.1: National disease Min: National disease surveillance plan understands its EDP disease surveillance plan. exists. and pathogen baselines. Obj: National active and passive xv surveillance is established and executed IAW WHO Recommended Surveillance Standards, Second Edition. Comprehensive MOE 5: Partner country 5.1: Disease Min: Disease surveillance system is capable disease surveillance system is surveillance system is of detecting endemic EDP suspect cases. capable of detecting and capable of detecting Obj: Disease surveillance system is capable reporting suspect EDP cases to EDP cases. of detecting all suspect EDP cases. those responsible for human 5.2: Epidemiological Min/Obj: Epidemiological EDP data from and animal health. data from EDP case case investigations are shared with those investigations are responsible for human and animal health shared with those within partner country’s own government responsible for human structure. and animal health. 5.3: Laboratory test Min/Obj: Laboratory test reports from EDP results are provided to cases are shared with those responsible for those responsible for human and animal health within partner human and animal country’s own government structure. health. 5.4: All reported Min/Obj: All events constituting a PHEIC human are reported to WHO IAW IHR regulations. epidemiological events constituting a Public Health Emergency of International Concern (PHEIC) are reported xvi to the WHO . 5.5: All animal Min/Obj: All events constituting OIE reportable diseases are animal reportable diseases are reported to xvii OIE IAW OIE guidelines. reported to OIE. 5.6: Case data is Min/Obj: Reporting systems meet xviii xix shared via appropriate WHO and OIE standards for early reporting systems. detection and timely reporting. IMPROVING METRICS FOR THE DOD CTR PROGRAM

OCR for page 65
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria Capability (MOE) Timeliness MOE 6: Partner country 6.1: Epidemiological Min/Obj: EDP case investigation reports xx disease surveillance system is data from EDP case are shared within 24 hours of capable of early detection and investigations are investigation. timely reporting of EDP cases promptly reported to to those responsible for human those responsible for and animal health. human and animal health. 6.2: Laboratory test Min/Obj: Laboratory test results are shared results are promptly within 24-hours. provided to those responsible for human and animal health. 6.3: All human Min/Obj: All PHEIC reports are shared epidemiological IAW WHO IHR timelines. events constituting a Public Health Emergency of International Concern (PHEIC) are promptly reported to the xxi WHO. 6.4: All animal Min/Obj: All reportable animal disease reportable diseases are reports are shared with OIE IAW OIE APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT promptly reported to timelines. xxii the OIE. Comprehensive MOE 7: Partner country is 7.1: Suspect EDP Min: Standardized case investigation form capable of investigating and cases are investigated is completed and complies with WHO/OIE diagnosing EDP cases. and documented by standard. those responsible for Obj: Standardized case investigation form human or animal is completed in an electronic system and xxiii health. complies with WHO/OIE standard. 7.2: Appropriate Min/Obj: Samples are collected and samples are collected transported IAW CBEP procedures. and promptly trans- ported under optimum xxiv conditions for 87

OCR for page 65
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria 88 Capability (MOE) laboratory confirmation of the diagnosis. 7.3: Partner country Min: Diagnosis is made IAW CBEP is able to diagnose procedures and demonstrated through endemic EDPs. proficiency testing program. Obj: Diagnosis is made IAW CBEP procedures and demonstrated in country during real time case investigation and diagnosis. 7.4: Partner country Min: Capable of shipping samples IAW is able to utilize IATA (International Airline Transport international reference Association) regulations and requirement laboratories when agreements of the international reference lab. there is no country Obj: Established agreement exists within diagnostic capability. international reference labs to which samples will be routinely shipped. Timeliness MOE 8: Suspect EDP cases 8.1: Suspect cases are Min/Obj: Within 48-hours of suspect case are promptly investigated and promptly investigated being reported to those responsible for diagnosed. and assessed by those human and animal health IAW WHO responsible for human IHR/OIE guidelines.xxv or animal health. 8.2: Appropriate Min: Initiate sample transport within 24- samples are collected hours of case investigation. and promptly Obj: Initiate sample transport immediately transported under following case investigation. optimum conditions for laboratory confirmation of the diagnosis. 8.3: Partner country Min: Initiate appropriate diagnostic testing promptly initiates protocols/SOPs within 24-hours of receipt of diagnostic testing of samples. endemic EDPs. Obj: Initiate appropriate diagnostic testing protocols/SOPs immediately upon receipt of samples. 8.4: Partner country Min/Obj: Ship samples to international is able to rapidly reference laboratory within 24-hours of IMPROVING METRICS FOR THE DOD CTR PROGRAM

OCR for page 65
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria Capability (MOE) utilize international acquiring samples. reference laboratories when there is no in- country diagnostic capability. Sustainability MOE 1: Partner country EDP 1.1: Sustainability Min/Obj: Plan identifies human, resource CBEP Objective 4: Capability 1: Ensure the Secure and collection methods and plan to physically and operational needs to maintain collection developed consolidate associated research are maintain collection. of viruses and bacteria over time to support capabilities are collections of sustainable. pathogen baseline and facilitate U.S. efforts designed to be EDP and their to mitigate biological risks. sustainable within associated each partner research. country/region’s Sustainability MOE 2: Partner country 2.1: Sustainment cost Min/Obj: Cost is within minimally Capability 2: current operating Enhance partner BS&S practices and of application and budgeted resources of the partner country budget. country BS&S regulations concerning EDPs enforcement of BS&S and in partnership with others. standards and are absorbable with respect to guidelines and practices. cost. regulations. Sustainability MOE 3: Partner country 3.1: Sustainment Min/Obj: Cost is within minimally Capability 3: Enhance partner disease detection, diagnosis, Cost. budgeted resources of the partner country country disease and reporting capabilities are and in partnership with others. detection, absorbable with respect to xxvi diagnosis, and cost. reporting capabilities APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT Sustainability MOE 4: Partner country 4.1 Trainee test Min/Obj: 70% of trained Capability 4: Enhance partner demonstrates an ability to results. biosafety/biosecurity officers can country ability to retain and/or train qualified demonstrate continued proficiency (by train/retain personnel to desired standards. means of either written or hands-on testing) personnel trained at 6, 12, and 18 months following to desired completion of training. biosafety/ biosecurity standards. 89

OCR for page 65
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria 90 Capability (MOE) Legitimacy MOE 1: Research objectives 1.1: Credible, Min: CBEP sponsored research projects are CBEP Capability 1: Engage partner of partner country scientists appropriate and technically reviewed and approved by CBEP Supplemental country life and institutes are aligned with executable EDP SMEs. Objective 5: Facilitate the science national and international EDP research projects. Obj: Country projects on EDPs are engagement of personnel in priorities. competitively peer reviewed and funded by partner areas of interest international review board. country’s/regional to partner Transparency MOE 2: Partner country 2.1: U.S. access to Min/Obj: U.S. has access to all data generated scientific and country/ region scientists and institutes openly research data funded by by joint research projects. technical personnel and the United share research information and CBEP. in research areas of States. copies of EDP strains. 2.2: Copies of EDP Min: Copies of all requested EDP strains from interest to both the strains transferred to joint research projects are transferred to the partner U.S. U.S. country/region and Obj: Copies of all requested EDP strains are the United States. transferred to the U.S. 2.3: Contribution to Min: Partner country scientists present BTRP international scientific research results at relevant international community. conferences, and contribute to research articles/presentations published/presented within the international community. Obj: Partner country scientists are invited to present their research results at relevant international conferences, and have lead role in publishing research articles and/or conducting presentations within the international community. Safety MOE 3: Work of partner 3.1: Biosafety Min: Consistent with the minimum country scientists and institutes guidelines. requirements of WHO IHR. with EDPs meets U.S. or Obj: Consistent with the requirements of international guidelines for BMBL 5 and subsequent editions for safety in biosafety. accordance with CTB-09-405 Memorandum. Security MOE 4: Partner country 4.1: Biosecurity Min: Consistent with the CWA Biorisk scientists and institutes work standards. Management requirements. with EDPs is consistent with Obj: Consistent with the requirements of standards for biosecurity. DoDI 5210.89 outlined in CTB-09-405 Memorandum. Comprehensive MOE 1: Disclosed BW-related 1.1: Extent of Min: Scale and scope consistent with peaceful CBEP Capability 1: Eliminate infrastructure, materials, and elimination purposes and BWC treaty. Supplemental bioweapons technologies are eliminated. Obj: 100% of disclosed BW-related Objective 6: IMPROVING METRICS FOR THE DOD CTR PROGRAM

OCR for page 65
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria Capability (MOE) Eliminate any BW- related infrastructure is eliminated. related infrastructure infrastructure and technologies and technologies. encountered in a partner/country region. i Collection is defined as any collections of U.S. Select agents in the same location/facility. ii Number of locations must effectively mitigate identified risks based on a threat/risk assessment. iii Confirmation of EDP diagnosis does not require EDP storage except at central repository. iv CBEP refers to baseline as no increase in number of EDP collections during pre-engagement. v An established legal framework should span all levels (local, regional, national, and international). vi Biocontainment is the containment of extremely pathogenic organisms (as viruses) usually by isolation in secure facilities to prevent their accidental release especially during scientific research. http://www.merriam-webster.com/dictionary/biocontainment vii Epidemic is the occurrence of more cases of disease than expected in a given area or among a specific group of people over a particular period of time. http://www.cdc.gov/excite/classroom/outbreak/objectives.htm viii BTRP refers to diagnosis as the identification and analysis of the nature and underlying cause of an illness or infection. Diagnosis involves laboratory confirmation of the presence or absence of infection and is made by a chemical, microscopic, microbiologic, immunologic, or pathologic study of samples collected from suspect case of infection. For BTRP, diagnosis does not refer to medical APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT diagnosis in which a medical condition or disease is identified/recognized by its outward signs and symptoms. ix Deleted reporting capability since not applicable to security. x Metric specifically references Pandemic Influenza Preparedness plans because these are the only pandemic plans being created globally. http://www.who.int/csr/disease/influenza/pipguidance2009/en/index.html xi World Health Organization (WHO) References CDC Emergency Preparedness and Response for templates on Bioterrorism Response Planning. http://www.bt.cdc.gov/planning/responseguide.asp xii Multi-hazard IHR hazards of relevance to BTRP include biological (infectious, zoonotic, and foodborne human pathogens). Source: International Health Regulations (IHR) 2005 Monitoring Framework for monitoring Progress in the implementation of IHR Core Capacities in State Parties. Annual Data Collection Tool (2010). http://www.who.int/ihr/checklist/en/index.html xiii "At the national level, state parties are required to assess all reports of urgent events within their territories within 48 hours by applying algorithm contained in Annex II of the IHR." "When a State Party identifies an event as notifiable, it must be notified to 91

OCR for page 65
92 WHO immediately*, i.e. within 24 hrs after having carried out the assessment of public health information related to the event. Such notification will include details of any health measure employed in response to the event as well as accurate and sufficiently detailed public health information available, including case definitions, laboratory results, and number of cases and deaths." "In addition to notification and consultation, state parties are required to inform WHO within 24 hrs of receipt of evidence of public health risks occurring outside their territory that may cause international disease spread." IHR (2005) defines immediately as: within 24 hrs. http://www.who.int/ihr/about/10things/en/index.html xiv FAO Animal Health Manual on the preparation of national animal disease emergency preparedness plans. http://www.fao.org/DOCREP/004/X2096E/X2096E00.htm xv Active surveillance includes activities in which those responsible for human and animal health identify new cases of disease (case finding) and/or involve interviewing patients, reviewing medical records to detect cases or after index case has been reported. Passive surveillance includes activities in which available data on reportable diseases is used or reporting is mandated or requested by those responsible for human and animal health. (Leon Gordis, Epidemiology Second Edition). WHO Recommended Surveillance Standards, Second edition. http://www.who.int/csr/resources/publications/surveillance/whocdscsrisr992.pdf xvi Events constituting a PHEIC are assessed by country using the decision instrument in Annex 2 of IHR. http://www.who.int/ihr/9789241596664/en/index.html xvii OIE reportable diseases http://www.oie.int/eng/maladies/en_classification2010.htm?e1d7 xviii WHO IHR Monitoring Framework for monitoring Progress in the implementation of IHR Core Capacities in State Parties. Annual Data Collection Tool (2010). http://www.who.int/ihr/checklist/en/index.html xix OIE World Animal Health Information System (WAHIS). http://www.oie.int/ENG/info/en_info.htm xx Immediate reporting of EDPs as required by CSTE and referenced by CDC for guidelines regarding nationally notifiable diseases. http://www.cste.org/dnn/LinkClick.aspx?fileticket=7CCxM20JUGg%3d&tabid=36&mid=1496 http://www.who.int/ihr/about/10things/en/index.html xxi Events constituting a PHEIC are assessed by country using the decision instrument in Annex 2 of the IHR. http://www.who.int/ihr/9789241596664/en/index.html xxii OIE reportable diseases http://www.oie.int/eng/maladies/en_classification2010.htm?e1d7 xxiii WHO/OIE standards http://www.who.int/ihr/IHR_Monitoring_Framework_Checklist_and_Indicators.pdf xxiv Optimum conditions include adherence to appropriate BS&S regulations, cold-chain maintenance, and universal precautions. xxv http://www.who.int/ihr/IHR_Monitoring_Framework_Checklist_and_Indicators.pdf xxvi Cost is the combined metric of multiple sustainment topics, including but not limited to: operations, maintenance, technology, infrastructure, knowledge, skills, abilities, educational, logistics, regulatory, cultural) etc. Operations cost is for biosecurity and collection facility. “Maintenance costs” is for biosecurity and collection facility. IMPROVING METRICS FOR THE DOD CTR PROGRAM