Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 65
APPENDIX B
REPRODUCTION OF THE DEPARTMENT OF DEFENSE
COOPERATIVE THREAT REDUCTION METRICS REPORT
Delivered by Department of Defense to Congress
September 2010
Introduction
This report is submitted in accordance with Section 1304 of the National Defense Authorization
Act for Fiscal Year 2010, Title XIII of Public Law 111-84. Section 1304 provides that the
Secretary of Defense shall develop and implement metrics to measure the impact and
effectiveness of activities of the Department of Defense (DoD) Nunn-Lugar Cooperative Threat
Reduction (CTR) Program. The metrics to be developed would measure the effectiveness of
projects and activities undertaken by DoD to prevent the proliferation of chemical, nuclear, and
biological weapons and weapons of mass destruction (WMD)-related materials, technologies,
and expertise.
Background
In the National Academy of Sciences’ (NAS) 2009 report, Global Security Engagement: A New
Model for Cooperative Threat Reduction, NAS affirmed the usefulness of the traditional metrics
of the DoD CTR Program for evaluating effectiveness in traditional CTR WMD elimination or
site security program areas, in which concrete actions such as chemical munitions, warhead
delivery systems, and launchers destroyed or chemical or nuclear warheads and weapons
materials secured clearly lead to threat reduction. Such activities break down into discrete events
with clear end points, and thus metrics that can be incrementally reported, as on the CTR
Scorecard (Appendix 1), continue to be appropriate for those types of programs. However, the
NAS report accurately noted that CTR program responses to new, often less quantifiable threats
in this century will require new methods to measure success. The challenge is to find
measureable performance indicators that capture the true value of program activities that may not
lend themselves readily to raw quantitative measurement, thus making it more difficult to
document the relative success of the activity.
In recent years, the CTR program has increasingly become involved in capacity-building efforts,
notably in the Cooperative Biological Engagement Program (CBEP), formerly the Biological
Threat Reduction Program (BTRP), and the WMD Proliferation Prevention Program (WMD-
PPP). In these programs, threat reduction is accomplished through building the capacity of the
partner state to conduct surveillance to deter or detect and report or respond to emerging
biological and WMD trafficking threats. Similarly, the CTR Program is building the capacity of
the Russian Federation to sustain upgraded physical protection systems at nuclear weapons
65
OCR for page 66
66 IMPROVING METRICS FOR THE DOD CTR PROGRAM
storage sites. Although increased partner state capacity makes an important contribution to a
reduction in the WMD threat to the United States, this is more difficult to quantify than numbers
of missiles or other forms of strategic offensive arms destroyed as in the traditional programs.
CTR’s prevention mission is difficult to quantify. Although a number of output measures can be
counted, they are proxies for the outcome of deterring terrorists and proliferators from accessing
WMD and related materials and expertise. We can measure the amount of equipment provided
and the number of training events conducted or scientists engaged; however, we need better
measures to show that these efforts actually result in changed practices or additional
effectiveness.
As an example that makes evident the findings in the NAS report and reflects a matter DoD has
recognized in connection with the CTR Program, a metric based on the number of Zonal
Diagnostic Laboratories built under the BTRP may not accurately reflect the partner country’s
enhanced capability for pathogen detection and reporting. Indeed, such a metric could create
unintended incentives to continue building additional laboratories that do not make a positive
contribution to threat reduction, when what is really needed might be additional training to make
better use of the existing laboratories’ capabilities for pathogen detection. In WMD-PPP, the
raw number of radars, sensors, and patrol boats provided for border security, although useful,
may not be as important as the effectiveness of the information sharing among those systems,
which is a function of system integration, training and effective decision making rather than of
the number of systems provided.
In light of these issues, Congress has mandated the development and implementation of metrics
to measure the impact and effectiveness of projects and activities of the CTR program to address
threats arising from the proliferation of chemical, nuclear, and biological weapons and WMD-
related materials, technologies, and expertise. Additionally, Section 1304 requires submission,
no later than 270 days after the enactment of the Act, of a report describing the metrics
developed and implemented. This report addresses these Section 1304 requirements.
Discussion
The metrics described in this report are designed to measure appropriately the impact of each
CTR program area, as the CTR program is currently constituted for Fiscal Year 2010, and does
not attempt to speculate on what metrics might be appropriate for future areas of programmatic
or geographic expansion. Neither do the metrics attempt to determine whether the activities of
the CTR program are the “right” activities. The metrics described in this report are intended to
best measure the effectiveness of the CTR Program in conducting these established programs.
These metrics are not intended to revise the method for establishing these objectives.
Figure 1 illustrates the organization of the Department of Defense as it relates to the CTR
program. CTR executes programs as directed in program guidance from the Deputy Assistant to
the Secretary of Defense (Treaties and Threat Reduction) (DATSD(T&TR)). This program
guidance provides instruction on how to implements policy guidance issued by the Office of the
Under Secretary of Defense for Policy (USD/P), which has been coordinated with all relevant
agencies within the Executive Branch.
OCR for page 67
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 67
Those responsible for each CTR program area, in concert with applicable stakeholders, reviewed
their specific program objectives and either established new program metrics or confirmed the
validity of existing program metrics. Although not culled out specifically in this report, except
as noted in the CTR Scorecard (Appendix 1), the metrics currently in use for the Strategic
Offensive Arms Elimination (SOAE) program will not change. The results of this review and
analysis process are provided in this report. Each report section provides a brief program
description, methodology for metrics development, and a description of metrics that includes
reporting and re-evaluation activities.
Conclusion
CTR program metrics serve as a means to evaluate how well program objectives have been
accomplished. Several traditional CTR program metrics are still very applicable and useful
measures of CTR Program threat reduction achievements. For those programs involved in
capacity-building efforts, notably the CBEP, basic metrics need to be specifically tailored and
evaluated individually at a project level. A number of program benefits that would be useful and
meaningful in the larger USG threat reduction effort and are routinely considered in establishing
execution plans, such as partner country contributions and commitments, or ability to leverage
multiple sources of expertise or funding, were considered but not recommended since these
benefits ultimately do not trace back directly to the CTR program objectives. As the CTR
program continues to evolve, we will continue to assess its metrics to ensure they best represent
achievement of new objectives.
Figure 1. The CTR program within the DoD organizational structure
Secretary of Defense
(OSD)
Deputy Secretary of Defense
(OSD)
Under Secretary of Defense Under Secretary of Defense
(Policy) (Acquisition, Technology, and Logistics)
(USD/P) (USD/AT&L)
Assistant to the Secretary of Defense for
Assistant Secretary of Defense
Nuclear and Chemical and Biological Defense Programs
Global Strategic Affairs
(ATSD(NCB) )
(ASD/GSA)
Policy Implementation
Guidance Guidance
Deputy Assistant to the Defense Threat Reduction Agency
&
Deputy Assistant Secretary of Defense for
Secretary of Defense for (DTRA)
Acquisition
Countering Weapons of Mass Destruction
Treaties and Threat Oversight
(DASD(CWMD))
Reduction
(DATSD(T&TR))
Associate Director, Operations
Cooperative Threat Reduction Policy
Enterprise
(CTR Policy)
(ADOP)
Cooperative Threat Reduction
Directorate
(OP-CT)
OCR for page 68
68 IMPROVING METRICS FOR THE DOD CTR PROGRAM
Cooperative Biological Engagement Program
Program Description
The Cooperative Biological Engagement Program (CBEP) has four main objectives that are
identified in the current policy and implementation guidance issued to the Defense Threat
Reduction Agency (DTRA) for implementation:
1. Secure and consolidate collections of especially dangerous pathogens (EDPs) and their
associated research at a minimum number of secure health and agricultural laboratories or
related facilities;
2. Enhance partner country/region’s capability to prevent the sale, theft, diversion, or
accidental release of biological weapons (BW)-related materials, technology, and
expertise by improving biological safety and security (BS&S) standards and procedures;
3. Enhance partner country/region’s capability to detect, diagnose, and report endemic and
epidemic, man-made or natural EDPs, bio-terror attacks, and potential pandemics; and
4. Ensure the developed capabilities are designed to be sustainable within each partner
country/region’s current operating budget.
The fourth objective identified above, sustainability, is a cross-cutting objective to ensure that
the capabilities provided by the CBEP are adapted and owned by the partner country. The
ultimate aim for CBEP implementation is that the partner country take ownership of a
sustainable capacity to conduct biosurveillance, meet BS&S standards, and conduct research
effectively on its own. Promoting partner country ownership is one of the critical program
objectives and one of the most significant CBEP challenges.
The CBEP’s supplemental objectives – objectives noted as supplemental in the guidance
received – include facilitating the engagement of partner country’s/regional scientific and
technical personnel in research areas of interest to both the partner country/region and the United
States and eliminating any BW infrastructure and technologies encountered in a partner
country/region. CBEP assistance also supports host country objectives in meeting the World
Health Organization (WHO) International Health Regulations (IHR).
CBEP implementation to meet these objectives includes collaborative biological research (CBR),
biosurveillance, and enhancement of BS&S. CBR efforts bring together U.S. and international
scientists in cooperative research of mutual interest in support of threat reduction priorities.
Biosurveillance projects include providing training to animal and human health experts on
disease recognition, diagnosis, and reporting; support for laboratory diagnostic equipment and
reagents; and implementation of the Electronic Integrated Disease Surveillance System (EIDSS).
BS&S enhancements include bioethics training, training on safe clinical and laboratory practices;
installation of security systems at laboratories; implementation of the Pathogen Asset Control
System (PACS) to track inventory; and support for development of biorisk management
processes and procedures.
Methodology for Development of New Metrics
Developing metrics for the CBEP presented several challenges. First, the program seeks to
achieve concrete improvements in disease detection, biological safety, and research systems in
an environment not fully under its control, with the result that accountability and analysis of
OCR for page 69
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 69
program impact are complex, difficult matters. Also, the metrics for the CBEP need to establish a
solid, traceable link from CBEP-funded efforts to concrete threat reduction achievements.
To address these challenges and establish actionable metrics, the CBEP adapted a Joint
Capabilities Integration and Development System-like process to develop mission and
performance measures. This effort took place over the course of one year of collaborative
development. The CBEP metrics team first developed an initial capabilities document that lays
out the key capabilities required in the program’s strategic guidance and the attributes supporting
these capabilities. From these attributes, the team then developed measures of effectiveness
(MOEs) to assess mission accomplishment. These MOEs are directly traceable to the program’s
policy and implementation guidance. Finally, the team developed measurement values to serve
as indicators that the critical MOEs have been achieved. The objective and minimum thresholds
for these measurement values indicate the desired end state and the minimally acceptable level of
achievement respectively.
Throughout the process, the CBEP engaged program stakeholders from across disciplines as
diverse as clinical practice, epidemiology, laboratory, biosafety, veterinary, biosecurity, systems
engineering, and training. The use of small working groups by discipline and progressive
reviews by outside parties and program leadership enhanced the accuracy, comprehensiveness,
and usefulness of the final product.
Description of New Metrics
The table in Appendix 2 depicts the linkages and associations across 6 program objectives, 5
program capabilities, 10 attributes, 20 MOEs, measurement values, and minimum/objective
values.
The CBEP will utilize these revised metrics throughout planning, implementation, and final
assessment of country/region engagements.
Using a systems engineering approach, all project officers will use metrics to plan and execute
work, including designing a concept of operations to support MOEs in a partner country or
region. The project implementation team will focus on achieving a capability in a partner
country or region as defined by the MOEs.
Interim performance against the metrics will be assessed through a seamless, collaborative test
and evaluation process that will include tabletop and field training exercises, as well as informal
assessments by CBEP collaborators. In addition, ongoing proficiency and competency testing
will assess capability levels, target areas for improvement, and build capacity on the micro level.
Results will also be fed back to the implementation teams, allowing them to make adjustments to
their execution plans. Feedback from CBEP collaborators is a part of each engagement activity
in each country or region, and the program holds an integrated country review at least every
quarter of the year to identify issues, necessary changes in approach, action items, and those
responsible for taking action. This iterative, ongoing assessment approach helps to ensure that
enhancements are taking place and that the implementation approach is delivering results.
OCR for page 70
70 IMPROVING METRICS FOR THE DOD CTR PROGRAM
In conducting formal, large-scale assessments of program achievements relative to the metrics,
the CBEP will leverage international authorities such as the WHO, the Food and Agriculture
Organization of the United Nations (FAO), or the World Organization for Animal Health (OIE)
to seek third-party validation of program accomplishments. These formal reviews will assess the
cumulative results of CBEP engagement and identify areas for continuing cooperation.
All of the above assessment efforts will document performance relative to the 20 MOEs
identified in Appendix 2. The use of multiple metrics allows for a thorough, broad evaluation of
program performance; however, it makes providing a succinct, top-level report of performance a
challenge. The CBEP plans to report the cumulative performance of each country against the
comprehensive metrics on an annual basis. This summary report would offer an overview of
each country’s “proficiency” relative to the CBEP objectives. Proficiency could be indicated in
some manner on a summary graphic; however, the complexity of the program does not lend itself
to simplification into a single scorecard metric.
Chemical Weapons Elimination
Program description
The original program task for the Chemical Weapons Elimination program was to assist the
Russian Federation in establishing a chemical weapons destruction facility (CWDF) near
Shchuch’ye, located in the Kurgan District, to destroy nerve agent-filled munitions located in the
Planovy CW storage facility in a safe, secure, and environmentally sound manner. The
assistance provided included the following:
• Site clearing, water drainage, and preparation of the land for construction;
• Evaluating, optimizing, and scaling-up of the Russian-developed nerve agent destruction
process;
• Developing, designing, fabricating, and testing of the munitions processing equipment;
• Integrating and systemizing the processing and associated support equipment and
facilities; and
• Commissioning the facilities as an acceptable industrial complex under Russian
Federation standards and laws.
The United States and other members of the Group of Eight Global Partnership against the
Spread of Weapons and Materials of Mass Destruction assisted with completion of this facility.
As construction of the first destruction building and other parts of the complex neared
completion, the Russian Federation requested continued U.S. technical assistance to monitor and
evaluate the agent destruction system and support equipment maintenance through the remaining
agent elimination efforts as a means of assuring continued full functionality of the systems and
equipment. The Russian Federation initiated destruction in the first building in March 2009.
The second destruction building is still under construction and is scheduled to be complete and in
operation in 2011.
OCR for page 71
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 71
Program-Level Metrics
Methodology for Development of New Program-Level Metrics
Technical assistance will primarily take the form of engineering advice to help resolve process
and equipment failures, identify potential future failures, and ensure replacement spares are on
hand. Quantifying how this advice/assistance provided by CTR contributes to the overall
availability of the facility to perform its agent destruction mission is the challenge. Two
quantifiable factors that can be identified and tracked from a strategic perspective are the amount
of funding invested in providing technical assistance and the amount of agent and munitions
destroyed by the facility.
Although it can be quantified, tracking funds expended on technical assistance provides no clear
measure of whether the technical assistance actually is contributing to the facility’s ability to
destroy nerve agent. Costs are incurred whether or not any deficiency is successfully identified,
isolated, and corrected. However, the number of munitions processed and the tonnage of nerve
agent destroyed does provide a valuable and useful link to determining how successful the
provided technical assistance has been toward keeping the facility functioning at capacity.
Progress in the destruction of agents and processing of munitions is clearly indicative of
continued facility functionality and operation. Therefore, we determined that using the amount
of agent destroyed and tracking the number of munitions processed would be the best metrics to
measure the success of the technical support presently being provided.
Description of New Program-Level Metrics
As nerve agent-filled munitions are brought to the facility for processing, the weight of nerve
agent extracted and neutralized is determined and the number of individual munitions bodies
processed is captured by the plant’s automated process control system. The Russian Federation’s
operations staff consolidates and reports the two quantities, and the U.S. on-site technical staff
obtains the data from the Russian Federation and maintains a cumulative total of both data
points. The on-site staff then provides the information weekly to the U.S. Project Management
Office.
Table 1 below provides an example of the metrics provided in the weekly report:
Table 1. CWE Metrics
Report Date Total Rounds Total Metric Tons
August 3, 2010 532,806 1461.8
However, since the number of munitions (total rounds) correlates closely to the amount of nerve
agent destroyed, for the purposes of reporting externally in the “CTR scorecard” only the metric
tons of agent destroyed will be reported. (Appendix 1),
OCR for page 72
72 IMPROVING METRICS FOR THE DOD CTR PROGRAM
Project-Level Metrics
Additional metrics will be maintained at a project level to assess how effectively the destruction
facility is being used in relation to performance of maintenance and repairs that help ensure the
safety of the destruction process.
Methodology for Development of New Project-Level Metrics
Reliability, Availability, and Maintainability (RAM) Analysis
When fully constructed, the Shchuch’ye CWDF will consist of two main munitions destruction
buildings, with two destruction process lines (DPLs) operating in each building. A RAM
analysis to establish the baseline threshold and objective capacities of the facility in this
configuration was completed in August 2005. That analysis determined that the design capacity
threshold for the completed facility is 1,000 metric tons of agent destruction per year, with a
design capacity objective of 1,700 metric tons of agent destruction per year (capacity for two
main destruction buildings with two DPLs operating in each).
Description of new project-level metrics:
Scheduled Facility Downtime
Scheduled Facility Downtime (SFD) tracks any days when it is planned that the facility will be
idle and not processing munitions. This can be due to scheduled maintenance, campaign
changeovers, or any administrative downtime where the idling of the process was planned in
advance. In the RAM analysis, the SFD was estimated to average 25 percent. For a facility of
this complexity, with its associated preventive maintenance requirements both for the destruction
process and all of the support systems, such an estimate is reasonable.
Minimizing down days, whether scheduled or unscheduled, is extremely important. Using
scheduled downtime efficiently and effectively by scheduling periodically required maintenance
actions for those periods can maintain overall production capabilities, as well as the safety of the
facility. U.S. technical support efforts help the Russian Federation to plan and optimize
scheduled maintenance outages and helps ensure that the scheduled outage does not extend
beyond the originally scheduled duration by having the right parts, equipment and vendor
support pre-staged to take full advantage of the outage.
Since the start of munitions destruction, with only one of the two destruction buildings in
operation, the cumulative SFD has been 19.25%. A rolling 12-month average helps identify
potential trends that might get masked over the long term. The Shchuch’ye CWDF’s current
rolling 12-month SFD average is 26.85%. This higher percentage is attributable to the recent
scheduled changeover from one type of munition to another.
In terms of the objectives and thresholds for agent tons per year, an objective SFD average of 20
percent per year would support meeting the 1,700 metric tons of agent per year average with two
OCR for page 73
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 73
main production facilities operational. Similarly, an SFD average of 35 percent would result in
meeting the threshold average of 1,000 metric tons per year.
Unscheduled Facility Downtime
A second key area for assessing overall throughput is Unscheduled Facility Downtime (UFD).
Efficient and effective technical support helps minimize the number of UFD days. This support
encompasses proactive failure root-cause analysis, analysis of spare parts consumption,
provisioning of the right amount and types of spare parts, and the selective replacement of items
before failure during scheduled maintenance outages.
Objective and threshold limits for this metric are being developed.
The Shchuch’ye CWDF’s current overall cumulative UFD percentage is 6.68%, with a 12-month
rolling average of 4.66%. The UFD for the Shchuch’ye CWDF is significantly lower than would
be expected for a complex CWDF, based upon experience with similar facilities in the U.S.
program, especially during the initial year of operation.
Facility Achieved Availability
The Achieved Availability metric encompasses the overall throughput of the facility achieved
versus the actual design capacity of the facility. During the current phase of operations, it is
based upon the one completed main destruction building (overall capacity based upon two
DPLs). The achieved availability with one main destruction building is a good indicator of what
could be expected with two main destruction buildings in terms of meeting the overall objective
of 1,700 metric tons per year for a facility operating with two main destruction facilities.
Based upon the RAM analysis, the estimated Achieved Availability was 35.4 percent, which
would result in an overall annual throughput rate of 1,382.5 metric tons for the facility with two
main destruction buildings in operation with two operational DPLs in each. In order to achieve
the objective of 1,700 metric tons per year, an overall Achieved Availability of approximately 44
percent would be required. Similarly, it is estimated that the 1,000 metric tons per year threshold
would be met with an Achieved Availability of 26 percent. To date, Building 101A has an
Achieved Availability of 54 percent, much higher than originally anticipated.
The Achieved Availability calculation takes into account Scheduled and Unscheduled Facility
Downtimes. Based on operations to date, these two components account for almost 26 percent
of the nearly 45 percent of lost production time for Building 101A associated with an Achieved
Availability of 54 percent. The remaining 19 percent of downtime is primarily due to unplanned
and planned maintenance of one or the other of the two DPLs while one remains in operation.
Summary
We are using three project-level metrics to track and evaluate the trend of the effectiveness of the
facility and the provided technical support. Based upon an analysis of available RAM reports,
OCR for page 74
74 IMPROVING METRICS FOR THE DOD CTR PROGRAM
the following are the project-level metrics, with associated objective and threshold values, as of
June 30, 2010:
Scheduled Facility Downtime (SFD):
Objective: 20 percent (RAM analysis rate of 1,700 metric tons/year)
Threshold: 35 percent (RAM analysis rate of 1,000 metric tons/year)
Cumulative SFD: 19.25 percent
Last 12 Months SFD: 26.85 percent
Unscheduled Facility Downtime (UFD):
Cumulative UFD: 6.7 percent
Last 12 Months UFD: 4.7 percent
Facility Achieved Availability (FAA):
Objective: 44 percent
Threshold: 26 percent
Cumulative FAA: 53.97 percent
Last 12 Months FAA: 52.51 percent
OCR for page 75
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT 75
Nuclear Weapons Safety and Security
Program Description
This program supports proliferation prevention by enhancing the security systems of nuclear
weapons storage sites using DoD nuclear security standards as a basis for design. It also trains
cadres of security systems operators, administrators, and maintenance and repair technicians and
provides capabilities, such as the personnel reliability program and guard force training systems.
In response to a request from Russia and former President Bush’s commitment at Bratislava,
Slovakia in February 2005, DoD and the Department of Energy (DOE) enhanced security
systems at requested locations that permanently or temporarily store strategic and non-strategic
(tactical) nuclear weapons. DoD upgraded a total of 24 sites and, additionally, supports
proliferation prevention by enhancing the security and safety of nuclear weapons during
shipment through the Nuclear Weapons Transportation Security (NWTS) Program.
Methodology for Development of New Metrics
Nuclear Weapons Safety and Security (NWS&S) projects have been focused on efforts
exclusively in Russia, and the metric used to measure program success reported the number of
nuclear weapons storage sites upgraded. This remains an appropriate measure of the impact and
effectiveness of program efforts in Russia and, therefore, this information will continue to be
reported on the CTR Scorecard (Appendix 1) in its present format.
Recently, however, with the potential global expansion of the Nuclear Weapons Storage Security
(NWSS) program to other partner countries, DTRA has been working to develop enhanced
metrics that better reflect how our efforts are contributing to overall threat reduction. These new
metrics are aimed at measuring the capability of Russia (and in the future, other countries) to
sustain the ability to store and transport nuclear materials safely and securely. The development
of these metrics presents a substantial challenge, as we need to develop measures that accurately
and correctly reflect an intangible capability. DoD and DOE have been working together on this
issue and have jointly developed eight principles to evaluate sustainment capabilities. Although
these principles are presently used as internal measures of effectiveness and are still being
refined, the NWSS program has already begun to apply these principles to the sustainment of the
site security projects in Russia.
Description of New Metrics
DoD and DOE jointly developed eight sustainment and logistics principles for the Russia NWSS
projects, and proposed metrics would follow these standards. These metrics also meet other
international standards, including those used by the International Atomic Energy Agency (IAEA)
for logistics and maintenance management of nuclear facilities. The metrics include:
• Performance/Capability Assurance,
• Configuration Management,
• Procedures and Processes,
OCR for page 82
82 IMPROVING METRICS FOR THE DOD CTR PROGRAM
OCR for page 83
Appendix 2: CBEP Objective, Capability and MOE Traceability
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
Capability (MOE)
Consolidation MOE 1: Partner country EDP 1.1: Number of EDP Min: No increase in the number of EDP
CBEP Objective 1: Capability 1:
i iv
Secure and Secure and collections.
collections and associated collections over baseline .
consolidate consolidate research are consolidated into Obj: No more than 1 EDP collection.
ii
collections of collections of a minimum number of
especially dangerous EDPs and their iii
locations.
pathogens (EDP) associated
and their associated research.
research at a Security MOE 2: Partner country 2.1: Biosecurity Min: Consistent with the European
minimum number of EDPs and associated research compliance. Committee for Standardization (CEN)
secure health and are secured in a manner Workshop Agreement (CWA) Biorisk
agricultural consistent with standards. Management requirements.
laboratories or Obj: Consistent with the requirements of
related facilities. DoDI 5210.89 outlined in CTB-09-405
Memorandum "Biological Threat Reduction
Program (BTRP) Biosafety and Security
Standards (BS&S) ,” dated November 20,
2009 (CTB-09-405 Memorandum).
Consolidation MOE 1: Same as Objective 1, 1.1: Same as Min: Same as Objective 1, Capability 1
CBEP Objective 2: Capability 1:
Enhance partner Secure and Capability 1 MOE 1 Objective 1, MOE 1.
country/region’s consolidate Capability 1 MOE 1. Obj: Same as Objective 1, Capability 1
capability to prevent collections of MOE 1.
the sale, theft, EDP and their Security MOE 2: Same as Objective 1 2.1: Same as Min: Same as Objective 1, Capability 1
diversion, or associated Capability 1 MOE 2 Objective 1, MOE 2.
accidental release of research. Capability 1 MOE 2 Obj: Same as Objective 1, Capability 1
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT
biological weapons MOE 2.
(BW)-related Enforceability MOE 3: Partner country has 3.1: Established legal Min: Partner country laws, guidelines, and
Capability 2:
materials, Enhance partner BS&S laws and regulations frameworkv. regulations demonstrate commitment and
technology, and country BS&S governing work with EDPs. progress towards compliance with
expertise by standards and international laws, regulations, and treaties
improving biological practices. (BWC, WHO, OIE, UN, etc.).
safety and security Obj: Partner country laws, guidelines, and
(BS&S) standards regulations are compliant with DODI
and procedures. 5210.89 to the maximum extent possible and
CDC/NIH “Biosafety in Microbiological and
Biomedical Laboratories, 5th edition, 2007”
(BMBL 5) and subsequent versions.
83
OCR for page 84
84
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
Capability (MOE)
3.2: Level of Min: Compliance at the national level.
Regulation. Obj: Compliance at the local, regional, and
national levels.
Safety MOE 4: Partner country 4.1: Biosafety Min: Consistent with the minimum
guidelines for work with EDPs guidelines. requirements of WHO IHR.
meet U.S. or international Obj: Consistent with the requirements of
guidelines for biosafety. BMBL 5 and subsequent editions for safety
in accordance with CTB-09-405
Memorandum.
4.2: Facility specific Min/Obj: Yes.
biosafety plans exist.
Security MOE 5: Partner country 5.1: Biosecurity Min: Consistent with the CWA Biorisk
standards for work with EDPs standards. Management requirements.
are consistent with standards Obj: Consistent with the requirements of
for biosecurity. DoDI 5210.89 outlined in CTB-09-405
Memorandum.
5.2: Facility specific Min/Obj: Yes.
biosecurity plans
exist.
Transparency MOE 6: Partner country 6.1: BS&S standards Min: Available to independent third party.
BS&S standards and practices are available. Obj: Available to U.S. Government.
are transparent.
6.2: Biosecurity Min/Obj: Immediate notification of the
Event Notification. theft or loss of an EDP to the appropriate
Ministry and law enforcement/security
organizations, as well as in accordance with
international obligations and treaties.
6.3: Biosafety Event Min/Obj: Immediate notification of the
Notification. release of an EDP causing occupational
exposure or release of an EDP outside of the
vi
primary barriers of the biocontainment area
to the appropriate Ministry and law
enforcement/security organizations, as well
as in accordance with international
obligations and treaties.
IMPROVING METRICS FOR THE DOD CTR PROGRAM
OCR for page 85
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
Capability (MOE)
MOE 1: Partner country 1.1: Biosafety Min: Consistent with the minimum
CBEP Objective 3: Capability 1:
Enhance partner Enhance partner Safety disease detection and diagnosis guidelines requirements of WHO IHR.
country/region’s country disease capability meets U.S. and/or Obj: Consistent with the requirements of
capability to detect, detection, international guidelines for BMBL 5 and subsequent editions for safety
viii
diagnose, and report biosafety. in accordance with CTB-09-405
diagnosis , and
endemic and Memorandum.
reporting
vii
epidemic , man- capabilities.
Security MOE 2: Partner country 2.1: Biosecurity Min: Consistent with the CWA Biorisk
made or natural
ix standards Management requirements.
EDPs, bio-terror disease detection and
Obj: Consistent with the requirements of
attacks, and potential diagnostic infrastructure
DoDI 5210.89 outlined in CTB-09-405
pandemics. complies with standards for
Memorandum.
biosecurity.
Comprehensive MOE 3: Partner country has 3.1: National Min: National Pandemic Influenza
preparedness and response Pandemic Influenza Preparedness and Response Plan has been
plans. Preparedness and developed.
x Obj: Plan conforms to WHO guidance
Response plan .
document for Pandemic Influenza
Preparedness and Response.
3.2: Bioterrorism Min: Bioterrorism preparedness and
preparedness and response are covered in a developed Public
response plan. Health Emergency/Bioterrorism Response
Plan.
Obj: Plan conforms to HHS/CDC
xi
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT
template .
xii Min: Multi-hazard National Public Health
3.3: Multi-Hazard
Emergency Preparedness and Response Plan
National Public Health
for Biological Hazards has been developed.
Emergency
Obj: Plan conforms to core biological
Preparedness and
xiii
capacity requirements IAW WHO/IHR
Response Plan.
Monitoring Framework.
3.4: National Min: National Emergency Preparedness and
Emergency Response Plan for animal diseases has been
Preparedness and developed.
xiv
Response Plan for Obj: Plan conforms to FAO/OIE manual .
animal diseases.
85
OCR for page 86
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
86
Capability (MOE)
Comprehensive MOE 4: Partner country 4.1: National disease Min: National disease surveillance plan
understands its EDP disease surveillance plan. exists.
and pathogen baselines. Obj: National active and passive
xv
surveillance is established and executed
IAW WHO Recommended Surveillance
Standards, Second Edition.
Comprehensive MOE 5: Partner country 5.1: Disease Min: Disease surveillance system is capable
disease surveillance system is surveillance system is of detecting endemic EDP suspect cases.
capable of detecting and capable of detecting Obj: Disease surveillance system is capable
reporting suspect EDP cases to EDP cases. of detecting all suspect EDP cases.
those responsible for human 5.2: Epidemiological Min/Obj: Epidemiological EDP data from
and animal health. data from EDP case case investigations are shared with those
investigations are responsible for human and animal health
shared with those within partner country’s own government
responsible for human structure.
and animal health.
5.3: Laboratory test Min/Obj: Laboratory test reports from EDP
results are provided to cases are shared with those responsible for
those responsible for human and animal health within partner
human and animal country’s own government structure.
health.
5.4: All reported Min/Obj: All events constituting a PHEIC
human are reported to WHO IAW IHR regulations.
epidemiological
events constituting a
Public Health
Emergency of
International Concern
(PHEIC) are reported
xvi
to the WHO .
5.5: All animal Min/Obj: All events constituting OIE
reportable diseases are animal reportable diseases are reported to
xvii OIE IAW OIE guidelines.
reported to OIE.
5.6: Case data is Min/Obj: Reporting systems meet
xviii xix
shared via appropriate WHO and OIE standards for early
reporting systems. detection and timely reporting.
IMPROVING METRICS FOR THE DOD CTR PROGRAM
OCR for page 87
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
Capability (MOE)
Timeliness MOE 6: Partner country 6.1: Epidemiological Min/Obj: EDP case investigation reports
xx
disease surveillance system is data from EDP case are shared within 24 hours of
capable of early detection and investigations are investigation.
timely reporting of EDP cases promptly reported to
to those responsible for human those responsible for
and animal health. human and animal
health.
6.2: Laboratory test Min/Obj: Laboratory test results are shared
results are promptly within 24-hours.
provided to those
responsible for human
and animal health.
6.3: All human Min/Obj: All PHEIC reports are shared
epidemiological IAW WHO IHR timelines.
events constituting a
Public Health
Emergency of
International Concern
(PHEIC) are promptly
reported to the
xxi
WHO.
6.4: All animal Min/Obj: All reportable animal disease
reportable diseases are reports are shared with OIE IAW OIE
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT
promptly reported to timelines.
xxii
the OIE.
Comprehensive MOE 7: Partner country is 7.1: Suspect EDP Min: Standardized case investigation form
capable of investigating and cases are investigated is completed and complies with WHO/OIE
diagnosing EDP cases. and documented by standard.
those responsible for Obj: Standardized case investigation form
human or animal is completed in an electronic system and
xxiii
health. complies with WHO/OIE standard.
7.2: Appropriate Min/Obj: Samples are collected and
samples are collected transported IAW CBEP procedures.
and promptly trans-
ported under optimum
xxiv
conditions for
87
OCR for page 88
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
88
Capability (MOE)
laboratory
confirmation of the
diagnosis.
7.3: Partner country Min: Diagnosis is made IAW CBEP
is able to diagnose procedures and demonstrated through
endemic EDPs. proficiency testing program.
Obj: Diagnosis is made IAW CBEP
procedures and demonstrated in country
during real time case investigation and
diagnosis.
7.4: Partner country Min: Capable of shipping samples IAW
is able to utilize IATA (International Airline Transport
international reference Association) regulations and requirement
laboratories when agreements of the international reference lab.
there is no country Obj: Established agreement exists within
diagnostic capability. international reference labs to which samples
will be routinely shipped.
Timeliness MOE 8: Suspect EDP cases 8.1: Suspect cases are Min/Obj: Within 48-hours of suspect case
are promptly investigated and promptly investigated being reported to those responsible for
diagnosed. and assessed by those human and animal health IAW WHO
responsible for human IHR/OIE guidelines.xxv
or animal health.
8.2: Appropriate Min: Initiate sample transport within 24-
samples are collected hours of case investigation.
and promptly Obj: Initiate sample transport immediately
transported under following case investigation.
optimum conditions
for laboratory
confirmation of the
diagnosis.
8.3: Partner country Min: Initiate appropriate diagnostic testing
promptly initiates protocols/SOPs within 24-hours of receipt of
diagnostic testing of samples.
endemic EDPs. Obj: Initiate appropriate diagnostic testing
protocols/SOPs immediately upon receipt of
samples.
8.4: Partner country Min/Obj: Ship samples to international
is able to rapidly reference laboratory within 24-hours of
IMPROVING METRICS FOR THE DOD CTR PROGRAM
OCR for page 89
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
Capability (MOE)
utilize international acquiring samples.
reference laboratories
when there is no in-
country diagnostic
capability.
Sustainability MOE 1: Partner country EDP 1.1: Sustainability Min/Obj: Plan identifies human, resource
CBEP Objective 4: Capability 1:
Ensure the Secure and collection methods and plan to physically and operational needs to maintain collection
developed consolidate associated research are maintain collection. of viruses and bacteria over time to support
capabilities are collections of sustainable. pathogen baseline and facilitate U.S. efforts
designed to be EDP and their to mitigate biological risks.
sustainable within associated
each partner research.
country/region’s Sustainability MOE 2: Partner country 2.1: Sustainment cost Min/Obj: Cost is within minimally
Capability 2:
current operating Enhance partner BS&S practices and of application and budgeted resources of the partner country
budget. country BS&S regulations concerning EDPs enforcement of BS&S and in partnership with others.
standards and are absorbable with respect to guidelines and
practices. cost. regulations.
Sustainability MOE 3: Partner country 3.1: Sustainment Min/Obj: Cost is within minimally
Capability 3:
Enhance partner disease detection, diagnosis, Cost. budgeted resources of the partner country
country disease and reporting capabilities are and in partnership with others.
detection, absorbable with respect to
xxvi
diagnosis, and cost.
reporting
capabilities
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT
Sustainability MOE 4: Partner country 4.1 Trainee test Min/Obj: 70% of trained
Capability 4:
Enhance partner demonstrates an ability to results. biosafety/biosecurity officers can
country ability to retain and/or train qualified demonstrate continued proficiency (by
train/retain personnel to desired standards. means of either written or hands-on testing)
personnel trained at 6, 12, and 18 months following
to desired completion of training.
biosafety/
biosecurity
standards.
89
OCR for page 90
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
90
Capability (MOE)
Legitimacy MOE 1: Research objectives 1.1: Credible, Min: CBEP sponsored research projects are
CBEP Capability 1:
Engage partner of partner country scientists appropriate and technically reviewed and approved by CBEP
Supplemental
country life and institutes are aligned with executable EDP SMEs.
Objective 5:
Facilitate the science national and international EDP research projects. Obj: Country projects on EDPs are
engagement of personnel in priorities. competitively peer reviewed and funded by
partner areas of interest international review board.
country’s/regional to partner Transparency MOE 2: Partner country 2.1: U.S. access to Min/Obj: U.S. has access to all data generated
scientific and country/ region scientists and institutes openly research data funded by by joint research projects.
technical personnel and the United share research information and CBEP.
in research areas of States. copies of EDP strains. 2.2: Copies of EDP Min: Copies of all requested EDP strains from
interest to both the strains transferred to joint research projects are transferred to the
partner U.S. U.S.
country/region and Obj: Copies of all requested EDP strains are
the United States. transferred to the U.S.
2.3: Contribution to Min: Partner country scientists present BTRP
international scientific research results at relevant international
community. conferences, and contribute to research
articles/presentations published/presented
within the international community.
Obj: Partner country scientists are invited to
present their research results at relevant
international conferences, and have lead role in
publishing research articles and/or conducting
presentations within the international
community.
Safety MOE 3: Work of partner 3.1: Biosafety Min: Consistent with the minimum
country scientists and institutes guidelines. requirements of WHO IHR.
with EDPs meets U.S. or Obj: Consistent with the requirements of
international guidelines for BMBL 5 and subsequent editions for safety in
biosafety. accordance with CTB-09-405 Memorandum.
Security MOE 4: Partner country 4.1: Biosecurity Min: Consistent with the CWA Biorisk
scientists and institutes work standards. Management requirements.
with EDPs is consistent with Obj: Consistent with the requirements of
standards for biosecurity. DoDI 5210.89 outlined in CTB-09-405
Memorandum.
Comprehensive MOE 1: Disclosed BW-related 1.1: Extent of Min: Scale and scope consistent with peaceful
CBEP Capability 1:
Eliminate infrastructure, materials, and elimination purposes and BWC treaty.
Supplemental
bioweapons technologies are eliminated. Obj: 100% of disclosed BW-related
Objective 6:
IMPROVING METRICS FOR THE DOD CTR PROGRAM
OCR for page 91
CBEP Objective CBEP Attribute Measure of Effectiveness Indicator Evaluation Criteria
Capability (MOE)
Eliminate any BW- related infrastructure is eliminated.
related infrastructure infrastructure
and technologies and technologies.
encountered in a
partner/country
region.
i
Collection is defined as any collections of U.S. Select agents in the same location/facility.
ii
Number of locations must effectively mitigate identified risks based on a threat/risk assessment.
iii
Confirmation of EDP diagnosis does not require EDP storage except at central repository.
iv
CBEP refers to baseline as no increase in number of EDP collections during pre-engagement.
v
An established legal framework should span all levels (local, regional, national, and international).
vi
Biocontainment is the containment of extremely pathogenic organisms (as viruses) usually by isolation in secure facilities to
prevent their accidental release especially during scientific research.
http://www.merriam-webster.com/dictionary/biocontainment
vii
Epidemic is the occurrence of more cases of disease than expected in a given area or among a specific group of people over a
particular period of time.
http://www.cdc.gov/excite/classroom/outbreak/objectives.htm
viii
BTRP refers to diagnosis as the identification and analysis of the nature and underlying cause of an illness or infection. Diagnosis
involves laboratory confirmation of the presence or absence of infection and is made by a chemical, microscopic, microbiologic,
immunologic, or pathologic study of samples collected from suspect case of infection. For BTRP, diagnosis does not refer to medical
APPENDIX B: REPRODUCTION OF THE DOD CTR METRICS REPORT
diagnosis in which a medical condition or disease is identified/recognized by its outward signs and symptoms.
ix
Deleted reporting capability since not applicable to security.
x
Metric specifically references Pandemic Influenza Preparedness plans because these are the only pandemic plans being created
globally. http://www.who.int/csr/disease/influenza/pipguidance2009/en/index.html
xi
World Health Organization (WHO) References CDC Emergency Preparedness and Response for templates on Bioterrorism
Response Planning. http://www.bt.cdc.gov/planning/responseguide.asp
xii
Multi-hazard IHR hazards of relevance to BTRP include biological (infectious, zoonotic, and foodborne human pathogens). Source:
International Health Regulations (IHR) 2005 Monitoring Framework for monitoring Progress in the implementation of IHR Core
Capacities in State Parties. Annual Data Collection Tool (2010). http://www.who.int/ihr/checklist/en/index.html
xiii
"At the national level, state parties are required to assess all reports of urgent events within their territories within 48 hours by
applying algorithm contained in Annex II of the IHR." "When a State Party identifies an event as notifiable, it must be notified to
91
OCR for page 92
92
WHO immediately*, i.e. within 24 hrs after having carried out the assessment of public health information related to the event. Such
notification will include details of any health measure employed in response to the event as well as accurate and sufficiently detailed
public health information available, including case definitions, laboratory results, and number of cases and deaths." "In addition to
notification and consultation, state parties are required to inform WHO within 24 hrs of receipt of evidence of public health risks
occurring outside their territory that may cause international disease spread." IHR (2005) defines immediately as: within 24 hrs.
http://www.who.int/ihr/about/10things/en/index.html
xiv
FAO Animal Health Manual on the preparation of national animal disease emergency preparedness plans.
http://www.fao.org/DOCREP/004/X2096E/X2096E00.htm
xv
Active surveillance includes activities in which those responsible for human and animal health identify new cases of disease (case
finding) and/or involve interviewing patients, reviewing medical records to detect cases or after index case has been reported.
Passive surveillance includes activities in which available data on reportable diseases is used or reporting is mandated or requested
by those responsible for human and animal health. (Leon Gordis, Epidemiology Second Edition). WHO Recommended Surveillance
Standards, Second edition. http://www.who.int/csr/resources/publications/surveillance/whocdscsrisr992.pdf
xvi
Events constituting a PHEIC are assessed by country using the decision instrument in Annex 2 of IHR.
http://www.who.int/ihr/9789241596664/en/index.html
xvii
OIE reportable diseases http://www.oie.int/eng/maladies/en_classification2010.htm?e1d7
xviii
WHO IHR Monitoring Framework for monitoring Progress in the implementation of IHR Core Capacities in State Parties. Annual
Data Collection Tool (2010). http://www.who.int/ihr/checklist/en/index.html
xix
OIE World Animal Health Information System (WAHIS). http://www.oie.int/ENG/info/en_info.htm
xx
Immediate reporting of EDPs as required by CSTE and referenced by CDC for guidelines regarding nationally notifiable diseases.
http://www.cste.org/dnn/LinkClick.aspx?fileticket=7CCxM20JUGg%3d&tabid=36&mid=1496
http://www.who.int/ihr/about/10things/en/index.html
xxi
Events constituting a PHEIC are assessed by country using the decision instrument in Annex 2 of the IHR.
http://www.who.int/ihr/9789241596664/en/index.html
xxii
OIE reportable diseases http://www.oie.int/eng/maladies/en_classification2010.htm?e1d7
xxiii
WHO/OIE standards http://www.who.int/ihr/IHR_Monitoring_Framework_Checklist_and_Indicators.pdf
xxiv
Optimum conditions include adherence to appropriate BS&S regulations, cold-chain maintenance, and universal precautions.
xxv
http://www.who.int/ihr/IHR_Monitoring_Framework_Checklist_and_Indicators.pdf
xxvi
Cost is the combined metric of multiple sustainment topics, including but not limited to: operations, maintenance, technology,
infrastructure, knowledge, skills, abilities, educational, logistics, regulatory, cultural) etc. Operations cost is for biosecurity and
collection facility. “Maintenance costs” is for biosecurity and collection facility.
IMPROVING METRICS FOR THE DOD CTR PROGRAM
