National Academies Press: OpenBook

Improving Metrics for the Department of Defense Cooperative Threat Reduction Program (2012)

Chapter: Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction

« Previous: Appendix B: Department of Defense Cooperative Threat Reduction Metrics Report
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

APPENDIX C

NATIONAL RESEARCH COUNCIL REPORTS DIRECTLY RELEVANT TO COOPERATIVE THREAT REDUCTION

In addition to the 2009 report Global Security Engagement: A New Model for Cooperation Threat Reduction, several National Research Council (NRC) reports have made findings and recommendations directly relevant to metrics for the Cooperative Threat Reduction (CTR) Program key points from those reports are summarized here.

UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT

In 2006, at the request of U.S. Agency for International Development (USAID), the NRC published the report titled The Fundamental Role of Science and Technology in International Development: An Imperative for the U.S. Agency for International Development. The report highlights many positive impacts for the United States and developing countries from the application of science and technology (S&T) to international development within U.S. foreign assistance activities. It emphasizes the strengthening of the S&T capacity of developing countries as a key metric in assessing the success of USAID programs, while underscoring self-reliance as the essential component of achieving sustainability. The report discusses three essential components of S&T capacity, namely, (a) institutions that generate, adapt, and disseminate knowledge and technology at international, national, and local levels; (b) technology transfer organizations and individuals that evaluate and adapt knowledge and technologies developed internationally to local conditions; and (c) local institutions that tap and contribute to the world’s knowledge supply.

The report stresses the importance of revitalizing the program evaluation capability of the agency. A robust capability to carry out rigorous evaluations of program effectiveness has long been essential to understand the reasons for success or failure of particular programs, according to the report. Also, lessons learned from many types of projects need to be retained, and a key metric in this regard is the number of qualified professionals assigned full-time to the task of program evaluation and who consider such assignments to be career enhancing.

Finally, the report emphasizes the importance of building an S&T culture which spreads to all Missions and all offices of the agency. Other cultures that permeate USAID are the “democracy culture” and the “emergency response” culture. The former has been driven in large measure by congressional earmarks, which over time have required the hiring of 500 USAID advocates who devote their time and skills to promotion of democracy. The latter reflects a widely shared interest at home and abroad, within and outside the agency, in the responsibility of USAID to respond to tragedies throughout the world.

The report argues that there is a widespread latent interest in S&T in all of USAID’s partner countries and that expanding staff capabilities from a few dozen to a few hundred would have a dramatic impact in harnessing U.S. S&T in the service of international development. The report

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

sets forth initial personnel hiring targets as an important metric in this regard. It underscores, however, that personnel upgrading must be accompanied by agency leadership, an agency-wide written policy on drawing on the nation’s S&T capabilities, available financial resources to prime the pumps of interest in development countries, and an organizational structure that helps ensure that enthusiasm for S&T is captured programmatically.

DEPARTMENT OF ENERGY

Material Protection Control & Accountability Programs

The NRC has carried out three studies supported by the Department of Energy (DOE), on the strengths and weaknesses of the department’s cooperative program with Russia on protection, control and accountability of weapons-grade nuclear material (MPC&A program). The reports were as follows: Proliferation Concerns: Assessing U.S. Efforts to Help Contain Nuclear and Other Dangerous Materials and Technologies in the Former Soviet Union (1997), Protecting Nuclear Weapons Materials in Russia (1999), and Strengthening Long-Term Nuclear Security: Protecting Weapon-Usable Material in Russia (2006). Over the lifetime of the MPC&A program, DOE has continuously calculated the quantity of nuclear material of concern that was locked down in secure Russian facilities as its primary metric in judging the success of the program and the impact on non-proliferation. The speed with which the lockdown took place at various facilities was a secondary metric. Eventually, the sustainability of the security upgrades that were installed under the guidance of U.S. experts became a third metric, with sustainability usually measured in terms of Russian financial commitments to supporting and continuing the program and the number of Russian specialists that participated in U.S.-organized training programs.

The first report highlighted the importance of also counting the number of discovered thefts and failed thefts of material as two significant metrics. There were a total of 23 discovered thefts in 1993 and 1994, but zero in 1995 and 1996 from the facilities of the Russian Federation Ministry of Atomic Energy. Also, subsequent reports emphasized the importance of obtaining data on the number of nuclear security violations reported by Russian inspectors of nuclear facilities as indicators of program effectiveness. DOE did not follow up on these recommendations.

Another important recommendation was for DOE department and its Russian counterpart to adopt a ten-year cost-sharing program as a bridge to sustainability. Under this concept, Russia eventually would pick up all costs for the $2 billion program, which had begun with the United States covering all of the costs. While DOE eventually embarked on a sustainability program that rested on cost-sharing, the long-term U.S. financial commitment was much higher than it would have been had the ten-year program been adopted.

Finally, one of the reports offered the following observations:

There are no good quantitative measures of the effectiveness of U.S. programs to support efforts to upgrade control of sensitive items. First, there is uncertainty as to the effectiveness of existing controls, the security conditions at production and research facilities, and the capabilities of security personnel. In short, there is no good baseline against which to measure progress. Second, it is not possible to separate contributions of the American participation in the program from progress that would have resulted without the Americans. Finally,

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

there are no reliable data sets concerning legal transfers of sensitive items out of the region, let alone contraband goods, which may not even be known to national authorities.

Reliance must be placed on qualitative assessments as to whether U.S. agencies are effectively using opportunities to upgrade regulatory and security systems. Surrogate indicators-the effectiveness and functioning of components of MPC&A systems and export control systems, for example—are important. Well-developed MPC&A systems in the west provide a basis for comparison in assessing activities in the countries of the former Soviet Union. Considerable attention has been given to opportunities to adapt American experience to different conditions in the target countries. Also, since effective control systems take years to develop, particular attention must be given to interim approaches that can contain leakages in the immediate future.

Protection of Radiation Sources

Hundreds of thousands of ionizing radiation sources, including hundreds and perhaps thousands of highly dangerous sources, were not under adequate control in Russia in the early 2000s. In a 2007 report prepared for the DOE titled U.S.-Russian Collaboration in Combating Radiological Terrorism, the NRC identified six key metrics for judging the effectiveness of Russian efforts in reducing both the terrorism and radiological safety aspects of inadequately controlled sources. They were:

•   quantification and characterization of existing inventories

•   prioritization of source recovery requirements and security enhancements

•   enhancement of security at user facilities, during transportation, and at temporary storage sites

•   enhancement of final disposition capabilities

•   development and implementation of management systems for improved accountability

•   development and implementation of research and development priorities in support of the foregoing activities

The report recognized the enormous difficulties and long timelines involved in making measurable progress in reaching each of these goals. At the same time, it urged a nation-wide effort to move forward in all of the areas. Major impediments included:

•   decentralized responsibilities in Moscow and throughout the country for undertaking and financing many relevant activities

•   chronic shortages of necessary funding, either from the government or from the custodians of sources, to correct security deficiencies

•   a legacy of security problems reflected in many inadequately protected radiation sources

Meanwhile, DOE continued its modest efforts to support Russian institutions in estimating the number and locations of sources throughout the country, collecting orphan sources, disposing of unneeded sources, and upgrading disposable areas. Major accomplishments were usually recognized through ribbon-cutting ceremonies. These modest efforts involving U.S. specialists were rapidly magnified through much larger efforts by Russian organizations, and this should have been more widely recognized with DOE as a very impressive accomplishment.

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

DEPARTMENT OF DEFENSE/DEFENSE THREAT REDUCTION AGENCY

The NRC has published four reports on the Department of Defense (DoD)/Biological Threat Reduction Program (BTRP). Metrics were explicitly emphasized in two of these reports, as indicated below, while many issues directly and indirectly-related to metrics were considered in each of the reports. A few of the relevant observations in the reports are set forth below.

Controlling Dangerous Pathogens: A Blueprint for U.S.-Russia Cooperation (1997)

The report recommended the objectives set forth below as the basis for a long-term cooperative research program, with research activities conducted primarily at sensitive Russian facilities. The measurement of the success in fulfilling research objectives and of the impact of research results was left to others after the research was completed using the initial objectives as a guide in establishing more specific metrics.

•   National security benefits: confidence-building, reducing incentives for proliferation, leading to opportunities for dismantlement of biological weapons BW capabilities, enhancing capabilities to combat terrorism

•   Public health benefits: improved understanding of characteristics of pathogens that threaten public health; strengthened capabilities to prevent, diagnose, and treat outbreaks of infectious diseases; and enhanced national and international communications concerning disease trends and outbreaks

•   Economic benefits: stabilizing Russian institutes, providing opportunities for U.S. private sector investments in Russia, and leveraging limited national financial resources

•   Scientific benefits: enhanced knowledge about pathogenesis and increased international availability of research results

Eight pilot research proposals were selected by the NRC authoring committee for financial support by the DoD, according to the following criteria which were to provide the basis for the metrics used to judge success of the specific projects:

•   scientific importance of the topic

•   quality of the proposal

•   quality or capacity of the principal investigator, research team, and facilities

•   provision for strong U.S. collaborator

•   engagement of former Soviet BW expertise

•   promotion of transparency

It was suggested that two additional criteria be introduced as the program expanded. They were (a) the likelihood of sustained support by attracting other sources of finance, and (b) the promotion of linkages between Russian institutions that had been involved in BW activities and those that had not.

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

Threat Agent Detection Response System Data Base (2006)

The report highlighted the following concerns that would complicate both (a) implementing the intended disease surveillance program, and (b) assessing the impact of the program:

•   the complex data collection system will challenge many clinicians in host countries who have only limited degrees of computer literacy

•   three sets of reporting requirements should be merged into a single system

•   stable funding will be difficult to ensure

•   training and retention of qualified personnel for the system will be a continuing concern

•   governments are apprehensive of U.S. control over all data

•   full compliance in providing all data will be difficult

•   reviews of large quantities of raw data will produce many false alarms

•   effective integration of human and animal disease-related activities will be difficult

The report then concluded that (a) the proposed surveillance network was reasonably well designed on paper, (b) sustainability of the network after DoD completes its efforts to establish the network is critical, and (c) an essential element of sustainability is the broadening of the network from the 16 “low probability” agents, classes of agents, and diseases of interest to DOD to include “high probability agents, classes of agents, and diseases of much greater interest to the host countries. The key metrics measure long-term, not short-term, progress and therefore they must be fully embraced by the host countries.

The Biological Threat Reduction Program of the Department of Defense: From Foreign Assistance to Sustainable Partnerships (2007)

This report set forth 54 recommendations for upgrading the BTRP. The overarching theme of the report was the importance of transitioning from a foreign assistance program designed in Washington to a cooperative program designed jointly with partner institutions in the host countries. Five priority recommendations, in addition to the principle of collaboration rather than assistance, addressed the following issues.

•   Strong and sustained support for BTRP and related programs is needed.

•   The White House should lead the effort to integrate BTRP with related programs of other government departments and agencies, with the authorizing legislation for BTRP requiring involvement of other government agencies.

•   BTRP should give greater attention to comprehensive multifaceted engagement for achieving biosecruity while focusing on long-term in addition to short-term payoffs.

•   Bioengagement with Russia should be re-invigorated through a variety of cost-shared activities.

•   Program management within BTRP needed improvement.

Among the metrics used by BTRP in the past have been the following:

•   Number of weapon scientists involved, including the number trained.

•   Number of sustainable jobs created.

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

•   Level of matching contributions by cooperating governments or other partners.

•   Follow-on contracts resulting from research projects.

•   Number of publications in internationally recognized journals.

•   Number of patents awarded.

•   Number of research products that have reached the market.

•   Number of companies that have been spawned.

These indicators were considered important; but they did not address the essence of the program, namely, to what extent has the likelihood of outbreaks of endemic and emerging diseases and the associated terrorist aspects been reduced? Related concerns are the timeliness, adequacy, and quality of responses to outbreaks should they occur. There have been many positive achievements by BTRP at the national and facility levels for addressing infectious diseases. Developing measures for evaluating the changes (e.g., ease of access to sensitive laboratories, response time in identifying outbreaks) could provide the foundation for useful metrics.

Countering Biological Threats: Challenges for the Department of Defense’s Nonproliferation Program Beyond the Former Soviet Union (2009)

In the sections focusing on metrics, the report made the following observations:

Inferring the impact of BTRP on a nation’s security from laboratory and field evaluations is the most difficult task. Such a task involves understanding the security situation when BTRP entered the scene (the baseline) and the unique contributions of BTRP to reducing biological threats. One approach is for BTRP to support continuing assessments of BTRP impact on risk reduction by both a group of specialists in Washington and a counterpart group of local specialists in the host country. They could develop either common or competing methodologies and then compare results of their assessments. Their different insights as to how BTRP can most effectively enhance security on a broad basis would be of considerable interest.

In short, as BTRP expands into other countries this concern with the impact of BTRP on reducing the threat of bioterrorism deserves greater emphasis. Evaluation efforts should begin from the outset of BTRP involvement in a country. After projects are completed, it will be too late to examine reliable indicators of risk reduction as a result of the projects. At the same time, current midcourse reviews of the Threat Agent Detection and Response [TADR] system in Georgia is a step in the right direction, even though the assumption that TADR is an appropriate approach is not being challenged by the external evaluators—a shortcoming that should be corrected in the future.

The report recommended that BTRP should continue to develop improved metrics that will help guide evaluations of the impact of BTRP, and provide information for setting priorities for activities designed to reduce proliferation of biological weapons as well as related risks from naturally occurring pathogens.

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

INTERAGENCY

Biological Engagement Programs

The NRC conducted a very broad assessment of U.S. bioengagement activities with Russia. The assessment was published in the report titled Biological Science and Biotechnology in Russia: Controlling Diseases and Enhancing Security (2006).

In setting the stage for recommending program approaches and measuring likely impacts, the report identified four U.S. interests in cooperation, namely:

•   high potential of Russia’s biological research and industrial complex to support both civilian and military programs

•   significance of efforts by the Russian government to revitalize research capabilities and apply them to solving social and economic problems

•   Russia’s vast ecological diversity, which offers unique research environments and provides opportunities for detecting early emergence and movement of dangerous diseases of global importance

•   marketing opportunities within Russia for U.S. companies that provide biological products and services

Emphasizing the importance of integrating the Russian health system into global networks that can respond to endemic and emerging diseases, the report stresses the importance of strengthening a broad range of policies and programs of Russia, including the following aspects:

•   focus on surveillance, diagnostics, and countermeasures (e.g., drugs and vaccines)

•   improve capabilities to detect and diagnose pathogens in both urban and rural environments with improved communication systems

•   integrate human and animal disease surveillance

•   monitor safety and acceptability of food and water supplies

•   strengthen basic research

•   strengthen commercialization programs within agriculture and public health

•   improve understanding of relationships between infectious agents and non-communicable chronic diseases

•   support the biotechnology sector

•   strengthen biosecurity procedures at hundreds of facilities

•   promote broad transparency

•   recruit, train, and retain an expanded cadre of scientists doctors, veterinarians, plant pathologists, and epidemiologists who have access to modern technology to deal with infectious diseases

Then the report focused on five areas where programs should be expanded, results observed, and impact measured:

•   improving surveillance and response

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×

•   meeting pathogen challenges

•   the promise of biotechnology

•   the human resource base

•   reshaping U.S. Cooperation

In summary, while the report does not explicitly address metrics, it identifies a host of indicators of impact that should lead to approaches in measuring progress toward improved bilateral cooperation that benefits both countries.

Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 93
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 94
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 95
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 96
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 97
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 98
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 99
Suggested Citation:"Appendix C: National Research Council Reports Directly Relevant to Cooperative Threat Reduction." National Academy of Sciences. 2012. Improving Metrics for the Department of Defense Cooperative Threat Reduction Program. Washington, DC: The National Academies Press. doi: 10.17226/13289.
×
Page 100
Next: Appendix D: Biographical Sketches of Committee Members »
Improving Metrics for the Department of Defense Cooperative Threat Reduction Program Get This Book
×
Buy Paperback | $38.00 Buy Ebook | $30.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Cooperative Threat Reduction (CTR) Program was created in 1991 as a set of support activities assisting the Former Soviet Union states in securing and eliminating strategic nuclear weapons and the materials used to create them. The Program evolved as needs and opportunities changed: Efforts to address biological and chemical threats were added, as was a program aimed at preventing cross-border smuggling of weapons of mass destruction. CTR has traveled through uncharted territory since its inception, and both the United States and its partners have taken bold steps resulting in progress unimagined in initial years. Over the years, much of the debate about CTR on Capitol Hill has concerned the effective use of funds, when the partners would take full responsibility for the efforts, and how progress, impact, and effectiveness should be measured.

Directed by Congress, the Secretary of Defense completed a report describing DoD's metrics for the CTR Program (here called the DoD Metrics Report) in September 2010 and, as required in the same law, contracted with the National Academy of Sciences to review the metrics DoD developed and identify possible additional or alternative metrics, if necessary. Improving Metrics for the DoD Cooperative Threat Reduction Program provides that review and advice.

Improving Metrics for the DoD Cooperative Threat Reduction Program identifies shortcomings in the DoD Metrics Report and provides recommendations to enhance DoD's development and use of metrics for the CTR Program. The committee wrote this report with two main audiences in mind: Those who are mostly concerned with the overall assessment and advice, and those readers directly involved in the CTR Program, who need the details of the DoD report assessment and of how to implement the approach that the committee recommends.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!