In addition to the 2009 report Global Security Engagement: A New Model for Cooperation Threat Reduction, several National Research Council (NRC) reports have made findings and recommendations directly relevant to metrics for the Cooperative Threat Reduction (CTR) Program key points from those reports are summarized here.
UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT
In 2006, at the request of U.S. Agency for International Development (USAID), the NRC published the report titled The Fundamental Role of Science and Technology in International Development: An Imperative for the U.S. Agency for International Development. The report highlights many positive impacts for the United States and developing countries from the application of science and technology (S&T) to international development within U.S. foreign assistance activities. It emphasizes the strengthening of the S&T capacity of developing countries as a key metric in assessing the success of USAID programs, while underscoring self-reliance as the essential component of achieving sustainability. The report discusses three essential components of S&T capacity, namely, (a) institutions that generate, adapt, and disseminate knowledge and technology at international, national, and local levels; (b) technology transfer organizations and individuals that evaluate and adapt knowledge and technologies developed internationally to local conditions; and (c) local institutions that tap and contribute to the world’s knowledge supply.
The report stresses the importance of revitalizing the program evaluation capability of the agency. A robust capability to carry out rigorous evaluations of program effectiveness has long been essential to understand the reasons for success or failure of particular programs, according to the report. Also, lessons learned from many types of projects need to be retained, and a key metric in this regard is the number of qualified professionals assigned full-time to the task of program evaluation and who consider such assignments to be career enhancing.
Finally, the report emphasizes the importance of building an S&T culture which spreads to all Missions and all offices of the agency. Other cultures that permeate USAID are the “democracy culture” and the “emergency response” culture. The former has been driven in large measure by congressional earmarks, which over time have required the hiring of 500 USAID advocates who devote their time and skills to promotion of democracy. The latter reflects a widely shared interest at home and abroad, within and outside the agency, in the responsibility of USAID to respond to tragedies throughout the world.
The report argues that there is a widespread latent interest in S&T in all of USAID’s partner countries and that expanding staff capabilities from a few dozen to a few hundred would have a dramatic impact in harnessing U.S. S&T in the service of international development. The report
sets forth initial personnel hiring targets as an important metric in this regard. It underscores, however, that personnel upgrading must be accompanied by agency leadership, an agency-wide written policy on drawing on the nation’s S&T capabilities, available financial resources to prime the pumps of interest in development countries, and an organizational structure that helps ensure that enthusiasm for S&T is captured programmatically.
DEPARTMENT OF ENERGY
Material Protection Control & Accountability Programs
The NRC has carried out three studies supported by the Department of Energy (DOE), on the strengths and weaknesses of the department’s cooperative program with Russia on protection, control and accountability of weapons-grade nuclear material (MPC&A program). The reports were as follows: Proliferation Concerns: Assessing U.S. Efforts to Help Contain Nuclear and Other Dangerous Materials and Technologies in the Former Soviet Union (1997), Protecting Nuclear Weapons Materials in Russia (1999), and Strengthening Long-Term Nuclear Security: Protecting Weapon-Usable Material in Russia (2006). Over the lifetime of the MPC&A program, DOE has continuously calculated the quantity of nuclear material of concern that was locked down in secure Russian facilities as its primary metric in judging the success of the program and the impact on non-proliferation. The speed with which the lockdown took place at various facilities was a secondary metric. Eventually, the sustainability of the security upgrades that were installed under the guidance of U.S. experts became a third metric, with sustainability usually measured in terms of Russian financial commitments to supporting and continuing the program and the number of Russian specialists that participated in U.S.-organized training programs.
The first report highlighted the importance of also counting the number of discovered thefts and failed thefts of material as two significant metrics. There were a total of 23 discovered thefts in 1993 and 1994, but zero in 1995 and 1996 from the facilities of the Russian Federation Ministry of Atomic Energy. Also, subsequent reports emphasized the importance of obtaining data on the number of nuclear security violations reported by Russian inspectors of nuclear facilities as indicators of program effectiveness. DOE did not follow up on these recommendations.
Another important recommendation was for DOE department and its Russian counterpart to adopt a ten-year cost-sharing program as a bridge to sustainability. Under this concept, Russia eventually would pick up all costs for the $2 billion program, which had begun with the United States covering all of the costs. While DOE eventually embarked on a sustainability program that rested on cost-sharing, the long-term U.S. financial commitment was much higher than it would have been had the ten-year program been adopted.
Finally, one of the reports offered the following observations:
There are no good quantitative measures of the effectiveness of U.S. programs to support efforts to upgrade control of sensitive items. First, there is uncertainty as to the effectiveness of existing controls, the security conditions at production and research facilities, and the capabilities of security personnel. In short, there is no good baseline against which to measure progress. Second, it is not possible to separate contributions of the American participation in the program from progress that would have resulted without the Americans. Finally,
there are no reliable data sets concerning legal transfers of sensitive items out of the region, let alone contraband goods, which may not even be known to national authorities.
Reliance must be placed on qualitative assessments as to whether U.S. agencies are effectively using opportunities to upgrade regulatory and security systems. Surrogate indicators-the effectiveness and functioning of components of MPC&A systems and export control systems, for example—are important. Well-developed MPC&A systems in the west provide a basis for comparison in assessing activities in the countries of the former Soviet Union. Considerable attention has been given to opportunities to adapt American experience to different conditions in the target countries. Also, since effective control systems take years to develop, particular attention must be given to interim approaches that can contain leakages in the immediate future.
Protection of Radiation Sources
Hundreds of thousands of ionizing radiation sources, including hundreds and perhaps thousands of highly dangerous sources, were not under adequate control in Russia in the early 2000s. In a 2007 report prepared for the DOE titled U.S.-Russian Collaboration in Combating Radiological Terrorism, the NRC identified six key metrics for judging the effectiveness of Russian efforts in reducing both the terrorism and radiological safety aspects of inadequately controlled sources. They were:
• quantification and characterization of existing inventories
• prioritization of source recovery requirements and security enhancements
• enhancement of security at user facilities, during transportation, and at temporary storage sites
• enhancement of final disposition capabilities
• development and implementation of management systems for improved accountability
• development and implementation of research and development priorities in support of the foregoing activities
The report recognized the enormous difficulties and long timelines involved in making measurable progress in reaching each of these goals. At the same time, it urged a nation-wide effort to move forward in all of the areas. Major impediments included:
• decentralized responsibilities in Moscow and throughout the country for undertaking and financing many relevant activities
• chronic shortages of necessary funding, either from the government or from the custodians of sources, to correct security deficiencies
• a legacy of security problems reflected in many inadequately protected radiation sources
Meanwhile, DOE continued its modest efforts to support Russian institutions in estimating the number and locations of sources throughout the country, collecting orphan sources, disposing of unneeded sources, and upgrading disposable areas. Major accomplishments were usually recognized through ribbon-cutting ceremonies. These modest efforts involving U.S. specialists were rapidly magnified through much larger efforts by Russian organizations, and this should have been more widely recognized with DOE as a very impressive accomplishment.
DEPARTMENT OF DEFENSE/DEFENSE THREAT REDUCTION AGENCY
The NRC has published four reports on the Department of Defense (DoD)/Biological Threat Reduction Program (BTRP). Metrics were explicitly emphasized in two of these reports, as indicated below, while many issues directly and indirectly-related to metrics were considered in each of the reports. A few of the relevant observations in the reports are set forth below.
Controlling Dangerous Pathogens: A Blueprint for U.S.-Russia Cooperation (1997)
The report recommended the objectives set forth below as the basis for a long-term cooperative research program, with research activities conducted primarily at sensitive Russian facilities. The measurement of the success in fulfilling research objectives and of the impact of research results was left to others after the research was completed using the initial objectives as a guide in establishing more specific metrics.
• National security benefits: confidence-building, reducing incentives for proliferation, leading to opportunities for dismantlement of biological weapons BW capabilities, enhancing capabilities to combat terrorism
• Public health benefits: improved understanding of characteristics of pathogens that threaten public health; strengthened capabilities to prevent, diagnose, and treat outbreaks of infectious diseases; and enhanced national and international communications concerning disease trends and outbreaks
• Economic benefits: stabilizing Russian institutes, providing opportunities for U.S. private sector investments in Russia, and leveraging limited national financial resources
• Scientific benefits: enhanced knowledge about pathogenesis and increased international availability of research results
Eight pilot research proposals were selected by the NRC authoring committee for financial support by the DoD, according to the following criteria which were to provide the basis for the metrics used to judge success of the specific projects:
• scientific importance of the topic
• quality of the proposal
• quality or capacity of the principal investigator, research team, and facilities
• provision for strong U.S. collaborator
• engagement of former Soviet BW expertise
• promotion of transparency
It was suggested that two additional criteria be introduced as the program expanded. They were (a) the likelihood of sustained support by attracting other sources of finance, and (b) the promotion of linkages between Russian institutions that had been involved in BW activities and those that had not.
Threat Agent Detection Response System Data Base (2006)
The report highlighted the following concerns that would complicate both (a) implementing the intended disease surveillance program, and (b) assessing the impact of the program:
• the complex data collection system will challenge many clinicians in host countries who have only limited degrees of computer literacy
• three sets of reporting requirements should be merged into a single system
• stable funding will be difficult to ensure
• training and retention of qualified personnel for the system will be a continuing concern
• governments are apprehensive of U.S. control over all data
• full compliance in providing all data will be difficult
• reviews of large quantities of raw data will produce many false alarms
• effective integration of human and animal disease-related activities will be difficult
The report then concluded that (a) the proposed surveillance network was reasonably well designed on paper, (b) sustainability of the network after DoD completes its efforts to establish the network is critical, and (c) an essential element of sustainability is the broadening of the network from the 16 “low probability” agents, classes of agents, and diseases of interest to DOD to include “high probability agents, classes of agents, and diseases of much greater interest to the host countries. The key metrics measure long-term, not short-term, progress and therefore they must be fully embraced by the host countries.
The Biological Threat Reduction Program of the Department of Defense: From Foreign Assistance to Sustainable Partnerships (2007)
This report set forth 54 recommendations for upgrading the BTRP. The overarching theme of the report was the importance of transitioning from a foreign assistance program designed in Washington to a cooperative program designed jointly with partner institutions in the host countries. Five priority recommendations, in addition to the principle of collaboration rather than assistance, addressed the following issues.
• Strong and sustained support for BTRP and related programs is needed.
• The White House should lead the effort to integrate BTRP with related programs of other government departments and agencies, with the authorizing legislation for BTRP requiring involvement of other government agencies.
• BTRP should give greater attention to comprehensive multifaceted engagement for achieving biosecruity while focusing on long-term in addition to short-term payoffs.
• Bioengagement with Russia should be re-invigorated through a variety of cost-shared activities.
• Program management within BTRP needed improvement.
Among the metrics used by BTRP in the past have been the following:
• Number of weapon scientists involved, including the number trained.
• Number of sustainable jobs created.
• Level of matching contributions by cooperating governments or other partners.
• Follow-on contracts resulting from research projects.
• Number of publications in internationally recognized journals.
• Number of patents awarded.
• Number of research products that have reached the market.
• Number of companies that have been spawned.
These indicators were considered important; but they did not address the essence of the program, namely, to what extent has the likelihood of outbreaks of endemic and emerging diseases and the associated terrorist aspects been reduced? Related concerns are the timeliness, adequacy, and quality of responses to outbreaks should they occur. There have been many positive achievements by BTRP at the national and facility levels for addressing infectious diseases. Developing measures for evaluating the changes (e.g., ease of access to sensitive laboratories, response time in identifying outbreaks) could provide the foundation for useful metrics.
Countering Biological Threats: Challenges for the Department of Defense’s Nonproliferation Program Beyond the Former Soviet Union (2009)
In the sections focusing on metrics, the report made the following observations:
Inferring the impact of BTRP on a nation’s security from laboratory and field evaluations is the most difficult task. Such a task involves understanding the security situation when BTRP entered the scene (the baseline) and the unique contributions of BTRP to reducing biological threats. One approach is for BTRP to support continuing assessments of BTRP impact on risk reduction by both a group of specialists in Washington and a counterpart group of local specialists in the host country. They could develop either common or competing methodologies and then compare results of their assessments. Their different insights as to how BTRP can most effectively enhance security on a broad basis would be of considerable interest.
In short, as BTRP expands into other countries this concern with the impact of BTRP on reducing the threat of bioterrorism deserves greater emphasis. Evaluation efforts should begin from the outset of BTRP involvement in a country. After projects are completed, it will be too late to examine reliable indicators of risk reduction as a result of the projects. At the same time, current midcourse reviews of the Threat Agent Detection and Response [TADR] system in Georgia is a step in the right direction, even though the assumption that TADR is an appropriate approach is not being challenged by the external evaluators—a shortcoming that should be corrected in the future.
The report recommended that BTRP should continue to develop improved metrics that will help guide evaluations of the impact of BTRP, and provide information for setting priorities for activities designed to reduce proliferation of biological weapons as well as related risks from naturally occurring pathogens.
Biological Engagement Programs
The NRC conducted a very broad assessment of U.S. bioengagement activities with Russia. The assessment was published in the report titled Biological Science and Biotechnology in Russia: Controlling Diseases and Enhancing Security (2006).
In setting the stage for recommending program approaches and measuring likely impacts, the report identified four U.S. interests in cooperation, namely:
• high potential of Russia’s biological research and industrial complex to support both civilian and military programs
• significance of efforts by the Russian government to revitalize research capabilities and apply them to solving social and economic problems
• Russia’s vast ecological diversity, which offers unique research environments and provides opportunities for detecting early emergence and movement of dangerous diseases of global importance
• marketing opportunities within Russia for U.S. companies that provide biological products and services
Emphasizing the importance of integrating the Russian health system into global networks that can respond to endemic and emerging diseases, the report stresses the importance of strengthening a broad range of policies and programs of Russia, including the following aspects:
• focus on surveillance, diagnostics, and countermeasures (e.g., drugs and vaccines)
• improve capabilities to detect and diagnose pathogens in both urban and rural environments with improved communication systems
• integrate human and animal disease surveillance
• monitor safety and acceptability of food and water supplies
• strengthen basic research
• strengthen commercialization programs within agriculture and public health
• improve understanding of relationships between infectious agents and non-communicable chronic diseases
• support the biotechnology sector
• strengthen biosecurity procedures at hundreds of facilities
• promote broad transparency
• recruit, train, and retain an expanded cadre of scientists doctors, veterinarians, plant pathologists, and epidemiologists who have access to modern technology to deal with infectious diseases
Then the report focused on five areas where programs should be expanded, results observed, and impact measured:
• improving surveillance and response
• meeting pathogen challenges
• the promise of biotechnology
• the human resource base
• reshaping U.S. Cooperation
In summary, while the report does not explicitly address metrics, it identifies a host of indicators of impact that should lead to approaches in measuring progress toward improved bilateral cooperation that benefits both countries.