The overview of recent program experiences continued with presentations on four recent evaluative activities.
Caroline Wagner (Critical Technologies Institute, RAND)
Ms. Caroline Wagner described a study on international research and development (R&D) cooperation on a worldwide basis undertaken by the Critical Technologies Institute (CTI) as part of their mission to provide objective, independent research and analysis support to OSTP. Conventional wisdom suggests that international cooperation might be leveraging R&D dollars in the United States, but it is unclear how much R&D spending goes to cooperative activities. Policymakers are being asked to justify international R&D spending, but there is no clear picture of the benefits of participating in cooperative R&D activities. Thus, the goal of the study was to assess the level, costs, and benefits of U.S. participation in international R&D.
The results of CTI's inventory indicated that in FY 1995, the U.S. government spent approximately $3.5 billion on international cooperation in R&D, which is about 5 percent of total federal R&D spending. Other S&T activities, including the Global Seismographic Network, Cooperative Threat Reduction program, and education and training transfers might total an additional $1.5 billion or more. The CTI study indicated that, when counted on a bilateral basis, Russia is the largest partner for the U.S. government in terms of dollars spent on coop-
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 12
--> 3 Selected Reviews of Programs The overview of recent program experiences continued with presentations on four recent evaluative activities. Assessing International Cooperation in Research and Development Caroline Wagner (Critical Technologies Institute, RAND) Ms. Caroline Wagner described a study on international research and development (R&D) cooperation on a worldwide basis undertaken by the Critical Technologies Institute (CTI) as part of their mission to provide objective, independent research and analysis support to OSTP. Conventional wisdom suggests that international cooperation might be leveraging R&D dollars in the United States, but it is unclear how much R&D spending goes to cooperative activities. Policymakers are being asked to justify international R&D spending, but there is no clear picture of the benefits of participating in cooperative R&D activities. Thus, the goal of the study was to assess the level, costs, and benefits of U.S. participation in international R&D. The results of CTI's inventory indicated that in FY 1995, the U.S. government spent approximately $3.5 billion on international cooperation in R&D, which is about 5 percent of total federal R&D spending. Other S&T activities, including the Global Seismographic Network, Cooperative Threat Reduction program, and education and training transfers might total an additional $1.5 billion or more. The CTI study indicated that, when counted on a bilateral basis, Russia is the largest partner for the U.S. government in terms of dollars spent on coop-
OCR for page 12
--> erative activities. Approximately $105 million was spent on bilateral projects with Russia in FY 1995; this figure does not include multinational projects in which Russia might be involved. The U.S. agency spending the most money on projects involving international cooperation in R&D is NASA. Here again, Russia is the largest bilateral partner in terms of dollars spent when only one partner can be identified. Ms. Wagner then described a case study of a cooperative project in seismology that was intended to assess some costs and benefits of international S&T collaboration. The methodology used for the case study involved identifying the national goals for the international cooperation; matching the goals to measures; choosing measures and conducting an assessment; and assessing lessons learned. The framework for applying the assessment tools looks first at the reasons for international cooperation, which in this case included the very large-scale equipment; the global nature of the subject; the unique foreign expertise or resources; and the U.S. government mission. The framework then considers the type of cooperation, which can be collaborative research, technical support, operational support, or standard database development. CTI then chose three tools to match goals and assessments in this case study: A survey of internationally co-authored papers in 1985 and 1995. The Science Citation Index was used to provide an indicator of whether there had been an increase or decrease in jointly published papers in this field of science. A survey of seismology equipment to determine the extent to which U.S. companies were setting the technical standards for seismology equipment. A survey of U.S. investigators working on cooperative projects with foreign counterparts to ask to what extent the foreign partner contributed monetary or in-kind contributions to the project. Among the findings of the assessment of this particular case study were: Jointly authored papers increased from 259 in 1985 to 351 in 1995, even while funding held constant in real terms. The United States is leveraging foreign research dollars approximately dollar-for-dollar. The United States is setting the standard in approximately 80 percent of the technical equipment used in seismology tools. The United States is gaining unprecedented access to unique resources not generally available to foreign researchers. Both qualitative and quantitative measures should be used in tandem to get a full picture of the costs and benefits of international collaborative research. These findings have since been published by RAND (see Appendix D).
OCR for page 12
--> Overview of Cooperative Programs: 1991-1996 Glenn Schweitzer (National Research Council) Mr. Glenn Schweitzer briefly described the findings in his recent book, Experiments in Cooperation, which reviewed U.S. collaborative activities with Russia during the period from 1991 to 1996. He considered those activities in which scientists and engineers played key roles. The scope of the activities involved government-funded activities costing the U.S. government about $2.5 billion and U.S. private-sector investments of about $2 billion. The activities included many types of programs, such as nonproliferation, exchanges, joint mission-oriented programs, support of science and education, foreign assistance, and commercial investments. His approach to the evaluation of the activities was to review program documents; review other U.S., Russian, and international publications; conduct a limited survey of knowledgeable Russian leaders; conduct interviews in Russia and the United States; and consider in more detail 33 case studies. His judgments on the relative success or failure of particular programs were based on (1) the views of funders and participants; (2) the impacts, in terms of achieving project objectives, of follow-on activities, impacts beyond participating individuals and institutions (i.e., leveraging), and impacts on national policies and reform; and (3) "footprints in the sand," or the sustainability of the impacts. As a result of his research, Mr. Schweitzer was able to identify specific negative reactions in Russia to some U.S. programs and activities. Among those reactions he cited is that of false expectations, that is, some U.S. activities led Russian participants to expect more money, more assistance, and more activity than was actually delivered. Another negative reaction from the Russian side is the inability of the United States to sustain successful projects. The U.S. budget process, decreasing budgets, and a variety of other factors contribute at times to a random stop-and-go approach. The Russians also complain of unfair intellectual property rights arrangements. Finally, there is the lack of appreciation on the U.S. side of Russian capabilities. He concluded by noting some past approaches in U.S. programs and activities that need attention and possibly revision: Emphasizing "assistance." Assistance is a donor-recipient concept that is not an appropriate basis for programs, particularly in science and technology, with Russia. Rather, programs should be based on cooperation. Lumping Russia with other states of the former Soviet Union. Russia has unique problems and capabilities and the models for the country to follow should be made in Russia for Russia. Moreover, programs that lump Russia together with the other states of the former Soviet Union, although administratively convenient, lead to distortions in the treatment of Russia.
OCR for page 12
--> Selecting "targets of opportunity" whether or not such activities are supportive of U.S. priorities or indeed divert attention to marginal concerns. Developing details of projects in the United States. The details of program implementation are critically important, and the Russians need to be involved. Supporting specialists at weak institutions. Even generous grants to scientists in weak institutions will not result in high-quality work. In the end, we are likely better off supporting scientists and institutions with the infrastructure to sustain high-quality research. Transferring wage payments to institutes. With few exceptions, only a small percentage of funds transferred to Russian institutes for joint projects reaches the researchers. The most effective way to avoid overhead, taxes, social funds, and other government payments (which can total up to 85 percent of the funds) in the absence of special arrangements with the Russian government is to concentrate financial support on maintaining infrastructure and providing equipment and covering costs of travel and per diem and other direct expenses. Cooperative Ventures Between U.S. Companies and Russian Defense Enterprises David Bernstein (Stanford University) Dr. David Bernstein reported on his work using case studies of cooperative ventures between U.S. industry and the Russian defense sector. He began by noting that his approach in which cases are selected for a particular reason, as compared with a statistical approach, leads to biases in data. His approach is intended to yield models, not statistics. Also, his cases are industrial ventures, and the results and conclusions do not necessarily extrapolate to other types of projects. He summarized some of his conclusions: The amount of U.S. investment into commercial ventures with Russian defense enterprises has been small. There are a few large projects, especially in the space propulsion sector, and many diverse small commercial ventures. Space propulsion may be a unique coincidence of performance, cost, and market growth. There are many factors affecting the decision to invest and the degree of success. Of course, all "success" at this stage is interim. Among the reasons to invest are the following: market penetration, cost reduction, technology,
OCR for page 12
--> human capital, and availability of funds from the U.S. government or international financial institutions. Dr. Bernstein cited numerous factors influencing the success or failure of a commercial joint venture. First are a number of parameters of the venture itself. These include the sector within which the venture is operating; the type of Russian asset utilized (e.g., human, technical, or physical); the output (e.g., product, service, or research); whether the market to which the venture is looking is foreign or domestic; whether the product is an end item; and the legal form and size of the venture. There are also operational factors. These include a long-term commitment; the relationship between the parties; flexibility; a strategic versus a purely financial alliance; phasing of the program; compatibility of objectives; and training and technical assistance provided by the Western parties. With these points in mind, he gave some observations on successful joint ventures: Well-nurtured, trustful alliances have the best chance of success. In a U.S.-to-U.S. venture, both partners understand the rules of the game and can look out for themselves. In a U.S.-to-Russian venture, the U.S. partners have to be sensitive to the fact that many of the rules are new to Russians. There is a need for more of a positive-sum philosophy for the venture to be sustainable over the long-term. There are successes in both manufacturing and in research, but success in research or software projects seems to be easier to achieve. Market-driven projects are usually more successful than technology-driven projects. Component projects or projects that provide equipment for further use appear to be more successful than end-product projects. Enterprises that are willing to carry out low- to medium-technology work are often better matches to the market and their partners' needs than high-technology organizations. Russian technologists, like their U.S. counterparts, can be challenged by semitechnical tasks such as improving efficiency, quality assurance, and marketing. Russian enterprises willing to decentralize, delegate, and spin off units are better at attracting and working with U.S. partners. Privatization is less important than these other factors. U.S. companies hold a variety of views on seeking help from the U.S. government. Many companies will not touch government programs; others find it a necessary condition for investing; still others are engaged in joint venture merely because of the availability of U.S. government funding.
OCR for page 12
--> U.S. government participation has helped to facilitate joint ventures in a variety of direct and indirect ways. To date, the government's participation is largely at the margins, but it can be significant. For example, the U.S. government should consider removing or reducing barriers, such as export controls, and offsetting some risks and costs to make it easier to find private capital. Government-to-government facilitation, as takes place under the Gore-Chernomyrdin Commission, is also very important. Finally, the U.S. government can provide logistical and informational help and technical assistance. Other less direct ways in which the U.S. government can assist a joint venture is through encouraging openness and access throughout the Russian industrial complex. Dr. Bernstein concluded by noting that there is no good benchmark for comparison in assessing the success of either government or industrial efforts with Russia. The projects of most value to building a Russian economy are those that give the Russians a business proprietorship. Evaluating Scientific Grants Programs Kelly Robbins (National Research Council) Taking the discussion from the aggregate to an individual program level, Ms. Kelly Robbins reported on evaluations of scientific grants programs. She focused in particular on the Collaboration in Basic Science and Engineering (COBASE) program, which is funded by the National Science Foundation and administered by the National Research Council. The program funds U.S. researchers in the basic sciences to host or visit colleagues in the former Soviet Union and Eastern Europe for developing and carrying out joint research. Grants vary in amount from just over $2,000 to $15,000, depending on the time period of the joint activity. Grantees' reports submitted at the end of the visits provide information on immediate results. To find out more long-term impacts and program results, however, Ms. Robbins began the practice of contacting COBASE grantees approximately 1 year after the end of their programs. The informal survey approach used is certainly not all-encompassing or highly scientific, but by asking a few open-ended questions, it prompts the grantees to provide valuable longer-term outcomes of their visits. The questions asked of grantees, and some of the results from recent program years, follow. Are you still in contact with your foreign colleague? This is a basic indicator of whether the U.S. grantee felt it was worthwhile to continue the interaction. Ms. Robbins reported that results had been surprisingly positive on this question, with well over 80 percent answering yes. Have you applied for, or received, follow-on funding for the joint collaboration? The results here have also been good, with approximately one-
OCR for page 12
--> third of grantees having received funding. Moreover, a large number of grantees report that they had continued their collaboration even without funding specifically intended for international collaboration. Have you and your colleague published any articles or made conference presentations? Any other results? This question prompts for specific outcomes as well as less quantifiable follow-on activities. Over one-half the respondents had publications. Ms. Robbins then went on to point out some other benefits of the grant-funded projects. Some, such as publications, patents, and presentations, are easy to quantify. Other benefits seen in the evaluation include curriculum enrichment, involvement of graduate students and junior researchers in international activities, access to unpublished foreign data, and establishment of lasting individual and institutional linkages. Finally, Ms. Robbins noted some trends that were evident in recent years in the grants programs. The number of female applicants and grantees was diminishing, which might be the result of budget cuts at foreign institutes affecting females first. As would be expected, there has been an increase in foreigners using the grants to emigrate to the United States. This can of course be considered positive or negative, depending on the perspective. The foreigner receives a stable job and vastly improved lifestyle and the U.S. host obtains a well-trained addition to his or her laboratory; both benefit from the emigration opportunities. In the larger picture, it is questionable as to whether such emigration is in the best interests of the United States or Russia. If some of these emigres return to Russia when it has restored its economy and started to rebuild its S&T capacity, the long stay abroad by the returnees could have real benefits for Russia. Regarding the duration of foreign visits, it is increasingly the case that good researchers from Russia and elsewhere cannot afford to be away from their laboratories for even 6 months. Grants programs such as COBASE might have to be flexible in splitting visits or using the model of another NRC program that allows back-and-forth visits by both the foreigner and the American over a longer period. Results from collaborative activities take time. Depending on the scientific field and the people involved, major results can be achieved in just several weeks or might take many years. Thus, program evaluations and measurements of success should not set rigid standards on duration and outcomes.