Assessing the Value of Research in the Chemical Sciences (1998)

Richard K. Koehn

University of Utah

The growth of university-industry-government collaborative research and development programs is the single most important development in the character of university technology transfer endeavors since the early 1980s.¹

There is no question of the widespread perception in both academia and business that partnerships between universities and industry, whether or not forged by the catalyst of government, are important new parts of the landscape of research universities.^2-4 Over the past several decades, we have seen the transformation of the U.S. economy from one reliant on agriculture and manufacturing to one opening up new industrial sectors, like information technology and biotechnology. These are industries that not only are driven by research and development, but also have their origins within the research communities of both universities and corporations. When we look to see where this wealth has developed most rapidly and forcefully, it is in places like Silicon Valley, the North Carolina Research Triangle, Massachusetts Route 128, and even the University of Utah Research Park—locations where major research universities have spawned new companies and industries and created new wealth for the local citizenry.

Our perception that wealth flows from university-industry partnerships is repeatedly reinforced. Former Vice President Walter Mondale, now U.S. ambassador to Japan, recently discussed this point in an editorial in Science (1996):⁵

Universities generally, under increasing scrutiny to devise measures of productivity of the academic enterprise, have taken up the argument that universities (specifically through their research missions) have significant economic impact. A recent summary published by the National Association of State Universities and Land Grant Colleges ⁶ emphasizes that its member institutions foster new business, create new jobs, promote innovation, enhance the work force, and improve the quality (that is, the standard) of life—all positive economic forces that generally fall within the public service mission of these institutions. The data offered to support these points are unfortunately anecdotal and in some instances questionable.

In theory, assessing the value of research based on the productivity of university-industry partnerships should be relatively simple: the desirable metric should be in simple economic terms, consistently structured, unbiased by geography, and easily available. Unfortunately, if you search for hard and simple evidence of any universal metric, it cannot be found. In a recent publication of a government-university-industry research roundtable on the subject of developing performance standards and output measures for the research enterprise, we read: "Although measurement lies at the heart of the scientific method, no universal [emphasis mine] metrics exist for assessing research."⁷ Recently, writing in the New York Times, William Broad noted:⁸

If a metric is not obvious to use in measuring the economic productivity of research generally, that is certainly also the case for measures of the benefits of university research partnerships. For example, referring specifically to these collaborations, the Government-University-Industry Research Roundtable stated in its annual report: "There is surprisingly little empirical information available about the probability of satisfaction or the actual benefits realized by those who engage in collaboration in cross sectors."⁹

So there we have it. Despite our sense that university-industry partnerships have produced significant economic impact, there is disagreement both on the point and on how to measure. We have even heard in other presentations in this Chemical Sciences Roundtable that any claims for a valid metric are simply fiction—hopeful fiction, perhaps, but fiction nevertheless.

The main point that I make in this presentation is that although there is no universal metric for assessing university-industry research collaboration productivity, there are, I believe, some individual metrics that can be used to measure important aspects of these partnerships. To be sure, there are many

metrics that measure little or nothing, and I will discuss these briefly. However, there is emerging evidence that university-industry partnerships are indeed productive in a variety of ways, though little systematic information is provided by the participating industries or universities.

Each year, the Association of University Technology Managers produces statistics on various measures of technology transfer activity at its member institutions. These measures include the number of disclosures filed, patents issued, and licenses granted, as well as university royalty income from licenses. To my knowledge, these are the only "metrics" that produce standard information for a large number of universities on the possible intensity or purported success of institutional technology transfer programs.

Unfortunately, none of these parameters are accurate measures of economic impact. Therefore, they do not serve as adequate metrics for meaningful measures of productivity of university-industry partnerships. The number of disclosures varies widely among institutions, even differing greatly among years within a single institution. First, assuming all else to be equal (which it is not), the number of disclosures reflects the level of research funding; that is, it is a reflection of the intensity of discovery. A rule of thumb is one or two disclosures per $1 million of research. Second, the number of disclosures at a particular institution reflects to some degree the technology transfer policies of an institution. Does the institution encourage innovation? Does it encourage disclosing potential discoveries for economic reasons? Does it offer an incentive to disclose? If one were to offer a "bounty" for disclosures (say $50 cash to faculty making disclosures, as some institutions have done), the number of disclosures would rise dramatically, at least temporarily. Yet they would falsely represent the potential economic impact of discoveries that ultimately find their way to the marketplace. In short, while tabulating the number of disclosures might be useful for an institution to monitor its own activities, this cannot serve as an accurate metric of any significant variable among institutions.

The number of patents is often claimed to reflect an institution's involvement with industry. However, a closer look at patents shows that "patents are a limited measure of the extent to which technology, much less scientific and technological knowledge, is being transferred to university-industry."¹⁰ A better metric might be the number of university patents paid for by industry. Only a small fraction of U.S. patents are issued to universities: about 3 percent currently, up from about 1 percent 25 years ago. More significantly, academic patents are concentrated in a few "utility classes" and have become more so in recent years with the emergence of biotechnology.

There is, of course, great variation in the cost of pursuing a patent, depending on its complexity. Although the rule of thumb might be a cost of about $15,000, the cold fusion patents pursued by the University of Utah in the late 1980s have cost many times that amount, and they were never issued in the United States! That is the point. The number of patents issued reflects not only an institution' s commitment to intellectual property ownership, but also its financial capacity to secure that ownership. Institutions that are active in technology transfer have recognized the necessity of implementing a process intended to identify those innovations with the highest probability for commercial exploitation, in order to minimize institutional patent costs. The raw data on numbers of patents measures nothing of particular relevance to the productivity of industrial partnerships.

Again, statistics on the number of licenses to companies for university intellectual property reflect an institution's interest in and commitment to technology transfer, but these numbers do not constitute a metric for economic impacts of the research or technology. Most licenses never result in a marketable product. There is great variation in the length of time between the issuance of a license and any revenue that may be generated—usually many years (on average about 8 years), but longer for biotechnologies. Sometimes the license involves a complicated technology that cannot be commercialized. Sometimes the technology doesn't work, corporate priorities change, or the market window closes. There is tremendous variation in the value of technologies commercialized through license agreements.

The last purported metric of technology transfer that is commonly used is the royalty income realized by universities from the licensing of intellectual property. Here again, a look below the surface tells us that royalty revenue is not a precise metric for our interest (though it may be a valid measure for a single institution through time). For example, in recent years the top 10 royalty revenue recipient universities in the United States collectively received a large majority of all revenues paid to all universities. Even within this revenue stream, a very small number of patents at any institution produce the majority of revenues. The Wisconsin Alumni Research Foundation, one of the oldest and most successful technology transfer institutions, received 90 percent of its royalty income from ten patents, and one of these, Vitamin D, dominated the royalty income.¹¹ Similarly, at Stanford, seven individual patents produced more than three-quarters of the institution's royalty revenue in recent years—patents dominated by the Cohen-Boyer gene sequencing patent. For the University of Utah, the data are similar; a single license produced almost 25 percent of the royalties received in FY 1996.

If these traditional university metrics that purportedly measure the productivity of industrial partnerships are not valid, what shall we use as an alternative? Not surprisingly, the answer depends on the focus of the question. For example, the metric applied by an individual faculty member to assess the potential benefits of a partnership with industry will differ from that applied by either the university or the state, regional, or federal government. The goals of each are very different.

To measure the potential benefits of a partnership with industry, an individual member of the faculty would measure the level of research support in relation to the workload. He or she might ask, What is the rate of publication and innovation from this project relative to the level of support? Is the project likely to create a significant outcome in scientific or technological terms? What could be the effect of this project on my scientific reputation? The metric here is reputation-enhancement, since research faculty operate on a credit economy and not necessarily a financial one. How a particular partnership plays into this credit economy is an important consideration and therefore an important metric to research faculty. There are other factors that influence this metric, including flexibility of the research support, exploratory or proprietary nature of the research, and deliverables that a sponsor might expect to receive from it.

The corporate university has very different values from those of the individual faculty who are part of it. Institutional measures of productivity are therefore different. Each institution will ultimately have to measure the scholarly and/or economic effects of these partnerships on the bottom line, either directly or indirectly. Despite the foregoing characterization, universities will undoubtedly continue to use the numbers of disclosures, patents, licenses, and royalty revenue in measuring institutional performance in technology transfer. However, I believe it is important to devise at least two new metrics that would be both more objective and more germane to the measure of university-industry partnership productivity.

First, a tabulation of the number of new companies created with university technologies, together with an objective measure of their economic impact, would provide a more accurate estimate of productivity as a metric directly comparable among institutions. It is the only metric that is important to local, regional, and state governments when they look to the university for technology innovation and a force in the economic development. The metric might be more sophisticated if it included measures of preproduction investment and jobs induced by these companies,¹² or even a highly comprehensive study of the economic impact of start-up companies such as been done for MIT by BankBoston.¹³ A similar study on Silicon Valley¹⁴ demonstrated that the number of jobs created and the tax revenue generated by such companies are important elements of an overall metric of economic impact.

Second, institutions need to develop metrics that reflect the amount of investment made in support of their own research from their partnerships with industry. A direct measure might be simply the amount of funds received for the support of research. More important indirect measures would include the creation of intramural funding programs for enhancing research and/or faculty competitiveness for extramural funds, jointly authored papers with industry, and so forth.

Research and education have been, and are likely to continue to be, the core mission of most universities. As such, the most efficient means of transferring technology is by graduating well-educated science and engineering students. Indeed, industry does not forge partnerships with universities so much for access to technologies as for access to students. Another metric for a university to apply to its industrial partnerships is the success that graduates have in entering the labor pool of those industries with which it collaborates.

There are, of course, many other issues that bear on a university's measure of the importance of industrial research partnerships—in particular, the trade-off between additional revenue from partnerships versus the additional management required for proper oversight.

University-industry partnerships have made the management of university research more complicated, including litigation over contractual disagreements, political exposure of faculty entrepreneurship in public universities, and faculty noncompliance with research misconduct and conflict-of-interest policies. Nevertheless, where the economic impact of the corporate spin-offs of university technology innovation has been carefully analyzed, research institutions are clearly forging highly productive partnerships with industry and, together, are forcefully driving economic growth of high-technology industrial sectors.