6
Measuring Outputs and Outcomes of Innovation

As described in Chapter 4, innovation output may be indicated by statistics on patents, papers, prizes, invention disclosures, and degrees awarded, while outcome proxies include patent and paper citations, expert evaluations, innovation counts, new product sales, measured productivity growth, and benefit/cost or rate-of-return estimates. The workshop discussion focused primarily on patent counts, patent citations, productivity measurement, valuation of R&D using financial reporting data, the representation of innovation output in the national accounts, and technology adoption surveys. Participants emphasized repeatedly that much of the outcome of innovation accrues to society as spillover effects that present considerable measurement challenges.

Patent Counts and Patent Citations

The number of patents issued and the technical and scientific literature citations on the patents are used to develop quantitative measures of innovative output and science-technology linkages. Adam Jaffe pointed out that among the advantages of patent data are that they are readily available, including on the Internet, contain considerable detail, and can be used to develop time series. The limitations of patent data are well known. Patents represent only a portion of innovative activity and none of the latter stage investments entailed in commercializing technology. Patents are of widely varying value to firms. Motivations for patenting vary across industries and technologies; equally important, they change over time. The strengths and limitations of patent count data as an indicator of innovation output are summarized in the accompanying text box.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 23
--> 6 Measuring Outputs and Outcomes of Innovation As described in Chapter 4, innovation output may be indicated by statistics on patents, papers, prizes, invention disclosures, and degrees awarded, while outcome proxies include patent and paper citations, expert evaluations, innovation counts, new product sales, measured productivity growth, and benefit/cost or rate-of-return estimates. The workshop discussion focused primarily on patent counts, patent citations, productivity measurement, valuation of R&D using financial reporting data, the representation of innovation output in the national accounts, and technology adoption surveys. Participants emphasized repeatedly that much of the outcome of innovation accrues to society as spillover effects that present considerable measurement challenges. Patent Counts and Patent Citations The number of patents issued and the technical and scientific literature citations on the patents are used to develop quantitative measures of innovative output and science-technology linkages. Adam Jaffe pointed out that among the advantages of patent data are that they are readily available, including on the Internet, contain considerable detail, and can be used to develop time series. The limitations of patent data are well known. Patents represent only a portion of innovative activity and none of the latter stage investments entailed in commercializing technology. Patents are of widely varying value to firms. Motivations for patenting vary across industries and technologies; equally important, they change over time. The strengths and limitations of patent count data as an indicator of innovation output are summarized in the accompanying text box.

OCR for page 23
--> Patent Count Data as an Indicator of Innovation Output Strengths Are available to the public, Contain considerable detail, and Are available over time, enabling time series analysis. Limitations Some important technologies are not patentable, such as software that is protected by copyright. Not all inventions are patented. Firms protect the returns to their investment in other ways such as through secrecy, lead-time advantages, and marketing. Firms patent for different reasons, not all of them related directly to commercial exploitation, for example, to protect an invention from imitation, to block competitors from patenting or pursuing a line of research, or to evaluate the productivity of their R&D activities. This adds considerable uncertainty to the interpretation of patent count data. Patents represent only the practical application of ideas, not more general advances in knowledge. Patents represent inventions, not activities and investments to commercialize new technology. Patents have widely varying commercial value and therefore significance with respect to innovation. One suggestion for improving the usefulness of patent count data was to ask companies to list on the RD-1 survey form the names of the entities under which they hold patents. With this information, one could match R&D expenditures to the patent data and create a series for patents by industry that corresponds to existing data series on R&D by industry. Some participants also noted current efforts to use the information gathered by innovation surveys (e.g., on the use of patents relative to other means of appropriating returns to innovation) and to weight patent counts by industry. The front page of U.S. patents contains citations of previous patents and other critical scientific literature purportedly representing the intellectual lineage or prior art of the patented invention and distinguishing it from previous inventions. In principle, patent citation data contain a wealth of micro-information on

OCR for page 23
--> Patent Citation Data as an Indicator of Knowledge Flows Strength Are available to the public, Contain considerable detail, including related inventions and research, and Are available over time, enabling time series analysis. Limitations Origins and innovations are uncertain; citations inserted by persons other than the inventor (e.g., the applicant's lawyer or the Patent Office examiner) may not reflect the actual flow of knowledge as much as a strategy for obtaining patent approval or the ease of literature searches with automated databases; and Citation practices may vary across patents, technologies, industries, countries, and so forth. This variance may bias quantitative analyses of citation counts. science and technology linkages by subject, geographic location, and source of investment. For example, they are being used by Adam Jaffe and Francis Narin of CHI Research, Inc., as proxies of technical knowledge flows to assess firms' reliance on science and technology of the same or foreign national origins, on publicly versus privately funded research, and on particular public research sponsors. The strengths and limitations of patent citation data as an indicator of knowledge flows are given in the text box. Aside from the limitations of patent data generally, this promising line of research is handicapped by a good deal of uncertainty about the origins and motivation of patent citations. It is known, for example, that some proportion of prior art citations on applications are inserted by applicants' attorneys or consultants and by examining officers in the Patent Office, presumably to strengthen the case for issuing a patent. That may weaken the inference that the inventor has in some measure relied on the work listed. New Measures of Innovative Output Throughout the workshop, participants commented on the need to develop new measures of innovative output. Better measures and a better understanding of the products and outcomes of innovative activity and of the economic and social value of this output are especially needed to inform public policy making.

OCR for page 23
--> Participants discussed using financial market information to estimate the economic value of R&D capacity and developing more direct output indicators in the construction of the national accounts. Financial Market Valuations of R&D Investments Although the outputs and outcomes of industrial R&D activities are often difficult to define, let alone value, financial markets do provide a valuation of firms' R&D investments. For example, researchers have used financial reports to link R&D investments to changes in company stock prices. Baruch Lev, professor of accounting and finance at New York University's Stern School of Business, reported on work of the Intangibles Research Project, which was established with support from the Securities and Exchange Commission (SEC). In a recent study of 1,500 R&D-intensive companies, Lev capitalized reported R&D and then analyzed the relationship between the R&D investments and changes in the companies' stock prices. Lev found a strong, positive correlation between new R&D outlays and stock price increases. Although financial accounting standards require full expensing of R&D outlays in financial reports of public corporations, the software industry has been required to capitalize part of software development costs. Here, too, studies have found R&D costs to be highly correlated with subsequent reported earnings. Another measure of the value of R&D is the reported value of assets in a corporate acquisition. Lev and his colleagues have found that firms have a tendency to expense all of the R&D in progress at the time of acquisition. A company's valuation of the R&D stock, which is based on the net present value of expected future cash flows, provides a quantitative measure of R&D output. These encouraging results of analyzing financial information suggest that there may indeed be a market for R&D and therefore market-based indicators of the value of intellectual capital. One implication of this analysis is that the R&D reporting requirements of the SEC should be reexamined. The Financial Accounting Standards Board decision in 1974 to require full expensing of R&D outlays rested on the lack of evidence of a relation between R&D expenditures and subsequent benefits. If the relationship is empirically demonstrable, full expensing may not be appropriate. Representation of Innovation Output in the National Accounts Steven Landefeld explained that the national accounts do not really measure the output of R&D or innovative activity because the value of R&D is represented solely by its cost, which is an input to innovation. Landefeld noted that more direct measures of R&D and technological change may require use of nonexpenditure data such as the number of patents, patent applications, R&D personnel, proxies for embodied technological change (e.g., average age of physical capital), and human capital stock estimates.

OCR for page 23
--> Dale Jorgenson, professor of economics at Harvard University, agreed that in the absence of innovation output measures in the national accounts, a component of national output or gross domestic product (GDP) that corresponds to the national income produced and then invested in the form of R&D is missing. Jorgenson observed, however, that Baruch Lev's work represents a way to measure the output of R&D activity at the firm level and questioned whether such firm-level financial data can be aggregated in a way that could be incorporated in the national accounts. Lev replied that aggregation may be possible, especially in periods during which the market for R&D is growing and firms are valuing R&D at its fair market value. Landefeld responded that the development of new, more direct measures of innovation and R&D output is an objective that the Bureau of Economic Analysis hopes to pursue in the future. Other participants cautioned, however, that because financial market valuations of R&D assets are based on highly volatile expectations, as a component of the national accounts they would be problematic. Furthermore, market transactions do not necessarily reflect the benefits to consumers and spillovers to other firms. Other New Output Measures Other measures of innovative output that received brief mention include innovation counts and bibliometrics. Databases at the Science Policy Research Unit at the University of Sussex, England, and the Small Business Administration are the result of literature searches to develop lists of innovations. Bibliometric studies use data on published papers and citations to assess the output of basic scientific research. Another approach to measuring technology output, called technometrics, involves measuring and comparing the dimensions of technical performance of a product or production process. In none of these cases is there an agreed upon methodology for weighing the dimensions of change and comparing them with costs. As a result, the development of new measures of innovative output was generally accorded high priority. Technology Use Survey Data The extent to which new technologies are used, the factors influencing their adoption, and the impact of their use on factors such as plant productivity, product quality, workforce skill requirements, and employment, are all outcomes of technological innovation as well as integral parts of the interactive process of innovation. Surveys of technology use and adoption are therefore an important element of national statistics on industrial innovation. At the initiative of the Department of Defense and with its support, the Census Bureau has conducted two surveys of technology use in defense-related manufacturing industries. The Survey of Manufacturing Technology (SMT), conducted

OCR for page 23
--> in 1988 and again in 1993, asked firms about their use of 17 advanced dual-use technologies. Such surveys have the potential to provide useful information on the adoption, use, diffusion, and effects of new process technologies, especially if they generate longitudinal data.5 Unfortunately, the SMT surveys were not administered with this as a priority. Of the 10,000 plants in the first SMT and the 7,000 plants in the second survey, fewer than 2,000 were in both samples. In any case, there are no plans to repeat the SMT, let alone extend it to a broader range of manufacturing industries or to the service sector. Statistics Canada, the centralized federal statistical agency, adopted the Census Bureau's technology use survey model but conducts its survey on a regular basis. Citing the Canadian example, several participants suggested repeating the SMT on a regular basis with a stable but expanding sample of firms. Extending the survey to service companies would enable researchers to assess the impact of their enormous recent investment in information technology (IT) and, in particular, the significance of the negligible service-sector productivity improvement from IT adoption. Productivity Effects and Product Improvement To understand the relationship between R&D and product growth requires matching expenditure data, collected at the level of the firm, and production performance data, generally linked to the enterprise level, a task not often attempted. Several workshop participants urged that steps be taken to enable more accurate and efficient matching of these data. Adam Jaffe noted that attempts to measure the productivity effects of R&D fail to account for the full impact because approximately two-thirds to three-quarters of R&D is devoted to new product development rather than to process improvement, underscoring the need to develop new measures of product improvement. Outputs and Outcomes Occurring as Spillovers A number of workshop participants emphasized the critical importance but enormous difficulty of accounting for the spillover effects of innovation—the benefits accruing to other firms and industries without compensation to the innovating firm. John Baldwin of Statistics Canada noted that knowledge flows can be observed with information from innovation surveys, but since the flows are measured largely through information on commercial transactions, they do not constitute spillovers and therefore do not represent market failure. Spillover effects can occur as productivity improvements, but other types of outputs and outcomes need to be measured as well. Participants emphasized that the measurement and analysis of spillover effects were central to the task of improving policy-relevant information on industrial innovation. Bill Long noted that ''if it weren't for spillovers, we wouldn't be holding this workshop."

OCR for page 23
--> To understand the nature and extent of innovation spillovers, input, output, and outcome data need to be matched at different levels of aggregation. This includes data with better geographic detail. It was suggested that if companies reported on the R&D survey forms the locations of their R&D activities, the data could be matched to Statistical Metropolitan Area (SMA) data to allow better analysis of the extent of local spillovers and the formation and dynamics of innovative, technology-intensive geographic regions.