6
Measuring Outputs and Outcomes of Innovation
As described in Chapter 4, innovation output may be indicated by statistics on patents, papers, prizes, invention disclosures, and degrees awarded, while outcome proxies include patent and paper citations, expert evaluations, innovation counts, new product sales, measured productivity growth, and benefit/cost or rate-of-return estimates. The workshop discussion focused primarily on patent counts, patent citations, productivity measurement, valuation of R&D using financial reporting data, the representation of innovation output in the national accounts, and technology adoption surveys. Participants emphasized repeatedly that much of the outcome of innovation accrues to society as spillover effects that present considerable measurement challenges.
Patent Counts and Patent Citations
The number of patents issued and the technical and scientific literature citations on the patents are used to develop quantitative measures of innovative output and science-technology linkages. Adam Jaffe pointed out that among the advantages of patent data are that they are readily available, including on the Internet, contain considerable detail, and can be used to develop time series. The limitations of patent data are well known. Patents represent only a portion of innovative activity and none of the latter stage investments entailed in commercializing technology. Patents are of widely varying value to firms. Motivations for patenting vary across industries and technologies; equally important, they change over time. The strengths and limitations of patent count data as an indicator of innovation output are summarized in the accompanying text box.
Patent Count Data as an Indicator of Innovation Output Strengths
Limitations
|
One suggestion for improving the usefulness of patent count data was to ask companies to list on the RD-1 survey form the names of the entities under which they hold patents. With this information, one could match R&D expenditures to the patent data and create a series for patents by industry that corresponds to existing data series on R&D by industry. Some participants also noted current efforts to use the information gathered by innovation surveys (e.g., on the use of patents relative to other means of appropriating returns to innovation) and to weight patent counts by industry.
The front page of U.S. patents contains citations of previous patents and other critical scientific literature purportedly representing the intellectual lineage or prior art of the patented invention and distinguishing it from previous inventions. In principle, patent citation data contain a wealth of micro-information on
Patent Citation Data as an Indicator of Knowledge Flows Strength
Limitations
|
science and technology linkages by subject, geographic location, and source of investment. For example, they are being used by Adam Jaffe and Francis Narin of CHI Research, Inc., as proxies of technical knowledge flows to assess firms' reliance on science and technology of the same or foreign national origins, on publicly versus privately funded research, and on particular public research sponsors. The strengths and limitations of patent citation data as an indicator of knowledge flows are given in the text box.
Aside from the limitations of patent data generally, this promising line of research is handicapped by a good deal of uncertainty about the origins and motivation of patent citations. It is known, for example, that some proportion of prior art citations on applications are inserted by applicants' attorneys or consultants and by examining officers in the Patent Office, presumably to strengthen the case for issuing a patent. That may weaken the inference that the inventor has in some measure relied on the work listed.
New Measures of Innovative Output
Throughout the workshop, participants commented on the need to develop new measures of innovative output. Better measures and a better understanding of the products and outcomes of innovative activity and of the economic and social value of this output are especially needed to inform public policy making.
Participants discussed using financial market information to estimate the economic value of R&D capacity and developing more direct output indicators in the construction of the national accounts.
Financial Market Valuations of R&D Investments
Although the outputs and outcomes of industrial R&D activities are often difficult to define, let alone value, financial markets do provide a valuation of firms' R&D investments. For example, researchers have used financial reports to link R&D investments to changes in company stock prices. Baruch Lev, professor of accounting and finance at New York University's Stern School of Business, reported on work of the Intangibles Research Project, which was established with support from the Securities and Exchange Commission (SEC). In a recent study of 1,500 R&D-intensive companies, Lev capitalized reported R&D and then analyzed the relationship between the R&D investments and changes in the companies' stock prices. Lev found a strong, positive correlation between new R&D outlays and stock price increases. Although financial accounting standards require full expensing of R&D outlays in financial reports of public corporations, the software industry has been required to capitalize part of software development costs. Here, too, studies have found R&D costs to be highly correlated with subsequent reported earnings.
Another measure of the value of R&D is the reported value of assets in a corporate acquisition. Lev and his colleagues have found that firms have a tendency to expense all of the R&D in progress at the time of acquisition. A company's valuation of the R&D stock, which is based on the net present value of expected future cash flows, provides a quantitative measure of R&D output.
These encouraging results of analyzing financial information suggest that there may indeed be a market for R&D and therefore market-based indicators of the value of intellectual capital. One implication of this analysis is that the R&D reporting requirements of the SEC should be reexamined. The Financial Accounting Standards Board decision in 1974 to require full expensing of R&D outlays rested on the lack of evidence of a relation between R&D expenditures and subsequent benefits. If the relationship is empirically demonstrable, full expensing may not be appropriate.
Representation of Innovation Output in the National Accounts
Steven Landefeld explained that the national accounts do not really measure the output of R&D or innovative activity because the value of R&D is represented solely by its cost, which is an input to innovation. Landefeld noted that more direct measures of R&D and technological change may require use of nonexpenditure data such as the number of patents, patent applications, R&D personnel, proxies for embodied technological change (e.g., average age of physical capital), and human capital stock estimates.
Dale Jorgenson, professor of economics at Harvard University, agreed that in the absence of innovation output measures in the national accounts, a component of national output or gross domestic product (GDP) that corresponds to the national income produced and then invested in the form of R&D is missing. Jorgenson observed, however, that Baruch Lev's work represents a way to measure the output of R&D activity at the firm level and questioned whether such firm-level financial data can be aggregated in a way that could be incorporated in the national accounts. Lev replied that aggregation may be possible, especially in periods during which the market for R&D is growing and firms are valuing R&D at its fair market value. Landefeld responded that the development of new, more direct measures of innovation and R&D output is an objective that the Bureau of Economic Analysis hopes to pursue in the future.
Other participants cautioned, however, that because financial market valuations of R&D assets are based on highly volatile expectations, as a component of the national accounts they would be problematic. Furthermore, market transactions do not necessarily reflect the benefits to consumers and spillovers to other firms.
Other New Output Measures
Other measures of innovative output that received brief mention include innovation counts and bibliometrics. Databases at the Science Policy Research Unit at the University of Sussex, England, and the Small Business Administration are the result of literature searches to develop lists of innovations. Bibliometric studies use data on published papers and citations to assess the output of basic scientific research. Another approach to measuring technology output, called technometrics, involves measuring and comparing the dimensions of technical performance of a product or production process. In none of these cases is there an agreed upon methodology for weighing the dimensions of change and comparing them with costs. As a result, the development of new measures of innovative output was generally accorded high priority.
Technology Use Survey Data
The extent to which new technologies are used, the factors influencing their adoption, and the impact of their use on factors such as plant productivity, product quality, workforce skill requirements, and employment, are all outcomes of technological innovation as well as integral parts of the interactive process of innovation. Surveys of technology use and adoption are therefore an important element of national statistics on industrial innovation.
At the initiative of the Department of Defense and with its support, the Census Bureau has conducted two surveys of technology use in defense-related manufacturing industries. The Survey of Manufacturing Technology (SMT), conducted
in 1988 and again in 1993, asked firms about their use of 17 advanced dual-use technologies. Such surveys have the potential to provide useful information on the adoption, use, diffusion, and effects of new process technologies, especially if they generate longitudinal data.5 Unfortunately, the SMT surveys were not administered with this as a priority. Of the 10,000 plants in the first SMT and the 7,000 plants in the second survey, fewer than 2,000 were in both samples. In any case, there are no plans to repeat the SMT, let alone extend it to a broader range of manufacturing industries or to the service sector.
Statistics Canada, the centralized federal statistical agency, adopted the Census Bureau's technology use survey model but conducts its survey on a regular basis. Citing the Canadian example, several participants suggested repeating the SMT on a regular basis with a stable but expanding sample of firms. Extending the survey to service companies would enable researchers to assess the impact of their enormous recent investment in information technology (IT) and, in particular, the significance of the negligible service-sector productivity improvement from IT adoption.
Productivity Effects and Product Improvement
To understand the relationship between R&D and product growth requires matching expenditure data, collected at the level of the firm, and production performance data, generally linked to the enterprise level, a task not often attempted. Several workshop participants urged that steps be taken to enable more accurate and efficient matching of these data. Adam Jaffe noted that attempts to measure the productivity effects of R&D fail to account for the full impact because approximately two-thirds to three-quarters of R&D is devoted to new product development rather than to process improvement, underscoring the need to develop new measures of product improvement.
Outputs and Outcomes Occurring as Spillovers
A number of workshop participants emphasized the critical importance but enormous difficulty of accounting for the spillover effects of innovation—the benefits accruing to other firms and industries without compensation to the innovating firm. John Baldwin of Statistics Canada noted that knowledge flows can be observed with information from innovation surveys, but since the flows are measured largely through information on commercial transactions, they do not constitute spillovers and therefore do not represent market failure.
Spillover effects can occur as productivity improvements, but other types of outputs and outcomes need to be measured as well. Participants emphasized that the measurement and analysis of spillover effects were central to the task of improving policy-relevant information on industrial innovation. Bill Long noted that ''if it weren't for spillovers, we wouldn't be holding this workshop."
To understand the nature and extent of innovation spillovers, input, output, and outcome data need to be matched at different levels of aggregation. This includes data with better geographic detail. It was suggested that if companies reported on the R&D survey forms the locations of their R&D activities, the data could be matched to Statistical Metropolitan Area (SMA) data to allow better analysis of the extent of local spillovers and the formation and dynamics of innovative, technology-intensive geographic regions.