National Academies Press: OpenBook

Industrial Research and Innovation Indicators: Report of a Workshop (1997)

Chapter: 6 Measuring Outputs and Outcomes of Innovation

« Previous: 5 Improving Information on Industrial R&D
Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×

6
Measuring Outputs and Outcomes of Innovation

As described in Chapter 4, innovation output may be indicated by statistics on patents, papers, prizes, invention disclosures, and degrees awarded, while outcome proxies include patent and paper citations, expert evaluations, innovation counts, new product sales, measured productivity growth, and benefit/cost or rate-of-return estimates. The workshop discussion focused primarily on patent counts, patent citations, productivity measurement, valuation of R&D using financial reporting data, the representation of innovation output in the national accounts, and technology adoption surveys. Participants emphasized repeatedly that much of the outcome of innovation accrues to society as spillover effects that present considerable measurement challenges.

Patent Counts and Patent Citations

The number of patents issued and the technical and scientific literature citations on the patents are used to develop quantitative measures of innovative output and science-technology linkages. Adam Jaffe pointed out that among the advantages of patent data are that they are readily available, including on the Internet, contain considerable detail, and can be used to develop time series. The limitations of patent data are well known. Patents represent only a portion of innovative activity and none of the latter stage investments entailed in commercializing technology. Patents are of widely varying value to firms. Motivations for patenting vary across industries and technologies; equally important, they change over time. The strengths and limitations of patent count data as an indicator of innovation output are summarized in the accompanying text box.

Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×

Patent Count Data as an Indicator of Innovation Output

Strengths

  • Are available to the public,
  • Contain considerable detail, and
  • Are available over time, enabling time series analysis.

Limitations

  • Some important technologies are not patentable, such as software that is protected by copyright.
  • Not all inventions are patented. Firms protect the returns to their investment in other ways such as through secrecy, lead-time advantages, and marketing.
  • Firms patent for different reasons, not all of them related directly to commercial exploitation, for example, to protect an invention from imitation, to block competitors from patenting or pursuing a line of research, or to evaluate the productivity of their R&D activities. This adds considerable uncertainty to the interpretation of patent count data.
  • Patents represent only the practical application of ideas, not more general advances in knowledge.
  • Patents represent inventions, not activities and investments to commercialize new technology.
  • Patents have widely varying commercial value and therefore significance with respect to innovation.

One suggestion for improving the usefulness of patent count data was to ask companies to list on the RD-1 survey form the names of the entities under which they hold patents. With this information, one could match R&D expenditures to the patent data and create a series for patents by industry that corresponds to existing data series on R&D by industry. Some participants also noted current efforts to use the information gathered by innovation surveys (e.g., on the use of patents relative to other means of appropriating returns to innovation) and to weight patent counts by industry.

The front page of U.S. patents contains citations of previous patents and other critical scientific literature purportedly representing the intellectual lineage or prior art of the patented invention and distinguishing it from previous inventions. In principle, patent citation data contain a wealth of micro-information on

Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×

Patent Citation Data as an Indicator of Knowledge Flows

Strength

  • Are available to the public,
  • Contain considerable detail, including related inventions and research, and
  • Are available over time, enabling time series analysis.

Limitations

  • Origins and innovations are uncertain; citations inserted by persons other than the inventor (e.g., the applicant's lawyer or the Patent Office examiner) may not reflect the actual flow of knowledge as much as a strategy for obtaining patent approval or the ease of literature searches with automated databases; and
  • Citation practices may vary across patents, technologies, industries, countries, and so forth. This variance may bias quantitative analyses of citation counts.

science and technology linkages by subject, geographic location, and source of investment. For example, they are being used by Adam Jaffe and Francis Narin of CHI Research, Inc., as proxies of technical knowledge flows to assess firms' reliance on science and technology of the same or foreign national origins, on publicly versus privately funded research, and on particular public research sponsors. The strengths and limitations of patent citation data as an indicator of knowledge flows are given in the text box.

Aside from the limitations of patent data generally, this promising line of research is handicapped by a good deal of uncertainty about the origins and motivation of patent citations. It is known, for example, that some proportion of prior art citations on applications are inserted by applicants' attorneys or consultants and by examining officers in the Patent Office, presumably to strengthen the case for issuing a patent. That may weaken the inference that the inventor has in some measure relied on the work listed.

New Measures of Innovative Output

Throughout the workshop, participants commented on the need to develop new measures of innovative output. Better measures and a better understanding of the products and outcomes of innovative activity and of the economic and social value of this output are especially needed to inform public policy making.

Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×

Participants discussed using financial market information to estimate the economic value of R&D capacity and developing more direct output indicators in the construction of the national accounts.

Financial Market Valuations of R&D Investments

Although the outputs and outcomes of industrial R&D activities are often difficult to define, let alone value, financial markets do provide a valuation of firms' R&D investments. For example, researchers have used financial reports to link R&D investments to changes in company stock prices. Baruch Lev, professor of accounting and finance at New York University's Stern School of Business, reported on work of the Intangibles Research Project, which was established with support from the Securities and Exchange Commission (SEC). In a recent study of 1,500 R&D-intensive companies, Lev capitalized reported R&D and then analyzed the relationship between the R&D investments and changes in the companies' stock prices. Lev found a strong, positive correlation between new R&D outlays and stock price increases. Although financial accounting standards require full expensing of R&D outlays in financial reports of public corporations, the software industry has been required to capitalize part of software development costs. Here, too, studies have found R&D costs to be highly correlated with subsequent reported earnings.

Another measure of the value of R&D is the reported value of assets in a corporate acquisition. Lev and his colleagues have found that firms have a tendency to expense all of the R&D in progress at the time of acquisition. A company's valuation of the R&D stock, which is based on the net present value of expected future cash flows, provides a quantitative measure of R&D output.

These encouraging results of analyzing financial information suggest that there may indeed be a market for R&D and therefore market-based indicators of the value of intellectual capital. One implication of this analysis is that the R&D reporting requirements of the SEC should be reexamined. The Financial Accounting Standards Board decision in 1974 to require full expensing of R&D outlays rested on the lack of evidence of a relation between R&D expenditures and subsequent benefits. If the relationship is empirically demonstrable, full expensing may not be appropriate.

Representation of Innovation Output in the National Accounts

Steven Landefeld explained that the national accounts do not really measure the output of R&D or innovative activity because the value of R&D is represented solely by its cost, which is an input to innovation. Landefeld noted that more direct measures of R&D and technological change may require use of nonexpenditure data such as the number of patents, patent applications, R&D personnel, proxies for embodied technological change (e.g., average age of physical capital), and human capital stock estimates.

Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×

Dale Jorgenson, professor of economics at Harvard University, agreed that in the absence of innovation output measures in the national accounts, a component of national output or gross domestic product (GDP) that corresponds to the national income produced and then invested in the form of R&D is missing. Jorgenson observed, however, that Baruch Lev's work represents a way to measure the output of R&D activity at the firm level and questioned whether such firm-level financial data can be aggregated in a way that could be incorporated in the national accounts. Lev replied that aggregation may be possible, especially in periods during which the market for R&D is growing and firms are valuing R&D at its fair market value. Landefeld responded that the development of new, more direct measures of innovation and R&D output is an objective that the Bureau of Economic Analysis hopes to pursue in the future.

Other participants cautioned, however, that because financial market valuations of R&D assets are based on highly volatile expectations, as a component of the national accounts they would be problematic. Furthermore, market transactions do not necessarily reflect the benefits to consumers and spillovers to other firms.

Other New Output Measures

Other measures of innovative output that received brief mention include innovation counts and bibliometrics. Databases at the Science Policy Research Unit at the University of Sussex, England, and the Small Business Administration are the result of literature searches to develop lists of innovations. Bibliometric studies use data on published papers and citations to assess the output of basic scientific research. Another approach to measuring technology output, called technometrics, involves measuring and comparing the dimensions of technical performance of a product or production process. In none of these cases is there an agreed upon methodology for weighing the dimensions of change and comparing them with costs. As a result, the development of new measures of innovative output was generally accorded high priority.

Technology Use Survey Data

The extent to which new technologies are used, the factors influencing their adoption, and the impact of their use on factors such as plant productivity, product quality, workforce skill requirements, and employment, are all outcomes of technological innovation as well as integral parts of the interactive process of innovation. Surveys of technology use and adoption are therefore an important element of national statistics on industrial innovation.

At the initiative of the Department of Defense and with its support, the Census Bureau has conducted two surveys of technology use in defense-related manufacturing industries. The Survey of Manufacturing Technology (SMT), conducted

Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×

in 1988 and again in 1993, asked firms about their use of 17 advanced dual-use technologies. Such surveys have the potential to provide useful information on the adoption, use, diffusion, and effects of new process technologies, especially if they generate longitudinal data.5 Unfortunately, the SMT surveys were not administered with this as a priority. Of the 10,000 plants in the first SMT and the 7,000 plants in the second survey, fewer than 2,000 were in both samples. In any case, there are no plans to repeat the SMT, let alone extend it to a broader range of manufacturing industries or to the service sector.

Statistics Canada, the centralized federal statistical agency, adopted the Census Bureau's technology use survey model but conducts its survey on a regular basis. Citing the Canadian example, several participants suggested repeating the SMT on a regular basis with a stable but expanding sample of firms. Extending the survey to service companies would enable researchers to assess the impact of their enormous recent investment in information technology (IT) and, in particular, the significance of the negligible service-sector productivity improvement from IT adoption.

Productivity Effects and Product Improvement

To understand the relationship between R&D and product growth requires matching expenditure data, collected at the level of the firm, and production performance data, generally linked to the enterprise level, a task not often attempted. Several workshop participants urged that steps be taken to enable more accurate and efficient matching of these data. Adam Jaffe noted that attempts to measure the productivity effects of R&D fail to account for the full impact because approximately two-thirds to three-quarters of R&D is devoted to new product development rather than to process improvement, underscoring the need to develop new measures of product improvement.

Outputs and Outcomes Occurring as Spillovers

A number of workshop participants emphasized the critical importance but enormous difficulty of accounting for the spillover effects of innovation—the benefits accruing to other firms and industries without compensation to the innovating firm. John Baldwin of Statistics Canada noted that knowledge flows can be observed with information from innovation surveys, but since the flows are measured largely through information on commercial transactions, they do not constitute spillovers and therefore do not represent market failure.

Spillover effects can occur as productivity improvements, but other types of outputs and outcomes need to be measured as well. Participants emphasized that the measurement and analysis of spillover effects were central to the task of improving policy-relevant information on industrial innovation. Bill Long noted that ''if it weren't for spillovers, we wouldn't be holding this workshop."

Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×

To understand the nature and extent of innovation spillovers, input, output, and outcome data need to be matched at different levels of aggregation. This includes data with better geographic detail. It was suggested that if companies reported on the R&D survey forms the locations of their R&D activities, the data could be matched to Statistical Metropolitan Area (SMA) data to allow better analysis of the extent of local spillovers and the formation and dynamics of innovative, technology-intensive geographic regions.

Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×
Page 23
Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×
Page 24
Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×
Page 25
Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×
Page 26
Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×
Page 27
Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×
Page 28
Suggested Citation:"6 Measuring Outputs and Outcomes of Innovation." National Research Council. 1997. Industrial Research and Innovation Indicators: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/5976.
×
Page 29
Next: 7 Innovation Surveys »
Industrial Research and Innovation Indicators: Report of a Workshop Get This Book
×
Buy Paperback | $47.00
MyNAP members save 10% online.
Login or Register to save!

Workshop participants offered a variety of suggestions to the National Science Foundation (NSF) and other federal agencies to improve the usefulness and relevance of data on industrial research and innovation, as well as to increase efficiency in collecting and processing the data. The suggestions dealt with the need to (1) clarify policy information needs; (2) improve the quality, coverage, and collection of existing data items; and (3) identify and collect new types of data.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!