National Academies Press: OpenBook
« Previous: 3 Measuring Human Capital
Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×

4

Measuring Innovation

 

CONCEPTS AND DEFINITIONS OF INNOVATION

The use of “innovation” in this report is taken from Schumpeter (1934, p. 66), who defined product innovation as “the introduction of a new good … or a new quality of a good,” and process innovation as ‘‘the introduction of a new method of production … or a new way of handling a commodity commercially.” This basic definition of innovation was codified by the Organisation for Economic Co-operation and Development-Eurostat (2005, p. 46):

…the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organizational method in business practices, workplace organisation or external relations.…This includes products, processes and methods that firms are the first to develop and those that have been adopted from other firms or organisations.

Innovations can be distinguished from inventions by the criterion that innovations are implemented in the marketplace. Innovation, therefore, is a new product or idea when it is commercialized, by definition. This definition does not mean that an innovation is necessarily widely distributed or diffused in a market. It does mean that development of a new product or process that is not marketed is not considered an innovation.

However, research and development (R&D) and other activities related to innovation are counted as innovation activity in the Oslo Manual (Organisation for Economic Co-operation and Development-Eurostat, 2005, p. 18):

…all scientific, technological, organisational, financial and commercial steps which actually, or are intended to, lead to the implementation of innovations. Some innovation activities are themselves innovative; others are not novel activities but are necessary for the implementation of innovations. Innovation activities also include R&D that is not directly related to the development of a specific innovation.

Since innovation is a term widely used in society, the National Center for Science and Engineering Statistics (NCSES) goes to great lengths to convey to its survey respondents what is meant by innovation or innovation activities. Yet it is important to recognize that such concepts as invention, innovation, and technological diffusion are on a continuum, and there is still some debate regarding their respective space on that continuum. As NCSES develops surveys, new datasets and new indicators of innovation activities, it will be important to try to establish

Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×

rigorous standards of definitions for these terms. NCSES’s role in the working group of National Experts on Science and Technology Indicators (NESTI) of the Organisation for Economic Cooperation and Development (OECD) gives the agency good opportunity over time to establish clearer definitions as revisions are made to the Frascati manual on R&D (Organisation for Economic Co-operation and Development, 2002), the Oslo manual on innovation (Organisation for Economic Co-operation and Development-Eurostat, 2005), and, possibly, the Canberra manual on human resources (Organisation for Economic Co-operation and Development, 1995). Although there is no mandate to review and revise the Community Innovation Survey (CIS), the OECD has received recommendations on how to better design innovation questions. This will have implications for CIS, BRDIS and other innovation surveys internationally. Several things affect the lack of comparability of United States and European data on the innovativeness of firms—the framing effect of using a lengthy R&D survey, sampling errors, and weighting issues, to name a few. NCSES and the OECD are actively collecting evidence to assess what factors may drive biases in international comparisons.

Since 2008, the National Science Foundation (NSF) has collected data on product and process innovation from the Business Research and Development and Innovation Survey (BRDIS). NCSES augmented its R&D survey to measure innovation activities, allowing for comparisons of innovation statistics across several countries.1 Although the 2008 BRDIS was a pilot survey, it did yield some data on the incidence of product and process innovation among firms by sector (including services), size class, and whether or not respondents reported R&D activity.2 BRDIS questions on innovation were augmented in the 2009 and 2010 versions of the survey; the 2011 version is currently under development.3 NCSES endeavors to gather more information on innovation activities, going beyond simple “yes/no” questions on whether a firm introduced or significantly improved goods, services or processes. These efforts have also included attempts to develop more comparability to key questions in the CIS and to ensure that the innovation questions are answered by firms that do not do R&D. Comparability of the BRDIS and CIS data also depends on surveying similar populations of firms and the techniques used to derive estimates from the data. BRDIS is still a work in progress. Complete cognitive testing of the innovation questions still remains to be done in both the United States and Europe. Nevertheless, the data are useful for preliminary snapshots.

POLICY RELEVANCE

One impediment to understanding and assessing the country’s innovation is the lack of comparability of U.S. STI indicators with those developed by other OECD nations and other countries. NCSES should develop more useful indicators of innovation—an outcome measure. The biennial publication of the National Science Board, Science and Engineering Indicators (SEI), now includes information on a range of factors, including science, technology, engineering, and mathematics (STEM) graduates, R&D, and patents, but these are intermediate inputs into or proxies for innovation. They are not indicators of innovation itself, which is the

____________

1It is widely known that the innovation statistics from BRDIS and the CIS lack comparability: see an explanation in Hall (2011, fn. 4).

2See InfoBrief 11-300 (October 2010): available: http://www.nsf.gov/statistics/infbrief/nsf11300/ [December 2011]. The data are based on the 2008 BRDIS, which was launched in January 2009.

3Tabulations on the 2009 data are due to be released in mid-2012, with an InfoBrief on the 2009 BRDIS data scheduled to be released in December 2011. The 2010 BRDIS data were still being collected in November 2011.

Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×

combination of new products and services (see Organisation for Economic Co-operation and Development-Eurostat, 2005, pp. 89-104; 2011, Chapter 5).

Ideally, one would like to measure the contribution of innovation to total output (gross domestic product, GDP) in the United States and have these measures in a form that allows for cross-country comparisons. However, the currently available measures are imperfect. Policy makers and citizens would be well served by useful proxies that could mature into more refined measures of innovation.

In this report we focus on the need for NCSES further to use the BRDIS data to answer such key policy questions as: How influential is R&D for innovation and growth (in manufacturing compared with services)? At times of economic crisis, what is the contribution of scientific research and innovation to economic recovery? How important are different kinds of entities for advancing national innovation: small firms, young firms, high-growth firms, government labs and procurement activities, and nonprofit organizations? How does innovation activity in a given firm at a given place contribute to that firm’s productivity, employment and growth, and perhaps also to these characteristics in the surrounding area?

NEEDED DATA

To answer these questions, four improvements seem feasible for NCSES in the near term, using BRDIS data: statistics that are more comparable with those of other OECD countries; better information on what firms characterize as innovation; data on high-growth firms and “gazelles;” and data on the relationship of new firms to innovation.

Comparable Statistics

Understanding the nation’s position on innovation would be greatly helped if NCSES developed statistics from its BRDIS data with the same cutoffs as those from other OECD countries. In addition to the number of innovative firms in various industries that have 5 or more employees, it would be helpful to have similar statistics for firms with 10 or more employees and firms with 20 or more employees. These data will show the size dependency of innovation activities.

Also needed for comparability are statistics using the same set of industries typically used in statistics for other countries. An example is the core set of industries used by Eurostat for comparison of innovation statistics among European Union countries, including: mining; manufacturing; and selected service industries, such as wholesale trade, transports, financial services, information technology (IT) services, R&D services, and business services. These data could be used to compile a simple indicator of the share of product-process innovative firms, defined as firms that have implemented a product or process innovation.

NCSES is developing another R&D and innovation survey on firms with fewer than five employees: MIST, Microbusiness, Innovation Science, and Technology. The survey is designed to yield data measuring the incidence of R&D activities among small businesses. NCSES has an interagency agreement with the Internal Revenue Service (IRS) to use IRS data for the sampling frame, for data quality review and to supplement the MIST data. Potential questions in MIST will query firms on: R&D and innovation funding; employment and owner characteristics; sales of new or significantly improved goods and services; technology transfer and knowledge diffusion; sources of technical knowledge; and measures of firms’ entrepreneurial effectiveness.

Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×

The MIST pilot is planned for 2012. Those data, together with BRDIS data for companies with 5+, 10+, and 20+ employees, will provide useful information on the size dependency of innovation (among other things).4

Understanding “Innovation”

Even with more comparable statistics on innovation, however, it will still not be clear to users that firms are representing the same or very similar things when they report product or process innovations. The survey questions are not transparent enough to provide a full understanding of what the resulting data and statistics mean. For example, users have no independent measure of whether the innovations of firms that innovate but do not do R&D are more or less important than those that do R&D. Users would have more confidence in and understanding of BRDIS innovation measures if they knew that knowledge input measures correlated with actual performance and even more confidence if they knew what some of the firms were calling innovation—how closely their reports matched the standards in the question. To obtain this information, NCSES could commission a study of a subset of firms to determine what they are measuring as innovation. It would be useful to have firms of different sizes, different sectors and different geographic locations represented in such a study.

Innovation and Firm Size

The data in BRDIS could be used to begin developing statistics on high-growth firms and “gazelles.” The Manual on Business Demography (Organisation for Economic Co-operation and Development-Eurostat 2008, Chapter 2) defines high-growth enterprises as “All enterprises with average annualised growth greater than 20% per annum, over a three year period…. Growth can be measured by the number of employees or by turnover…. A size threshold has been suggested as 10 employees at the beginning of the growth period.” Gazelles are the subset of high-growth enterprises that are up to 5 years old. These thresholds are arbitrary and only based on convention. NCSES could conduct its own sensitivity analysis to fine-tune the definitions of high-growth firms and gazelles.5 Ditte Rude Petersen and Nadim Ahmad show a technique of conducting this type of analysis in “High-Growth Enterprises and Gazelles—Preliminary and Summary Sensitivity Analysis (Organisation for Economic Co-operation and Development, 2007).

During the panel’s workshop, several speakers (Howard Alper, University of Ottawa; Robert Atkinson, Information Technology and Innovation Foundation; John Haltiwanger, University of Maryland; Hugo Hollanders, U.N. University’s Maastricht Economic and Social Research Institute on Innovation and Technology [UNU-MERIT]; and Brian MacAulay, National Endowment for Science, Technology, and the Arts) mentioned the importance of tracking trends of sustainability of jobs in these types of firms during economic downturns (even if total employment is small). Atkinson said there was an interest in having these data at the national, state, and at finer geographical levels.

It would also be useful to have those data to determine over time whether high-growth firms or gazelles in particular have a higher incidence of innovation activity. The connection

____________

4Clearly, other data, together with more thorough regression analysis, would be necessary to determine causal relationships.

5See Petersen and Ahmad (2007) for a technique of conducting this type of analysis.

Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×

between high-growth firms and innovation is complex. These data would enable researchers better to determine the relationship between the presence of high-growth firms and innovation. At the June 2011 roundtable workshop lead by Andrew Reamer (George Washington University) on innovation measures, B.K. Atrostic, Cheryl Grim, and Javier Miranda from the Center for Economic Studies at the U.S. Census Bureau suggested the development of a database of business dynamics statistics, which would provide information about births and deaths of firms, as well as the distribution of growth rates of gazelles and other types of high-growth firms. The suggested database would have information on firm size, age, location, and sector. This information would allow examination of the connection between innovative industries’ job creation and high-growth firms in the context of economies at various geographical scales.

In his presentation, at the panel’s July 2011 workshop, Hollanders showed that high-growth firms are significantly more innovative than other firms in his dataset. The statistics on high-growth firms and gazelles could, therefore, be used to answer the following question: Are these the types of firms that drive economic and job growth? A simple table could show high-growth firms and other firms that are and are not innovative to compare economic characteristics, ideally over time.

At the September 2011 panel meeting in Washington, DC, representatives from the Bureau of Labor Statistics (BLS), the U.S. Census Bureau, and the Bureau of Economic Analysis (BEA) mentioned that linking certain datasets among them would yield reasonable numbers on gazelles. Such a table could be added to the SEI or become the foundation of an InfoBrief. The following indicators could be produced by starting with the BRDIS data on high-growth firms and gazelles: rate of high-growth enterprises (number of high-growth enterprises as a percentage of the total population of active enterprises with at least n-number of employees); rate of gazelles among newly born enterprises (number of gazelles as a percentage of all active enterprises with at least n-number of employees that were born 4 or 5 years ago). These indicators would be comparable to those produced in several other countries, thus increasing users’ understanding of the comparative position of the United States on an aspect of the country’s innovative capacity.

Clearly, getting publishable statistics on high-growth firms and gazelles is a multistage task that will require data acquisition and linking in addition to the data available in BRDIS. A good first step would be for NCSES to explore the matchup of their BRDIS data with data on firm dynamics from BLS. The panel’s final report will address the complexities of data linking, particularly with the view to international comparability of the resulting statistics.

Relationship of New Firms and Innovation

New indicators that allow users to determine to what extent births of new firms are driving innovation would also be useful. At the panel’s workshop, John Haltiwanger presented a compelling presentation on what he and other researchers have developed using Census Bureau data on firm dynamics—that is, firm births and deaths over time. These data can provide the groundwork to answer important questions from policy makers and researchers, such as: Does innovation come disproportionately from new firms? Do start-ups and young businesses create net new jobs and increase productivity growth?

NCSES has a unique set of data in BRDIS, which, when combined with other datasets, can be instrumental in answering these and other important questions. Integrating firm dynamism (and the related employment effect) will take time and resources. During his presentation,

Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×

Haltiwanger described three Census Bureau datasets that, together with BRDIS data, would allow NCSES to develop its business dynamics indicators:

  • Longitudinal Business Database—tracks all establishments and firms with at least one employee including startups from 1976 to present;
  • Integrated Longitudinal Business Database—tracks all nonemployer firms and integrated with employer firms from 1994 to present; and
  • Longitudinal Employer-Household Dynamics—tracks longitudinally all employer-employee matches and transitions (hires, separations, job creation, and job destruction) from 1990 to present.6

Questions from the Census Bureau’s 2007 and 2012 Economic Census, Company Organization Survey, and Management and Organizational Practices Survey will also yield useful information on R&D and other innovation activities for establishments. Infrastructure datasets can also track relationships between start-up and young high-growth firms and large, mature firms, and they can be linked further to patent and citation data. It is important as well to link the data on firm dynamics to those on innovation outputs, such as patent and citation data.

In the near term, NCSES could begin to match up its BRDIS data to datasets at the Census and BLS to create indicators of firm dynamism. Haltiwanger proposed that indicators track dynamics by geography, industry, business size, and business age. Hollanders noted that European countries and other OECD members are continuing to fine-tune their measures of firm dynamism. NCSES’s indicators on this dimension could further the international comparability of its STI indicators. Beginning to build the foundations on firm dynamics using BRDIS and other datasets would give NCSES a productive platform for developing several STI indicators that are policy relevant.

There are many other avenues that NCSES could take to developing a complete suite of indicators on innovation. In the panel’s final report, we will offer specific recommendations on how NCSES could develop statistics on expenditures on intangible assets and life lengths (as suggested by Haskel). For many of these measures, research is still needed to determine proper weighting methods and deflators and how to structure questionnaires (or modules to be placed on existing questionnaires).

RECOMMENDATIONS

RECOMMENDATION 2: The National Center for Science and Engineering Statistics should develop new indicators on innovation, based on data from its Business Research and Development and Innovation Survey (BRDIS). The agency should develop comparative statistics with the same cutoffs used by countries in the Organisation for Economic Co-operation and Development for its BRDIS data.

RECOMMENDATION 3: The National Center for Science and Engineering Statistics should begin to match its Business Research and Development and Innovation Survey data to data from ongoing surveys at the U.S. Census Bureau and the Bureau of Labor Statistics to create indicators of firm dynamism. This is a

____________

6These data would also enable NCSES to create indicators on job mobility (see further discussion below).

Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×

necessary first step for developing data linkages that yield measures of activities by high-growth firms, and on births and deaths of businesses linked to innovation outputs. These measures should be established by geographic and industry sectors and by business size and business age. Such measures would be an important step in furthering international comparability on innovation indicators. NCSES should conduct its own sensitivity analysis to fine tune meaningful age categories of high-growth firms.

Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×
Page 20
Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×
Page 21
Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×
Page 22
Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×
Page 23
Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×
Page 24
Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×
Page 25
Suggested Citation:"4 Measuring Innovation." National Research Council. 2012. Improving Measures of Science, Technology, and Innovation: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/13358.
×
Page 26
Next: 5 Measuring Research and Development Services »
Improving Measures of Science, Technology, and Innovation: Interim Report Get This Book
×
Buy Paperback | $31.00 Buy Ebook | $24.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Center for Science and Engineering Statistics (NCSES), at the U.S. National Foundation, is 1 of 14 major statistical agencies in the federal government, of which at least 5 collect relevant information on science, technology, and innovation activities in the United States and abroad. The America COMPETES Reauthorization Act of 2010 expanded and codified NCSES's role as a U.S. federal statistical agency. Important aspects of the agency's mandate include collection, acquisition, analysis, and reporting and dissemination of data on research and development trends, on U.S. competitiveness in science, technology, and research and development, and on the condition and progress of U.S. science, technology, engineering, and mathematics (STEM) education.

Improving Measures of Science, Technology and Innovation: Interim Report examines the status of the NCSES's science, technology, and innovation (STI) indicators. This report assesses and provides recommendations regarding the need for revised, refocused, and newly developed indicators designed to better reflect fundamental and rapid changes that are reshaping global science, technology and innovation systems. The book also determines the international scope of STI indicators and the need for developing new indicators that measure developments in innovative activities in the United States and abroad, and Offers foresight on the types of data, metrics and indicators that will be particularly influential in evidentiary policy decision-making for years to come.

In carrying out its charge, the authoring panel undertook a broad and comprehensive review of STI indicators from different countries, including Japan, China, India and several countries in Europe, Latin America and Africa. Improving Measures of Science, Technology, and Innovation makes recommendations for near-term action by NCSES along two dimensions: (1) development of new policy-relevant indicators that are based on NCSES survey data or on data collections at other statistical agencies; and (2) exploration of new data extraction and management tools for generating statistics, using automated methods of harvesting unstructured or scientometric data and data derived from administrative records.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!