Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 20
4
Measuring Innovation
CONCEPTS AND DEFINITIONS OF INNOVATION
The use of “innovation” in this report is taken from Schumpeter (1934, p. 66), who
defined product innovation as “the introduction of a new good . . . or a new quality of a good,”
and process innovation as ‘‘the introduction of a new method of production . . . or a new way of
handling a commodity commercially.” This basic definition of innovation was codified by the
Organisation for Economic Co-operation and Development-Eurostat (2005, p. 46):
…the implementation of a new or significantly improved product (good or
service), or process, a new marketing method, or a new organizational method in
business practices, workplace organisation or external relations.…This includes
products, processes and methods that firms are the first to develop and those that
have been adopted from other firms or organisations.
Innovations can be distinguished from inventions by the criterion that innovations are
implemented in the marketplace. Innovation, therefore, is a new product or idea when it is
commercialized, by definition. This definition does not mean that an innovation is necessarily
widely distributed or diffused in a market. It does mean that development of a new product or
process that is not marketed is not considered an innovation.
However, research and development (R&D) and other activities related to innovation are
counted as innovation activity in the Oslo Manual (Organisation for Economic Co-operation and
Development-Eurostat, 2005, p. 18):
. . . all scientific, technological, organisational, financial and commercial steps
which actually, or are intended to, lead to the implementation of innovations.
Some innovation activities are themselves innovative; others are not novel
activities but are necessary for the implementation of innovations. Innovation
activities also include R&D that is not directly related to the development of a
specific innovation.
Since innovation is a term widely used in society, the National Center for Science and
Engineering Statistics (NCSES) goes to great lengths to convey to its survey respondents what is
meant by innovation or innovation activities. Yet it is important to recognize that such concepts
as invention, innovation, and technological diffusion are on a continuum, and there is still some
debate regarding their respective space on that continuum. As NCSES develops surveys, new
datasets and new indicators of innovation activities, it will be important to try to establish
20
OCR for page 21
rigorous standards of definitions for these terms. NCSES’s role in the working group of National
Experts on Science and Technology Indicators (NESTI) of the Organisation for Economic Co-
operation and Development (OECD) gives the agency good opportunity over time to establish
clearer definitions as revisions are made to the Frascati manual on R&D (Organisation for
Economic Co-operation and Development, 2002), the Oslo manual on innovation (Organisation
for Economic Co-operation and Development-Eurostat, 2005), and, possibly, the Canberra
manual on human resources (Organisation for Economic Co-operation and Development, 1995).
Although there is no mandate to review and revise the Community Innovation Survey (CIS), the
OECD has received recommendations on how to better design innovation questions. This will
have implications for CIS, BRDIS and other innovation surveys internationally. Several things
affect the lack of comparability of United States and European data on the innovativeness of
firms—the framing effect of using a lengthy R&D survey, sampling errors, and weighting issues,
to name a few. NCSES and the OECD are actively collecting evidence to assess what factors
may drive biases in international comparisons.
Since 2008, the National Science Foundation (NSF) has collected data on product and
process innovation from the Business Research and Development and Innovation Survey
(BRDIS). NCSES augmented its R&D survey to measure innovation activities, allowing for
comparisons of innovation statistics across several countries.1 Although the 2008 BRDIS was a
pilot survey, it did yield some data on the incidence of product and process innovation among
firms by sector (including services), size class, and whether or not respondents reported R&D
activity.2 BRDIS questions on innovation were augmented in the 2009 and 2010 versions of the
survey; the 2011 version is currently under development.3 NCSES endeavors to gather more
information on innovation activities, going beyond simple “yes/no” questions on whether a firm
introduced or significantly improved goods, services or processes. These efforts have also
included attempts to develop more comparability to key questions in the CIS and to ensure that
the innovation questions are answered by firms that do not do R&D. Comparability of the
BRDIS and CIS data also depends on surveying similar populations of firms and the techniques
used to derive estimates from the data. BRDIS is still a work in progress. Complete cognitive
testing of the innovation questions still remains to be done in both the United States and Europe.
Nevertheless, the data are useful for preliminary snapshots.
POLICY RELEVANCE
One impediment to understanding and assessing the country’s innovation is the lack of
comparability of U.S. STI indicators with those developed by other OECD nations and other
countries. NCSES should develop more useful indicators of innovation—an outcome measure.
The biennial publication of the National Science Board, Science and Engineering Indicators
(SEI), now includes information on a range of factors, including science, technology,
engineering, and mathematics (STEM) graduates, R&D, and patents, but these are intermediate
inputs into or proxies for innovation. They are not indicators of innovation itself, which is the
1
It is widely known that the innovation statistics from BRDIS and the CIS lack comparability: see an explanation
in Hall (2011, fn. 4).
2
See InfoBrief 11-300 (October 2010): available: http://www.nsf.gov/statistics/infbrief/nsf11300/ [December
2011]. The data are based on the 2008 BRDIS, which was launched in January 2009.
3
Tabulations on the 2009 data are due to be released in mid-2012, with an InfoBrief on the 2009 BRDIS data
scheduled to be released in December 2011. The 2010 BRDIS data were still being collected in November 2011.
21
OCR for page 22
combination of new products and services (see Organisation for Economic Co-operation and
Development-Eurostat, 2005, pp. 89-104; 2011, Chapter 5).
Ideally, one would like to measure the contribution of innovation to total output (gross
domestic product, GDP) in the United States and have these measures in a form that allows for
cross-country comparisons. However, the currently available measures are imperfect. Policy
makers and citizens would be well served by useful proxies that could mature into more refined
measures of innovation.
In this report we focus on the need for NCSES further to use the BRDIS data to answer
such key policy questions as: How influential is R&D for innovation and growth (in
manufacturing compared with services)? At times of economic crisis, what is the contribution of
scientific research and innovation to economic recovery? How important are different kinds of
entities for advancing national innovation: small firms, young firms, high-growth firms,
government labs and procurement activities, and nonprofit organizations? How does innovation
activity in a given firm at a given place contribute to that firm’s productivity, employment and
growth, and perhaps also to these characteristics in the surrounding area?
NEEDED DATA
To answer these questions, four improvements seem feasible for NCSES in the near term,
using BRDIS data: statistics that are more comparable with those of other OECD countries;
better information on what firms characterize as innovation; data on high-growth firms and
“gazelles;” and data on the relationship of new firms to innovation.
Comparable Statistics
Understanding the nation’s position on innovation would be greatly helped if NCSES
developed statistics from its BRDIS data with the same cutoffs as those from other OECD
countries. In addition to the number of innovative firms in various industries that have 5 or more
employees, it would be helpful to have similar statistics for firms with 10 or more employees and
firms with 20 or more employees. These data will show the size dependency of innovation
activities.
Also needed for comparability are statistics using the same set of industries typically used
in statistics for other countries. An example is the core set of industries used by Eurostat for
comparison of innovation statistics among European Union countries, including: mining;
manufacturing; and selected service industries, such as wholesale trade, transports, financial
services, information technology (IT) services, R&D services, and business services. These data
could be used to compile a simple indicator of the share of product-process innovative firms,
defined as firms that have implemented a product or process innovation.
NCSES is developing another R&D and innovation survey on firms with fewer than five
employees: MIST, Microbusiness, Innovation Science, and Technology. The survey is designed
to yield data measuring the incidence of R&D activities among small businesses. NCSES has an
interagency agreement with the Internal Revenue Service (IRS) to use IRS data for the sampling
frame, for data quality review and to supplement the MIST data. Potential questions in MIST
will query firms on: R&D and innovation funding; employment and owner characteristics; sales
of new or significantly improved goods and services; technology transfer and knowledge
diffusion; sources of technical knowledge; and measures of firms’ entrepreneurial effectiveness.
22
OCR for page 23
The MIST pilot is planned for 2012. Those data, together with BRDIS data for companies with
5+, 10+, and 20+ employees, will provide useful information on the size dependency of
innovation (among other things).4
Understanding “Innovation”
Even with more comparable statistics on innovation, however, it will still not be clear to
users that firms are representing the same or very similar things when they report product or
process innovations. The survey questions are not transparent enough to provide a full
understanding of what the resulting data and statistics mean. For example, users have no
independent measure of whether the innovations of firms that innovate but do not do R&D are
more or less important than those that do R&D. Users would have more confidence in and
understanding of BRDIS innovation measures if they knew that knowledge input measures
correlated with actual performance and even more confidence if they knew what some of the
firms were calling innovation—how closely their reports matched the standards in the question.
To obtain this information, NCSES could commission a study of a subset of firms to determine
what they are measuring as innovation. It would be useful to have firms of different sizes,
different sectors and different geographic locations represented in such a study.
Innovation and Firm Size
The data in BRDIS could be used to begin developing statistics on high-growth firms and
“gazelles.” The Manual on Business Demography (Organisation for Economic Co-operation and
Development-Eurostat 2008, Chapter 2) defines high-growth enterprises as “All enterprises with
average annualised growth greater than 20% per annum, over a three year period…. Growth can
be measured by the number of employees or by turnover…. A size threshold has been suggested
as 10 employees at the beginning of the growth period.” Gazelles are the subset of high-growth
enterprises that are up to 5 years old. These thresholds are arbitrary and only based on
convention. NCSES could conduct its own sensitivity analysis to fine-tune the definitions of
high-growth firms and gazelles.5 Ditte Rude Petersen and Nadim Ahmad show a technique of
conducting this type of analysis in “High-Growth Enterprises and Gazelles—Preliminary and
Summary Sensitivity Analysis (Organisation for Economic Co-operation and Development,
2007).
During the panel’s workshop, several speakers (Howard Alper, University of Ottawa;
Robert Atkinson, Information Technology and Innovation Foundation; John Haltiwanger,
University of Maryland; Hugo Hollanders, U.N. University’s Maastricht Economic and Social
Research Institute on Innovation and Technology [UNU-MERIT]; and Brian MacAulay,
National Endowment for Science, Technology, and the Arts) mentioned the importance of
tracking trends of sustainability of jobs in these types of firms during economic downturns (even
if total employment is small). Atkinson said there was an interest in having these data at the
national, state, and at finer geographical levels.
It would also be useful to have those data to determine over time whether high-growth
firms or gazelles in particular have a higher incidence of innovation activity. The connection
4
Clearly, other data, together with more thorough regression analysis, would be necessary to determine causal
relationships.
5See Petersen and Ahmad (2007) for a technique of conducting this type of analysis.
23
OCR for page 24
between high-growth firms and innovation is complex. These data would enable researchers
better to determine the relationship between the presence of high-growth firms and innovation.
At the June 2011 roundtable workshop lead by Andrew Reamer (George Washington University)
on innovation measures, B.K. Atrostic, Cheryl Grim, and Javier Miranda from the Center for
Economic Studies at the U.S. Census Bureau suggested the development of a database of
business dynamics statistics, which would provide information about births and deaths of firms,
as well as the distribution of growth rates of gazelles and other types of high-growth firms. The
suggested database would have information on firm size, age, location, and sector. This
information would allow examination of the connection between innovative industries’ job
creation and high-growth firms in the context of economies at various geographical scales.
In his presentation, at the panel’s July 2011 workshop, Hollanders showed that high-
growth firms are significantly more innovative than other firms in his dataset. The statistics on
high-growth firms and gazelles could, therefore, be used to answer the following question: Are
these the types of firms that drive economic and job growth? A simple table could show high-
growth firms and other firms that are and are not innovative to compare economic
characteristics, ideally over time.
At the September 2011 panel meeting in Washington, DC, representatives from the
Bureau of Labor Statistics (BLS), the U.S. Census Bureau, and the Bureau of Economic Analysis
(BEA) mentioned that linking certain datasets among them would yield reasonable numbers on
gazelles. Such a table could be added to the SEI or become the foundation of an InfoBrief. The
following indicators could be produced by starting with the BRDIS data on high-growth firms
and gazelles: rate of high-growth enterprises (number of high-growth enterprises as a percentage
of the total population of active enterprises with at least n-number of employees); rate of gazelles
among newly born enterprises (number of gazelles as a percentage of all active enterprises with
at least n-number of employees that were born 4 or 5 years ago). These indicators would be
comparable to those produced in several other countries, thus increasing users’ understanding of
the comparative position of the United States on an aspect of the country’s innovative capacity.
Clearly, getting publishable statistics on high-growth firms and gazelles is a multistage
task that will require data acquisition and linking in addition to the data available in BRDIS. A
good first step would be for NCSES to explore the matchup of their BRDIS data with data on
firm dynamics from BLS. The panel’s final report will address the complexities of data linking,
particularly with the view to international comparability of the resulting statistics.
Relationship of New Firms and Innovation
New indicators that allow users to determine to what extent births of new firms are
driving innovation would also be useful. At the panel’s workshop, John Haltiwanger presented a
compelling presentation on what he and other researchers have developed using Census Bureau
data on firm dynamics—that is, firm births and deaths over time. These data can provide the
groundwork to answer important questions from policy makers and researchers, such as: Does
innovation come disproportionately from new firms? Do start-ups and young businesses create
net new jobs and increase productivity growth?
NCSES has a unique set of data in BRDIS, which, when combined with other datasets,
can be instrumental in answering these and other important questions. Integrating firm dynamism
(and the related employment effect) will take time and resources. During his presentation,
24
OCR for page 25
Haltiwanger described three Census Bureau datasets that, together with BRDIS data, would
allow NCSES to develop its business dynamics indicators:
Longitudinal Business Database—tracks all establishments and firms with at least one
employee including startups from 1976 to present;
Integrated Longitudinal Business Database—tracks all nonemployer firms and
integrated with employer firms from 1994 to present; and
Longitudinal Employer-Household Dynamics—tracks longitudinally all employer-
employee matches and transitions (hires, separations, job creation, and job
destruction) from 1990 to present.6
Questions from the Census Bureau’s 2007 and 2012 Economic Census, Company Organization
Survey, and Management and Organizational Practices Survey will also yield useful information
on R&D and other innovation activities for establishments. Infrastructure datasets can also track
relationships between start-up and young high-growth firms and large, mature firms, and they
can be linked further to patent and citation data. It is important as well to link the data on firm
dynamics to those on innovation outputs, such as patent and citation data.
In the near term, NCSES could begin to match up its BRDIS data to datasets at the
Census and BLS to create indicators of firm dynamism. Haltiwanger proposed that indicators
track dynamics by geography, industry, business size, and business age. Hollanders noted that
European countries and other OECD members are continuing to fine-tune their measures of firm
dynamism. NCSES’s indicators on this dimension could further the international comparability
of its STI indicators. Beginning to build the foundations on firm dynamics using BRDIS and
other datasets would give NCSES a productive platform for developing several STI indicators
that are policy relevant.
There are many other avenues that NCSES could take to developing a complete suite of
indicators on innovation. In the panel’s final report, we will offer specific recommendations on
how NCSES could develop statistics on expenditures on intangible assets and life lengths (as
suggested by Haskel). For many of these measures, research is still needed to determine proper
weighting methods and deflators and how to structure questionnaires (or modules to be placed on
existing questionnaires).
RECOMMENDATIONS
RECOMMENDATION 2: The National Center for Science and Engineering
Statistics should develop new indicators on innovation, based on data from its
Business Research and Development and Innovation Survey (BRDIS). The agency
should develop comparative statistics with the same cutoffs used by countries in the
Organisation for Economic Co-operation and Development for its BRDIS data.
RECOMMENDATION 3: The National Center for Science and Engineering
Statistics should begin to match its Business Research and Development and
Innovation Survey data to data from ongoing surveys at the U.S. Census Bureau
and the Bureau of Labor Statistics to create indicators of firm dynamism. This is a
6
These data would also enable NCSES to create indicators on job mobility (see further discussion below).
25
OCR for page 26
necessary first step for developing data linkages that yield measures of activities by
high-growth firms, and on births and deaths of businesses linked to innovation
outputs. These measures should be established by geographic and industry sectors
and by business size and business age. Such measures would be an important step in
furthering international comparability on innovation indicators. NCSES should
conduct its own sensitivity analysis to fine tune meaningful age categories of high-
growth firms.
26
