8

Key Themes and Possible Next Steps

Scott Stern and Bronwyn Hall closed the workshop with comments on what they saw as some of the main points presented and discussed. Stern noted three broad questions were addressed, in some cases directly and in others indirectly:

1. What can be done to facilitate productive updating of the Oslo Manual, which defines innovation and prescribes an approach to its measurement, going forward?
2. How should metrics be developed and modernized to capture the components of innovation—including its inputs, outputs, and dynamics over time—more accurately and more comprehensively?
3. How are new (or revised) measures and datasets changing the understanding of innovation?

Stern noted that an objective of the workshop organizing committee was to envision how the range of measures could potentially be improved and expanded to more fully capture, on a reliable basis, aspects of innovation processes that are currently missed or not completely measured. This goal requires first taking stock of the work already being done by the National Center for Science and Engineering Statistics (NCSES) and other statistical agencies. The value of their contributions, made possible by significant public investments, improve the understanding of innovation and advance its measurement and was clearly evident at many steps

along the way during the proceedings. While highlighting the value for new kinds of data and new analytic constructs, Stern credited workshop participants for recognizing that the current apparatus—for example, the Community Innovation Survey (CIS) and the Business R&D and Innovation Survey (BRDIS)—contains invaluable tools that benefit from a balance between standardization (across countries and time) and experimentation (to uncover new phenomena and local circumstance). Stern made the point that planners should be explicit about what parts of a survey and resultant data products require standardization (because the capacity to make direct comparisons across countries and time has value) and what parts could be opened up to experimentation.

While progress in the field was acknowledged, several workshop participants sought to identify what Ben Martin (Chapter 2) termed the “dark matter”—aspects of innovation that are not measured and, consequently, not well understood. Over the course of the presentations, Stern observed, it became clear that shedding light on this dark matter will entail individual and organizational initiatives on domains outside the current scope of the Oslo Manual and the CIS upon which it is based. In some cases, the comparative advantage for advancing measurement will be in collaboration with the statistical agencies, while in other cases, researchers will need to take the lead.

Regarding revision of the Oslo Manual, Stern pointed out that the definition of innovation matters. Fred Gault (Chapter 2) commented that the definition is central because it is the anchor determining what is in scope and what is out of scope. Thus, to some extent, a narrow definition will confine inquiry and a broad definition will stimulate wider inquiry.

A major theme that emerged from this workshop—reiterated in the presentation by Hall—was that, to be useful, any framework must be capable of reflecting the complex and diverse processes and interactions inherent in innovation:

Ideally, Hall argued, a measurement framework would specify a system of indicators that, taken together, are helpful in monitoring these processes and informing policies designed to foster, facilitate, and enhance them. The conceptual framework drives (implicitly or explicitly) data collection and data organization, as well as what measurements based on those data will be illuminating. Subsequently, when measurements are used in research, a feedback mechanism is created that further shapes the framework and helps to identify data and indicator gaps that, if filled, could help answer key questions about the pace, location, and nature of innovation activity (Hall and Jaffe, 2012).

Though the idea of a single unified indicator framework has allure, it is not possible if the goal is to promote research on the many ways that innovation is fostered and impacts the economy and society. This point was raised in the presentations during the workshop session on “regional innovation models and data needs.” At subnational levels of analysis, it becomes especially clear that multiple approaches are needed to serve the many kinds of users of innovation (and science and technology) indicators and to address the wide range of policy questions posed. No one set or system of indicators will suit all monitoring, benchmarking, evaluating, and forecasting purposes—a unified framework is probably not a realistic goal. The range of objectives across stakeholders represented at the workshop dictates that multiple approaches may be considered when designing data collection and indicator development programs. For example, as noted above, since some potential benefits of innovation—such as better health, greater happiness, and improved environment—are not market based, other concepts of output (and sets of indicators to capture their trends) are also needed.

Accounts of innovation are also increasingly capturing a broader range of sources and administrative units. As discussed by Owen-Smith, von Hippel, and Winston Smith, among others (see Chapters 5 and 6), innovation can be tracked at the industry and sectoral levels but, in principle, it can also be traced back to where it originates, which may be businesses, households, individual entrepreneurs, universities, nonprofit organizations, or the public sector (or some combination of these). The Oslo Manual was created largely for the production of national-level and industry-level innovation statistics, which are important. Yet several presenters (see Chapter 6) demonstrated that measurement can be applied with illuminating effect at more localized levels—levels that are often more relevant in terms of understanding the sources of innovation (such as the human capital and geographic clustering of activities) and their impact on economic and social outcomes.

Additionally, measurement frameworks should reflect linkages in innovation processes so that flows of inputs, outcomes, and broader eco-

nomic and societal impacts can be tracked. Charles Edquist (Chapter 5) noted the importance of distinguishing innovations from their determinants. He viewed entrepreneurship and research, for example, as drivers of innovation. He said these determinants need to be measured, as do the innovations themselves and the impact they have, in a detailed and coordinated way.

Regarding modernizing innovation measurement and metrics, Edquist reiterated that the scope of data collection must continue to broaden beyond R&D and patents. Stern noted the modal notion is that R&D is undertaken in a company, which then leverages the knowledge generated to produce one or more innovations—suggesting an alignment between R&D as the input with the output being new or significantly changed goods and services produced by the company. This perception may have been sufficiently accurate at the time when a number of key surveys were designed, but the interaction of different players is now more complex. To portray innovation fully, data systems should be capable of detecting knowledge linkages between people and companies—and business and household interactions generally—over time (as noted by Javier Miranda, Chapter 3), and the division of labor, which means that people engaged in idea development are often different than those commercializing the innovations (Wesley Cohen and Ashish Arora, Chapter 3). In addition, Stern concluded that the role of “nontraditional” but prominent aspects of value creation, such as design (Bruce Tether, Chapter 3), digitization, and robotics (Rob Seamans, Chapter 3) is necessary. These complexities are only compounded by the changing innovation landscape that includes ever-increasing internationalization, globalization, networking, and co-invention. Similarly, Robbins (2016, p. 11) in the background paper prepared for the workshop noted, “the increasingly open nature of innovation, including the role of suppliers, customers, licensed technology, and collaborations and joint projects, suggest that the activities of firms need to be understood in the context of their relationships to other institutions, including other firms, universities, non-profits, and government entities.”

In her summary comments, Hall noted that a comprehensive portrayal requires acknowledging that innovators (and not just the innovations they produce) matter: that is, a recognition that knowledge flows via individuals. Innovation is undertaken by a person or teams of persons, which makes maintaining individual identifiers in datasets crucial to tracking these linkages and for anticipating where action may occur. This point was made by Eric von Hippel (Chapter 6), Sallie Keller (Chapter 7), and others: People are innovators. Scientists, engineers, and designers, as well as people in the broader community, can be identified as engaging in a distinctive human trait, which is making tools, using tools, problem solving, and developing novel creations. Winston Smith commented that

this notion of tracking innovators as well as innovations is important at both the level of measurement (e.g., suggesting a need for data on the migration of science, technology, engineering, and mathematics graduates to firms) and at the level of the questions asked.

Ben Martin (Chapter 2) reinforced discussion to move beyond traditional indicators. In particular, he argued that indicators based on patent data are often only distant proxies for the true measurement objective. Jeff Oldham (Chapter 7) also demonstrated that patent data sources could be used more effectively, by linking them to nonpatent data, to shed light on the impact of patented ideas on sales, quality of life, educational effectiveness, and other areas. Several participants argued it was time to broaden sectoral coverage of innovation. Services and experiences increasingly dominate value-added and employment in the United States and other advanced economies, they pointed out, but there is still a manufacturing orientation in many measures of innovation. Martin noted, for example, that if more had been known about the characteristics of innovation in the financial sector, policy makers and researchers would have been better placed to anticipate the events leading to the 2007 crisis. This also highlights that not all innovation is socially desirable.

Several presentations highlighted that comprehensive conceptualization and measurement of innovation also requires recognizing activities (and outcomes) that take place in households (Eric von Hippel, Chapter 5), in the public and nonprofit sectors (Kay Husbands Fealing, Chapter 5), and in communities (all presenters in Chapter 6). Stern remarked that a goal for the future should be to better capture—in both survey-based and administrative datasets as well as the broader programs of the National Center for Science and Engineering Statistics, OECD, and others—the scope of innovation in a way that integrates leading-edge research findings (which breaks new ground) into the statistical machinery (which facilitates consensus and common understanding).

Including the public sector in “official” innovation measurement raises an important issue (which also applies to some other sectors), which is the dearth of data on outcomes. Hall pointed out that measurement of outcomes for public-sector innovation requires thinking about how survey questions should be asked. The wording of survey instruments has to change not only for the public sector and not only for the innovation question, but also possibly for other questions, she noted. Measurement of the innovation and public goods link (which may take place in the public or private sector) requires connecting indicators of the inputs (mainly dollars and human capital) all the way through to social outcomes in the ecosystems in which they take place. As von Hippel noted (Chapter 5), much of the current focus is on economic outcomes or tangible outputs, which leads to an overemphasis on patents and other readily available

data; many of the useful innovations that bring high utility for the public good are not “patented ideas.”

The development of innovation measurement will, to a great degree, be driven by the availability of new kinds of data—where, as Stern phrased it, the administrative data revolution meets the digitization revolution. The administrative data revolution discussed by Javier Miranda (Chapter 3) reflects at least two developments. One is an increasing capacity at the statistical agencies and others to link different administrative and survey data sources; Stern cited the work by Chetty et al. (2014) combining IRS earnings records and other sources to examine determinants of intergenerational economic mobility as one high-profile example. Elsewhere, there are data on entrepreneurship, around health records, around studies of schools and teacher value added, and others. In addition, Stern observed, the digitization revolution allows researchers to uncover new ways to understand company activities, new products, and new services that reflect the process of innovation. Except for a few examples, these two developments have so far existed as separate tracks within the innovation community. Sallie Keller (Chapter 7) provided an overview of the role of nonsurvey data for statistical use, and of how statistical and data-gathering techniques are already being used to advance research interests.

As articulated by Robbins (2016), innovation-related data priorities for the statistical agencies can be informed by evaluating the needs of the data-using community against three factors: (1) data currently available from NCSES and other statistical agencies, (2) increasing opportunities provided by nonsurvey data, and (3) opportunities for linking existing datasets. Further, capabilities of the data-collecting agencies can be enhanced for these priorities by leveraging partnerships, collaborations, and supporting research. Several participants identified enhancing the ability to link data across sources as a prerequisite to analytic and measurement progress. One example among the many kinds of new data that will aid a greater understanding of innovation are those tracking angel funding and crowd funding, but currently, the data are still not sufficiently detailed and available to enable sustained high-quality research on new innovation funding.

Regarding what has been learned and continuing gaps, Stern observed that researchers are able to measure some aspects of innovation extremely well. However, he said, the share of what is being missed—the portion identified by Martin’s dark matter analogy—is still largely unknown, except perhaps in a highly aggregated sense as reflected in statistics on productivity (as described by Dan Sichel, Chapter 3). And productivity measurement is essentially a black box—a residual measure of what is not explained by changes in the levels of inputs of production. Even assessing the amount missed on the basis of productivity measures (without claim-

ing to identify the sources) is controversial, Stern noted, as it requires anchoring one’s belief that the input-output relationships are captured accurately, which is not universally accepted.

Stern described the following areas for future research identified by workshop participants as key to shedding light on some of the innovation “dark matter”:

• The division of innovative labor: Research by Cohen, Arora, and others is demonstrating that a surprisingly high fraction of important innovations come from outside the firm, albeit sometimes cooperatively, Stern said. The linkages among innovating individuals and organizations need to be better understood.
• Design, digitization, robotics, household users and entrepreneurship: Workshop discussants (specifically, Tether, Chapter 3) noted that aspects of innovation such as design could be systematized enough to be measured at the level of a “design index.” Rob Seamans’ presentation (Chapter 3) on innovations in the area of automation illustrated that there is not a single use dataset that currently captures job-creating and job-replacing phenomena adequately.
• Centrality of the regional systems approach: A considerable amount of new work is seeking to understand entrepreneurship and other innovative activities at the regional level. Practitioners and policy makers have become accustomed to the idea that innovation statistics are produced at the national level. In the 1990s, outside of a few state-level aggregates, science and engineering indicators were based at the level of countries. Stern observed that at least in a large country such as the United States, many processes can be better understood at the regional level. Presentations by Feldman, Fazio, Ziedonis, and Winston Smith (Chapter 6) made it clear that there is probably more to be learned at the subnational unit of analysis than at the national level, especially because both research communities and policy communities are often regional in character.

The last area that Stern noted as emerging from the workshop concerned the skewed distributions of discovery, invention, innovation, and entrepreneurship and the implications for data and research. Stern referred to an empirical phenomenon that a large percentage of profoundly influential innovative activity—for example, returns on initial public offerings or the impact of ideas on profits, products, and job creation—occurs from a small proportion of the distribution. This means that, for some measurement purposes, detailed statistics based on 1 percent of activity (if it is

the right 1%, or perhaps 2 or 3%) are very useful, while information on the full distribution of total number of papers or patents or research and development that a country or a region produced is inefficient in producing insights. At an aggregate level, he said, it may be more important to know if a country or a region is reliably producing innovations that appear in the top 0.5 percent of the distribution measured along dimension x than to know their totals. An important research priority for the near-term horizon, then, is to evaluate what statistics can be produced at the nth percentile as part of the information infrastructure. The Census Bureau’s Business Dynamics Statistics group has done pioneering work to reveal the analogous skewedness of job creation and destruction, firm survival, and other areas. Stern suggested that this work might provide a blueprint for further work on the topic. Just as focusing on the top 1 percent in the area of income changed the conversation of what people studied, focusing more centrally on these skewed distributions in areas related to innovation is important for advancing understanding of the relationship between innovation and its sources and outcomes.