Advisers to the Nation on Science, Engineering, and Medicine

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.

The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.

www.national-academies.org

This page intentionally left blank.

This report contains the proceedings of a one-day workshop organized by the National Research Council’s Board on Science, Technology, and Economic Policy (STEP), in conjunction with a study by a panel of the NRC Committee on National Statistics (CNSTAT). This combined activity was commissioned by the Science Resources Statistics Division (SRS) of the National Science Foundation (NSF) to recommend improvements in the Foundation’s portfolio of surveys of research and development spending by the federal government, state governments, private industry, the nation’s universities and colleges, and other nonprofit institutions

The purpose of the workshop was to inform the CNSTAT panel and the Foundation about the adequacy of national and regional R&D data from the perspective of evaluating economic performance and formulating public policies to enhance it. Not designed to review the entire portfolio of R&D surveys, the workshop examined several areas where specific concerns have been expressed Thus, after first considering some users’ data needs and recent changes in the surveys, the agenda primarily focused on the distribution of R&D spending among research fields and industry sectors and across geographic regions, the nature of R&D activity in the service sector, and the extent of cross-national and collaborative R&D investment by large firms. The CNSTAT panel’s recommendations on the broader topic of NSF’s portfolio of R&D data surveys are contained in the report, Measuring Research and Development Expenditures in the U.S. Economy (National Research Council, 2004).

The workshop was the third in a series of meetings convened by the STEP Board to assess national indicators and data on innovation processes encompassing but not limited to formal research and development. The first meeting, sponsored by the National Science Foundation, was held in February 1997 and was intended to generate suggestions for improving the measurement of industrial research and innovation activity. It resulted in the report, Industrial Research and Innovation Indicators (National Research Council, 1997). The second meeting, sponsored by the National Aeronautics and Space Administration, National Institutes of Health, and Department of Energy, was held in November 1999 to consider how data on scientists, engineers, and other professionals--their training, employment and mobility, structure of work and affiliations, and productivity–could be used to illuminate trends in industrial innovation. It resulted in the report, Using Human Resource Data to Track Innovation (National Research Council, 2002).

The STEP program has not been concerned solely with private sector innovation. It has also examined the United States’ public and private investment in research to generate new scientific and technical knowledge and the relationship of that research investment to the training of scientists and engineers at the post-graduate level. In a 2001 report, Trends in Federal Support of Research and Graduate Education (National Research Council, 2001) a STEP committee examined the 1993-2000 trends in research spending in 23 disciplines and the corresponding trends in graduate enrollment (by source of financial support) in those disciplines, primarily to determine the magnitude of the shift in spending and enrollments away from many physical science and engineering fields in the aftermath of the Cold War. This analysis relied primarily on NSF’s Survey of Federal Funds for Research and Development but it also considered data from the Survey of Research and Development Expenditures at Universities and

Colleges, the Survey of Industrial Research and Development, the one-time (FY 1995) Survey of State Research and Development Expenditures, and the Survey of R&D Expenditures by Nonprofit Institutions as well as the Foundation’s surveys of science and engineering personnel.

Research and development are embedded in two broader intersecting processes, both of which are important to economic growth, productivity advances, and employment creation as well as to noneconomic aspects of national welfare–physical security, environmental preservation, intellectual understanding, and cultural creativity. The first process is technological innovation, which refers to the invention, commercialization, and diffusion of new products, processes, and services. Innovation is an important determinant of aggregate productivity and economic growth, countries’ comparative advantages, and firms’ competitiveness vis-à-vis their rivals. The second process is creation and diffusion of new knowledge, which may or may not have near-term market-based applications but which represents a pool of information from which many different kinds of users draw. To understand these processes, conceptual and data needs extend well beyond current national R&D statistics, whose focus is narrowly on how much money is spent by whom on the performance of what types of research and development in which institutions.

Consider first industrial innovation. Although theoretical understanding of the process is not far advanced, it is possible to elaborate a simple conceptual framework that encompasses the aspects that it is important to measure if we are to have a full picture of the status of national innovation. Four broad categories of innovation information are (1) innovation inputs such as the level and direction of the research effort but also activities that occur outside of formal R&D laboratories, for example, changes in management practices or on the manufacturing line; (2) determinants of innovative activity such as the means and effectiveness of appropriating returns from innovation, market structure and conditions, the nature and extent of knowledge flows among firms and other institutions, technological opportunities, etc., (3) measures of innovative output – newly commercialized products, processes, and services and their characteristics and diffusion; and (4) outcomes or the impact of innovation on firms, regions, workers, and the economy as a whole. On some of these important variables hardly any data are currently collected on a national basis and regular schedule for major sectors of the economy; on other variables information is fragmentary.

• Systematically collected data are generally lacking on new product, process, and service introductions and their use and diffusion, although some intermediate outputs, including patents and their citations, are well documented.

• Apart from formal R&D, we have little idea how much innovative activity goes unmeasured.

• There are large gaps in data on the many channels (e.g., meetings, personal communications, technical publications, patents, etc.) through which knowledge moves within firms, between customers and suppliers, and across competing firms, along with other determinants of innovative activity.

• Macro-level data on GDP and productivity growth are readily available but micro-level data on firm performance and worker characteristics likely to be influenced by innovation are not.

These deficiencies are a principal rationale for the efforts – extensive, publicly supported, and internationally coordinated in Europe, highly detailed in Canada and Australia, but much more modest in the United States – to mount “innovation surveys,” independent of R&D reporting, to fill some of these gaps. It was part of the CNSTAT panel’s mandate to evaluate

these efforts and make recommendations to NSF on conducting research and surveys on innovation (NRC, 2004, Chapter 4).

For scientific and engineering knowledge creation the principal data sources are surveys of the training and deployment of scientists and engineers, publications of science and engineering research results, and formal and informal channels apart from publications by which new knowledge is communicated – licenses, meetings, consulting arrangements, personal communications, etc. The quality and robustness of the data collected and reported by NSF are greatest for the first two categories – training and publications -- at least for the individuals and journals with recognized standing in science and engineering (e.g., individuals with degrees from accredited institutions and journals with professional society affiliations). Data are more limited for contractual relationships such as licenses and consulting and poorest for informal channels of communication.

Research and development merit close attention because they are critical inputs to both innovation and knowledge creation and diffusion. Moreover, because of budgeting, accounting, and tax conventions, data on R&D expenditures are easier to collect than are data directly measuring innovation and knowledge creation. For these reasons, R&D expenditures are often treated as surrogates for knowledge creation and innovation effort. In addition, some administrative data¹ exist on industrial R&D collaborations among companies and between companies and federal laboratories.

Notwithstanding the extensive portfolio of R&D expenditure data, from time to time several concerns about their adequacy and comprehensiveness have been articulated. Some of the concerns are longstanding while others derive from recent perceptions that the process of innovation and characteristics of R&D are changing. These concerns motivated the Foundation’s request. They relate to

• the comprehensiveness of data on some payers and performers of R&D (e.g., state and local government and nonprofit institutions that are surveyed irregularly);

• the classification of R&D by type (basic research, applied research and development), by industry, sector or activity, and by field of science and engineering as well as consistency of taxonomies across surveys;

• the geographical location of R&D, both internationally and by locality within the United States;

• Characteristics of R&D performance (locus within the institution or firm, time horizon, etc.);

• inconsistencies in what are ostensibly the same data reported by different sources (e.g., differences between federal spending in universities as reported by federal agencies and performing institutions);

• timeliness of R&D data availability; and

• Survey methodology (e.g., sample construction, imputation, etc.).

This workshop addressed selected issues in the first four categories.

Reflected in all three of the workshops organized under STEP’s auspices is a perception that these and other issues of R&D data collection have become more important because of the likelihood that the pattern, determinants, and effects of innovation are changing, sometimes remarkably quickly and radically and with potentially significant consequences for the nation’s economic performance. These perceived changes include shifts in the sectoral, industry, and

technology distribution of innovative activity, shifts in the time horizon of innovative effort and investment, changes in the organizational structure of innovative activity, and changes in the location of activity both within the United States and globally.

Officially reported national data should not be expected to capture all of the R&D trends of interest, let alone tell us all that we want to know about knowledge creation and innovation processes. Throughout the workshop there were descriptions of private data collection efforts of great utility, some of them of them repeated periodically, others limited to a single project. These are essential complements to NSF’s continuing survey time series and often necessary to illuminate discontinuities and emerging changes.

The workshop was planned by a diverse committee including experts in statistics, economics, policy analysis, and policymaking. Their institutional experience is also wide-ranging. Two are veterans of a federal statistical agency, two of congressional staffs, one of a federal R&D agency, and one of the White House staff. State government, international business, and international organizations were also represented, and two panel members have spent their careers primarily in academic research.

The meeting was organized in seven parts. In the first session several government officials described their R&D data needs for national economic accounting and program planning and evaluation purposes. An official of the National Science Foundation’s Office of Science Resources Statistics followed with a detailed description of recent changes in the R&D survey data portfolio – some of them responsive to new needs and criticisms, others made for reasons of practicality and reliability. There were two discussions of the composition of R&D, one focused on the field distribution of public sector basic and applied research and the other focused on the distribution of business R&D activity. The adequacy of data on R&D collaborations across firms was discussed. Finally, there were two sessions on locational issues – one addressing cross-national R&D investment and the other examining the domestic distribution of R&D activity.

The views expressed in the STEP workshop were, of course, those of the individual speakers and do not necessarily represent positions of the National Academies. Nevertheless, information and insights from the presentations were incorporated in the CNSTAT panel report along with information from its own two-day workshop and many other sources.

Bronwyn H. Hall

Stephen A. Merrill

This volume has been reviewed in draft form by individuals chosen for their technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for quality. The review comments and draft manuscript remain confidential to protect the integrity of the process.

We wish to thank the following individuals for their review of the preface: Lawrence Brown, University of Pennsylvania; Wesley Cohen, Duke University; and Thomas Plewes, consultant. Although the reviewers provided constructive comments and suggestions, they were not asked to endorse the content of the preface. Responsibility for the final content of this proceedings rests with the editors and the individual workshop presenters.

We also thank Marisa Gerstein, Tanya Lee, and Lee Pollack of the staff of the NRC’s Committee on National Statistics for their initial editing of the workshop transcript.

This page intentionally left blank.

This page intentionally left blank.