Measuring Research and Development Expenditures IN THE U.S. ECONOMY
Lawrence D. Brown, Thomas J. Plewes, and Marisa A. Gerstein, Editors
THE NATIONAL ACADEMIES PRESS
THE NATIONAL ACADEMIES PRESS
500 FIFTH STREET, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract No. SRS-0244598 between the National Academy of Sciences and the National Science Foundation. Support of the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (Number SBR-0112521). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
Library of Congress Cataloging-in-Publication Data
Measuring research and development expenditures in the U.S. economy / editors, Lawrence D. Brown, Thomas J. Plewes, Marisa A. Gerstein.
Panel to Review Research and Development Statistics at the National Science Foundation. The panel was convened in January 2002 by the Committee on National Statistics of the National Academies at the request of the National Science Foundation to conduct an indepth and broad-based study of the Research and Development Statistics program of the NSF Science Resources Statistics (SRS) Division. The study was later mandated by Congress in the National Science Foundation Authorization Act of 2002.
Includes bibliographical references.
ISBN 0-309-09320-1 (pbk.) — ISBN 0-309-54595-1 (pdf)
1. Scientific surveys—United States. 2. Science and industry—United States. 3. Research, Industrial—United States. 4. Research—Economic aspects—United States. 5. Research, Industrial—Economic aspects—United States. 6. Science and industry—Government policy—United States. 7. Research, Industrial—Government policy—United States. I. Brown, Lawrence D. II. Plewes, Thomas J. III. Gerstein, Marisa A. IV. National Research Council (U.S.). Panel to Review Research and Development Statistics at the National Science Foundation.
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Printed in the United States of America.
Copyright 2005 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council. (2005). Measuring Research and Development Expenditures in the U.S. Economy. Panel on Research and Development Statistics at the National Science Foundation, Lawrence D. Brown, Thomas J. Plewes, and Marisa A. Gerstein, Editors. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
PANEL ON RESEARCH AND DEVELOPMENT STATISTICS AT THE NATIONAL SCIENCE FOUNDATION
LAWRENCE D. BROWN (Chair),
Department of Statistics, The Wharton School, University of Pennsylvania
JOHN L. ADAMS,
RAND Corporation, Santa Monica, California
WESLEY M. COHEN,
Fuqua School of Business, Duke University
Science, Innovation, and Electronic Information Division, Statistics Canada, Ottawa
Jimmy Carter Library and Museum, Atlanta
BRONWYN H. HALL,
Department of Economics, University of California, Berkeley
CHRISTOPHER T. HILL,
Public Policy and Technology, George Mason University, Fairfax, VA
Department of Economics and Social Science, Carnegie Mellon University
Department of Finance and Department of Entrepreneurial Management, Harvard Business School
Stern School of Business, New York University
Department of Mathematics and Statistics, Oakland University, Rochester, MI
NORA CATE SCHAEFFER,
Department of Sociology, University of Wisconsin, Madison
Joint Program in Statistical Methodology, University of Michigan
THOMAS J. PLEWES, Study Director
MARISA A. GERSTEIN, Research Assistant
TANYA M. LEE, Program Assistant
BARBARA A. BAILAR, Consultant
MICHAEL McGEARY, Consultant
WILLIAM F. LONG, Consultant
COMMITTEE ON NATIONAL STATISTICS 2003-2004
JOHN E. ROLPH (Chair),
Marshall School of Business, University of Southern California
JOSEPH G. ALTONJI,
Institute for Policy Research and Department of Economics, Northwestern University
AT&T Research Laboratories, Florham Park, NJ
LAWRENCE D. BROWN,
Department of Statistics, The Wharton School, University of Pennsylvania
Department of Statistics, Carnegie Mellon University
ROBERT M. GROVES,
Survey Research Center, University of Michigan
Department of Economics, University of Maryland
Educational Testing Service, Princeton, NJ
JOEL L. HOROWITZ,
Department of Economics, Northwestern University
Survey Research Unit, Department of Biostatistics, University of North Carolina
School of Public Policy and Social Research, University of California, Los Angeles
Department of Statistics, Department of Industrial and Operations Engineering, University of Michigan
Google, New York, NY
Russell Sage Foundation, New York, NY
NORA CATE SCHAEFFER,
Department of Sociology, University of Wisconsin, Madison
CONSTANCE F. CITRO, Director
The nation’s commitment to research and development (R&D) has a history that goes back to the earliest days of the United States; some date that commitment to the patent clause in the Constitution, which set the stage for an interplay between the government and the scientific community that continues to this day. Scientific discoveries, technical applications, and, more recently, federal R&D programs have been integral to the nation’s development. The role of R&D in the U.S. economy has been characterized as essential to national prosperity. The statement “Research and development widely recognized as being key to economic growth” begins the discussion of trends in R&D expenditures in the National Science Board’s report, Science and Engineering Indicators—2002 (p. 4). The report goes on to suggest that R&D decision making is critical to the future of the U.S. economy and national well-being.
Recognizing the importance of research and development to national prosperity, the United States and other countries have developed systems that conceptualize, define, and measure research and development activities so as to understand and to quantify the influence of R&D on the economy. It is claimed that the level of expenditures and the composition of those expenditures in R&D may be considered a proxy measure of national and sectoral commitment to economic growth and development, and that expenditures are an indication of the “perceived economic importance of R&D relative to all other economic activities” (National Science Board, 2002:4).
Despite this high level of attention to the role of R&D in the economy, the U.S. government devotes little of its treasury to the measurement of
R&D. If the relative spending of the federal government on the measurement of R&D expenditures can be viewed as a measure of its commitment to R&D, that commitment is problematic. In fiscal year 2003, the federal government spent approximately $4.2 million on the National Science Foundation programs to measure R&D expenditures, a minuscule part of the nearly $4.7 billion national investment in statistical programs.
The fast-changing environment for research and development severely impacts the R&D Statistics Program managed by the Science Resources Statistics Division (SRS) at the National Science Foundation (NSF). The NSF mission includes being a source of information for policy formation by other agencies of the federal government and providing data and analysis on a broad policy area for public- and private-sector constituents. SRS products also inform congressional policy makers and NSF itself.
As a major organization in the National Science Foundation, SRS has a requirement for a 5-year review of its programs by an NSF Committee of Visitors.
The review requirement has been partially met in the recent past by such activities as a U.S. Census Bureau advisory committee that recently reviewed the statistical foundation of the R&D industry survey; an academic R&D advisory panel that contributed guidance on collection of information on R&D activities in colleges and universities; and a series of workshops, such as a federal agency R&D administrator workshop in 1998, that have been periodically convened to examine certain aspects of program operations. The R&D surveys have also been reviewed in several recent studies by the National Research Council (2000, 2001c) and also by both the U.S. General Accounting Office (2001) and the Congressional Research Service (1999).
These reviews identified problems with specific R&D surveys, with inconsistencies in data among surveys, and with the inability of the current portfolio to address important policy questions and research issues generated by changes in how R&D is carried out in the United States today. Examples of these problems include high nonresponse rates to items in the industrial R&D survey; discrepancies of several billion dollars between the amount of money that federal agencies reported as R&D support and the amount that the performers of the R&D work reported spending; inconsistent classification of R&D by field across surveys; and the failure of existing surveys to capture interdisciplinary research and collaborations among firms, between research institutions, and across sectors.
In view of the special role of SRS, NSF leadership decided that a review of the R&D statistics programs by a panel of the National Research Council’s Committee on National Statistics (CNSTAT) would meet the internal requirement for a periodic program review. A review of the entire portfolio of R&D data collection activities of SRS is timely to identify its
strengths and weaknesses and to recommend ways to improve data quality and relevance. In addition to fulfilling the requirement for a periodic outside evaluation, the NSF leadership sponsored this review as a means of developing a comprehensive, integrated review of concepts, definitions, survey methodology, and dissemination.
As result of this interest, the Committee on National Statistics convened the Panel on Research and Development at the National Science Foundation in early 2003 to conduct an in-depth and broad-based study to look at the Research and Development Statistics Program of the Science Resources Statistics Division. The goal of the panel has been to look at how R&D surveys are currently conducted and how they should be conducted to capture the country’s R&D activities over the coming decade.
The CNSTAT committee has conducted its work in cooperation with a separately appointed panel of the Board on Science, Technology, and Economic Policy (STEP). The STEP panel focused on the issues of composition, structure, sourcing, and location, particularly in the context of the industrial R&D and federal funds surveys, covering the majority of U.S. funding and performance. The STEP panel’s workshop, held in April 2003, informed the deliberations of this panel.
Our panel represents expertise in the fields of survey methodology, data analysis, statistics, economics, research analysis, and the organization and conduct of R&D in the public and private sectors. In selecting the members of the panel, the National Research Council sought to include representation from data users in the fields of science and technology policy and the analysis of technological change, as well as data providers from industry and academia, the two major sources of raw data. Because many of the practices and procedures for providing data depend on tax law, panel expertise in the valuation of intangible assets and the treatment of R&D in the tax code was also considered essential.
In conducting this review, the panel examined existing R&D data collection systems and relevant literature, commissioned appropriate papers, identified gaps in current methodology, and held a workshop on R&D measurement methodology. The panel held five meetings beginning in January 2003, as well as a workshop on July 24-25, 2003. The papers the panel commissioned address specific issues. In focusing on issues of statistical accuracy and reliability, the panel had the benefit of advice and analysis from two meetings with staff of NSF’s Division of Science Resources Statistics and the public and private organizations directly responsible for data collection and compilation. In addition, subject-matter experts from the panel met to explore issues of statistical methodology and cognitive aspects of data collection.
The panel issued an interim report in March 2004. This interim report presented the panel’s findings and conclusions regarding the present array
of surveys on matters of statistical accuracy and reliability, as well as interim recommendations on near-term improvements that should be considered and could be implemented by NSF in developing plans and making resource decisions for the next several years. The interim report’s major findings and recommendations have been fully incorporated into this report.
The interim report found that significant progress has been made by the Science Resources Statistics Division in fostering an environment for improvement of data quality. The panel expressed hope that these recent initiatives, buttressed by additional resources and supplemented by further initiatives such as those outlined in this report, will lay a basis for further improvements in the future.
The interim report focused on four basic methodological issues: web-based collection, the practice of providing prior-year data to survey respondents, the designation of respondents, and nonresponse adjustment and imputation.
The very newness of web-based collection in the academic and federal government surveys suggests both opportunities and challenges. Web-based collection can afford efficiencies and economies and promises to improve such functions as editing and imputation. However, in reviewing how web-based collection is now implemented, the panel raised several cautions, suggesting that additional research is needed on such issues as self-imputation that is forced by demanding a data entry in each cell.
The panel considered the practice of providing data collected in prior-year responses to respondents. Although we have some confidence that the practice does not generate significant errors, the panel urged NSF to sponsor research on the effect of imprinting prior-period data on the industrial R&D survey in conjunction with testing the introduction of web-based data collection.
The interim report found that the industrial R&D survey gives limited attention to interaction with survey respondents, beyond the largest reporters, and then only when questions or issues of nonresponse are encountered. The panel supported the initiative to identify individual respondents in companies as a first and necessary step toward developing an educational interaction with respondents, so as to improve response rates and the quality of responses. The panel also strongly recommended that NSF and the U.S. Census Bureau resume a program of field observation staff visits to a sampling of reporters to examine record-keeping practices and conduct research on how respondents fill out the forms.
The R&D surveys have very different approaches to the treatment of nonresponse and the imputation of missing values. The panel recommended that NSF revise its Statistical Guidelines for Surveys and Publications to set standards for treatment of unit nonresponse and to direct the computation of response rates for each item, prior to sample weighting.
The interim report also looked at and made recommendations in regard to each of the surveys. The major findings of the interim report are reflected in the discussions of each of these surveys in this final report.
The Panel on Research and Development Statistics at the National Science Foundation gratefully acknowledges the contributions of many individuals and organizations which supported our activities over the 2-year study period. Not all of those who gave so freely of their time, energy, and knowledge can be adequately acknowledged. As we conducted our work, it was obvious that the topic of measurement of research and development is of intense interest to a wide-ranging community of government, academic, private, and private nonprofit individuals and organizations. Without their assistance and encouragement, we could not have completed this work.
Our appreciation begins with the sponsor of the study, the Science Resources Statistics Division of the National Science Foundation. The director of this division, Lynda Carlson, was instrumental in identifying the need for the study and assisting the panel in framing the scope and approach to the study. Her insightful comments on the occasion of the first meeting of the panel presaged a very successful undertaking. She ensured that appropriate members of her staff were available to assist the panel as we approached our task, and invariably recognized the independence of the panel.
Special recognition should be given to John Jankowski, program director, Research and Development Statistics Program, who was patient, supportive, and invariably helpful in fielding the many requests for information about the programs from the panel members and staff. His key survey managers and support staff, including Rich Bennof, Leslie Christovich, Mage Machen, Ron Meeks, Francisco Moris, Brandon Schackleford, and Ray Wolfe, provided support and assistance beyond the expected. Other members of the staff of the SRS division likewise contributed their time and talents to this product. The division’s assistant director, Mary Frase, as well as Fran Featherstone, Ron Fesco, Rolf Lemming, and Jeri Mulrow, assisted in laying out issues of statistical methodology and data quality. The constant interest and support of Norman Bradburn, the head of NSF’s Directorate for Social, Behavioral, and Economic Sciences, was absolutely essential to the success of the enterprise. Although we recognize the crucial assistance of all the members of the NSF staff, we want to emphasize that the deliberations and recommendations of the panel are the panel’s own.
Almost uniquely among federal statistical agencies, SRS relies on the services of the U.S. Census Bureau, as well as private contractors, especially QRC Macro Corporation, to perform many key functions in the surveys, including data collection. The panel, in turn, relied on the knowledgeable staff of these organizations to assist in understanding the survey operations. Staff of these organizations gave presentations at both of the panel’s open
meetings and participated fully in two “mini-meetings” involving members of the panel who focused on specific aspects of survey operations. The Census Bureau staff who were invariably helpful included William Bostic, Stacey Cole, Paul Hsen, Kimberly Moore, John Slanta, and Julius Smith. Adding strength and perspective to the discussion of cognitive issues in survey design at the mini-meeting on cognitive issues was Don Dillman of Washington State University.
The panel thanks QRC Macro staff members Susan Akin, Dan McMaster, Mark Morgan, and Michael Rossi, and Jim Smith of WESTAT, who freely assisted us in understanding the strengths and limitations of their operations.
The panel is also indebted to many others who made presentations on data and methodological issues, assisting us to focus on all aspects of the surveys, from concepts to uses. In addition to Lynda Carlson and John Jankowski, who participated in the initial meeting of the panel, the panel benefited from the work of Michael Bordt, Statistics Canada; Donna Fossum, RAND Corporation; Barbara Fraumeni, Bureau of Economic Analysis; Michael Gallaher, Research Triangle Institute; August Goetzfried, EUROSTAT; David Goldston, House Science Committee; Tomorhio Ijichi, University of Tokyo; Anita K. Jones, University of Virginia; Key Koizumi, American Association for the Advancement of Science; Anna Larsson, EUROSTAT; Gregory Tassey, National Institute of Science and Technology; and John Walsh, University of Tokyo.
The study director conducted many interviews with several other prominent users, providers, and producers of data on research and development. In addition to those who gave presentations at the open meetings of the panel, the panel recognizes the contributions of David Appler, Federal Laboratory Consortium for Technology Transfer; William Bonvillian, Office of Senator Lieberman; Rick Cheston, Government Accountability Office; Michael Davey, Congressional Research Service; Floyd DesChamps, Senate Committee on Commerce, Science, and Technology; Mark Herbst, Office of the Secretary of Defense; Ken LaSala, Senate Committee on Commerce, Science, and Technology; Chan Lieu, Senate Committee on Commerce, Science, and Technology; Charles Ludlam, Office of Senator Lieberman; Bob Palmer, House Committee on Science; David Radzanowski, U.S. Office of Management and Budget; Maurice Swinton, Small Business Administration; Jean Tol-Eisen, Senate Committee on Commerce, Science, and Technology; and David Trinkle, U.S. Office of Management and Budget.
The panel wishes to thank the senior staff of the General Motors Research and Development Center, which hosted members of the panel at its facilities in October 2003 for a wide-ranging discussion of the company’s organization for research and development and its use of federal R&D