Capturing Change in Science,
Technology, and Innovation
IMPROVING INDICATORS TO INFORM POLICY
Panel on Developing Science, Technology, and Innovation Indicators for the Future
Robert E. Litan, Andrew W. Wyckoff, and Kaye Husbands Fealing, Editors
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
and
Board on Science, Technology, and Economic Policy
Division of Policy and Global Affairs
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
THE NATIONAL ACADEMIES PRESS • 500 Fifth Street, NW • Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
The project that is the subject of this report was supported by grant number SES-0453930 between the National Science Foundation and the National Academy of Sciences. Support for the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (grant number SES-1024012). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-29744-8
International Standard Book Number-10: 0-309-29744-3
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2014 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Cover credit: Image of U.S. Capitol from Architect of the Capitol. All other images from iStock cover images. Cover designed by Jacqui Sovde of the National Research Council.
Suggested citation: National Research Council. (2014). Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Panel on Developing Science, Technology, and Innovation Indicators for the Future, R.E. Litan, A.W. Wyckoff, and K.H. Fealing, Editors. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Board on Science, Technology, and Economic Policy, Division of Policy and Global Affairs. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
PANEL ON DEVELOPING SCIENCE, TECHNOLOGY, AND INNOVATION INDICATORS FOR THE FUTURE
ROBERT E. LITAN (Cochair), Bloomberg Government, Washington, DC
ANDREW W. WYCKOFF (Cochair), OECD, Paris, France
CARTER BLOCH, Danish Center for Studies in Research and Research Policy, University of Aarhus, Denmark
NICHOLAS R. CHRISMAN, Geospatial Science, RMIT University, Melbourne, Australia
CARL J. DAHLMAN, Edmund A. Walsh School of Foreign Service, Georgetown University, Washington, DC
GEOFF M. DAVIS, Quantitative Insights Group, Google, Inc., Mountain View, CA
KATHARINE G. FRASE, Industries Research, IBM, Yorktown Heights, NY
BARBARA M. FRAUMENI, Muskie School of Public Service, University of Southern Maine
RICHARD B. FREEMAN, Department of Economics, Harvard University
FREDERICK D. GAULT, Economic and Social Research and Training Centre on Innovation and Technology, United Nations University, Maastricht, The Netherlands
DAVID GOLDSTON, Government Affairs, Natural Resources Defense Council, Washington, DC
MICHAEL MANDEL, Wharton School, University of Pennsylvania
JOHN E. ROLPH, Marshall School of Business, University of Southern California
LELAND WILKINSON, Data Visualization, Skytree, Inc., San Jose, CA
KAYE HUSBANDS FEALING, Study Director
ESHA SINHA, Associate Program Officer
ANTHONY S. MANN, Program Coordinator
COMMITTEE ON NATIONAL STATISTICS 2013-2014
LAWRENCE D. BROWN (Chair), Department of Statistics, Wharton School, University of Pennsylvania
JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University
MARY ELLEN BOCK, Department of Statistics, Purdue University
DAVID CARD, Department of Economics, University of California, Berkeley
ALICIA CARRIQUIRY, Department of Statistics, Iowa State University
MICHAEL E. CHERNEW, Department of Health Care Policy, Harvard Medical School
CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University
JAMES S. HOUSE, Survey Research Center, Institute for Social Research, University of Michigan
MICHAEL HOUT, Department of Sociology, New York University
SALLIE ANN KELLER, Virginia Bioinformatics Institute, Virginia Polytechnic Institute and State University
LISA LYNCH, The Heller School for Social Policy and Management, Brandeis University
COLM A. O’MUIRCHEARTAIGH, Harris Graduate School of Public Policy Studies, University of Chicago
RUTH D. PETERSON, Department of Sociology (emerita) and Criminal Justice Research Center, Ohio State University
EDWARD H. SHORTLIFFE, Departments of Biomedical Informatics, Columbia University and Arizona State University
HAL S. STERN, Donald Bren School of Information and Computer Sciences, University of California, Irvine
CONSTANCE F. CITRO, Director
JACQUELINE R. SOVDE, Program Associate
BOARD ON SCIENCE, TECHNOLOGY, AND ECONOMIC POLICY
PAUL L. JOSKOW (Chair), Alfred P. Sloan Foundation, New York
ERNST R. BERNDT, Sloan School of Management, Massachusetts Institute of Technology
JEFF BINGAMAN, U.S. Senate (former senator), New Mexico
ELLEN R. DULBERGER, Ellen Dulberger Enterprises, LLC
ALAN M. GARBER, Office of the Provost, Harvard University
RALPH E. GOMORY, Stern School of Business, New York University
JOHN L. HENNESSY, Office of the President, Stanford University
WILLIAM H. JANEWAY, Warburg Pincus, LLC, New York
RICHARD K. LESTER, Department of Nuclear Science and Engineering, Massachusetts Institute of Technology
DAVID T. MORGENTHALER, Morgenthaler Ventures, Palo Alto, CA
LUIS M. PROENZA, Office of the President, University of Akron
KATHRYN L. SHAW, Graduate School of Business, Stanford University
LAURA D’ANDREA TYSON, Haas School of Business, University of California, Berkeley
HAL R. VARIAN, Google, Inc., Mountain View, CA
ALAN WM. WOLFF, McKenna, Long, & Aldridge, LLP, Washington, DC
RALPH J. CICERONE, Ex Officio, President, National Academy of Sciences, Washington, DC
VICTOR J. DZAU, Ex Officio, President, Institute of Medicine, Washington, DC
C. D. MOTE, Jr., Ex Officio, President, National Academy of Engineering, Washington, DC
RICHARD E. BISSELL, Acting Director
This page intentionally left blank.
Preface
Indicators are a scaffolding of statistics to which decision makers can relate other elements needed to make decisions. Indicators often are used to tell an end-to-end story on a policy-relevant topic. Science, technology, and innovation (STI) indicators are at a transition point in the formation of mainstream policy, especially in the current economic environment, in which a premium is placed on sustainable growth (e.g., growth that is not dependent on speculative bubbles). The United States has always played a leadership role internationally in the development of standards for statistical measurement of STI activities. This report is intended to offer the National Center for Science and Engineering Statistics (NCSES) at the U.S. National Science Foundation guidance that will help keep it at the forefront of this endeavor.
The primary audience for this report is the sponsor—NCSES—as well as similar statistical agencies that produce data and statistics on innovative activities worldwide. The report assesses the demand for STI indicators from different perspectives: national, international, subnational, and sectoral. Although STI indicators often are retrospective—measuring stocks, flows, and networks within the system—some bellwethers can show prospective trends in science, technology, engineering, and mathematics (STEM) talent; new areas of scientific exploration; potentially vibrant regions of innovation activity; and arcs toward the international rise or decline of countries based on their technological capabilities. STI indicators, therefore, can be used to report macro-level STI activities, outputs, outcomes, and linkages, as well as micro-level metrics related to actors and intermediate outputs in the system. New indicators can be developed by layering, linking, or blending existing data and indicators.
The users of NCSES data and statistics also are expected to be an important audience for this report. The user community for NCSES’s STI indicators is diverse and includes the National Science Board; staff at other statistical agencies in the United States and abroad; local, state, federal, and international policy makers and government administrators; academic researchers and administrators; and business managers and consultants. Users want indicators that relay timely information about drivers, trends, advances, vulnerabilities, culture/climate, and distributions related to the STI system. Users also are eager to explore nontraditional sources of data from which indicators are derived and want to be informed about the reliability of those data. Lastly, qualitative or descriptive information should accompany quantitative measures to explain the economic tenets, social norms, regulatory constructs, and political atmosphere with which the STI system engages. Qualitative information and even case studies allow for deeper insights into not only what happened but also why.
The user community relies on STI indicators for answering key questions regarding the global science and technology enterprise, including: What are direct measures of innovation, and what drives innovation? Where is leadership in science and engineering knowledge generation trending? What is the status of STEM talent around the world? What is the portfolio of spending and other support by governments and private firms and organizations for STI activities, particularly those at universities? What institutions, networks, and regulations facilitate or impede advances in science, technology, and entrepreneurship? What is the trend in online learning in the United States and abroad, and what impact will that have on university finance, operations and recruitment? What are important subnational collaborative activities that promote innovation and economic growth? What are the general perceptions about science and the public value of science in the general population in the United States and abroad? Dozens more such questions are enumerated in this report.
Given its broad disciplinary scope, the study was conducted by a panel of experts that collectively represent more than a dozen fields, including computer science, economics, education, engineering, finance, geography, mathematics,
physics, political science, psychology, statistics, and visual analytics. The panel also reflects the international nature of the topic, with members from Canada, Denmark, France, and the Netherlands.
In undertaking this study, the panel first relied on users, experts, and written reports and peer-reviewed articles to establish current and anticipated user needs for STI indicators. Second, the panel recognized that no one model informs the types of indicators NCSES needs to produce. Policy questions served as an important guide to the panel’s review, but the study was also informed by systems approaches and international comparability. Third, it was important to identify data resources and tools NCSES could exploit to develop its indicators program. Understanding the network of inputs—including data from NCSES surveys, other federal agencies, international organizations, and the private sector—that can be tapped in the production of indicators gave rise to a set of recommendations for working with other federal agencies and public and private organizations. Fourth, the panel did not limit its recommendations to
indicators but also addressed processes for prioritizing data development and the production of indicators in the future, because it was clear that the changing environment in which NCSES operates is a key determinant of the agency’s priorities from year to year. Internal processes that are observant, networked, and statistically and analytically balanced are important for NCSES’s indicators program.
On request of the sponsor, an interim report was published in February 2012, summarizing the panel’s early findings and recommendations. The recommendations offered in this report expand on those of the interim report. They are intended to serve as the basis for a strategic program of work that will enhance NCSES’s ability to produce indicators that capture change in STI to inform policy and optimally meet the needs of its user community.
Robert E. Litan and Andrew W. Wyckoff, Cochairs
Panel on Developing Science, Technology,
and Innovation Indicators for the Future
Acknowledgments*
It is with extreme gratitude that the panel thanks the many people who made contributions to this study on science, technology, and innovation (STI) indicators. The staff of the National Center for Science and Engineering Statistics (NCSES) at the National Science Foundation, under the directorship of John Gawalt and formerly Lynda Carlson, gracefully provided invaluable input and insights, including clear direction on what they wanted to learn from the study, as well as useful sources of information from their division and other resources. Robert Bell, Lawrence Burton, John Jankowski, Nirmala Kannankutty, Beethika Khan, Rolf Lehming, Francisco Moris, Jeri Mulrow, Christopher Pece, and Emilda Rivers all contributed their knowledge and expertise to answer our questions.
The panel’s work benefited greatly from presenters and attendees at our open meetings. The insights of the following individuals were critical for the framing of policy issues that are relevant to this study: Jeff Alexander (SRI International), Patrick Clemins (formerly of the American Association for the Advancement of Science), Mark Doms (U.S. Department of Commerce), Matthew Gerdin (U.S. State Department), Kei Koizumi (U.S. Office of Science and Technology Policy), Christine Matthews (Congressional Research Service), Amber Hartman Scholz (President’s Council of Advisors on Science and Technology), Dahlia Sokolov (U.S. House of Representatives), and D. Greg Tassey (formerly of the National Institute of Standards and Technology). Conceptual frameworks for STI indicators were presented by Michelle Alexopoulos (University of Toronto), Bronwyn Hall (University of California, Berkeley), and Adam Jaffe (Brandeis University).
Opportunities for advances in STI data collections and statistics, particularly among U.S. federal agencies, were discussed with panel members by B.K. Atrostic, Cheryl Grim, Richard Hough, Dave Kinyon, Erika McEntarfer, and Mary Potter (U.S. Census Bureau); Ana Aizcorbe, Maria Borga, and Carol Robbins (Bureau of Economic Analysis); Laurie Salmon, Jim Spletzer, and David Talan (Bureau of Labor Statistics); David McGranahan and Tim Wojan (U.S. Department of Agriculture); Daniel McGrath, Jessica Shedd, Matthew Soldner, and Tom Weko (National Center for Education Statistics); Stuart Graham (U.S. Patent and Trademark Office); and George Chacko and Walter Schaffer (National Institutes of Health). We also thank Rochelle (Shelly) Martinez and her colleagues and Katherine Wallman at the Office of Management and Budget for an engaging discussion regarding synergies in the federal statistical system with respect to measures of STI activities.
Because international comparability is an important aspect of this study, the panel convened two workshops of international researchers and practitioners who use STI indicators. The first workshop, in July 2011, covered STI measures and described opportunities and obstacles that NCSES should anticipate as it further develops its STI indicators program. We thank the workshop presenters: Shinichi Akaike (Hitotsubashi University, Tokyo), Howard Alper (Canada’s Science Technology and Innovation Council), Jayanta Chatterjee (Indian Institute of Technology), Gustavo Crespi (Inter-American Development Bank), Matthieu Delescluse (European Commission), Changlin Gao (Chinese Academy of Science and Technology), Jonathan Haskel (Imperial College Business School), Hugo Hollanders (United Nations University-Maastricht Economic and Social Research Institute on Innovation and Technology), Brian MacAulay (National Endowment for Science, Technology and the Arts, London), and Philippe Mawoko (The New Partnership for Africa’s Development).
The second workshop, in June 2012, included participants from the OECD-National Experts on Science and Technology Indicators (NESTI) representing nations, economic regions, and international institutions, including the African Union, Australia, Austria, Belgium, Brazil, Chile, the Czech Republic, Estonia, the European Union, Germany, Hungary, India, Israel, Italy, Japan, Korea, La Red
____________________
*All listed affiliations are as of February 2014.
Iberoamericana e Interamericana de Indicadores de Ciencia y Tecnología (RICYT), Luxembourg, the Netherlands, Norway, Poland, Portugal, Russia, the Slovak Republic, South Africa, Spain, Sweden, Switzerland, Turkey, the United Kingdom, and the United States. The panel chair, study director, and panel member Fred Gault, who is also chair of the NESTI advisory board, discussed the interim report from the panel and solicited comments from attendees, primarily on which indicators would be most useful to policy makers in their nations or regions. We thank the presenters, who discussed prioritization methods and specific indicators that are important for international comparisons: Alessandra Colecchia, Dominique Guellec, and Joaquim Oliveria Martins (OECD); Matthieu Delescluse (European Commission); Almamy Konté (African Union); Leonid Gokhberg (National Research University, Russia); and Veijo Ritola (Eurostat).
During the course of its work, the panel also obtained input from several other science and technology policy experts, including Aaron Chatterji (Duke University, formerly of the Council of Economic Advisers); Bhavya Lal and Stephanie Shipp (Institute for Defense Analyses–Science and Technology Policy Institute); Donna Ginther (University of Kansas); and Alessandra Colecchia, Gili Greenberg, and Fernando Galindo-Rueda (OECD). The panel explored the use of microdata, particularly administrative records and web tools, to create STI statistics. We heard from several experts in this diverse field of study at the July 2011 workshop, including Carl Bergstrom (University of Washington), Stefano Bertuzzi (National Institutes of Health and the STAR METRICS Program), Erik Brynjolfsson (Massachusetts Institute of Technology), Lee Giles (Pennsylvania State University), John Haltiwanger (University of Maryland), Richard Price (Academia.edu), and Alicia Robb (Kauffman Foundation).
The development of STI indicators at subnational levels was also an important topic for this study. At the July 2011 workshop, Rob Atkinson (Information Technology and Innovation Foundation), Maryann Feldman (University of North Carolina), Andrew Reamer (George Washington University), and Robert Samors and David Winwood (Association of Public and Land-grant Universities) presented options for measuring STI activities at a variety of geographic scales.
Nicholas Donofrio (IBM) participated in a roundtable discussion with panel members during the workshop. We greatly appreciate his insights from a business perspective on measuring research and development and innovation. His comments reminded us that the role of multinational corporations in the global STI system should be examined carefully and that entrepreneurial activities at firms of various sizes deserve careful measure.
Because the National Science Board (NSB) is a primary user of NCSES’s STI indicators—the biennial Science and Engineering Indicators volumes are published by NSB—the panel conducted two rounds of interviews with board members. We thank Ray Bowen, Kelvin K. Droegemeier, José-Marie Griffiths, Arthur Reilly, and Arnold F. Stancell for in-depth and insightful responses to our questions, as well as the board’s staff members Jennie Moehlmann, Michael Van Woert, and Matthew Wilson, for facilitating the meetings.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study’s charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Henry Brady, Office of the Dean, Goldman School of Public Policy, University of California, Berkeley; Michael Conlon, Clinical and Translation Science Institute, University of Florida; Martin Fleming, chief economist and vice president, Business Performance Services, International Business Machines Corporation; Jacques S. Gansler, Center for Public Policy and Private Enterprise, School of Public Policy, University of Maryland; Christopher T. Hill, Emeritus Department of Public Policy and Technology, School of Public Policy, George Mason University; Graham G. Kalton, Westat, Inc., Rockville, Maryland; Jason Owen-Smith, Barger Leadership Institute, University of Michigan; Georgine M. Pion, Department of Psychology and Human Development, Vanderbilt University; Hal Salzman, Bloustein School of Planning and Public Policy and Heldrich Center for Workforce Development, Rutgers University; Phillip Swagel, School of Public Policy, University of Maryland; Albert H. Teich, director, Science and Policy Programs, American Association for the Advancement of Science; and Ward Ziarko, Scientific and Technical Information Service, Belgian Federal Science Policy.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the report’s conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Chuck Phelps, provost emeritus, University of Rochester, and John Haltiwanger, Department of Economics, University of Maryland. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring panel and the institution.
We extend special thanks to staff of the Committee on National Statistics. The study director, Kaye Husbands Fealing, provided invaluable assistance to the panel in orga-
nizing the meetings and preparing this report. Connie Citro, Tom Plewes, and Michael Cohen gave excellent guidance to the panel. Stephen Merrill, director of the Board on Science, Technology, and Economic Policy, contributed to panel meetings. Esha Sinha assembled and developed a public use databank of STI indicators from several international sources (see http://sites.nationalacademies.org/DBASSE/CNSTAT/Science_Technology_and_Innovation_Indicators/index.htm [May 2014]. In collaboration with panel member Leland Wilkinson, she also conducted the cluster analysis and heat map exercises for this study and cowrote the accompanying data appendix (Appendix F). Anthony Mann provided outstanding administrative and logistical support to the panel. Our Mirzayan fellow, Daniel Grady, drew on his expertise in systems dynamics and web tools to benefit the panel’s work.
Robert E. Litan and Andrew W. Wyckoff, Cochairs
Panel on Developing Science, Technology,
and Innovation Indicators for the Future
This page intentionally left blank.
Acronyms, Abbreviations, and Descriptions
ACRONYMS AND ABBREVIATIONS
America COMPETES Act |
America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science |
APLU |
Association of Public and Land-grant Universities |
BEA |
Bureau of Economic Analysis, U.S. Department of Commerce |
BERD |
business enterprise expenditure on research and development |
BLS |
Bureau of Labor Statistics, U.S. Department of Labor |
BOP |
balance of payments |
BRDIS |
Business Research and Development and Innovation Survey |
CICEP |
Commission on Innovation, Competitiveness, and Economic Prosperity |
CIS |
Community Innovation Survey |
ERS |
Economic Research Service, U.S. Department of Agriculture |
ETA |
Employment and Training Administration, U.S. Department of Labor |
EU |
European Union |
FTE |
full-time equivalent |
GBAORD |
government budget appropriations or outlays for research and development |
GDP |
gross domestic product |
GERD |
gross domestic expenditure on research and development |
GO→SPIN |
Global Observatory on Science, Technology and Innovation Policy Instruments |
GOVERD |
government intramural expenditure on research and development |
HERD |
higher education expenditure on research and development or Higher Education Research and Development Survey |
ICT |
information and communication technology |
IMF |
International Monetary Fund |
IPEDS |
Integrated Postsecondary Education Data System |
MEP |
Manufacturing Extension Partnership |
MIST |
Microbusiness, Innovation Science, and Technology |
MOOC |
massive open online course |
NAICS |
North American Industry Classification System |
NCES |
National Center for Education Statistics, U.S. Department of Education |
NCSES |
National Center for Science and Engineering Statistics, U.S. National Science Foundation |
NEPAD |
New Partnership for Africa’s Development |
NESTI |
National Experts on Science and Technology Indicators |
NIH |
National Institutes of Health, U.S. Department of Health and Human Services |
NIST |
National Institute of Standards and Technology, U.S. Department of Commerce |
NLM |
National Library of Medicine, U.S. Department of Health and Human Services |
NSCG |
National Survey of College Graduates |
NSF |
U.S. National Science Foundation |
NSRCG |
National Survey of Recent College Graduates |
OECD |
Organisation for Economic Co-operation and Development |
OMB |
U.S. Office of Management and Budget |
OSTP |
U.S. Office of Science and Technology Policy |
PISA |
Programme for International Student Assessment |
R&D |
research and development |
S&E |
science and engineering |
S&T |
science and technology |
SBIR |
Small Business Innovation Research |
SDR |
Survey of Doctorate Recipients |
SED |
Survey of Earned Doctorates |
SEI |
Science and Engineering Indicators |
SESTAT |
Science and Engineering Statistical Data System |
SIBS |
Survey of Innovation and Business Strategy |
STAR METRICS |
Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science |
STEM |
science, technology, engineering, and mathematics |
STI |
science, technology, and innovation |
STTR |
Small Business Technology Transfer |
UNESCO |
United Nations Educational, Scientific and Cultural Organization |
UNU-MERIT |
United Nations University’s Maastricht Economic and Social Research Institute on Innovation and Technology |
USDA |
U.S. Department of Agriculture |
DESCRIPTIONS
BERD: Research and development expenditure in the business enterprise sector in a given year at the regional level.
GBAORD: Government budget appropriations or outlays for research and development, a way of measuring government support for research and development activities.
GERD: Gross domestic expenditure on research and development, defined as total intramural expenditure on research and development performed on the national territory during a given period.
GO→SPIN: Global Observatory on Science, Technology and Innovation Policy Instruments. Contains not only indicators but also an inventory of science, technology, and innovation (STI) national system descriptions; STI legal frameworks (with access to text of the acts and decrees); and an inventory of operational policy instruments that promote STI activities in a country. Developed by a group at the United Nations Educational, Scientific and Cultural Organization (UNESCO) Regional Bureau for Science for Latin America and the Caribbean.
HERD: Research and development expenditure in the higher education sector in a given year at the national and different subnational geographic scales.
Human capital: The ability, knowledge, and skill base that are typically acquired or enhanced by an individual through education and training.
Innovation: The implementation of a new or significantly improved product (good or service) or process; a new marketing method; or a new organizational method in business practices, workplace organization, or external relations (OECD-Eurostat, 2005, p. 46). A common feature of an innovation is that it must have been implemented. A new or improved product is implemented when it is introduced on the market. New processes, marketing methods, or organizational methods are implemented when they are brought into actual use in a firm’s operations (OECD-Eurostat, 2005, p. 47).
NLSY: National Longitudinal Survey of Youth, conducted by the Bureau of Labor Statistics.
Research and development (R&D): Comprises creative work undertaken on a systematic basis to increase the stock of knowledge, including knowledge of man, culture, and society, and the use of this stock of knowledge to devise new applications (OECD, 2002a, p. 30).
Science and technology (S&T): A broad concept that includes science and technology activities, defined as follows: “For statistical purposes, Scientific and Technological Activities (STA) can be defined as all systematic activities which are closely concerned with the generation, advancement, dissemination, and application of scientific and technical knowledge in all fields of science and technology, that is the natural sciences, engineering and technology, the medical and the agricultural sciences (NS), as well as the social sciences and humanities (SSH)” (United Nations Educational, Scientific and Cultural Organization, 1984, p. 17). Also included are Scientific and Technological Services (STS) and Scientific and Technological Education and Training (STET), the definitions of which are found in UNESCO (1978). Research and development is included in STA. The OECD Frascati Manual (OECD, 2002a, p. 19) notes that “R&D (defined similarly by UNESCO and the OECD) is thus to be distinguished from both STET and STS.” In the Frascati definition, R&D includes basic research, applied research, and experimental development.
SIBS: Survey of Innovation and Business Strategy, conducted by Statistics Canada.
Statistic: A numerical fact or datum, especially one computed from a sample.
Statistical data: Data from a survey or administrative source used to produce statistics (OECD, 2002b, pp. 205-230).
Statistical indicator: A statistic, or combination of statistics, providing information on some aspect of the state of a system or of its change over time. For example, gross domestic product (GDP) provides information on the level of value added in the economy, and its change over time is an indicator of the economic state of the nation. The decline of GDP for two quarters is indicative of a recession. The ratio of gross domestic expenditure on research and development (GERD) to GDP is an indicator of the formal generation of new knowledge and is used both for international comparisons and for the setting of targets, such as the Lisbon target for the European Union of 3 percent. There are also composite indicators involving many component indicators. A single indicator is indicative but not definitive in its description of the system. As an example, GDP per capita provides one piece of information about an economy and may be indicative of wealth or productivity, but the income distribution for the country, another indicator summarized in a Gini coefficient, provides complementary information on income inequality. Employment is yet another indicator of the state of the economy.
Statistical information: Statistical data, or a statistic, placed in a context. As an example, the number of people making less than a dollar a day in a country is a statistic populated by statistical data that may result from estimation based on a sample. The context is the analysis of poverty, and in that context, the statistic provides information on poverty, but it is only one dimension.
This page intentionally left blank.
Contents
2 CONCEPTS AND USES OF INDICATORS
Desirable Attributes of Indicators
Cautions, Possibilities, and Limitations
3 DATA RESOURCES FOR INDICATORS
Gaps in STI Indicators That NCSES Should Fill
Policy Relevance of Innovation Measures
The Role of Innovation Surveys
Use of Nontraditional Methodologies
5 MEASURING THE THREE K’S: KNOWLEDGE GENERATION, KNOWLEDGE NETWORKS, AND KNOWLEDGE FLOWS
Measuring Science and Technology: Major Gaps in International Comparability
Traditional Indicators of the Three K’s
Business R&D: R&D Services and Intangible Assets
NCSES’s Existing Human Capital Indicators
Potential for New Data Sources
Revised and New Human Capital Indicators
7 A PARADIGM SHIFT IN DATA COLLECTION AND ANALYSIS
Indicators from Frontier Tools: Example of the Data Science Discipline
Annex 7-1: Potential Data Sources to Explore
8 INFORMING THE STRATEGIC PLANNING PROCESS
Panel Recommendations: Chapters 2-7
A Biographical Sketches of Panel Members and Staff
D OECD-National Experts on Science and Technology Indicators (NESTI) Workshop and Attendees
E National Center for Science and Engineering Statistics Research Abstracts 2012
F Science, Technology, and Innovation Databases and Heat Map Analysis