National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

Capturing Change in Science,
Technology, and Innovation

IMPROVING INDICATORS TO INFORM POLICY

Panel on Developing Science, Technology, and Innovation Indicators for the Future

Robert E. Litan, Andrew W. Wyckoff, and Kaye Husbands Fealing, Editors


Committee on National Statistics
Division of Behavioral and Social Sciences and Education

and

Board on Science, Technology, and Economic Policy
Division of Policy and Global Affairs

NATIONAL RESEARCH COUNCIL
                              OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS

Washington, D.C.

www.nap.edu

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.

The project that is the subject of this report was supported by grant number SES-0453930 between the National Science Foundation and the National Academy of Sciences. Support for the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (grant number SES-1024012). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.

International Standard Book Number-13: 978-0-309-29744-8
International Standard Book Number-10: 0-309-29744-3

Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2014 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Cover credit: Image of U.S. Capitol from Architect of the Capitol. All other images from iStock cover images. Cover designed by Jacqui Sovde of the National Research Council.

Suggested citation: National Research Council. (2014). Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Panel on Developing Science, Technology, and Innovation Indicators for the Future, R.E. Litan, A.W. Wyckoff, and K.H. Fealing, Editors. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Board on Science, Technology, and Economic Policy, Division of Policy and Global Affairs. Washington, DC: The National Academies Press.

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.

The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine.

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.

www.national-academies.org

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

PANEL ON DEVELOPING SCIENCE, TECHNOLOGY, AND INNOVATION INDICATORS FOR THE FUTURE

ROBERT E. LITAN (Cochair), Bloomberg Government, Washington, DC

ANDREW W. WYCKOFF (Cochair), OECD, Paris, France

CARTER BLOCH, Danish Center for Studies in Research and Research Policy, University of Aarhus, Denmark

NICHOLAS R. CHRISMAN, Geospatial Science, RMIT University, Melbourne, Australia

CARL J. DAHLMAN, Edmund A. Walsh School of Foreign Service, Georgetown University, Washington, DC

GEOFF M. DAVIS, Quantitative Insights Group, Google, Inc., Mountain View, CA

KATHARINE G. FRASE, Industries Research, IBM, Yorktown Heights, NY

BARBARA M. FRAUMENI, Muskie School of Public Service, University of Southern Maine

RICHARD B. FREEMAN, Department of Economics, Harvard University

FREDERICK D. GAULT, Economic and Social Research and Training Centre on Innovation and Technology, United Nations University, Maastricht, The Netherlands

DAVID GOLDSTON, Government Affairs, Natural Resources Defense Council, Washington, DC

MICHAEL MANDEL, Wharton School, University of Pennsylvania

JOHN E. ROLPH, Marshall School of Business, University of Southern California

LELAND WILKINSON, Data Visualization, Skytree, Inc., San Jose, CA

KAYE HUSBANDS FEALING, Study Director

ESHA SINHA, Associate Program Officer

ANTHONY S. MANN, Program Coordinator

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

COMMITTEE ON NATIONAL STATISTICS 2013-2014

LAWRENCE D. BROWN (Chair), Department of Statistics, Wharton School, University of Pennsylvania

JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University

MARY ELLEN BOCK, Department of Statistics, Purdue University

DAVID CARD, Department of Economics, University of California, Berkeley

ALICIA CARRIQUIRY, Department of Statistics, Iowa State University

MICHAEL E. CHERNEW, Department of Health Care Policy, Harvard Medical School

CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University

JAMES S. HOUSE, Survey Research Center, Institute for Social Research, University of Michigan

MICHAEL HOUT, Department of Sociology, New York University

SALLIE ANN KELLER, Virginia Bioinformatics Institute, Virginia Polytechnic Institute and State University

LISA LYNCH, The Heller School for Social Policy and Management, Brandeis University

COLM A. O’MUIRCHEARTAIGH, Harris Graduate School of Public Policy Studies, University of Chicago

RUTH D. PETERSON, Department of Sociology (emerita) and Criminal Justice Research Center, Ohio State University

EDWARD H. SHORTLIFFE, Departments of Biomedical Informatics, Columbia University and Arizona State University

HAL S. STERN, Donald Bren School of Information and Computer Sciences, University of California, Irvine

CONSTANCE F. CITRO, Director

JACQUELINE R. SOVDE, Program Associate

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

BOARD ON SCIENCE, TECHNOLOGY, AND ECONOMIC POLICY

PAUL L. JOSKOW (Chair), Alfred P. Sloan Foundation, New York

ERNST R. BERNDT, Sloan School of Management, Massachusetts Institute of Technology

JEFF BINGAMAN, U.S. Senate (former senator), New Mexico

ELLEN R. DULBERGER, Ellen Dulberger Enterprises, LLC

ALAN M. GARBER, Office of the Provost, Harvard University

RALPH E. GOMORY, Stern School of Business, New York University

JOHN L. HENNESSY, Office of the President, Stanford University

WILLIAM H. JANEWAY, Warburg Pincus, LLC, New York

RICHARD K. LESTER, Department of Nuclear Science and Engineering, Massachusetts Institute of Technology

DAVID T. MORGENTHALER, Morgenthaler Ventures, Palo Alto, CA

LUIS M. PROENZA, Office of the President, University of Akron

KATHRYN L. SHAW, Graduate School of Business, Stanford University

LAURA D’ANDREA TYSON, Haas School of Business, University of California, Berkeley

HAL R. VARIAN, Google, Inc., Mountain View, CA

ALAN WM. WOLFF, McKenna, Long, & Aldridge, LLP, Washington, DC

RALPH J. CICERONE, Ex Officio, President, National Academy of Sciences, Washington, DC

VICTOR J. DZAU, Ex Officio, President, Institute of Medicine, Washington, DC

C. D. MOTE, Jr., Ex Officio, President, National Academy of Engineering, Washington, DC

RICHARD E. BISSELL, Acting Director

Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

Preface

Indicators are a scaffolding of statistics to which decision makers can relate other elements needed to make decisions. Indicators often are used to tell an end-to-end story on a policy-relevant topic. Science, technology, and innovation (STI) indicators are at a transition point in the formation of mainstream policy, especially in the current economic environment, in which a premium is placed on sustainable growth (e.g., growth that is not dependent on speculative bubbles). The United States has always played a leadership role internationally in the development of standards for statistical measurement of STI activities. This report is intended to offer the National Center for Science and Engineering Statistics (NCSES) at the U.S. National Science Foundation guidance that will help keep it at the forefront of this endeavor.

The primary audience for this report is the sponsor—NCSES—as well as similar statistical agencies that produce data and statistics on innovative activities worldwide. The report assesses the demand for STI indicators from different perspectives: national, international, subnational, and sectoral. Although STI indicators often are retrospective—measuring stocks, flows, and networks within the system—some bellwethers can show prospective trends in science, technology, engineering, and mathematics (STEM) talent; new areas of scientific exploration; potentially vibrant regions of innovation activity; and arcs toward the international rise or decline of countries based on their technological capabilities. STI indicators, therefore, can be used to report macro-level STI activities, outputs, outcomes, and linkages, as well as micro-level metrics related to actors and intermediate outputs in the system. New indicators can be developed by layering, linking, or blending existing data and indicators.

The users of NCSES data and statistics also are expected to be an important audience for this report. The user community for NCSES’s STI indicators is diverse and includes the National Science Board; staff at other statistical agencies in the United States and abroad; local, state, federal, and international policy makers and government administrators; academic researchers and administrators; and business managers and consultants. Users want indicators that relay timely information about drivers, trends, advances, vulnerabilities, culture/climate, and distributions related to the STI system. Users also are eager to explore nontraditional sources of data from which indicators are derived and want to be informed about the reliability of those data. Lastly, qualitative or descriptive information should accompany quantitative measures to explain the economic tenets, social norms, regulatory constructs, and political atmosphere with which the STI system engages. Qualitative information and even case studies allow for deeper insights into not only what happened but also why.

The user community relies on STI indicators for answering key questions regarding the global science and technology enterprise, including: What are direct measures of innovation, and what drives innovation? Where is leadership in science and engineering knowledge generation trending? What is the status of STEM talent around the world? What is the portfolio of spending and other support by governments and private firms and organizations for STI activities, particularly those at universities? What institutions, networks, and regulations facilitate or impede advances in science, technology, and entrepreneurship? What is the trend in online learning in the United States and abroad, and what impact will that have on university finance, operations and recruitment? What are important subnational collaborative activities that promote innovation and economic growth? What are the general perceptions about science and the public value of science in the general population in the United States and abroad? Dozens more such questions are enumerated in this report.

Given its broad disciplinary scope, the study was conducted by a panel of experts that collectively represent more than a dozen fields, including computer science, economics, education, engineering, finance, geography, mathematics,

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

physics, political science, psychology, statistics, and visual analytics. The panel also reflects the international nature of the topic, with members from Canada, Denmark, France, and the Netherlands.

In undertaking this study, the panel first relied on users, experts, and written reports and peer-reviewed articles to establish current and anticipated user needs for STI indicators. Second, the panel recognized that no one model informs the types of indicators NCSES needs to produce. Policy questions served as an important guide to the panel’s review, but the study was also informed by systems approaches and international comparability. Third, it was important to identify data resources and tools NCSES could exploit to develop its indicators program. Understanding the network of inputs—including data from NCSES surveys, other federal agencies, international organizations, and the private sector—that can be tapped in the production of indicators gave rise to a set of recommendations for working with other federal agencies and public and private organizations. Fourth, the panel did not limit its recommendations to

indicators but also addressed processes for prioritizing data development and the production of indicators in the future, because it was clear that the changing environment in which NCSES operates is a key determinant of the agency’s priorities from year to year. Internal processes that are observant, networked, and statistically and analytically balanced are important for NCSES’s indicators program.

On request of the sponsor, an interim report was published in February 2012, summarizing the panel’s early findings and recommendations. The recommendations offered in this report expand on those of the interim report. They are intended to serve as the basis for a strategic program of work that will enhance NCSES’s ability to produce indicators that capture change in STI to inform policy and optimally meet the needs of its user community.

Robert E. Litan and Andrew W. Wyckoff, Cochairs
Panel on Developing Science, Technology,
and Innovation Indicators for the Future

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

Acknowledgments*

It is with extreme gratitude that the panel thanks the many people who made contributions to this study on science, technology, and innovation (STI) indicators. The staff of the National Center for Science and Engineering Statistics (NCSES) at the National Science Foundation, under the directorship of John Gawalt and formerly Lynda Carlson, gracefully provided invaluable input and insights, including clear direction on what they wanted to learn from the study, as well as useful sources of information from their division and other resources. Robert Bell, Lawrence Burton, John Jankowski, Nirmala Kannankutty, Beethika Khan, Rolf Lehming, Francisco Moris, Jeri Mulrow, Christopher Pece, and Emilda Rivers all contributed their knowledge and expertise to answer our questions.

The panel’s work benefited greatly from presenters and attendees at our open meetings. The insights of the following individuals were critical for the framing of policy issues that are relevant to this study: Jeff Alexander (SRI International), Patrick Clemins (formerly of the American Association for the Advancement of Science), Mark Doms (U.S. Department of Commerce), Matthew Gerdin (U.S. State Department), Kei Koizumi (U.S. Office of Science and Technology Policy), Christine Matthews (Congressional Research Service), Amber Hartman Scholz (President’s Council of Advisors on Science and Technology), Dahlia Sokolov (U.S. House of Representatives), and D. Greg Tassey (formerly of the National Institute of Standards and Technology). Conceptual frameworks for STI indicators were presented by Michelle Alexopoulos (University of Toronto), Bronwyn Hall (University of California, Berkeley), and Adam Jaffe (Brandeis University).

Opportunities for advances in STI data collections and statistics, particularly among U.S. federal agencies, were discussed with panel members by B.K. Atrostic, Cheryl Grim, Richard Hough, Dave Kinyon, Erika McEntarfer, and Mary Potter (U.S. Census Bureau); Ana Aizcorbe, Maria Borga, and Carol Robbins (Bureau of Economic Analysis); Laurie Salmon, Jim Spletzer, and David Talan (Bureau of Labor Statistics); David McGranahan and Tim Wojan (U.S. Department of Agriculture); Daniel McGrath, Jessica Shedd, Matthew Soldner, and Tom Weko (National Center for Education Statistics); Stuart Graham (U.S. Patent and Trademark Office); and George Chacko and Walter Schaffer (National Institutes of Health). We also thank Rochelle (Shelly) Martinez and her colleagues and Katherine Wallman at the Office of Management and Budget for an engaging discussion regarding synergies in the federal statistical system with respect to measures of STI activities.

Because international comparability is an important aspect of this study, the panel convened two workshops of international researchers and practitioners who use STI indicators. The first workshop, in July 2011, covered STI measures and described opportunities and obstacles that NCSES should anticipate as it further develops its STI indicators program. We thank the workshop presenters: Shinichi Akaike (Hitotsubashi University, Tokyo), Howard Alper (Canada’s Science Technology and Innovation Council), Jayanta Chatterjee (Indian Institute of Technology), Gustavo Crespi (Inter-American Development Bank), Matthieu Delescluse (European Commission), Changlin Gao (Chinese Academy of Science and Technology), Jonathan Haskel (Imperial College Business School), Hugo Hollanders (United Nations University-Maastricht Economic and Social Research Institute on Innovation and Technology), Brian MacAulay (National Endowment for Science, Technology and the Arts, London), and Philippe Mawoko (The New Partnership for Africa’s Development).

The second workshop, in June 2012, included participants from the OECD-National Experts on Science and Technology Indicators (NESTI) representing nations, economic regions, and international institutions, including the African Union, Australia, Austria, Belgium, Brazil, Chile, the Czech Republic, Estonia, the European Union, Germany, Hungary, India, Israel, Italy, Japan, Korea, La Red

____________________

*All listed affiliations are as of February 2014.

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

Iberoamericana e Interamericana de Indicadores de Ciencia y Tecnología (RICYT), Luxembourg, the Netherlands, Norway, Poland, Portugal, Russia, the Slovak Republic, South Africa, Spain, Sweden, Switzerland, Turkey, the United Kingdom, and the United States. The panel chair, study director, and panel member Fred Gault, who is also chair of the NESTI advisory board, discussed the interim report from the panel and solicited comments from attendees, primarily on which indicators would be most useful to policy makers in their nations or regions. We thank the presenters, who discussed prioritization methods and specific indicators that are important for international comparisons: Alessandra Colecchia, Dominique Guellec, and Joaquim Oliveria Martins (OECD); Matthieu Delescluse (European Commission); Almamy Konté (African Union); Leonid Gokhberg (National Research University, Russia); and Veijo Ritola (Eurostat).

During the course of its work, the panel also obtained input from several other science and technology policy experts, including Aaron Chatterji (Duke University, formerly of the Council of Economic Advisers); Bhavya Lal and Stephanie Shipp (Institute for Defense Analyses–Science and Technology Policy Institute); Donna Ginther (University of Kansas); and Alessandra Colecchia, Gili Greenberg, and Fernando Galindo-Rueda (OECD). The panel explored the use of microdata, particularly administrative records and web tools, to create STI statistics. We heard from several experts in this diverse field of study at the July 2011 workshop, including Carl Bergstrom (University of Washington), Stefano Bertuzzi (National Institutes of Health and the STAR METRICS Program), Erik Brynjolfsson (Massachusetts Institute of Technology), Lee Giles (Pennsylvania State University), John Haltiwanger (University of Maryland), Richard Price (Academia.edu), and Alicia Robb (Kauffman Foundation).

The development of STI indicators at subnational levels was also an important topic for this study. At the July 2011 workshop, Rob Atkinson (Information Technology and Innovation Foundation), Maryann Feldman (University of North Carolina), Andrew Reamer (George Washington University), and Robert Samors and David Winwood (Association of Public and Land-grant Universities) presented options for measuring STI activities at a variety of geographic scales.

Nicholas Donofrio (IBM) participated in a roundtable discussion with panel members during the workshop. We greatly appreciate his insights from a business perspective on measuring research and development and innovation. His comments reminded us that the role of multinational corporations in the global STI system should be examined carefully and that entrepreneurial activities at firms of various sizes deserve careful measure.

Because the National Science Board (NSB) is a primary user of NCSES’s STI indicators—the biennial Science and Engineering Indicators volumes are published by NSB—the panel conducted two rounds of interviews with board members. We thank Ray Bowen, Kelvin K. Droegemeier, José-Marie Griffiths, Arthur Reilly, and Arnold F. Stancell for in-depth and insightful responses to our questions, as well as the board’s staff members Jennie Moehlmann, Michael Van Woert, and Matthew Wilson, for facilitating the meetings.

This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study’s charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Henry Brady, Office of the Dean, Goldman School of Public Policy, University of California, Berkeley; Michael Conlon, Clinical and Translation Science Institute, University of Florida; Martin Fleming, chief economist and vice president, Business Performance Services, International Business Machines Corporation; Jacques S. Gansler, Center for Public Policy and Private Enterprise, School of Public Policy, University of Maryland; Christopher T. Hill, Emeritus Department of Public Policy and Technology, School of Public Policy, George Mason University; Graham G. Kalton, Westat, Inc., Rockville, Maryland; Jason Owen-Smith, Barger Leadership Institute, University of Michigan; Georgine M. Pion, Department of Psychology and Human Development, Vanderbilt University; Hal Salzman, Bloustein School of Planning and Public Policy and Heldrich Center for Workforce Development, Rutgers University; Phillip Swagel, School of Public Policy, University of Maryland; Albert H. Teich, director, Science and Policy Programs, American Association for the Advancement of Science; and Ward Ziarko, Scientific and Technical Information Service, Belgian Federal Science Policy.

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the report’s conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Chuck Phelps, provost emeritus, University of Rochester, and John Haltiwanger, Department of Economics, University of Maryland. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring panel and the institution.

We extend special thanks to staff of the Committee on National Statistics. The study director, Kaye Husbands Fealing, provided invaluable assistance to the panel in orga-

Page xiii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

nizing the meetings and preparing this report. Connie Citro, Tom Plewes, and Michael Cohen gave excellent guidance to the panel. Stephen Merrill, director of the Board on Science, Technology, and Economic Policy, contributed to panel meetings. Esha Sinha assembled and developed a public use databank of STI indicators from several international sources (see http://sites.nationalacademies.org/DBASSE/CNSTAT/Science_Technology_and_Innovation_Indicators/index.htm [May 2014]. In collaboration with panel member Leland Wilkinson, she also conducted the cluster analysis and heat map exercises for this study and cowrote the accompanying data appendix (Appendix F). Anthony Mann provided outstanding administrative and logistical support to the panel. Our Mirzayan fellow, Daniel Grady, drew on his expertise in systems dynamics and web tools to benefit the panel’s work.

Robert E. Litan and Andrew W. Wyckoff, Cochairs
Panel on Developing Science, Technology,
and Innovation Indicators for the Future

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

Acronyms, Abbreviations, and Descriptions

ACRONYMS AND ABBREVIATIONS

America COMPETES Act

America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science

APLU

Association of Public and Land-grant Universities

   
BEA

Bureau of Economic Analysis, U.S. Department of Commerce

BERD

business enterprise expenditure on research and development

BLS

Bureau of Labor Statistics, U.S. Department of Labor

BOP

balance of payments

BRDIS

Business Research and Development and Innovation Survey

   
CICEP

Commission on Innovation, Competitiveness, and Economic Prosperity

CIS

Community Innovation Survey

   
ERS

Economic Research Service, U.S. Department of Agriculture

ETA

Employment and Training Administration, U.S. Department of Labor

EU

European Union

   
FTE

full-time equivalent

   
GBAORD

government budget appropriations or outlays for research and development

GDP

gross domestic product

GERD

gross domestic expenditure on research and development

GO→SPIN

Global Observatory on Science, Technology and Innovation Policy Instruments

GOVERD

government intramural expenditure on research and development

   
HERD

higher education expenditure on research and development or Higher Education Research and Development Survey

ICT

information and communication technology

IMF

International Monetary Fund

IPEDS

Integrated Postsecondary Education Data System

   
MEP

Manufacturing Extension Partnership

MIST

Microbusiness, Innovation Science, and Technology

MOOC

massive open online course

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
NAICS

North American Industry Classification System

NCES

National Center for Education Statistics, U.S. Department of Education

NCSES

National Center for Science and Engineering Statistics, U.S. National Science Foundation

NEPAD

New Partnership for Africa’s Development

NESTI

National Experts on Science and Technology Indicators

NIH

National Institutes of Health, U.S. Department of Health and Human Services

NIST

National Institute of Standards and Technology, U.S. Department of Commerce

NLM

National Library of Medicine, U.S. Department of Health and Human Services

NSCG

National Survey of College Graduates

NSF

U.S. National Science Foundation

NSRCG

National Survey of Recent College Graduates

   
OECD

Organisation for Economic Co-operation and Development

OMB

U.S. Office of Management and Budget

OSTP

U.S. Office of Science and Technology Policy

   
PISA

Programme for International Student Assessment

   
R&D

research and development

   
S&E

science and engineering

S&T

science and technology

SBIR

Small Business Innovation Research

SDR

Survey of Doctorate Recipients

SED

Survey of Earned Doctorates

SEI

Science and Engineering Indicators

SESTAT

Science and Engineering Statistical Data System

SIBS

Survey of Innovation and Business Strategy

STAR METRICS

Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science

STEM

science, technology, engineering, and mathematics

STI

science, technology, and innovation

STTR

Small Business Technology Transfer

   
UNESCO

United Nations Educational, Scientific and Cultural Organization

UNU-MERIT

United Nations University’s Maastricht Economic and Social Research Institute on Innovation and Technology

USDA

U.S. Department of Agriculture

DESCRIPTIONS

BERD: Research and development expenditure in the business enterprise sector in a given year at the regional level.

GBAORD: Government budget appropriations or outlays for research and development, a way of measuring government support for research and development activities.

GERD: Gross domestic expenditure on research and development, defined as total intramural expenditure on research and development performed on the national territory during a given period.

GO→SPIN: Global Observatory on Science, Technology and Innovation Policy Instruments. Contains not only indicators but also an inventory of science, technology, and innovation (STI) national system descriptions; STI legal frameworks (with access to text of the acts and decrees); and an inventory of operational policy instruments that promote STI activities in a country. Developed by a group at the United Nations Educational, Scientific and Cultural Organization (UNESCO) Regional Bureau for Science for Latin America and the Caribbean.

Page xvii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

HERD: Research and development expenditure in the higher education sector in a given year at the national and different subnational geographic scales.

Human capital: The ability, knowledge, and skill base that are typically acquired or enhanced by an individual through education and training.

Innovation: The implementation of a new or significantly improved product (good or service) or process; a new marketing method; or a new organizational method in business practices, workplace organization, or external relations (OECD-Eurostat, 2005, p. 46). A common feature of an innovation is that it must have been implemented. A new or improved product is implemented when it is introduced on the market. New processes, marketing methods, or organizational methods are implemented when they are brought into actual use in a firm’s operations (OECD-Eurostat, 2005, p. 47).

NLSY: National Longitudinal Survey of Youth, conducted by the Bureau of Labor Statistics.

Research and development (R&D): Comprises creative work undertaken on a systematic basis to increase the stock of knowledge, including knowledge of man, culture, and society, and the use of this stock of knowledge to devise new applications (OECD, 2002a, p. 30).

Science and technology (S&T): A broad concept that includes science and technology activities, defined as follows: “For statistical purposes, Scientific and Technological Activities (STA) can be defined as all systematic activities which are closely concerned with the generation, advancement, dissemination, and application of scientific and technical knowledge in all fields of science and technology, that is the natural sciences, engineering and technology, the medical and the agricultural sciences (NS), as well as the social sciences and humanities (SSH)” (United Nations Educational, Scientific and Cultural Organization, 1984, p. 17). Also included are Scientific and Technological Services (STS) and Scientific and Technological Education and Training (STET), the definitions of which are found in UNESCO (1978). Research and development is included in STA. The OECD Frascati Manual (OECD, 2002a, p. 19) notes that “R&D (defined similarly by UNESCO and the OECD) is thus to be distinguished from both STET and STS.” In the Frascati definition, R&D includes basic research, applied research, and experimental development.

SIBS: Survey of Innovation and Business Strategy, conducted by Statistics Canada.

Statistic: A numerical fact or datum, especially one computed from a sample.

Statistical data: Data from a survey or administrative source used to produce statistics (OECD, 2002b, pp. 205-230).

Statistical indicator: A statistic, or combination of statistics, providing information on some aspect of the state of a system or of its change over time. For example, gross domestic product (GDP) provides information on the level of value added in the economy, and its change over time is an indicator of the economic state of the nation. The decline of GDP for two quarters is indicative of a recession. The ratio of gross domestic expenditure on research and development (GERD) to GDP is an indicator of the formal generation of new knowledge and is used both for international comparisons and for the setting of targets, such as the Lisbon target for the European Union of 3 percent. There are also composite indicators involving many component indicators. A single indicator is indicative but not definitive in its description of the system. As an example, GDP per capita provides one piece of information about an economy and may be indicative of wealth or productivity, but the income distribution for the country, another indicator summarized in a Gini coefficient, provides complementary information on income inequality. Employment is yet another indicator of the state of the economy.

Statistical information: Statistical data, or a statistic, placed in a context. As an example, the number of people making less than a dollar a day in a country is a statistic populated by statistical data that may result from estimation based on a sample. The context is the analysis of poverty, and in that context, the statistic provides information on poverty, but it is only one dimension.

Page xviii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

Traditional Indicators of the Three K’s

Business R&D: R&D Services and Intangible Assets

Summary

6       MEASURING HUMAN CAPITAL

What Users Want

NCSES’s Existing Human Capital Indicators

The Survey Problem

Potential for New Data Sources

Revised and New Human Capital Indicators

Key Opportunities

Summary

7       A PARADIGM SHIFT IN DATA COLLECTION AND ANALYSIS

New Methods, New Data Sources

Implications for NCSES

Indicators from Frontier Tools: Example of the Data Science Discipline

A New Direction for NCSES

Next Steps

Summary

Annex 7-1: Potential Data Sources to Explore

8       INFORMING THE STRATEGIC PLANNING PROCESS

NCSES’s Accomplishments

Processes for Change

A Program of Work

Panel Recommendations: Chapters 2-7

REFERENCES

APPENDIXES

A      Biographical Sketches of Panel Members and Staff

B      Users of Science, Technology, and Innovation (STI) Data and Indicators and Their Questions and Requests for STI Indicators

C      Workshop on Developing Science, Technology, and Innovation Indicators for the Future Agenda and Participants

D      OECD-National Experts on Science and Technology Indicators (NESTI) Workshop and Attendees

E      National Center for Science and Engineering Statistics Research Abstracts 2012

F      Science, Technology, and Innovation Databases and Heat Map Analysis

G      2011 BRDIS Results

H      2011 BRDIS Table 46

I       2011 BRDIS Table 47

J       2011 BRDIS Table 48

K      2011 BRDIS Table 49

Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R1
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R2
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R3
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R4
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R5
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R6
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R8
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R9
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R10
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R11
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R12
Page xiii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R13
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R14
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R15
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R16
Page xvii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R17
Page xviii Cite
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R18
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R19
Suggested Citation:"Front Matter." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page R20
Next: Summary »
Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy Get This Book
×
Buy Paperback | $64.00 Buy Ebook | $49.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since the 1950s, under congressional mandate, the U.S. National Science Foundation (NSF) - through its National Center for Science and Engineering Statistics (NCSES) and predecessor agencies - has produced regularly updated measures of research and development expenditures, employment and training in science and engineering, and other indicators of the state of U.S. science and technology. A more recent focus has been on measuring innovation in the corporate sector. NCSES collects its own data on science, technology, and innovation (STI) activities and also incorporates data from other agencies to produce indicators that are used for monitoring purposes - including comparisons among sectors, regions, and with other countries - and for identifying trends that may require policy attention and generate research needs. NCSES also provides extensive tabulations and microdata files for in-depth analysis.

Capturing Change in Science, Technology, and Innovation assesses and provides recommendations regarding the need for revised, refocused, and newly developed indicators of STI activities that would enable NCSES to respond to changing policy concerns. This report also identifies and assesses both existing and potential data resources and tools that NCSES could exploit to further develop its indicators program. Finally, the report considers strategic pathways for NCSES to move forward with an improved STI indicators program. The recommendations offered in Capturing Change in Science, Technology, and Innovation are intended to serve as the basis for a strategic program of work that will enhance NCSES's ability to produce indicators that capture change in science, technology, and innovation to inform policy and optimally meet the needs of its user community.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!