The availability of relevant, accurate, timely, and objective information on the health of the science, technology, and innovation (STI) enterprise in the United States is critical to addressing vital policy questions for the nation. Just some of these questions are
- How is the contribution of STI to productivity, employment, and growth in the broader U.S. economy changing in a world of economic globalization?
- What are the drivers of innovation that benefit the economy and society?
- Does the United States have the STI knowledge capital it needs to move the nation forward, address its social challenges, and maintain competitiveness with other countries?
- What effect does federal expenditure on research and development (R&D) and on science and engineering education have on innovation, economic health, and social welfare, and over what time frame?
- What characteristics of industries and geographic areas facilitate productive innovation?
Since the 1950s, under congressional mandate, the U.S. National Science Foundation (NSF)—through its National Center for Science and Engineering Statistics (NCSES) and predecessor agencies—has produced regularly updated measures of R&D expenditures, employment and training in science and engineering, and other indicators of the state of U.S. science and technology. A more recent focus has been on measuring innovation in the corporate sector. NCSES collects its own data on STI activities and also incorporates data from other agencies to produce indicators that are used for monitoring purposes—including comparisons among sectors, regions, and with other countries—and for identifying trends that may require policy attention and generate research needs. NCSES also provides extensive tabulations and microdata files for in-depth analysis.
Changes in the structure of the U.S. and global economies and in NCSES’s mandate and budget environment pose not only significant challenges but also opportunities for its efforts to monitor STI activities in the United States. On the challenge side, what used to be the relatively simple task of tracking domestic R&D spending by a small number of U.S. manufacturers has evolved into the need to monitor STI activities across the globe and across a wide range of industrial and commercial sectors. Similarly challenging are the increasing velocity and changing character of the innovation system. Yet another challenge is the constrained budget environment, which means that NCSES, like other federal agencies, is trying to do more with less.
Offering both challenge and opportunity is the recent broadening of NCSES’s statistical mission by the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (America COMPETES) Reauthorization Act of 2010. Section 505 of the act expanded and codified NCSES’s role as a U.S. federal statistical agency charged with collecting, acquiring, analyzing, reporting, and disseminating data on R&D trends; the science and engineering workforce; U.S. competitiveness in science, technology, and R&D; and the condition and progress of U.S. science, technology, engineering, and mathematics (STEM) education. The act also charged NCSES with supporting research that uses its data and with improving its statistical methods.
Further affording both challenge and opportunity is the emergence of new types of information with which to track innovation, R&D, and the STEM workforce. Historically, statistical agencies such as NCSES have relied on sample surveys and censuses to collect consistent and unbiased information in these and other areas. In recent years, however, the amount of raw data available online has soared, creating possibilities for new STI indicators. Microdata from administrative records and other sources are increasingly being used to produce measures of capacities and trends in
the global STI system. Also, frontier methods are emerging for monitoring phenomena such as the number of new product introductions through sophisticated web-scraping algorithms; for tracking innovation activities through help-wanted ads; and for tracing networks of scientists engaged in research through textual analysis of grant abstracts, patent applications, and online working papers and publications. At present, such data sources, although promising, are largely untested and hence of highly uncertain quality, which means they will require careful evaluation to determine those that may be suited for statistical use.
These new challenges and opportunities raise questions about whether NCSES’s current STI statistical activities are properly focused to produce the information needed by policy makers, researchers, and businesses. Such questions have become especially acute given tightening fiscal measures and the importance of innovation for economic growth and other aspects of social well-being.
STUDY PURPOSE AND METHODOLOGY
In response to a request from NCSES, the Committee on National Statistics and the Board on Science, Technology, and Economic Policy of the National Research Council convened the Panel on Developing Science, Technology, and Innovation Indicators for the Future. The panel was charged to assess and provide recommendations regarding the need for revised, refocused, and newly developed indicators of STI activities that would enable NCSES to respond to changing policy concerns. In carrying out its charge, the panel reviewed STI indicators from the United States as well as Australia, Canada, China, India, Japan, Russia, South Korea, and several countries in Africa, Europe, and Latin America. It consulted with a broad range of users of STI indicators in the academic, public, and private sectors and with statistical agencies and organizations that produce relevant information in the United States and abroad. Although its focus was on indicators, the panel also believed it important to identify and assess both existing and potential data resources and tools that NCSES could exploit to further develop its indicators program. Finally, the panel considered strategic pathways for NCSES to move forward with an improved STI indicators program.
The panel recognized that no one model informs the types of indicators NCSES needs to produce. Policy questions for the United States served as an important guide to the panel’s review, but the study was also informed by considerations of international comparability. In addition, the panel recognized the need to balance the introduction of new indicator series against the need to maintain long-standing bellwether indicators that require continual monitoring over long periods of time—for example, to ascertain the rise or decline of countries based on their technological capabilities.
KEY FINDINGS
The United States has historically played a leadership role internationally in the development of STI indicators, particularly in the areas of human capital and R&D expenditures. NCSES specifically has displayed areas of strength and innovation in its data collection and indicator development, outreach to the user community, and collaboration with statistical agencies in the United States and abroad. This report is intended to offer NCSES guidance that will help keep it at the forefront of providing accurate, relevant, timely, and objective STI indicators that can inform policy makers about the health of the STI enterprise, signal trends of policy concern, and suggest questions for in-depth research.
In reviewing STI indicators around the globe, the panel found a depth and breadth of indicator programs that is truly remarkable—many countries are placing a high priority on collecting information on innovation and related activities, and they are gathering high-quality data. Nevertheless, after hearing presentations from different countries ranging across the African, Asian, and European continents, the panel was unable to identify any proven STI indicators or methodologies used by other countries that NCSES lacks and could easily and inexpensively adopt for its own program. Given the global nature of STI, however, it is essential for NCSES to remain aware of the experimentation currently under way and to benefit from the lessons learned from these experiments.
The panel also identified a number of ways in which NCSES could improve its current STI indicators program with relatively little new investment in original data collection. Examples include increasing comparability with international classifications and concepts and improving the usefulness of the agency’s Business Research and Development and Innovation Survey (BRDIS) to policy makers and researchers.
Finally, changes in the economy have made it necessary to develop new concepts and measures of STI and its economic and social impacts. Economic changes also have given rise to new methodologies for data collection and analysis that have the potential to lower costs and expand the usefulness of STI indicators while enabling NCSES to maintain and enhance its global leadership. NCSES may find it difficult to fund and supervise the development of new STI measures and methodologies, especially while continuing its current program of STI indicators. Nonetheless, continued production of only the traditional STI measures will provide an incomplete and possibly misleading indication of how well or poorly the economies of the United States and other countries are performing in generating the innovations in products, services, and production and delivery chains that lead to improved living standards.
STRATEGIC RECOMMENDATIONS
To help NCSES deal with the implications of the above findings, the panel offers five recommendations that together form a strategy for moving forward with an improved STI indicators program. The first strategic recommendation entails according priority to data quality, broadly defined, which is a principal objective of a statistical agency. The remaining recommendations offer four strategic pathways for development: working with other agencies to share data and link databases to produce indicators that would not be possible if the agencies worked independently; using existing grants and fellowship programs to support relevant methodological research; making data holdings available to researchers on a timely basis for substantive work on new and revised STI indicators, in addition to methodological advances; and establishing a position of chief analyst to link policy and research needs more effectively with analytical concepts and data collection methods.
NCSES will first need to assign relative priorities to the above four strategic pathways and then use the other 26 more specific recommendations in this report to flesh out a program of work along each pathway. NCSES has already made significant strides in each of these areas. Strengthening its program of work by incorporating the panel’s recommendations should ensure that NCSES is well positioned to maintain and enhance its domestic and international leadership in the production and interpretation of STI indicators needed to inform policy.
The panel’s five strategic recommendations are briefly outlined below, with examples of selected specific recommendations (numbered according to the chapter of the main text in which they appear). The panel’s specific recommendations also are summarized and linked to the strategic recommendations in Chapter 8.
Data Quality (Recommendation 8-1)
The panel recommends that NCSES, as a statistical agency, place data quality at the top of its priority list. The panel defines data quality to include accuracy, relevance, timeliness, and accessibility. To make data quality central, NCSES should review its data quality framework, establish a set of quality indicators for all of its surveys, and publish the results of its review at least annually.
The overriding need with respect to data quality for NCSES’s STI indicators program is for the indicators, and the data used to generate them, to accurately reflect policy-relevant dimensions of the underlying processes of science, technology, and innovation that contribute to economic growth and societal well-being. To be relevant, indicators should also be available on a timely basis and widely accessible to the data user community. A concern with quality means that indicators should be carefully evaluated for their conceptual soundness, and all feasible steps should be taken to minimize measurement error. Quality considerations should inform decisions to revise, drop, or add indicators. Indicators that are highly relevant but do not meet measurement standards may be published as experimental or research series, accompanied by clear qualifications.
Data Linkage and Sharing (Recommendation 8-2)
The panel recommends that NCSES work with other federal agencies bilaterally and in interagency statistical committees to share data, link databases where feasible, and produce products that would not be possible if the agencies worked independently. The use of data from outside the federal system, where appropriate, should be part of this process. This strategic pathway is central to enable NCSES to develop selected new policy-relevant, internationally comparable indicators that are based on existing NCSES survey data and on data collections of other statistical agencies and organizations, both within and outside the government. Selected recommendations for implementation via this pathway include the following:
- NCSES has many data series that have not been fully analyzed but have great potential to help answer questions posed by users. Recommendation 3-1 stresses the importance of developing new STI indicators from existing data.
- In particular, existing BRDIS data should be exploited by, for example,
– developing innovation-related tabulations from BRDIS data for comparison purposes using the same cutoffs for firm size used by other OECD countries and cross-tabulations from BRDIS data that link innovation indicators to a variety of business characteristics, including the amount of R&D spending by U.S.-based companies outside of the United States (Recommendation 4-2);
– matching existing BRDIS data to ongoing surveys and administrative records at the U.S. Census Bureau and Bureau of Labor Statistics to create measures of activities by high-growth firms and of firm dynamics, such as births and deaths of businesses, linked to innovation outputs (Recommendation 4-4);
– making greater use of BRDIS data to provide indicators of payments and receipts for R&D services purchased from and sold to other countries, an effort that would require continued collaboration with the U.S. Bureau of Economic Analysis on the linked dataset (Recommendation 5-2); and
– developing a suite of indicators that track the development and diffusion of general-purpose technologies, including information and communication technologies, biotechnology, nanotechnology, and green technologies, using BRDIS data
as well as patent and bibliometric data (Recommendation 5-4).
- Better access to BRDIS data by NCSES staff is imperative for the timely distribution of new and improved BRDIS-based indicators (Recommendation 4-3).
- NCSES should also draw on longitudinal datasets on occupations and education levels to create indicators of labor mobility. Data from the Survey of Doctorate Recipients, the Longitudinal Employer-Household Dynamics Study, and the Baccalaureate and Beyond Longitudinal Study would be particularly useful for understanding the correspondence between supply and demand for skill sets in science and technology sectors and worker mobility (Recommendations 6-1 and 6-2).
- An important prerequisite for linking data from different sources is the development of a consistent taxonomy of science and engineering fields and occupations (including the health and social sciences). NCSES should engage with other statistical agencies, including but not limited to the Bureau of Labor Statistics, the U.S. Census Bureau, the National Center for Education Statistics, and the National Institutes of Health, to develop this taxonomy and to establish a process for updating it as needed (Recommendation 2-2).
Methodological and Substantive Research Through a Community of Practice (Recommendations 8-3 and 8-4)
The panel recommends that NCSES build a community of practice around existing and emerging methodological issues so it can update its data acquisition and analysis techniques to support new and revised STI indicators. Such a community should include not only NCSES staff but also, given constrained staff resources, outside researchers. NCSES should leverage its existing grants and fellowship programs to support methodological research that addresses its specific needs.
Relatedly, the panel recommends that NCSES make its data holdings available to external researchers on a timely basis to facilitate their research while protecting confidentiality and privacy. The timeliness with which NCSES delivers indicators to user communities depends on its own access to data resources, primarily surveys and censuses, but increasingly other sources as well, such as databases that involve text processing.
Outside researchers can help NCSES address methodological issues entailed in improving the accuracy and timeliness of new and revised STI indicators and evaluating the usefulness of new kinds of data for indicator production. External researchers can also help NCSES develop new and improved indicators that are relevant to policy on such topics as the following (relevant recommendations are in Chapters 4 through 7):
- organizational and market innovations, as well as innovations in training and design;
- hindrances to the innovation process;
- new measures of innovation based on business practice data obtained through administrative records and web-based data;
- knowledge networks that contribute to innovation;
- improved measures of labor mobility, career paths, stay rates for students at various levels of education, wages and salaries by skill set, and demand and supply of skill sets in various industries;
- international trade in technological goods and services;
- emerging regions for entrepreneurial activity in science and technology; and
- precommercialized inventions, to shed light on the early stages of the innovation process.
Chief Analyst (Recommendation 8-5)
The panel recommends that NCSES establish a unit to manage data quality assurance, cooperation with other organizations, and focused analysis to support the development of its indicators program and underlying datasets. Establishment of such a unit would enable NCSES to more fully embody an important practice for federal statistical agencies, which is to have an active research program on the substantive issues for which the agency compiles information and to understand how that information is used for policy and decision making (National Research Council, 2013b, p. 22). NCSES should include within this unit a new position of chief analyst, whose role would be to interface with users of NCSES data, including indicators; provide other NCSES staff with periodic updates on areas of change that may affect the agency’s statistical operations; and assess the utility of new types of datasets and tools that NCSES could use either in house or by contractual arrangement.
The panel believes a chief analyst would better enable NCSES to keep up to date with changing demand for STI indicators and other data products; with socioeconomic changes that have implications for STI indicators; and with new ways of collecting, acquiring, analyzing, and disseminating the most relevant, accurate, and timely indicators for policy and research use. Through adopting this recommendation, along with the other strategic and specific recommendations in the panel’s report, NCSES will be well positioned to carry out its role from the America COMPETES Reauthorization Act of 2010 to “collect, acquire, analyze, report, and disseminate statistical data related to the science and engineering enterprise in the United States and other nations that is relevant and useful to practitioners, researchers, policymakers, and the public.”
