The availability of relevant, accurate, timely, and objective information on the health of the science, technology, and innovation (STI) enterprise in the United States is critical to addressing vital policy questions for the nation. Just some of these questions are
Since the 1950s, under congressional mandate, the U.S. National Science Foundation (NSF)—through its National Center for Science and Engineering Statistics (NCSES) and predecessor agencies—has produced regularly updated measures of R&D expenditures, employment and training in science and engineering, and other indicators of the state of U.S. science and technology. A more recent focus has been on measuring innovation in the corporate sector. NCSES collects its own data on STI activities and also incorporates data from other agencies to produce indicators that are used for monitoring purposes—including comparisons among sectors, regions, and with other countries—and for identifying trends that may require policy attention and generate research needs. NCSES also provides extensive tabulations and microdata files for in-depth analysis.
Changes in the structure of the U.S. and global economies and in NCSES’s mandate and budget environment pose not only significant challenges but also opportunities for its efforts to monitor STI activities in the United States. On the challenge side, what used to be the relatively simple task of tracking domestic R&D spending by a small number of U.S. manufacturers has evolved into the need to monitor STI activities across the globe and across a wide range of industrial and commercial sectors. Similarly challenging are the increasing velocity and changing character of the innovation system. Yet another challenge is the constrained budget environment, which means that NCSES, like other federal agencies, is trying to do more with less.
Offering both challenge and opportunity is the recent broadening of NCSES’s statistical mission by the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (America COMPETES) Reauthorization Act of 2010. Section 505 of the act expanded and codified NCSES’s role as a U.S. federal statistical agency charged with collecting, acquiring, analyzing, reporting, and disseminating data on R&D trends; the science and engineering workforce; U.S. competitiveness in science, technology, and R&D; and the condition and progress of U.S. science, technology, engineering, and mathematics (STEM) education. The act also charged NCSES with supporting research that uses its data and with improving its statistical methods.
Further affording both challenge and opportunity is the emergence of new types of information with which to track innovation, R&D, and the STEM workforce. Historically, statistical agencies such as NCSES have relied on sample surveys and censuses to collect consistent and unbiased information in these and other areas. In recent years, however, the amount of raw data available online has soared, creating possibilities for new STI indicators. Microdata from administrative records and other sources are increasingly being used to produce measures of capacities and trends in
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Summary The availability of relevant, accurate, timely, and objec- Changes in the structure of the U.S. and global econo- tive information on the health of the science, technology, and mies and in NCSES’s mandate and budget environment innovation (STI) enterprise in the United States is critical to pose not only significant challenges but also opportunities addressing vital policy questions for the nation. Just some for its efforts to monitor STI activities in the United States. of these questions are On the challenge side, what used to be the relatively simple task of tracking domestic R&D spending by a small number • How is the contribution of STI to productivity, of U.S. manufacturers has evolved into the need to monitor employment, and growth in the broader U.S. econ- STI activities across the globe and across a wide range of omy changing in a world of economic globalization? industrial and commercial sectors. Similarly challenging are • What are the drivers of innovation that benefit the the increasing velocity and changing character of the innova- economy and society? tion system. Yet another challenge is the constrained budget • Does the United States have the STI knowledge environment, which means that NCSES, like other federal capital it needs to move the nation forward, address agencies, is trying to do more with less. its social challenges, and maintain competitiveness Offering both challenge and opportunity is the recent with other countries? broadening of NCSES’s statistical mission by the America • What effect does federal expenditure on research and Creating Opportunities to Meaningfully Promote Excel- development (R&D) and on science and engineering lence in Technology, Education, and Science (America education have on innovation, economic health, and COMPETES) Reauthorization Act of 2010. Section 505 of social welfare, and over what time frame? the act expanded and codified NCSES’s role as a U.S. federal • What characteristics of industries and geographic statistical agency charged with collecting, acquiring, analyz- areas facilitate productive innovation? ing, reporting, and disseminating data on R&D trends; the science and engineering workforce; U.S. competitiveness in Since the 1950s, under congressional mandate, the U.S. science, technology, and R&D; and the condition and prog- National Science Foundation (NSF)—through its National ress of U.S. science, technology, engineering, and mathemat- Center for Science and Engineering Statistics (NCSES) ics (STEM) education. The act also charged NCSES with and predecessor agencies—has produced regularly updated supporting research that uses its data and with improving its measures of R&D expenditures, employment and training in statistical methods. science and engineering, and other indicators of the state of Further affording both challenge and opportunity is the U.S. science and technology. A more recent focus has been emergence of new types of information with which to track on measuring innovation in the corporate sector. NCSES innovation, R&D, and the STEM workforce. Historically, collects its own data on STI activities and also incorporates statistical agencies such as NCSES have relied on sample data from other agencies to produce indicators that are used surveys and censuses to collect consistent and unbiased for monitoring purposes—including comparisons among information in these and other areas. In recent years, how- sectors, regions, and with other countries—and for identify- ever, the amount of raw data available online has soared, ing trends that may require policy attention and generate creating possibilities for new STI indicators. Microdata from research needs. NCSES also provides extensive tabulations administrative records and other sources are increasingly and microdata files for in-depth analysis. being used to produce measures of capacities and trends in 1
OCR for page 1
2 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION the global STI system. Also, frontier methods are emerg- KEY FINDINGS ing for monitoring phenomena such as the number of new The United States has historically played a leadership product introductions through sophisticated web-scraping role internationally in the development of STI indicators, algorithms; for tracking innovation activities through help- particularly in the areas of human capital and R&D expen- wanted ads; and for tracing networks of scientists engaged ditures. NCSES specifically has displayed areas of strength in research through textual analysis of grant abstracts, patent and innovation in its data collection and indicator develop- applications, and online working papers and publications. At ment, outreach to the user community, and collaboration present, such data sources, although promising, are largely with statistical agencies in the United States and abroad. untested and hence of highly uncertain quality, which means This report is intended to offer NCSES guidance that will they will require careful evaluation to determine those that help keep it at the forefront of providing accurate, relevant, may be suited for statistical use. timely, and objective STI indicators that can inform policy These new challenges and opportunities raise questions makers about the health of the STI enterprise, signal trends of about whether NCSES’s current STI statistical activities policy concern, and suggest questions for in-depth research. are properly focused to produce the information needed by In reviewing STI indicators around the globe, the panel policy makers, researchers, and businesses. Such questions found a depth and breadth of indicator programs that is truly have become especially acute given tightening fiscal mea- remarkable—many countries are placing a high priority on sures and the importance of innovation for economic growth collecting information on innovation and related activities, and other aspects of social well-being. and they are gathering high-quality data. Nevertheless, after hearing presentations from different countries ranging across STUDY PURPOSE AND METHODOLOGY the African, Asian, and European continents, the panel was unable to identify any proven STI indicators or methodolo- In response to a request from NCSES, the Committee on gies used by other countries that NCSES lacks and could National Statistics and the Board on Science, Technology, easily and inexpensively adopt for its own program. Given and Economic Policy of the National Research Council the global nature of STI, however, it is essential for NCSES convened the Panel on Developing Science, Technology, and to remain aware of the experimentation currently under Innovation Indicators for the Future. The panel was charged way and to benefit from the lessons learned from these to assess and provide recommendations regarding the need experiments. for revised, refocused, and newly developed indicators of The panel also identified a number of ways in which STI activities that would enable NCSES to respond to chang- NCSES could improve its current STI indicators program ing policy concerns. In carrying out its charge, the panel with relatively little new investment in original data col- reviewed STI indicators from the United States as well as lection. Examples include increasing comparability with Australia, Canada, China, India, Japan, Russia, South Korea, international classifications and concepts and improving the and several countries in Africa, Europe, and Latin America. usefulness of the agency’s Business Research and Develop- It consulted with a broad range of users of STI indicators in ment and Innovation Survey (BRDIS) to policy makers and the academic, public, and private sectors and with statistical researchers. agencies and organizations that produce relevant information Finally, changes in the economy have made it necessary to in the United States and abroad. Although its focus was on develop new concepts and measures of STI and its economic indicators, the panel also believed it important to identify and and social impacts. Economic changes also have given rise assess both existing and potential data resources and tools to new methodologies for data collection and analysis that that NCSES could exploit to further develop its indicators have the potential to lower costs and expand the usefulness program. Finally, the panel considered strategic pathways of STI indicators while enabling NCSES to maintain and for NCSES to move forward with an improved STI indica- enhance its global leadership. NCSES may find it difficult tors program. to fund and supervise the development of new STI measures The panel recognized that no one model informs the types and methodologies, especially while continuing its current of indicators NCSES needs to produce. Policy questions for program of STI indicators. Nonetheless, continued produc- the United States served as an important guide to the panel’s tion of only the traditional STI measures will provide an review, but the study was also informed by considerations incomplete and possibly misleading indication of how well of international comparability. In addition, the panel recog- or poorly the economies of the United States and other nized the need to balance the introduction of new indicator countries are performing in generating the innovations in series against the need to maintain long-standing bellwether products, services, and production and delivery chains that indicators that require continual monitoring over long peri- lead to improved living standards. ods of time—for example, to ascertain the rise or decline of countries based on their technological capabilities.
OCR for page 1
SUMMARY 3 STRATEGIC RECOMMENDATIONS to minimize measurement error. Quality considerations should inform decisions to revise, drop, or add indicators. To help NCSES deal with the implications of the above Indicators that are highly relevant but do not meet measure- findings, the panel offers five recommendations that together ment standards may be published as experimental or research form a strategy for moving forward with an improved STI series, accompanied by clear qualifications. indicators program. The first strategic recommendation entails according priority to data quality, broadly defined, which is a principal objective of a statistical agency. The Data Linkage and Sharing (Recommendation 8-2) remaining recommendations offer four strategic pathways The panel recommends that NCSES work with other for development: working with other agencies to share data federal agencies bilaterally and in interagency statistical and link databases to produce indicators that would not committees to share data, link databases where feasible, and be possible if the agencies worked independently; using produce products that would not be possible if the agencies existing grants and fellowship programs to support relevant worked independently. The use of data from outside the methodological research; making data holdings available to federal system, where appropriate, should be part of this researchers on a timely basis for substantive work on new process. This strategic pathway is central to enable NCSES and revised STI indicators, in addition to methodological to develop selected new policy-relevant, internationally com- advances; and establishing a position of chief analyst to link parable indicators that are based on existing NCSES survey policy and research needs more effectively with analytical data and on data collections of other statistical agencies concepts and data collection methods. and organizations, both within and outside the government. NCSES will first need to assign relative priorities to the Selected recommendations for implementation via this path- above four strategic pathways and then use the other 26 way include the following: more specific recommendations in this report to flesh out a program of work along each pathway. NCSES has already • NCSES has many data series that have not been fully made significant strides in each of these areas. Strengthening analyzed but have great potential to help answer its program of work by incorporating the panel’s recom- questions posed by users. Recommendation 3-1 mendations should ensure that NCSES is well positioned to stresses the importance of developing new STI indi- maintain and enhance its domestic and international leader- cators from existing data. ship in the production and interpretation of STI indicators • In particular, existing BRDIS data should be exploited needed to inform policy. by, for example, The panel’s five strategic recommendations are briefly – developing innovation-related tabulations from outlined below, with examples of selected specific recom- BRDIS data for comparison purposes using the mendations (numbered according to the chapter of the main same cutoffs for firm size used by other OECD text in which they appear). The panel’s specific recom- countries and cross-tabulations from BRDIS data mendations also are summarized and linked to the strategic that link innovation indicators to a variety of busi- recommendations in Chapter 8. ness characteristics, including the amount of R&D spending by U.S.-based companies outside of the Data Quality (Recommendation 8-1) United States (Recommendation 4-2); – matching existing BRDIS data to ongoing surveys The panel recommends that NCSES, as a statistical and administrative records at the U.S. Census agency, place data quality at the top of its priority list. The Bureau and Bureau of Labor Statistics to create panel defines data quality to include accuracy, relevance, measures of activities by high-growth firms and timeliness, and accessibility. To make data quality central, of firm dynamics, such as births and deaths of NCSES should review its data quality framework, establish businesses, linked to innovation outputs (Recom- a set of quality indicators for all of its surveys, and publish mendation 4-4); the results of its review at least annually. – making greater use of BRDIS data to provide indi- The overriding need with respect to data quality for cators of payments and receipts for R&D services NCSES’s STI indicators program is for the indicators, and purchased from and sold to other countries, an the data used to generate them, to accurately reflect policy- effort that would require continued collaboration relevant dimensions of the underlying processes of science, with the U.S. Bureau of Economic Analysis on the technology, and innovation that contribute to economic linked dataset (Recommendation 5-2); and growth and societal well-being. To be relevant, indicators – developing a suite of indicators that track the should also be available on a timely basis and widely acces- development and diffusion of general-purpose sible to the data user community. A concern with quality technologies, including information and commu- means that indicators should be carefully evaluated for their nication technologies, biotechnology, nanotech- conceptual soundness, and all feasible steps should be taken nology, and green technologies, using BRDIS data
OCR for page 1
4 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION as well as patent and bibliometric data (Recom- topics as the following (relevant recommendations are in mendation 5-4). Chapters 4 through 7): • Better access to BRDIS data by NCSES staff is imperative for the timely distribution of new and • organizational and market innovations, as well as improved BRDIS-based indicators (Recommenda- innovations in training and design; tion 4-3). • hindrances to the innovation process; • NCSES should also draw on longitudinal datasets on • new measures of innovation based on business prac- occupations and education levels to create indicators tice data obtained through administrative records and of labor mobility. Data from the Survey of Doctorate web-based data; Recipients, the Longitudinal Employer-Household • knowledge networks that contribute to innovation; Dynamics Study, and the Baccalaureate and Beyond • improved measures of labor mobility, career paths, Longitudinal Study would be particularly useful for stay rates for students at various levels of education, understanding the correspondence between supply wages and salaries by skill set, and demand and sup- and demand for skill sets in science and technology ply of skill sets in various industries; sectors and worker mobility (Recommendations 6-1 • international trade in technological goods and and 6-2). services; • An important prerequisite for linking data from dif- • emerging regions for entrepreneurial activity in sci- ferent sources is the development of a consistent ence and technology; and taxonomy of science and engineering fields and occu- • precommercialized inventions, to shed light on the pations (including the health and social sciences). early stages of the innovation process. NCSES should engage with other statistical agencies, including but not limited to the Bureau of Labor Sta- Chief Analyst (Recommendation 8-5) tistics, the U.S. Census Bureau, the National Center for Education Statistics, and the National Institutes of The panel recommends that NCSES establish a unit to Health, to develop this taxonomy and to establish a manage data quality assurance, cooperation with other orga- process for updating it as needed (Recommendation nizations, and focused analysis to support the development 2-2). of its indicators program and underlying datasets. Estab- lishment of such a unit would enable NCSES to more fully embody an important practice for federal statistical agencies, Methodological and Substantive Research Through a which is to have an active research program on the substan- Community of Practice (Recommendations 8-3 and 8-4) tive issues for which the agency compiles information and The panel recommends that NCSES build a community to understand how that information is used for policy and of practice around existing and emerging methodological decision making (National Research Council, 2013b, p. 22). issues so it can update its data acquisition and analysis NCSES should include within this unit a new position of techniques to support new and revised STI indicators. Such chief analyst, whose role would be to interface with users a community should include not only NCSES staff but of NCSES data, including indicators; provide other NCSES also, given constrained staff resources, outside researchers. staff with periodic updates on areas of change that may affect NCSES should leverage its existing grants and fellowship the agency’s statistical operations; and assess the utility of programs to support methodological research that addresses new types of datasets and tools that NCSES could use either its specific needs. in house or by contractual arrangement. Relatedly, the panel recommends that NCSES make its The panel believes a chief analyst would better enable data holdings available to external researchers on a timely NCSES to keep up to date with changing demand for STI basis to facilitate their research while protecting confidential- indicators and other data products; with socioeconomic ity and privacy. The timeliness with which NCSES delivers changes that have implications for STI indicators; and with indicators to user communities depends on its own access to new ways of collecting, acquiring, analyzing, and dissemi- data resources, primarily surveys and censuses, but increas- nating the most relevant, accurate, and timely indicators for ingly other sources as well, such as databases that involve policy and research use. Through adopting this recommenda- text processing. tion, along with the other strategic and specific recommenda- Outside researchers can help NCSES address meth- tions in the panel’s report, NCSES will be well positioned odological issues entailed in improving the accuracy and to carry out its role from the America COMPETES Reau- timeliness of new and revised STI indicators and evaluating thorization Act of 2010 to “collect, acquire, analyze, report, the usefulness of new kinds of data for indicator production. and disseminate statistical data related to the science and External researchers can also help NCSES develop new engineering enterprise in the United States and other nations and improved indicators that are relevant to policy on such that is relevant and useful to practitioners, researchers, poli- cymakers, and the public.”