National Academies Press: OpenBook
« Previous: 7 A Paradigm Shift in Data Collection and Analysis
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

8

Informing the Strategic Planning Process

Collectively, the recommendations on science, technology, and innovation (STI) indicators offered in Chapters 2-7 constitute a program of work for the National Center for Science and Engineering Statistics (NCSES) that accords with NCSES’s obligations as a statistical agency and the requirements of the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (America COMPETES) Reauthorization Act of 2010 (see Chapter 1). The panel understands that resource constraints will not permit all of its recommendations to be pursued to the same extent or on the same time table, so that it will be necessary for NCSES to develop a strategic plan (as recommended in Recommendation 3-1 in Chapter 3), followed by an implementation plan.

This chapter presents five recommendations intended to inform NCSES’s strategic planning process for the future of its program of STI indicators. The first strategic recommendation is to accord priority to data quality, broadly defined. Then come recommendations for four strategic processes or pathways for development: data linkage and sharing, methodological research, user access to NCSES data and collaborative research on new and revised STI indicators, and establishment of a chief analyst position at NCSES. Although data quality is the principal priority for a statistical agency, the setting of relative priorities among the remaining four strategic recommendations must be done by NCSES. Once strategic priorities have been developed, NCSES can use the recommendations from Chapters 2-7 to lay out the requirements and priorities for the implementation plan that should accompany the strategic plan.

This chapter begins by noting the many strengths of the current program of STI indicators at NCSES. It then outlines pathways for moving the program forward. The final section of the chapter maps the recommendations in Chapters 2-7 to the five strategic recommendations to form a program of work that will enable NCSES to develop policy-relevant STI indicators to better meet user needs.

NCSES’S ACCOMPLISHMENTS

NCSES has displayed several areas of strength in its data and statistical products, its interactions with domestic and foreign statistical agencies, and its outreach to the research community. The panel offers the following specific observations about NCSES’s accomplishments:

  • NCSES provides comprehensive information to the National Science Board, whose members find this service highly useful. A broad range of users (academic and government) also rely on STI indicators produced by NCSES that these users believe they cannot obtain reliably from other sources.
  • The panel identified no little-known, proven STI indicators and methodologies used by other countries that could easily and inexpensively be adopted by NCSES. The panel identified new and revised indicators to recommend, but essential indicators are covered in the National Science Board’s Science and Engineering Indicators biennial volume prepared by NCSES and other NCSES publications.
  • NCSES is recognized worldwide by statistical agencies and organizations in other countries as a leader in the development and curation of human resources statistics. This is a well-deserved reputation. NCSES might consider publishing periodic highlights (similar to highlights published by the research arm of the National Science Foundation [NSF]), with specific references to data series and statistics that have been shown to be pivotal in policy decisions. These highlights might include lists of peer-reviewed published articles that use NCSES’s microdatasets or published statistics.
  • NCSES has undertaken numerous productive collaborative efforts with other federal statistical agencies, including the use of data on global multinational research and development (R&D) activities from
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
  • the U.S. Bureau of Economic Analysis (BEA) and a wide range of statistics from the National Center for Education Statistics.
  • The recent development of the Business Research and Development and Innovation Survey (BRDIS) instrument has produced needed statistics on national and international R&D expenditures and performance. BRDIS is a first step toward obtaining data on the diffuse nature of innovation and R&D in the United States and around the world.
  • NCSES took the initiative to rationalize its human resources surveys by eliminating the National Survey of Recent College Graduates (NSRCG), relying instead on a sampling frame from the American Community Survey to incorporate information on recent science and engineering graduates into the National Survey of College Graduates (NSCG). This was a necessary and well-designed cost reduction strategy.1
  • The U.S. Census Bureau established a microdata Survey Sponsor Data Center at NCSES’s location, giving NCSES staff ready access to microdata collected for NCSES by the Census Bureau, including from BRDIS and the NSCG. This should reduce the coordination costs of accessing the data and, once the process for accessing the data is fully operational, enable NCSES staff to accomplish more timely analysis and reporting of analytical statistics and indicators. NCSES staff have also worked to facilitate greater access by outside researchers to its surveys of science and engineering personnel through a secure data enclave operated by NORC at the University of Chicago.
  • NCSES staff have been leaders in various international forums, especially OECD, where NCSES officials have consistently provided leadership to the Committee on Scientific and Technological Policy’s expert statistical working party, the National Experts on Science and Technology Indicators (NESTI). In this role, NCSES staff have led work to improve the international comparability of a variety of STI indicators, including patent-based indicators, the careers of doctorate holders, statistics on Chinese STI performance, and cognitive testing of surveys on R&D and innovation.

PROCESSES FOR CHANGE

NCSES products, especially the biennial Science and Engineering Indicators volume for the National Science Board and the agency’s InfoBriefs, are highly regarded and widely used. However, they reflect a fraction of the information present in NCSES’s microdata holdings, let alone what would be available if these microdata were linked to the holdings of other federal agencies to expand their analytical potential. NCSES thus has an opportunity to produce much more output without having to invest in major new data collection. Taking advantage of this opportunity would enable NCSES to increase its contribution to the public good and provide users of its data, expert or novice, with deeper insights into the workings of the science and engineering enterprise in the United States.

For these new products to be produced, decisions will have to be made and priorities set, as accomplishing this will likely require a somewhat different skill mix from that currently present in NCSES, and there are other resource implications as well. Given that NCSES is limited in the number of staff it can employ, the production of new products will require a clear vision and a commitment to resource reallocation over a significant period of time. The necessary decisions cannot be made in isolation from other decisions that are part of managing the agency and the expectations of the users of its data. These decisions must be part of a strategic plan that provides a vision for the organization and lays out how the goals that support that vision are to be achieved.

This panel cannot provide a strategic plan for NCSES. Formulating such a plan requires priority setting, resource reallocation, and monitoring processes that only NSF can undertake using information on likely resource levels and other factors to which NSF alone is privy. However, the panel can, and does, make recommendations (presented later in this chapter) that can inform the strategic planning process. As outlined earlier, these recommendations deal with data quality (Recommendation 8-1), which is multidimensional but fundamental for a statistical agency; data linkage and sharing with other agencies, both inside and outside of government (Recommendation 8-2); the need for a program of methodological research (Recommendation 8-3); the building of a community of practice engaged in the use of data managed by NCSES, both its own and from other sources, which can support not only methodological but also substantive research on new and revised STI indicators (Recommendation 8-4); and the establishment of the position of a chief analyst who would interface with the users of NCSES’s products to monitor changes in demand and would make recommendations on methods and data sources for the STI indicators program (Recommendation 8-5).

It should be made clear that the panel’s recommendations in this chapter are hardly new in the sense that most of these areas have been touched on by previous National Research Council (NRC) reports. NCSES is encouraged to review the NRC (2005) report Measuring Research and Development Expenditures in the U.S. Economy. That report, although focused on R&D expenditures, deals with many of the issues that are addressed in this chapter and quotes earlier reports examining similar issues.

What makes the recommendations in this chapter even more important now is that the global position of the United States has changed, particularly with the emergence of

____________________

1The last year of data collection using the NSRCG was 2010.

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

competencies in R&D and innovation and of human capital skills in science and technology within emerging economies around the world. As the United States competes in a global economy, its policy makers, business managers, educators, and residents need to understand what is happening to the science and engineering enterprise so they have the informed capacity to advise, consent, and act to improve it. That understanding can come, in part, from more and better use of NCSES data and statistics, as well as other metrics produced by federal agencies and other organizations.

Data Quality

As NCSES is a statistical agency, it is bound by data quality considerations, and that is where the recommendations in this chapter begin. The above-referenced NRC report notes four components of data quality—accuracy, relevance, timeliness, and accessibility (National Research Council, 2005, p. 11). That report focuses on accuracy; this panel regards all four elements as important and interrelated.2 Accuracy is essential for all indicators, but relevance can be lost if the data release is not timely. Accessibility matters as it allows researchers from other institutions to work with the data, leveraging NCSES’s limited analytical capacity, but also imposing an obligation for NCSES to protect the confidentiality of the data and the need to train external users in how to access and manipulate the data.

A special emphasis on timeliness is warranted because users whose views were solicited for this study were emphatic that certain types of indicators lose relevance if they are not made available within a short time after the date of observation—providing indicators that are 2-3 years out of date is not helpful. For NCSES, greater timeliness could be achieved in several ways, including (1) the release of preliminary data and statistics, which is common for leading economic indicators from other statistical agencies, although users will need to be cognizant of the greater uncertainty of the estimates; (2) nowcasting and other techniques discussed in Chapter 7 of this report, although their use will require changes in skill sets or contractual relationships at NCSES; and (3) allocating resources to more timely release of final products by reengineering every component of the process from survey development and sourcing of raw data through publication processes and media for distribution. NCSES’s strategic planning process for its STI indicators program should specifically address methods to be used to respond to users’ needs for timeliness.

The panel is aware that NCSES surveys are contracted to other organizations, but the production of indicators to meet NCSES’s data quality requirements, including timeliness, should be a contractual obligation. Data quality indicators are necessary to enable the identification of quality deficiencies and to develop methods for data improvement. As earlier NRC reports provide specific advice about what kinds of quality indicators NCSES should monitor and disseminate for its data collection programs, including unit nonresponse, item nonresponse, and population coverage, the following recommendation does not go into further detail.3

RECOMMENDATION 8-1: Given the fundamental importance of data quality and reporting of quality measures, the National Center for Science and Engineering Statistics should review its data quality framework, establish a set of indicators for all of its surveys, and publish the results at regular intervals and at least annually.

Data Linkages and Sharing

Once data are seen to conform to quality assurance indicators, analysis of the data can follow, leading to new, relevant, and timely products addressing issues of importance to the policy community and to other users of the information. Datasets need not come only from NCSES; other agencies produce data that fall within the charge given to NCSES by the America COMPETES Act, and there are data from nongovernmental sources as well.

The usefulness of a single dataset can be increased if it is linked to other datasets of comparable quality, whether within NCSES or in collaboration with other agencies. Collaboration with other agencies on data linkages activities, which in some cases is already taking place, would lead to working-level knowledge of how other agencies manage data. In due course, the ongoing exchange of information would enable NCSES to assume the clearinghouse function required under the America COMPETES Act, which mandates that NCSES “shall serve as a central Federal clearinghouse for the collection, interpretation, analysis, and dissemination of objective data on science, engineering, technology, and research and development.”

Better integration of data sources is needed to develop more robust STI indicators. At the panel’s July 2011 workshop, John Haltiwanger of the University of Maryland described how infrastructure datasets could be fully integrated to track the career histories of innovators and entrepreneurs and the relationships between start-up, young, high-growth firms and large, mature firms (see Chapter 4 for more detail). These infrastructure datasets could be fully integrated with all existing Census Bureau business surveys and other data. For example, one could integrate economic censuses and annual surveys to measure productivity, business practices, and R&D, linked to patent, citation, and

____________________

2See also National Research Council (2013b).

3One example of a useful tool for communicating various elements of the quality of data and statistics to users is the U.S. Census Bureau’s website for the American Community Survey (U.S. Census Bureau, 2013). The panel recognizes that web-based links like this are rare, and that NCSES and other statistical agencies typically give this type of information in technical notes that accompany data releases.

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

other information about innovators from the U.S. Patent and Trademark Office.

Any new STI indicators that are developed will need to be integrated into the existing infrastructure (if not at the person/business level, then at some level of disaggregation). Data sharing and synchronization would permit even richer integration of Bureau of Labor Statistics (BLS) and BEA firm data. At the panel’s July 2011 workshop, Matthieu Delescluse of the European Commission remarked that the European Union (EU) is commissioning the linking of patent data with company databases to support the development of new indicators. For example, this type of linking will make it possible to track the relationship between small and medium firms and the number of patents over time. The EU is also using data from Community Innovation Survey Business Registers for member countries to determine the international sourcing of R&D. This statistic could also be developed in the United States through the linking of Census Bureau and BEA data. Employment dynamics, including worker mobility trends in science and engineering occupations, could be developed by linking Census Bureau, BLS, and BEA data. Existing research data centers or data enclaves could facilitate platforms for data integration, potentially making the data comparable with those of other nations that have similar data administration policies while protecting the confidentiality of the information.

RECOMMENDATION 8-2: The National Center for Science and Engineering Statistics should work with other federal agencies bilaterally and in interagency statistical committees to share data, link databases where feasible, and produce products that would not be possible if the agencies worked independently. The use of data from outside the federal system, where appropriate, should be part of this process.

Methodological Research

The production of datasets that conform to quality standards and meet user needs will almost certainly necessitate research on methodological questions within NCSES and in the wider community to provide optimum solutions. A critical way in which to identify important methodological issues is to work with the data, which typically leads to the discovery of errors, inconsistencies, and gaps. This is why data analysis by staff responsible for the data matters—it is a quality issue.

Not all of the methodological issues raised by data analysis—whether by staff or outside researchers—can be solved by staff or by methodologists in other agencies. NCSES has an opportunity, through its own and other NSF granting programs, to support research on the methodology of data production, including survey methods, data linkage, estimation, and quality control. NCSES could also issue contracts to address specific requirements, but the advantage of an NSF methodological research program that met the immediate needs of NCSES is that it would, over the years, result in a community of methodologists familiar with the agency’s work and serve as an invaluable resource for staffing and peer review of new initiatives.

RECOMMENDATION 8-3: The National Center for Science and Engineering Statistics should use its existing Grants and Fellowships Program and related programs at the National Science Foundation to support methodological research that addresses the agency’s specific needs.

Building of a Community of Practice in the Use of NCSES and Other Data

If NCSES is to advance its analytical capacity beyond the interpretation of tables and management of its biennial indicators report, then it will have to make greater use of its own data. In addition, these data can and should be used by researchers outside of NCSES to produce more analytical material and to build a community of practice that is knowledgeable about NCSES datasets. Such a community of practice could expand the range of outputs based on these datasets, including new and revised STI indicators; contribute to methodological improvements; and enhance the evident public good of NCSES’s surveys. Including other federal agencies in such collaboration would contribute to the knowledge and relationships NCSES needs to move toward assuming the clearinghouse function mandated by the America COMPETES Act.

Researchers would have to be trained in the use of NCSES’s datasets and their work monitored to ensure that there would be no breach of confidentiality or privacy. Again, this would require resource allocation and a commitment to working with outside communities. The panel stresses the importance of greater use of the data, or greater accessibility, to use the data quality term. A good start has been made in that direction for the science and engineering personnel surveys.

RECOMMENDATION 8-4: The National Center for Science and Engineering Statistics should make its data holdings available to external researchers to facilitate their research while protecting confidentiality and privacy.

Development of a Chief Analyst Position

The four recommendations offered thus far in this chapter relate to data quality, broadly defined, as a bedrock of NCSES’s STI indicators program, data linkages and sharing to permit the development of new and improved indicators, methodological research on indicators and the underlying data, and the building of a community of practice to leverage

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

the work of NCSES staff. The need to manage these activities suggests the need for a separate unit within NCSES led by a senior staff member who would be responsible for these activities, including building the internal team to carry them out and managing the necessary links with other organizations. This unit would monitor new indicators emerging across the globe and would have the capacity to absorb the knowledge needed to bring these new indicators into NCSES and present them to users for their evaluation. A close relationship and regular communication between members of this analytical unit and the survey units in NCSES would be necessary.

The new unit should include a position of chief analyst, analogous to the current position of chief statistician but with a different portfolio. The chief analyst should not manage the unit but instead provide substantive leadership and communication across units within NCSES and with other agencies and organizations to develop NCSES’s STI indicators program in ways that are most useful for the policy and research communities.

Taking these steps would facilitate NCSES’s ability to more fully embody Practice 10 in Principles and Practices for a Federal Statistical Agency, Fifth Edition. Practice 10 calls for a federal statistical agency to have an active research program, including not only methodological research, but also research “on the substantive issues for which the agency’s data are compiled, … [for] the identification and creation of new statistical measures, … [and] to understand how the agency’s information is used, in order to make the data more relevant to policy concerns, and more useful for policy research and decision making” (National Research Council, 2013b, p. 22).

RECOMMENDATION 8-5: The National Center for Science and Engineering Statistics (NCSES) should establish a unit to manage data quality assurance, cooperation with other institutions, and analysis for its science, technology, and innovation indicators and related programs. NCSES should develop a new position of chief analyst within this unit whose role would be to (1) interface periodically with users of NCSES data and statistics, including indicators, so the agency can remain up to date on changing demand for its products; (2) provide NCSES staff with periodic updates on areas of change that are likely to have an impact on the agency’s statistical operations; and (3) assess the utility of new types of datasets and tools that NCSES could use either in house or by contractual arrangement to develop and improve indicators.

A PROGRAM OF WORK

Once the strategic planning proposed in Recommendation 3-1 has been undertaken and informed by the five recommendations offered in this chapter, those five recommendations can be linked to the recommendations in the previous chapters to constitute a program of work for NCSES for its STI indicators. Implementing the panel’s recommendations would support the development of the capacity within NCSES to meet the requirements of the America COMPETES Act, especially SEC. 505 (b) (2) and (3) and (c). Table 8-1 shows which recommendations in Chapters 2-7 fall under the recommendations in this chapter; the full text of the recommendations in Chapters 2-7 is provided in the boxes at the end of this chapter.

Considerations for Prioritization

The panel believes that all pathways outlined in this chapter are important for NCSES to include in its strategic plan for its STI indicators program, although the agency will undoubtedly need to prioritize them with respect to the pace and specifics of implementation. As part of implementing the strategic plan, NCSES will also need to assign priorities for the detailed program of work along each pathway. Prioritization requires gauging which policy issues are likely to be important in the near, medium, and longer terms; assessing which policy issues might be informed by specific STI indicators; estimating the benefit versus the cost of developing indicators in house or obtaining them through contractual relationships or from scholarly research; and focusing on those indicators that would cost-effectively shed light on important policy questions. The key to this prioritization process is the recognition that not all indicators need to be sourced through traditional means such as surveys.

While, as noted earlier, the panel is not in a position to set priorities for NCSES’s STI indicators program, Table 8-1 reveals that some recommendations are pertinent for progress along more than one strategic pathway, which suggests a relative importance. Likewise, some of the recommendations are contingent on the implementation of others, again suggesting a relative importance. Linkages between the pathways and the programmatic recommendations are summarized below.

Pathways and Programmatic Recommendations

As is appropriate for a statistical agency, all the panel’s recommendations relate to data quality, Recommendation 8-1. At the same time, the recommendations in Chapters 2-7 entail actions by NCSES along four strategic pathways, corresponding to Recommendations 8-2 through 8-5, respectively.

Data Sharing and Linkage (Recommendation 8-2)

First is the development of new policy-relevant, internationally comparable indicators that are based on existing NCSES survey data and on data collections at other statistical

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

TABLE 8-1 Linking the Recommendations in Chapters 2-7 to the Strategic Pathways in Chapter 8 to Form a Strategic Program of Work

Chapter 8 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Chapter 7
Recommendation 8-2—Data Linking and Sharing Rec. 2-2 Rec. 3-1 Recs. 4-2 4-3, 4-4 Recs. 5-1, 5-2, 5-3, 5-4 Recs. 6-2, 6-4, 6-5, 6-6 Rec. 7-5
Recommendation 8-3—Methodological Research Rec. 2-1 Rec. 3-1 Recs. 4-1, 4-2, 4-4, 4-5 Recs. 5-1, 5-2, 5-4 Recs. 6-1, 6-2, 6-3, 6-4, 6-5, 6-6, 6-7 Recs. 7-1, 7-2, 7-4, 7-6, 7-7
Recommendation 8-4—Use of NCSES and Other Data Rec. 2-1 Rec. 3-1 Rec. 4-2   Rec. 6-1 Recs. 7-3, 7-5
Recommendation 8-5—Chief Analyst   Rec. 3-1 Rec. 4-1   Recs. 6-2, 6-3 Recs. 7-2, 7-7

NOTE: All recommendations link to Recommendation 8-1 on data quality indicators. NCSES = National Center for Science and Engineering Statistics.

agencies, both inside and outside the government, using data sharing and linkage techniques as appropriate.

 

  • A critical aspect of this effort is the development of integrating processes to leverage synergies within NCSES and collaborations outside NCSES at the same time that new data extraction and management methods for generating statistics are explored. A prerequisite for linking data from different sources is the development of a consistent taxonomy of science and engineering fields and occupations (Recommendation 2-2).
  • NCSES has many data series that have not been analyzed but have great potential to help answer questions posed by users. Recommendation 3-1 stresses the importance of developing new measures from existing data. This is very apparent for the innovation indicators.
  • BRDIS data already provide the input that could allow for analysis of comparative statistics on innovation by the same cutoffs for firm size used by the OECD countries (Recommendation 4-2); for cross-tabs on R&D and innovation spending in the United States and abroad, by firm characteristics (Recommendation 4-2); and for analysis of yield measures of activities by high-growth firms and of firm dynamics in terms of births and deaths of businesses linked to innovation outputs (Recommendation 4-4). Although some of the results require linkage with data from other agencies, these types of outputs would be welcomed by the user community. Better access to BRDIS data by NCSES staff is imperative for the timely distribution of such statistics to the users of STI indicators (Recommendation 4-3).

Methodological Research (Recommendation 8-3)

Second is the need for NCSES to build a community of practice around existing and emerging methodological issues if it is to update its data acquisition and analysis techniques.

  • Given resource constraints, and to avail itself of methodological best practices, NCSES needs to use existing grants programs and input from researchers and practitioners (Recommendations 2-1 and 3-1, respectively) to learn about the usefulness of its own data and new techniques for developing more useful indicators.
  • New measures are needed, including specific cutoffs for innovation statistics; measures of organizational and market innovations, as well as innovations in training and design; measures for understanding hindrances in the innovation process; and the use of business practice data (e.g., administrative records, web-based data) to produce new, timely indicators of innovation (Recommendations 4-1, 4-2, 4-4, and 4-5). Users of indicators also want to see improved measures of knowledge networks (Recommendation 5-1) and payments and receipts for R&D services (Recommendation 5-2), as well as indicators that track the development and diffusion of general-purpose technologies (Recommendation 5-4).
  • New and revised human capital indicators for labor mobility, career paths, stay rates for students at various levels of education, wages and salaries by skill set, and demand and supply of skill sets in various industries (Recommendations 6-1 through 6-7) also will depend on the cultivation of existing datasets and the development of new techniques for using business practice data (Recommendations 7-1, 7-2, 7-4, 7-6, and 7-7).
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

Wider Use of Data (Recommendation 8-4)

The development of new measures and methods for STI indicators will require research and implementation strategies that should be developed collaboratively among NCSES staff and the research and practitioner communities. NCSES will need to expand its mechanisms for providing researchers access to data so it can leverage its own limited staff resources. NCSES will also need to address directly the issue of timeliness, given that in many cases, the utility of indicators and the underlying databases to user communities is inversely related to the lag between when the data were sourced and when the indicators are publicly released. Building another community of practice engaged in the use of data managed by NCSES, both its own and from other sources, is important to address this timeliness issue, as well as to contribute to the methodological research described under Recommendation 8-3 above.

  • The timeliness with which NCSES delivers indicators to user communities depends on its own access to data resources, primarily from surveys, but increasingly from other sources as well (Recommendations 2-1, 3-1, 4-2, 6-2, 7-3, and 7-5).
  • Nontraditional methods hold promise for delivering information to users more quickly, but the panel recognizes that many of these methods are still in the exploratory stage. NCSES could play an important role in supporting research to advance the utility of these methods (see in particular Recommendations 7-3 and 7-5).

Chief Analyst (Recommendation 8-5)

Fourth and last, establishment of a chief analyst position would improve NCSES’s interface with the users of indicators, allowing the agency to monitor changes in demand. The chief analyst would also engage with users to observe and make recommendations on methods and data sources of particular relevance for the STI policy community (Recommendations 3-1, 4-1, 6-2, 6-3, 7-2, and 7-7). This role implies the forging of a balanced approach to governance of NCSES activities that span data collection and analytical processes. The goal is a process of feedback and improvement for NCSES’s STI indicators program—and the data collections on which it draws—to assist policy makers and other users in understanding the evolving U.S. science and engineering enterprise.

PANEL RECOMMENDATIONS: CHAPTERS 2-7

Chapter 2: Concepts and Uses of Indicators—Outreach Recommendations

RECOMMENDATION 2-1: The National Center for Science and Engineering Statistics should continue its Grants and Fellowships Program for using its datasets, maintaining the high National Science Foundation standards for peer-reviewed award decisions.

RECOMMENDATION 2-2: The National Center for Science and Engineering Statistics should engage with other statistical agencies, including but not limited to the Bureau of Labor Statistics, the U.S. Census Bureau, the National Center for Education Statistics, and the National Institutes of Health, to develop a consistent taxonomy of science and engineering fields and occupations (including the health and social sciences). There should also be an established process for performing updates of this taxonomy as needed.

Chapter 3: Data Resources for Indicators—Prioritization Recommendation

RECOMMENDATION 3-1: In the near term, the National Center for Science and Engineering Statistics (NCSES) should work to produce new and revised science, technology, and innovation indicators in a few key areas, using existing data from the Business Research and Development and Innovation Survey and the Scientists and Engineers Statistical Data System or from productive collaborations with other statistical agencies in the United States and abroad. Over time, NCSES should build capacity in house and through its Grants and Fellowships Program to develop measures that are high priority for users but that require deeper knowledge to obtain statistically valid data or to use frontier methods appropriately. NCSES should also develop a strategic plan for adding new indicators or case studies because doing so may require curtailing the frequency of some of its current measures.

Chapter 4: Measuring Innovation—Recommendations

RECOMMENDATION 4-1: The National Center for Science and Engineering Statistics should develop additional indicators for measuring innovation outcomes that would complement existing data on patents, inputs to innovation

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

activities, and broader measures of economic performance.

RECOMMENDATION 4-2: The National Center for Science and Engineering Statistics should build on its Business Research and Development and Innovation Survey (BRDIS) to improve its suite of innovation indicators in the following ways:

  • tabulate the results from BRDIS using the same cutoffs for firm size (as well as comparable industry sectors) that are used by OECD countries in order to facilitate international comparisons;
  • fund research exploring precisely what companies mean when they report an innovation or report no innovation on BRDIS—such research would help inform current policy debates;
  • broaden the innovations tracked by BRDIS to encompass organizational and marketing innovations, as well as new data algorithms;
  • consider adding a section to BRDIS on unmarketed innovations, giving respondents the opportunity to cite the main reason these innovations have not yet been marketed or implemented;
  • as funds permit, extend BRDIS to gather information on innovation-related expenditures in such areas as training and design; and
  • publish more results from BRDIS that link innovation to business characteristics, including the amount of research and development spending by U.S.-based companies outside of the United States. Production and distribution of such cross-tabulations should be timely, and they should address contemporary policy questions.

RECOMMENDATION 4-3: The Survey Sponsor Data Center at the National Science Foundation should house the Business Research and Development and Innovation Survey data, improving access to the data for National Center for Science and Engineering Statistics staff who develop the research and development statistics.

RECOMMENDATION 4-4: The National Center for Science and Engineering Statistics (NCSES) should begin a project to match its Business Research and Development and Innovation Survey data to data from ongoing surveys at the U.S. Census Bureau and the Bureau of Labor Statistics. It should use the resulting data linkages to develop measures of activities by high-growth firms, births and deaths of businesses linked to innovation outputs, and other indicators of firm dynamics, all of which should be tabulated by geographic and industry sector and by business size and business age to facilitate comparative analyses. NCSES should conduct a sensitivity analysis to fine-tune meaningful age categories for high-growth firms.

RECOMMENDATION 4-5: The National Center for Science and Engineering Statistics should make greater use of business practice data to track research and development spending and innovation-related jobs at a more detailed geographic and occupational level than is possible with government survey data.

Chapter 5: Measuring the Three K’s: Knowledge Generation, Knowledge Networks, and Knowledge Flows—Recommendations

RECOMMENDATION 5-1: The National Center for Science and Engineering Statistics should expand its current set of bibliometric indicators to develop additional measures of knowledge flows and networking patterns. Data on both coauthorship and citations should be exploited to a greater extent than is currently the case.

RECOMMENDATION 5-2: The National Center for Science and Engineering Statistics (NCSES) should make greater use of data from its Business Research and Development and Innovation Survey to provide indicators of payments and receipts for research and development services purchased from and sold to other countries. For this purpose, NCSES should continue collaboration with the U.S. Bureau of Economic Analysis on the linked dataset.

RECOMMENDATION 5-3: The National Center for Science and Engineering Statistics (NCSES) should continue to report statistics on knowledge-based capital and intangible assets obtained from other agencies as part of its data repository function. In addition, NCSES should seek to use data from the Business Research and Development and Innovation Survey on research and development and potentially also on innovation-related expenditures as valuable inputs to ongoing work in this area.

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

RECOMMENDATION 5-4: The National Center for Science and Engineering Statistics (NCSES) should develop a suite of indicators that can be used to track the development and diffusion of general-purpose technologies, including information and communication technologies, biotechnology, nanotechnology, and green technologies. NCSES should attempt to make greater use of data from the Business Research and Development and Innovation Survey for this purpose while also exploring the use of other sources, such as patent and bibliometric data.

Chapter 6: Measuring Human Capital—Recommendations

RECOMMENDATION 6-1: The National Center for Science and Engineering Statistics (NCSES) should do more to exploit existing longitudinal data. Specifically, NCSES should exploit the longitudinal panel structure of the Survey of Doctorate Recipients (SDR) in the following ways:

  • create indicators of researcher mobility over time, by constructing longitudinal weights for the SDR that take account of changes in the sample and target population over time—these weights should be constructed both for subsequent survey cycles and for existing data;
  • create a dynamic database for researcher use in which data from the SDR over time would be linked at the level of the individual; and
  • enhance coverage of recent doctorate recipients to better track their initial employment and career path in the first years after they receive their Ph.D, which could potentially be accomplished by including an additional module in the SDR or by exploiting that survey’s longitudinal capacities or both.

RECOMMENDATION 6-2: The National Center for Science and Engineering Statistics (NCSES) should draw on the Longitudinal Employer-Household Dynamics Program (occupations) and the Baccalaureate and Beyond Longitudinal Study (education levels) to create indicators of labor mobility. NCSES should focus in particular on industries that have been experiencing high growth and/or those in which the United States has a strong competitive advantage. Also relevant would be examining skill sets of firms with high growth.

RECOMMENDATION 6-3: The National Center for Science and Engineering Statistics (NCSES) should provide indicators for individual science, technology, engineering, and mathematics groups such as early-career doctorate recipients, master’s degree holders, and community college graduates. NCSES already distinguishes between bachelor’s and master’s degree holders in many of its statistics. Stay rates at different education levels by demographic characteristics such as gender, race/ethnicity, disability, and country of origin should be included.

RECOMMENDATION 6-4: The National Center for Science and Engineering Statistics should explore whether questions can be included in the National Survey of College Graduates and the American Community Survey that would allow the identification of community college graduates or of holders of higher university degrees who have attended a community college.

RECOMMENDATION 6-5: The National Center for Science and Engineering Statistics should explore methods for exploiting the full-text resources of dissertation databases to create indicators on selected topics both within and across scientific fields and on the relatedness of different fields.

RECOMMENDATION 6-6: The National Center for Science and Engineering Statistics should consider using American Community Survey data to produce indicators that can be used to track the salaries of science, technology, engineering, and mathematics occupations and/or college graduates receiving degrees in different fields and at different degree levels.

RECOMMENDATION 6-7: The National Center for Science and Engineering Statistics should consider adding questions to the Business Research and Development and Innovation Survey on the types of skill sets used by businesses to develop and implement innovations. The results would provide data on and indicators of innovative firms’ demand for skills.

Chapter 7: A Paradigm Shift in Data Collection and Analysis—Recommendations

RECOMMENDATION 7-1: The National Center for Science and Engineering Statistics (NCSES) should use research awards to support

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×

the development and experimental use of new sources of data to understand the broad spectrum of innovation activities and to develop new measures of science, technology, and innovation. NCSES should also support the development of new datasets to measure changing public perceptions of science, international trade in technological goods and services, new regions for entrepreneurial activity in science and technology, and precommercialized inventions.

RECOMMENDATION 7-2: The National Center for Science and Engineering Statistics (NCSES) should pursue the use of text processing for developing science, technology, and innovation indicators in the following ways:

  • explore synergies with National Science Foundation directorates that fund research on text processing; and
  • enable NCSES staff to attend and develop workshops that bring together researchers working on text processing and on understanding the science, technology, engineering, and mathematics ecosystem.

RECOMMENDATION 7-3: The National Center for Science and Engineering Statistics should use its grants program to encourage the sharing of new datasets and extracted metadata among researchers working on understanding innovation in science, technology, engineering, and mathematics.

RECOMMENDATION 7-4: The National Center for Science and Engineering Statistics should coordinate with directorates at the National Science Foundation in supporting exploratory research designed to validate new sources of data related to innovation in science, technology, engineering, and mathematics.

RECOMMENDATION 7-5: The National Center for Science and Engineering Statistics should explore the use of university-industry exchanges as a mechanism for giving researchers access to promising datasets and industry teams access to new research techniques.

RECOMMENDATION 7-6: The National Center for Science and Engineering Statistics (NCSES) should collaborate with university researchers on the use of data science techniques to understand the science, technology, engineering, and mathematics ecosystem, using a mechanism similar to existing National Science Foundation university-industry partnerships. One or two graduate students or postdoctoral fellows could alternate between working at NCSES and at their home institution for up to 2 years, with the specific goal of contributing new findings to NCSES’s data and indicators programs.

RECOMMENDATION 7-7: The National Center for Science and Engineering Statistics (NCSES) should explore methods of mining the awards database at the National Science Foundation as one means of discovering leading pathways for transformational scientific discoveries. NCSES should engage researchers in this exploratory activity, using its grants program. NCSES should develop mechanisms for using the tools and metadata developed in the course of this activity for the development of leading indicators of budding science and engineering fields.

Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 101
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 102
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 103
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 104
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 105
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 106
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 107
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 108
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 109
Suggested Citation:"8 Informing the Strategic Planning Process." National Research Council. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18606.
×
Page 110
Next: References »
Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy Get This Book
×
Buy Paperback | $64.00 Buy Ebook | $49.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since the 1950s, under congressional mandate, the U.S. National Science Foundation (NSF) - through its National Center for Science and Engineering Statistics (NCSES) and predecessor agencies - has produced regularly updated measures of research and development expenditures, employment and training in science and engineering, and other indicators of the state of U.S. science and technology. A more recent focus has been on measuring innovation in the corporate sector. NCSES collects its own data on science, technology, and innovation (STI) activities and also incorporates data from other agencies to produce indicators that are used for monitoring purposes - including comparisons among sectors, regions, and with other countries - and for identifying trends that may require policy attention and generate research needs. NCSES also provides extensive tabulations and microdata files for in-depth analysis.

Capturing Change in Science, Technology, and Innovation assesses and provides recommendations regarding the need for revised, refocused, and newly developed indicators of STI activities that would enable NCSES to respond to changing policy concerns. This report also identifies and assesses both existing and potential data resources and tools that NCSES could exploit to further develop its indicators program. Finally, the report considers strategic pathways for NCSES to move forward with an improved STI indicators program. The recommendations offered in Capturing Change in Science, Technology, and Innovation are intended to serve as the basis for a strategic program of work that will enhance NCSES's ability to produce indicators that capture change in science, technology, and innovation to inform policy and optimally meet the needs of its user community.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!