3

Overview of National Center for Science and Engineering Statistics Data Collections, Data Products, and Its Previous Work in This Area

Gary Anderson (National Center for Science and Engineering Statistics [NCSES]) opened his presentation by describing the work that NCSES has done in the past decade, particularly the work that relates to definitions. In that time, several definitional issues arose as academic institutions and disciplines have developed their own definitions to characterize research that reaches across disciplines. Anderson noted the transition from the use of the terms “disciplinary,” “multidisciplinary,” and “interdisciplinary,” to the term “convergence.” Each term created definitional issues and brought attention to new aspects of research that drew on expertise from various disciplines. NCSES has worked through these definitional issues in various forums including both survey development and workshops.

To describe his evolution, Anderson said that the terms unidisciplinarity, multidisciplinarity, and transdisciplinarity can reflect different degrees of integration, transdisciplinarity having the highest level of integration. The levels of integration include how the results produced relate to their individual components, how individual researchers work together, and how the process can create a more valuable product. Based on definitions in the literature, Anderson and colleagues use the following definition of convergence: an approach to problem solving that cuts across disciplinary boundaries. Two major areas are most salient for defining convergence: the deep integration of disciplines and problem-oriented research.

Deep integration can be described as researchers becoming a whole unit rather than working alone or side by side. Problem orientation can be described as an emphasis on an endpoint problem that needs to be solved, be it societal or scientific. Anderson expressed a desire to further explore

these issues as convergence research continues and noted the potential it holds for broader societal impacts, as opposed to (uni)disciplinary advances.

Anderson and colleagues shared the findings of their initial background research with NCSES leadership and posed two broad questions: (1) What have NCSES leaders done, and what are they currently doing? (2) Is there a level of rigor and standardization that can be used nationally? These questions informed development of this workshop’s agenda and were further divided into five more granular questions:

1. What has NCSES done that contributes to the measurement of convergence and its impact?
2. Are there NCSES programs, experiments, or pilots related to measuring convergence work that might be used as a basis for workshop sessions?
3. Is the current working definition adequate for the creation of national indicators using current surveys and reports?
4. Are current methodological approaches for NCSES surveys and indicators appropriate for the measurement of convergence?
5. Are indicators based on nonsurvey techniques (bibliometrics and scientometrics of citations, as well as network analysis and modeling) adequate for assessing convergence research and its impact, and can or should the NCSES seek to develop them?

To investigate what NCSES has done and is doing, Anderson began with decades of emails, decision documents, white papers, conference reports, and formal publications on science and engineering indicators. Regarding where NCSES is going, Anderson believes that now is a good time to re-investigate convergence and its definition. As research becomes increasingly problem-oriented rather than merely discipline-oriented, focuses on societal impacts, and leans toward deep integration, NCSES leadership realizes the importance of measuring the impact of interdisciplinary research and of convergence.

Anderson discussed areas where NCSES is considering incorporating measures of interdisciplinarity through taxonomies and classifications of research. A 2012 workshop to discuss this possibility considered the need for NCSES to measure interdisciplinarity in the context of what the current empirical literature says about this, focusing on bibliometric datasets. The subjects discussed included methods for producing valid measurements and whether the characterizations were exhaustive and mutually exclusive, which is in turn related to apportionment issues involved with interdisciplinary research. Anderson then made a distinction between a top-down approach, which relies on existing taxonomies to measure disciplinarity (e.g., whether the authors represent various disciplines), versus

a bottom-up approach that is driven by data science and machine-learning methods such as topic modeling to identify emerging fields. He mentioned the possibility of a hybrid approach that would leverage and then combine topic modeling with the priorities of the administration or Congress to produce valid measures that investigate interdisciplinarity. By the end of that workshop, attendees concluded that interdisciplinary research and development (R&D) is difficult to detect, describe, and measure, and that the existing taxonomy of disciplines does not easily accommodate an exploration of emerging outcomes. Despite these difficulties, Anderson reiterated the contemporary importance of grappling with these issues.

NCSES has been deeply involved with developing guidance for international research and design standards for interdisciplinary work. It contributed extensively to the 2015 Organisation for Economic Co-operation and Development (OECD) revision of the Frascati Manual,¹ which provides guidance on the collection and reporting of R&D performance and funding data. The manual describes fields of research and development, providing guidance on the use of taxonomies to determine levels of interdisciplinarity in a project. The manual also offers a caveat that academia may find this guidance easier to work with than other sectors of the economy. Although interdisciplinary, the experimental development work of businesses, for example, might be difficult to measure.

Anderson explained that over the course of several years, NCSES has updated the taxonomies that are used to measure research to better reflect the types of R&D conducted at educational institutions. NCSES revised its survey taxonomy to increase standardization across statistical agencies and to ensure that research fields remain up to date. Maintaining a current list of taxonomies would allow users to find their respective field more easily within a long list of disciplines presented in a survey. Other changes included adding new disciplines within examples, reclassifying some disciplines into different fields, and adding four distinctly new fields. Despite these revisions, the classification of interdisciplinary R&D has remained generally unchanged and is referred to as Other Sciences or Other Nonscience and Engineering Fields. Other Sciences is used for R&D that involves at least one science and engineering (S&E) field and also involves research expenditures that cannot be attributed to specific fields. Other Non-S&E Fields is used to categorize research and involves multiple Non-S&E Fields when a specific field cannot be reported.

Anderson also presented results from the Higher Education Research and Development (HERD) survey and explained how the survey’s 2009

___________________

¹ Organisation for Economic Co-operation and Development. (2015). Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development. Paris: OECD Publishing. doi:10.1787/9789264239012-en.

redesign investigated the possible inclusion of an interdisciplinary measure. This effort began with conversations with individual data users and universities that revealed the importance of interdisciplinarity. This led NCSES to develop a pilot survey, debrief with respondents, and create a final version, which was shown to workshop participants. To enable measurement of interdisciplinarity, the final version of the survey provided the following definition of interdisciplinary R&D:

Unfortunately, the operationalization of interdisciplinary R&D did not produce a consistent measure. The inconsistencies manifested themselves in two major ways. First, different respondents within a university understood the concept of interdisciplinarity differently. Second, record-keeping methods by universities did not match the survey’s definition of the concept. Rather than abandoning the effort to measure interdisciplinarity, the HERD survey instead differentiated the measurements by discipline or by broader categories. Consequently, although there is a measure of interdisciplinarity, Anderson explained that the levels of interdisciplinary research cannot be fully characterized because of differences in how interdisciplinarity is measured across disciplines.

Anderson then used the Business and Enterprise Research and Development (BERD) Survey to explain successes NCSES had with analyzing data on topics that fit its definitions of either interdisciplinary or convergence research. The BERD Survey, previously called the Business R&D Innovation Survey (BRDIS), maintains the use of definitions, but ones that are specific to a field such as biotechnology or nanotechnology. By following the guidance provided in the Frascati Manual, the survey allows NCSES to produce internationally comparable data to estimate the number of companies involved with interdisciplinary or convergent research.

The BERD survey successfully measured convergence within biotechnology and nanotechnology, but not for artificial intelligence (AI). The initial iteration of the survey raised concerns about the quality of the AI data, which could stem from the definitions provided, imperfect record keeping by respondents, or another factor. NCSES is working in collaboration with the Census Bureau to identify the source of these data quality issues.

Anderson also described the findings from the NCSES Survey of Earned Doctorates (SED). One survey question directly asked respondents whether their dissertations were based on interdisciplinary research. If yes, the survey asked respondents to identify the primary field and up to three additional fields involved in their research. Using the Frascati Manual and a 20-year time series of earned doctorates survey data, Anderson is building a measure of interdisciplinarity based on existing, field-based taxonomies. Since 2001, 24 to 45 percent of respondents indicated that at least two fields are included in their dissertation research.

In addition to taxonomies and survey revisions, Anderson investigated bibliometric data and the development of indicators for interdisciplinarity. Attempts to develop indicators in 2010 were based solely on bibliometrics and proved unsatisfactory for management and policy purposes.² The effort yielded inconsistent results and was thus deemed premature.

Anderson described his conclusions on the updated ability of NCSES to produce indicators and measures of inputs, outputs, and processes involved in convergence research. He began with a caveat that researchers will likely find that respondents have definitions of and experiences with convergence that differ from their own. He emphasized that the transition from articulating a good definition of convergence to operationalizing a statistical definition of convergence is thus difficult.

Anderson stated that certain research topics of convergence can be successfully measured, as evidenced by a decade of research, and that the impacts of interdisciplinary research may hold greater importance for society rather than the scientific community. The most successful example is the BERD survey on biotechnology and nanotechnology, arguably two particular topics that meet the definition of convergent research. Additionally, as was recommended in an earlier NCSES-sponsored workshop, it may be possible to implement hybrid approaches that combine top-down definitions with bottom-up approaches such as topic modeling. This hybrid approach may be able to successfully identify convergent activities within certain priority areas or topics of interest. As a highlight, Anderson referenced a partnership with researchers at the University of Virginia, who leveraged topic modeling strategies to investigate pandemics, with a particular focus on issues related to COVID-19. By extracting recent Federal RePORTER data prior to 2020, the group has been able to detect emerging trends with respect to pandemic and coronavirus research and to track topics using a convergent approach. Anderson also found that building up measures of convergence from existing taxonomies may be the most successful in the context of survey research.

___________________

² National Science Board. (2010). Science & Engineering Indicators 2010. Arlington, VA: National Science Foundation.

In summary, NCSES considers efforts to measure convergence research activities to be an important topic, and successes have been achieved through establishment surveys such as BERD and by leveraging existing taxonomies as has been done in current NCSES human resources statistics surveys. Further, hybrid (top-down/bottom-up) approaches may offer a potential path forward to the development of novel measurement approaches.

DISCUSSION

Nora Cate Schaeffer (University of Wisconsin–Madison) asked Anderson why data quality concerns were only attributed to AI and not nanotechnology and biotechnology. Based on administrative data and additional sources, Anderson’s research team suspected that some respondents who self-reported that they were working in AI were not engaged in work that the team considered to be AI. John Jankowski (NCSES) added that nanotechnology and biotechnology also posed data quality issues initially, but the definitions were improved over time, and successive data collections were more consistent. Jankowski is hopeful that later iterations, including AI, will produce higher-quality data.

Entwisle and Ben Jones (Northwestern University) asked whether NCSES is tracking information about diversity in the R&D enterprise. Anderson responded that NCSES does try to measure diversity in the S&E enterprise, however, it remains unclear whether existing measures of diversity can capture either interdisciplinarity or convergence. Anderson also clarified that NCSES conducts surveys of enterprises (universities, government agencies, businesses) rather than individual research projects. Emilda Rivers (NCSES) commented that tracking at the project level could involve qualitative as well as quantitative data collection, which can be difficult to generalize for a national audience. She noted that measuring the impact of convergence on diversity and on ways of learning in educational institutions can speak to the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act for short),³ which requires the formulation of evidence-based policy. Rivers noted that NCSES explores these impacts of convergence primarily on a regional level and offers national-level information only to help a user navigate the regional data.

Entwisle clarified that as a vice chancellor for research, she had to aggregate project-level data in order to respond to a HERD survey. Jankowski confirmed that the BERD and HERD surveys aggregate project-level data that are further aggregated at the national level. He noted that additional questions regarding demographic information have been added

___________________

³ U.S. Congress, House of Representatives. (2018). Foundations for Evidence-Based Policymaking Act of 2018 to accompany H.R. 4174. 115th Cong., 1st sess., pp. 115–411.

to the BERD survey, but although the survey asks questions about sociodemographic characteristics (sex, educational level), many respondents do not provide the level of data many investigators desire.

Joshua Schnell (Clarivate) asked whether NCSES has explored the problem-oriented aspect of convergence through bibliometric data analysis. Anderson referred this question to Beethika Khan (NCSES), who stated that no such investigation had been conducted. That investigation would require a deeper understanding of the problem or research question than such an analysis would address. She explained, though, that NCSES is attempting to link journal-level bibliometric data with data from SED to facilitate deeper demographic analysis of publications. The data currently provided by journal administrators do not allow for such demographic analysis. In responding to a follow-up question, Khan said that the data-linking process is ongoing, and so not all data are fully available to the National Science Foundation (NSF). She offered to connect interested researchers with her statistician colleagues for more information. Khan added that bibliometric data allow her team to see the ways in which countries specialize in a discipline or a topic and the kinds of collaborations that are taking place, including international collaborations and their locations. The levels of collaboration between academic institutions and businesses can also be measured.

Jankowski commented on his hope to move forward from this workshop with additional experimentation with topic modeling through natural language processing of grant documentation (in part through a partnership with the University of Virginia). He suggested that these analytic methods may reveal more about convergence than a survey instrument would be likely to do.

Khargonekar asked whether topic modeling is capable of distinguishing between problems or whether all problem-oriented research is combined, regardless of topic, in NSF’s analyses. Jankowski answered that NCSES began by generating relevant search terms clustered into general topics that fall under the rubric of a selected problem. The next step is to automate this process to allow the machine to identify subtopics. Anderson added that the research team is still learning how to automate the labeling of the orientation of clusters toward problems, techniques, and other focuses.

Rivers closed the session by expressing her gratitude to Anderson for his presentation’s depth and breadth and thanked all participants for their robust participation and engaging questions during the discussion.

This page intentionally left blank.