Appendix VI

NAKFI’S Use of Bibliometric Approaches to Demonstrating Impact

BIBLIOMETRIC ANALYSIS OF PUBLICATIONS FROM SMART PROSTHETICS GRANTS

The early years of the National Academies Keck Futures Initiative (NAKFI) evaluation involved the use of bibliometric methods. In 2010, the first seed grant review working group decided to assess the value and impact of NAKFI’s seed grants by combining bibliometric analyses with a peer review and dialogue assessment process. Bibliometric methods have been used in the evaluation of large-scale initiatives and can be especially useful when combined with other approaches such as conducting personal interviews, content analysis, or peer reviews of research papers. Bibliometric analysis involves quantitative examination of scientific publications, the works they cite, and citations of them. It has emerged as an important way of examining scientific influence and impact because the approach can be applied to small and large sets of papers at one point in time or longitudinally.

NAKFI evaluators developed and tested a computed measure of scholarly integration based on the references in a paper (Porter, Roessner, & Heberger, 2008). NAKFI used the Web of Science as a data source. The Web of Science Core Collection indexes more than 20,000 journals, books, and conference proceedings across 251 subject categories. Journals are assigned to subject categories based on a combination of expert reviews of journal content and citation patterns (Porter, Roessner, & Heberger, 2008). Subject categories are one way of classifying the vast and complex world of science. They can be thought of as “bodies of specialized knowl-

edge or research practice,” the integration of which is an important part of NAKFI’s definition of interdisciplinary research (see Chapter 1). Counting the subject categories of the journals in a reference list does not capture that some subject categories are more closely related than others. More subject categories do not necessarily correlate with more interdisciplinary integration if the subject categories are closely related. The computed measure of integration developed for NAKFI’s use results in a higher score for papers that reference works from diverse subject categories. In this section we describe the bibliometric characteristics of the journal articles generated from the Smart Prosthetics research projects, which were funded in 2007 and completed in 2009.

APPROACH

Web of Knowledge was searched for papers produced from NAKFI Smart Prosthetics seed grant projects in December 2011. We located 17 documents from 7 of the 14 Smart Prosthetics projects. Three of these publications were conference abstracts and were removed from subsequent analyses. The remaining articles were published in journals associated with subject categories such as Engineering, Neurosciences & Neurology, Urology & Nephrology, Biology & Molecular Biology, Chemistry, Life Sciences & Biomedicine-Other Topics, Materials Science, and Physics. Ten of the 14 papers acknowledged NAKFI funding. Table VI-1 lists the journals where the articles were published and the subject categories associated with those journals.

The papers made 720 references to 65 different subject categories. Neuroscience was the most frequently cited subject category and made up 28% of the references. Table VI-2 lists the 20 subject categories most frequently cited by these papers. Authors also drew on publications in similar areas including Engineering-Biomedical, Physiology, Multidisciplinary Sciences, Ophthalmology, Materials Science-Biomaterials, Clinical Neurology, and Computer Science-Artificial Intelligence. The expert reviewers indicated evidence of many of these disciplines in the final reports grantees submitted to NAKFI.

TABLE VI-1 Journals and Associated Subject Categories Where Smart Prosthetics Grantees Published

BIBLIOMETRIC RESULTS

The 14 journal articles scored on the higher end of NAKFI’s computed measure of interdisciplinarity called Integration. Figure VI-1 shows a scatterplot of each paper’s Integration score and the number of citations. Details of the development and testing of this measure can be found elsewhere (Porter et al., 2007; Porter, Roessner, & Heberger, 2008). Essentially, the more diverse the fields of knowledge that a paper draws on in its references, the higher the Integration score will be. Integration has a range of values between 0 (low) to 1 (high). The mean Integration score for this group of

TABLE VI-2 Top 20 Subject Categories Cited in Articles Published by Smart Prosthetics Grantees

NOTES: Bold indicates subject categories of journals in which Smart Prosthetics grantees published their research articles. Italics indicate disciplines identified by expert reviewers using the Written Products Protocol.

papers was 0.69. Four papers had an Integration score above 0.80. In 2005 NAKFI conducted an analysis to benchmark the integration score of 22 subject categories. A random sample of 100 papers published in the Neuroscience subject category had an Integration score of 0.60 and a sample of Engineering-Biomedical papers had an average Integration score of 0.69 (Porter, Roessner, & Heberger, 2008). Based on this benchmarking, the integration found in this group of papers may not be exceptional, but certain papers display higher than average levels of integration. More extensive profiling would be necessary to determine if these scores are indeed high.

The Integration score is positively correlated with the number of times a paper is cited by others. Papers with higher Integration scores tend to be cited more frequently, and the correlation is a modest 0.40. The scatter plot shows the variability in the number of times these papers had been cited in 2011, and shows the dispersion of the Integration scores. This analysis does not take into account the important variable of the amount of time that has passed since a paper was published.

Combining the bibliometric data with results from the expert reviews provides additional insight into these projects. One project conducted by Potter and Carmena called “Testbed for Developing Direct Cortical Feedback of Proprioception for Smart Limb Prostheses” provides a case of where

the bibliometric data and expert reviews agreed that the project was successful and scientifically important. The project was reviewed in 2011 shortly after three papers had been published. The project received high marks from reviewers for creativity, scientific impact, potential societal impact, and overall quality. The project also received high marks for Integration as described on the scoring rubric. Potter’s three papers scored 0.74, 0.68, and 0.62 on the computed bibliometric measure of Integration, slightly higher scores than were found in the benchmarked sample of biomedical engineering papers. In 2011 the three papers had already been cited 31 times; as of August 2018, they had been cited 151 times, a five-fold increase in citations.

By comparison, a grant by Gerald Loeb titled “Biomimetic Tactile Sensors and Grip Control Strategies” received high marks (rated 3 or 4 out of 5) from reviewers on creativity, scientific impact, potential societal impact, and overall quality. However, reviewers did not assess it as being an especially integrative project. The two papers Loeb published from this project received higher scores on the bibliometric measure of Integration (0.74 and 0.84) than the average score for the group of papers (0.69).

Web of Science can display citations to a paper visually. The Loeb paper, which had been cited 24 times in 2011, is a good candidate for visual display. Figure VI-2 shows the subject categories of the papers that Loeb cited in his paper on the left and those that have cited the paper on the right. The turquoise box in the center represents Loeb’s paper. Each color represents a subject category and each box represents a separate citation. We can see from the first figure that the colors on either side of the image

are different. Loeb’s paper is being cited by papers written in journals from different subject categories than he referred to in this paper. The red on the map represents papers with no data. The cobalt blue boxes appearing on both sides of the figure represent papers published in the journal Sensors and Actuators, which is associated with two subject categories, Engineering and Instruments & Instrumentation.

Figure VI-3 explores the subject categories Loeb referenced by zooming in on one section of the map. It also obscures papers that have no data (in red). The green boxes on the left hand side represent citations Loeb made to

papers published in the Neurology & Neurosciences subject category. This color does not appear on the right side of the map. Though Loeb drew heavily on this subject category in his paper, authors publishing in this category had not yet referenced his paper in 2011.

Though Loeb’s paper was published in a journal associated with the Robotics subject category (turquoise, center), he did not reference any Robotics papers. Figure VI-4 shows that 4 of the 24 papers referring to Loeb’s paper were published in Robotics journals. Many citations came from papers published in various engineering subject categories (green, blue, and maroon). There were also citations from outside engineering. One paper was published in the Medicine-General and Internal subject category and another in Biotechnology.

Taken together these examples indicate that bibliometrics and expert analysis are complementary approaches. Bibliometrics is useful in answering certain questions about scientific publications, including in what subject areas do grantees publish papers; what bodies of knowledge do grantees draw on in their publications; and what areas of science are referring to work published by grantees. However, not every research study will result in a publication. This is especially true for risky and bold research topics like the ones NAKFI funded. The publication process is also slow and it can take many years for a paper to be indexed and become cited. Citation is also only one indicator of impact. Using a rubric, expert reviewers can apply their professional knowledge and experience to published as well as unpublished research products. They have specialized insight into the subject areas explored by NAKFI projects and are well suited to recognize

creativity, scientific quality, and potential societal benefits of research in their area of expertise.

One powerful aspect of bibliometics is that the approach can be applied at various scales, from a deep-dive into a single paper to analyzing trends in science as a whole using hundreds of thousands of papers. Web of Science can also be used to explore bibliometric results of all journal articles listing NAKFI as a funding source.

BIBLIOMETRIC ANALYSIS OF PUBLICATIONS CITING NAKFI FUNDING

Previous evaluations of NAKFI seed grants revealed that grantees are inconsistent in acknowledging NAKFI funding in their publications. Nevertheless, Web of Science includes “Funding Agency” as a search field. In August of 2018, Web of Science was searched for all papers citing “Keck Futures” or “NAKFI” as a funding source, resulting in a list of 179 papers.

The years 2012–2015 show the most papers citing NAKFI funding, with fewer papers acknowledging funding before and after that time. The lack of citations in and prior to 2008 is mostly due to the timing of when Web of Science introduced the funding agency search field. Since Web of Science is continually updating its database, the 2017 and 2018 data may be incomplete.

NAKFI-funded articles appear in journals associated with 89 different subject categories; Web of Science has 251. The top 25 subject categories match what one would expect based on the topics covered by Futures Conferences. Twenty-nine papers (16%) appeared in Multidisciplinary Sciences

journals and 21 papers (11%) were published in journals associated with Biomedical Research Methods. The Multidisciplinary Sciences subject category includes journals such as Science, Nature, Proceedings of the National Academy of Sciences, Astrophysical Journal Letters, and Biotechnology Advances. Examining the subject categories with less concentration of papers shows the breadth of journals where NAKFI-funded papers appeared, including Sociology, Psychology, Paleontology, Mycology, Health Policy Services, Gastroenterology, Hepatology, Communication, History, Philosophy of Science, Ethics, Nutrition Dietetics, Women’s Studies, and Urban Studies.

These records also show that most of the papers acknowledge funding from multiple sources, especially the federal government. The National Science Foundation and the National Institutes of Health are sponsors of NAKFI grantees’ research, as are other governmental funding agencies including DARPA, NASA, the Army Research Office, and the Office of Naval Research.

REACH OF NAKFI-SPONSORED RESEARCH

All research builds on previous knowledge. It is possible to examine how others have used the work of NAKFI researchers. The 179 NAKFI-sponsored publications in Web of Science have been cited 6,442 times by 5,776 different articles, an average of more than 36 citations per paper.

Though NAKFI only funded research by U.S.-based investigators, the citations to NAKFI researchers’ publications reflect the global nature of science today. Though most of the citations are from authors based in the United States, 16% are from China, 9.5% are from Germany, 8% are from France, and 5% are from Canada.