National Academies Press: OpenBook

Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines (2018)

Chapter: Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact

« Previous: Appendix V: Five- and Ten-Year Review Summaries and Panel Members
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×

Appendix VI

Image

NAKFI’S Use of Bibliometric Approaches to Demonstrating Impact

BIBLIOMETRIC ANALYSIS OF PUBLICATIONS FROM SMART PROSTHETICS GRANTS

The early years of the National Academies Keck Futures Initiative (NAKFI) evaluation involved the use of bibliometric methods. In 2010, the first seed grant review working group decided to assess the value and impact of NAKFI’s seed grants by combining bibliometric analyses with a peer review and dialogue assessment process. Bibliometric methods have been used in the evaluation of large-scale initiatives and can be especially useful when combined with other approaches such as conducting personal interviews, content analysis, or peer reviews of research papers. Bibliometric analysis involves quantitative examination of scientific publications, the works they cite, and citations of them. It has emerged as an important way of examining scientific influence and impact because the approach can be applied to small and large sets of papers at one point in time or longitudinally.

NAKFI evaluators developed and tested a computed measure of scholarly integration based on the references in a paper (Porter, Roessner, & Heberger, 2008). NAKFI used the Web of Science as a data source. The Web of Science Core Collection indexes more than 20,000 journals, books, and conference proceedings across 251 subject categories. Journals are assigned to subject categories based on a combination of expert reviews of journal content and citation patterns (Porter, Roessner, & Heberger, 2008). Subject categories are one way of classifying the vast and complex world of science. They can be thought of as “bodies of specialized knowl-

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×

edge or research practice,” the integration of which is an important part of NAKFI’s definition of interdisciplinary research (see Chapter 1). Counting the subject categories of the journals in a reference list does not capture that some subject categories are more closely related than others. More subject categories do not necessarily correlate with more interdisciplinary integration if the subject categories are closely related. The computed measure of integration developed for NAKFI’s use results in a higher score for papers that reference works from diverse subject categories. In this section we describe the bibliometric characteristics of the journal articles generated from the Smart Prosthetics research projects, which were funded in 2007 and completed in 2009.

APPROACH

Web of Knowledge was searched for papers produced from NAKFI Smart Prosthetics seed grant projects in December 2011. We located 17 documents from 7 of the 14 Smart Prosthetics projects. Three of these publications were conference abstracts and were removed from subsequent analyses. The remaining articles were published in journals associated with subject categories such as Engineering, Neurosciences & Neurology, Urology & Nephrology, Biology & Molecular Biology, Chemistry, Life Sciences & Biomedicine-Other Topics, Materials Science, and Physics. Ten of the 14 papers acknowledged NAKFI funding. Table VI-1 lists the journals where the articles were published and the subject categories associated with those journals.

The papers made 720 references to 65 different subject categories. Neuroscience was the most frequently cited subject category and made up 28% of the references. Table VI-2 lists the 20 subject categories most frequently cited by these papers. Authors also drew on publications in similar areas including Engineering-Biomedical, Physiology, Multidisciplinary Sciences, Ophthalmology, Materials Science-Biomaterials, Clinical Neurology, and Computer Science-Artificial Intelligence. The expert reviewers indicated evidence of many of these disciplines in the final reports grantees submitted to NAKFI.

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×

TABLE VI-1 Journals and Associated Subject Categories Where Smart Prosthetics Grantees Published

Publishing Journal Subject Categories Assigned to Journal
IEEE Transactions on Neural Systems and Rehabilitation Engineering Engineering, Rehabilitation
Neurology & Urodynamics Urology & Nephrology
PLoS ONE Life Sciences & Biomedicine-Other
Advanced Materials Chemistry, Materials Science, Physics, Science & Technology-Other Topics
Advanced Robotics Robotics
IEEE Transactions on Biomedical Engineering Engineering
IEEE-ASME Transactions on Mechatronics Engineering, Automation & Control Systems
Investigative Ophthalmology & Visual Science Ophthalmology
Journal of Neural Engineering Engineering, Neurosciences & Neurology
Journal of Neurophysiology Neurosciences & Neurology
Journal of Neuroscience Methods Neurosciences & Neurology, Biochemistry & Molecular Biology
Journal of Urology Urology & Nephrology
PLoS Computational Biology Biochemistry & Molecular Biology, Mathematical & Computational Biology
Soft Matter Chemistry, Materials Science, Physics, Polymer Science

BIBLIOMETRIC RESULTS

The 14 journal articles scored on the higher end of NAKFI’s computed measure of interdisciplinarity called Integration. Figure VI-1 shows a scatterplot of each paper’s Integration score and the number of citations. Details of the development and testing of this measure can be found elsewhere (Porter et al., 2007; Porter, Roessner, & Heberger, 2008). Essentially, the more diverse the fields of knowledge that a paper draws on in its references, the higher the Integration score will be. Integration has a range of values between 0 (low) to 1 (high). The mean Integration score for this group of

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×

TABLE VI-2 Top 20 Subject Categories Cited in Articles Published by Smart Prosthetics Grantees

Neurosciences Engineering, Electrical & Electronic
Engineering, Biomedical Biotechnology & Applied Microbiology
Physiology Biophysics
Multidisciplinary Sciences Zoology
Ophthalmology Biochemistry & Molecular Biology
Materials Science-Biomaterials Robotics
Clinical Neurology Surgery
Biochemical Research Methods Biology
Computer Science- Artificial Intelligence Biotechnology & Applied Microbiology
Polymer Science Chemistry, Multidisciplinary

NOTES: Bold indicates subject categories of journals in which Smart Prosthetics grantees published their research articles. Italics indicate disciplines identified by expert reviewers using the Written Products Protocol.

papers was 0.69. Four papers had an Integration score above 0.80. In 2005 NAKFI conducted an analysis to benchmark the integration score of 22 subject categories. A random sample of 100 papers published in the Neuroscience subject category had an Integration score of 0.60 and a sample of Engineering-Biomedical papers had an average Integration score of 0.69 (Porter, Roessner, & Heberger, 2008). Based on this benchmarking, the integration found in this group of papers may not be exceptional, but certain papers display higher than average levels of integration. More extensive profiling would be necessary to determine if these scores are indeed high.

The Integration score is positively correlated with the number of times a paper is cited by others. Papers with higher Integration scores tend to be cited more frequently, and the correlation is a modest 0.40. The scatter plot shows the variability in the number of times these papers had been cited in 2011, and shows the dispersion of the Integration scores. This analysis does not take into account the important variable of the amount of time that has passed since a paper was published.

Combining the bibliometric data with results from the expert reviews provides additional insight into these projects. One project conducted by Potter and Carmena called “Testbed for Developing Direct Cortical Feedback of Proprioception for Smart Limb Prostheses” provides a case of where

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Image
FIGURE VI-1 Scatter plot of integration score and number of times cited for 14 articles produced by Smart Prosthetics grantees, as of 2011.

the bibliometric data and expert reviews agreed that the project was successful and scientifically important. The project was reviewed in 2011 shortly after three papers had been published. The project received high marks from reviewers for creativity, scientific impact, potential societal impact, and overall quality. The project also received high marks for Integration as described on the scoring rubric. Potter’s three papers scored 0.74, 0.68, and 0.62 on the computed bibliometric measure of Integration, slightly higher scores than were found in the benchmarked sample of biomedical engineering papers. In 2011 the three papers had already been cited 31 times; as of August 2018, they had been cited 151 times, a five-fold increase in citations.

By comparison, a grant by Gerald Loeb titled “Biomimetic Tactile Sensors and Grip Control Strategies” received high marks (rated 3 or 4 out of 5) from reviewers on creativity, scientific impact, potential societal impact, and overall quality. However, reviewers did not assess it as being an especially integrative project. The two papers Loeb published from this project received higher scores on the bibliometric measure of Integration (0.74 and 0.84) than the average score for the group of papers (0.69).

Web of Science can display citations to a paper visually. The Loeb paper, which had been cited 24 times in 2011, is a good candidate for visual display. Figure VI-2 shows the subject categories of the papers that Loeb cited in his paper on the left and those that have cited the paper on the right. The turquoise box in the center represents Loeb’s paper. Each color represents a subject category and each box represents a separate citation. We can see from the first figure that the colors on either side of the image

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Image
FIGURE VI-2 Subject categories of the papers cited in one grantee’s publication (left) and those that have cited the paper (right).
SOURCE: ISI Web of Knowledge Citation Analysis conducted by NAKFI staff in March 2012.

are different. Loeb’s paper is being cited by papers written in journals from different subject categories than he referred to in this paper. The red on the map represents papers with no data. The cobalt blue boxes appearing on both sides of the figure represent papers published in the journal Sensors and Actuators, which is associated with two subject categories, Engineering and Instruments & Instrumentation.

Figure VI-3 explores the subject categories Loeb referenced by zooming in on one section of the map. It also obscures papers that have no data (in red). The green boxes on the left hand side represent citations Loeb made to

Image
FIGURE VI-3 An expanded view of the left side of Figure VI-2 showing the paper’s frequent references to publications associated with the Neurosciences & Neurology subject category.
SOURCE: ISI Web of Knowledge Citation Analysis conducted by NAKFI staff in March 2012.
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Image
FIGURE VI-4 An expanded view of the right side of Figure VI-2 showing the that papers referencing this work appear in journals associated with a wide variety of subject categories.
SOURCE: ISI Web of Knowledge Citation Analysis conducted by NAKFI staff in March 2012.

papers published in the Neurology & Neurosciences subject category. This color does not appear on the right side of the map. Though Loeb drew heavily on this subject category in his paper, authors publishing in this category had not yet referenced his paper in 2011.

Though Loeb’s paper was published in a journal associated with the Robotics subject category (turquoise, center), he did not reference any Robotics papers. Figure VI-4 shows that 4 of the 24 papers referring to Loeb’s paper were published in Robotics journals. Many citations came from papers published in various engineering subject categories (green, blue, and maroon). There were also citations from outside engineering. One paper was published in the Medicine-General and Internal subject category and another in Biotechnology.

Taken together these examples indicate that bibliometrics and expert analysis are complementary approaches. Bibliometrics is useful in answering certain questions about scientific publications, including in what subject areas do grantees publish papers; what bodies of knowledge do grantees draw on in their publications; and what areas of science are referring to work published by grantees. However, not every research study will result in a publication. This is especially true for risky and bold research topics like the ones NAKFI funded. The publication process is also slow and it can take many years for a paper to be indexed and become cited. Citation is also only one indicator of impact. Using a rubric, expert reviewers can apply their professional knowledge and experience to published as well as unpublished research products. They have specialized insight into the subject areas explored by NAKFI projects and are well suited to recognize

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×

creativity, scientific quality, and potential societal benefits of research in their area of expertise.

One powerful aspect of bibliometics is that the approach can be applied at various scales, from a deep-dive into a single paper to analyzing trends in science as a whole using hundreds of thousands of papers. Web of Science can also be used to explore bibliometric results of all journal articles listing NAKFI as a funding source.

BIBLIOMETRIC ANALYSIS OF PUBLICATIONS CITING NAKFI FUNDING

Previous evaluations of NAKFI seed grants revealed that grantees are inconsistent in acknowledging NAKFI funding in their publications. Nevertheless, Web of Science includes “Funding Agency” as a search field. In August of 2018, Web of Science was searched for all papers citing “Keck Futures” or “NAKFI” as a funding source, resulting in a list of 179 papers.

The years 2012–2015 show the most papers citing NAKFI funding, with fewer papers acknowledging funding before and after that time. The lack of citations in and prior to 2008 is mostly due to the timing of when Web of Science introduced the funding agency search field. Since Web of Science is continually updating its database, the 2017 and 2018 data may be incomplete.

NAKFI-funded articles appear in journals associated with 89 different subject categories; Web of Science has 251. The top 25 subject categories match what one would expect based on the topics covered by Futures Conferences. Twenty-nine papers (16%) appeared in Multidisciplinary Sciences

Image
FIGURE VI-5 Papers citing NAKFI funding, by publication year (n = 179).
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Image
FIGURE VI-6 Tree diagram showing the top 25 subject categories associated with journals in which NAKFI-funded articles appear.
SOURCE: ISI Web of Knowledge Citation Analysis conducted by NAKFI staff in August 2018.

journals and 21 papers (11%) were published in journals associated with Biomedical Research Methods. The Multidisciplinary Sciences subject category includes journals such as Science, Nature, Proceedings of the National Academy of Sciences, Astrophysical Journal Letters, and Biotechnology Advances. Examining the subject categories with less concentration of papers shows the breadth of journals where NAKFI-funded papers appeared, including Sociology, Psychology, Paleontology, Mycology, Health Policy Services, Gastroenterology, Hepatology, Communication, History, Philosophy of Science, Ethics, Nutrition Dietetics, Women’s Studies, and Urban Studies.

These records also show that most of the papers acknowledge funding from multiple sources, especially the federal government. The National Science Foundation and the National Institutes of Health are sponsors of NAKFI grantees’ research, as are other governmental funding agencies including DARPA, NASA, the Army Research Office, and the Office of Naval Research.

REACH OF NAKFI-SPONSORED RESEARCH

All research builds on previous knowledge. It is possible to examine how others have used the work of NAKFI researchers. The 179 NAKFI-sponsored publications in Web of Science have been cited 6,442 times by 5,776 different articles, an average of more than 36 citations per paper.

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×

Though NAKFI only funded research by U.S.-based investigators, the citations to NAKFI researchers’ publications reflect the global nature of science today. Though most of the citations are from authors based in the United States, 16% are from China, 9.5% are from Germany, 8% are from France, and 5% are from Canada.

Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 365
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 366
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 367
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 368
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 369
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 370
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 371
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 372
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 373
Suggested Citation:"Appendix VI: NAKFI'S Use of Bibliometric Approaches to Demonstrating Impact." National Research Council. 2018. Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/25239.
×
Page 374
Next: Appendix VII: In Practice: How to Think Like NAKFI »
Collaborations of Consequence: NAKFI’s 15 Years Igniting Innovation at the Intersections of Disciplines Get This Book
×
Buy Paperback | $115.00 Buy Ebook | $89.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This publication represents the culmination of the National Academies Keck Futures Initiative (NAKFI), a program of the National Academy of Sciences, the National Academy of Engineering, and the National Academy of Medicine supported by a 15-year, $40 million grant from the W. M. Keck Foundation to advance the future of science through interdisciplinary research. From 2003 to 2017, more than 2,000 researchers and other professionals across disciplines and sectors attended an annual “think-tank” style conference to contemplate real-world challenges. Seed grants awarded to conference participants enabled further pursuit of bold, new research and ideas generated at the conference.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!