National Academies Press: OpenBook

A New Vision for Center-Based Engineering Research (2017)

Chapter: 4 Outcomes and Metrics

« Previous: 3 People
Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×

4

Outcomes and Metrics

According to the National Science Foundation’s (NSF’s) engineering research center (ERC) website, “The goal of the ERC Program is to integrate engineering research and education with technological innovation to transform national prosperity, health, and security.”1 The committee resonates with this existing goal and suggests that the top-level goal of future convergent engineering research centers (CERCs) should be to solve critical societal problems with engineering research and to advance fundamental knowledge on how to deliver those solutions to society. The goal of maximizing societal benefit will in most cases lead to the creation of great economic value as well. An example is the Internet, which began as a research tool but has transformed both the social and economic landscape.

Various metrics have been used to judge the performance of center-based research programs, such as number of students graduated, number of scientific articles published, number of industry participants, number of patents, number of startups spawned, or products commercialized. According to NSF, since the ERC Program’s inception in 1985, it has created 193 spin-off companies, disclosed more than 2,200 inventions, and has been awarded 739 patents, which resulted in 1,339 licenses.2

The problem with many of these metrics is that they measure outputs that say little about the degree to which the research has achieved center goals. Outputs are indicators of potential impact and are comparatively easy to measure (papers, patents, etc.) Outcomes are related to impacts (e.g., transformational changes). It is outcomes or impacts that NSF should be most concerned about. And some of these output measures—such as the number of participating companies, number of patents, or number of students involved—can be “gamed.” Above all, the metrics of center performance should not foster a “box-checking” mentality. Appropriate metrics are discussed below and in Chapter 6.

ELEMENTS OF SOCIETAL BENEFIT

Under the top-level goal of delivering societal benefit, the committee believes the following four outcomes will be critical to the success of the CERCs:

___________________

1 National Science Foundation (NSF), “Engineering Research Centers (ERC),” https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5502, accessed October 10, 2016.

2 NSF, 2015, Creating New Knowledge, Innovators, and Technologies for Over 30 Years, https://www.nsf.gov/eng/multimedia/NSF_ERC_30th_Anniversary.pdf.

Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×
  • Students with the skills to be innovators and leaders,
  • New ideas and paradigm-shifting research,
  • Development and use of work products of durable intellectual value, and
  • Creation of economic value.

Students with the Skills to be Innovators and Leaders

Top-notch undergraduate and graduate engineering students are not only key to the successful operation of CERCs, as was pointed out in Chapter 3, they are also an important indicator of center success. Center students should have the opportunity to (1) engage in significant, multidisciplinary technical training, (2) experience deep collaboration, (3) take on leadership roles, and (4) gain meaningful exposure to industry practices and culture. See the Chapter 3 section “Engineering Education.”

FINDING 4-1: Superbly educated and innovation-savvy engineering student leaders capable of addressing current and future societal challenges are likely to be the most important long-term outcome of CERCs.

RECOMMENDATION 4-1: A key objective of the convergent engineering research centers should be capacity-building through student training and development, at both the graduate and undergraduate level, and direct engagement with industry.

Metrics that might be used to measure student outcomes can include tracking of student placement in industry, university, government, or nonprofit positions and contributions made in those positions, compared with a non-CERC control student population. As discussed further in Chapter 6, emerging collaboration platforms allow real-time tracking and longitudinal follow-up of center research activities and of students who have been engaged at the centers—all with fewer burdens on the centers. With only a limited number of cohorts of students graduated with degrees during the typical 10-year NSF ERC funding lifetime, this metric may need to have a retrospective component; see further discussion in Chapter 6.

New Ideas and Paradigm-Shifting Research

Successful centers produce new research ideas and innovative technical approaches to the problems being addressed. In the case of a CERC devoted to a grand-challenge-like problem, a successful outcome may be just understanding how best to address the grand challenge. The center may also produce paradigm-shifting research. An example from the past was the pioneering work of W.E. Deming on statistical process control—a process innovation.3 It created an entirely new, more productive way to create value for society though lower cost and profoundly higher product quality. A modern example might be the marriage of artificial intelligence, machine learning, and big data technologies to usher in an era of lower-cost, easy-access cognitive health care.4

Metrics for this category of outcomes could be evidence of widespread adoption of new research techniques or processes developed.

Work Products of Durable Intellectual Value

The Internet makes it possible for scientists and engineers to not only to record their own work, but also to create durable work products and platforms that can be instantly shared with others, thereby directly accelerating the progress of science and helping to develop innovative engineering and manufacturing systems. These durable work products can include prototypes, innovative manufacturing processes, software applications, tools, and

___________________

3 D.W. Edwards, 1993, The New Economics for Industry, Government, and Education, MIT Press, Boston, Mass., p. 132.

4 See, for example, S. Smith, 2015, 5 ways the IBM Watson is changing health care, from diagnosing disease to treating it, Medical Daily, December 17, http://www.medicaldaily.com/5-ways-ibm-watson-changing-health-care-diagnosing-disease-treating-it-364394.

Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×

libraries; data and data repositories; cloud computing platforms and services; physical engineering systems with remote access; international standards; and other enabling technologies with broad application. Their development is enabled by the availability of distributed software development platforms such as GitHub, cloud computing platforms such as Microsoft Azure and Amazon AWS, the maturation of IoT (Internet of Things); hardware, software, and data aggregation tools; and the broad use of standardized REST Internet interface tools for connecting physical systems to applications and services. Examples of such systems include software such as the BLAST tool for comparing primary biological sequence information; the World Wide Telescope, which provides astronomers with new ways to visualize observational data from the world’s great telescopes; and data repositories such as the Network for Earthquake Engineering Simulation Hub (NEEShub), which provide a web-based gateway for earthquake engineering results and information.

Metrics for measuring impact in this category might include widespread use of center-developed software, tools, or standards, or the frequency with which data repositories are accessed.

FINDING 4-2: The development and dissemination of engineering and intellectual work products with durable value accelerates innovation and scientific progress.

RECOMMENDATION 4-2: Future convergent engineering research centers should be encouraged to produce broadly accessible engineering prototypes, tools, data repositories, platforms, and enabling technologies that foster broad scientific, engineering, and manufacturing innovation. Such work products might form useful interim deliverables from large-scale projects.

Development of research and education products can be shared widely through collaboration platforms. NSF should explore ways of disseminating CERC innovation, student training, and entrepreneurship activities that scale beyond the centers and participating institutions. Universities frequently develop tools of this type, but these may only be usable by their developers, because they do not put in the large amount of effort required to make them user-friendly. To make these tools usable by a wider audience, NSF may need to make funds available for professionals to refine them. As appropriate, NSF should consider leveraging existing federal initiatives, such as the Small Business Technology Transfer program, to help disseminate these tools.

Economic Value

From its inception, the ERC program has had the goal of enhancing U.S. industrial competitiveness by transferring intellectual value and technology developed in the centers into the commercial sphere. NSF has also sought to create economic value indirectly through the training of a diverse population of students with the skills to innovate.

The challenges associated with the commercialization goal for ERCs are often underappreciated. For example, it is generally not realistic to expect intellectual property developed in a center working in biology and materials sciences to become a commercial product in the center’s typical 10-year life span. Most medical or materials-based start-up companies take about 10 years before they produce products at sufficient scale to generate profits. However, information technologies can be developed much faster and, as recommended here, centers using value-creation best practices can engage industry much earlier in valuable commercial initiatives.

Today, companies often engage with centers to gain access to new knowledge and future employees and do not expect to develop specific processes or products as a result of these relationships.5 This is reasonable for ERCs, given that they initially sit at the early stages of the continuum of technology readiness levels (see Figure 1.1).

___________________

5 I. Feller, C.P. Ailes, and J.D. Roessner, 2002, Impacts of research universities on technological innovation in industry: Evidence from engineering research centers, Research Policy 31(3):457-474.

Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×

Researchers have documented considerable direct and indirect economic impact of selected ERCs.6,7,8 While commercial successes do occur (Box 4.1), they are relatively rare. Orin Herskowitz, director of Columbia University’s Columbia Technology Ventures, told the committee that about 85 percent of the technology transfer activities of U.S. universities lose money, with the bulk of licensing revenues accruing to a small subset of institutions from a small number of blockbuster products.9

Part of the challenge is that some center-related economic impacts are very difficult to quantify, while others may not be evident until many years after a center has ceased operation. External center funding, increased employment, and improvements in the technical workforce comprise the larger categories of quantifiable economic value.

Over time, one way to measure the success of delivering social benefit is that the results (e.g., intellectual work products) of the center are picked up by industry and then industry makes economic advances that can be traced back to the centers. Thus, one can use economic value delivered as one metric—but not the only one—to determine if CERCs (or the NSF centers generally) have delivered societal benefit.

The committee is agnostic about whether the larger goal of delivering maximum societal benefit is served by centers seeking to translate proprietary technologies to the private sector by either licensing or forming startups, or by giving away their intellectual content through open sourcing. As one example, the Linux operating system is open source, but has created huge economic value.

Metrics for measuring economic value are relatively well developed and quantitative and are not discussed further here.

FINDING 4-3: Metrics currently used to evaluate centers tend to focus on numbers of students graduated, papers published, patents awarded, and so on. These output numbers do not necessarily measure the true impact of the center and the metrics can be gamed and may encourage a box-checking mentality.

___________________

6 SRI International, 2004, The Economic Impact on Georgia of Georgia Tech’s Packaging Research Center, Arlington, Va.

7 SRI International, 2008, National and Regional Economic Impacts of Engineering Research Centers: A Pilot Study. Final Report, Arlington, Va., November.

8 D. Roessner, L. Manrique, and J. Park, 2010, The economic impact of engineering research centers: Preliminary results of a pilot study, Journal of Technology Transfer 35(5), 475-493.

9 National Academies of Sciences, Engineering, and Medicine, 2016, A Vision for the Future of Center-Based Multidisciplinary Engineering Research: Proceedings of a Symposium, The National Academies Press, Washington, D.C.

Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×

RECOMMENDATION 4-3: The National Science Foundation should develop metrics that track the impacts of center activities, not just the outputs. Examples might include the placement of graduated students in positions of influence, or evidence that intellectual value developed in the center is widely used.

OPPORTUNITIES FOR CENTERS TO ENHANCE THE DELIVERY OF SOCIETAL VALUE

While the considerable achievements of the ERC program have been recently documented,10 the committee believes that a qualitative improvement in the delivery of societal value is possible in future centers through dissemination and systematic use of team-research and value-creation best practices among all center personnel (Boxes 2.4 and 2.5, respectively). In the committee’s experience, this is not occurring now. Programs such as NSF’s Innovation Corps (I-Corps)11 are a good start, but more will be needed to fully immerse students and faculty in an entrepreneurial environment (see Chapter 6).

___________________

10 NSF, 2015, Creating New Knowledge, Innovators, and Technologies for Over 30 Years, https://www.nsf.gov/eng/multimedia/NSF_ERC_30th_Anniversary.pdf.

11 NSF, “NSF Innovation Corps (I-Corps),” https://www.nsf.gov/news/special_reports/i-corps/, accessed August 3, 2016.

Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×
Page 39
Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×
Page 40
Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×
Page 41
Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×
Page 42
Suggested Citation:"4 Outcomes and Metrics." National Academies of Sciences, Engineering, and Medicine. 2017. A New Vision for Center-Based Engineering Research. Washington, DC: The National Academies Press. doi: 10.17226/24767.
×
Page 43
Next: 5 Structure and Operations »
A New Vision for Center-Based Engineering Research Get This Book
×
Buy Paperback | $46.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The future security, economic growth, and competitiveness of the United States depend on its capacity to innovate. Major sources of innovative capacity are the new knowledge and trained students generated by U.S. research universities. However, many of the complex technical and societal problems the United States faces cannot be addressed by the traditional model of individual university research groups headed by a single principal investigator. Instead, they can only be solved if researchers from multiple institutions and with diverse expertise combine their efforts. The National Science Foundation (NSF), among other federal agencies, began to explore the potential of such center-scale research programs in the 1970s and 1980s; in many ways, the NSF Engineering Research Center (ERC) program is its flagship program in this regard.

The ERCs are “interdisciplinary, multi-institutional centers that join academia, industry, and government in partnership to produce transformational engineered systems and engineering graduates who are adept at innovation and primed for leadership in the global economy. To ensure that the ERCs continue to be a source of innovation, economic development, and educational excellence, A New Vision for Center-Based Engineering Research explores the future of center-based engineering research, the skills needed for effective center leadership, and opportunities to enhance engineering education through the centers.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!