National Academies Press: OpenBook
« Previous: 2 The Overall Context of the Materials Research Science and Engineering Centers Program
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

3
Assessment of Research and Facilities Impact

In carrying out its impact assessment task, the committee first analyzed the issues broadly in several different categories: research, education and outreach, collaboration with other sectors, and other areas. After the following introductory remarks, an analysis of the impact of Materials Research Science and Engineering Centers’ (MRSECs’) research is presented in this chapter.

INTRODUCTION

The Materials Research Science and Engineering Centers program (MRSEC program) was established by the National Science Foundation (NSF) in its Division of Materials Research (DMR) in 1994. As described in Chapter 2, the MRSEC program was born out of the decision to transform the Materials Research Laboratory (MRL) and Materials Research Group (MRG) programs to the current structure. The goal of this new initiative was to provide focused support for complex interdisciplinary materials research and education at the university level. To receive a MRSEC award, an institution must:

[demonstrate] outstanding research quality and intellectual breadth, provide support for research infrastructure and flexibility in responding to new opportunities, and strongly emphasize the integration of research and education. These centers foster active collaboration between universities and other sectors, including industry, and they constitute a national network of university-based centers in materials research. MRSECs address problems of a scope or complexity requiring the

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

advantages of scale and interdisciplinary interaction provided by a campus-based research center.1

Awards granted under the program provide support for a 6-year period; during the last 2 years of this period there is an external review under recompetition requirements in the program’s language. After the original competition in 1994, additional competitions occurred in 1996, 1998, 2000, 2002, and 2005. At the inception of the program in 1994, 30 full proposals were submitted, and 11 awards were given to 9 universities. Owing to the phase difference in the transition between programs, 13 new MRSEC awards were granted 2 years later.

The program currently funds about 29 MRSECs (26 active MRSECs and 3 on phase-out funding), which split a total of about $51 million with a range of $1.0 million to $5.0 million per institution per year, as shown in Figure 3.1. The awards are fully competed every 6 years but are staggered on the basis of the date of first award. An institution that does not receive continued MRSEC funding after recompetition is provided with phase-out support. The total number of MRSECs funded since the program’s inception in 1994 is as follows:

Year:

1994

1996

1998

2000

2002

2004

No.

11

24

25

27

27

27

A MRSEC provides a forum for researchers to come together and to share thoughts and ideas. Researchers participate because they realize the great advantages of working in an interdisciplinary team with exciting colleagues. The long-term nature of MRSEC support is welcomed because it allows researchers to pursue high-risk but potentially transformative ideas. Those ideas may lead to a new research direction for the MRSEC or may gain funding from other sources. MRSECs also provide a context for pursuing fundamental research that may not have immediately obvious payoffs but that is critical to future discovery. Students working within a MRSEC have a unique opportunity to learn from multiple mentors and to gain experience with techniques and ideas outside their own immediate field. Speaking to the committee, Harvard University MRSEC Director David Weitz emphasized these points by stating, “The most important products of the MRSEC are ideas (science, start-ups, etc.) and well-trained people.”

Evaluating MRSEC research is a daunting task. The committee considered several strategies, realizing that the MRSEC program contributes to the NSF mission in multiple ways even though “short-term research results” are usually considered the primary objective (see Box 3.1).

1

National Science Foundation, Program Solicitation for Materials Research Science and Engineering Centers, NSF 04-580, Washington, D.C., 2004.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.1 Distribution of annual Materials Research Science and Engineering Center budgets in the 1990s and in the current decade. The average (median) center budget in the late 1990s was $1.5 million ($1.2 million in inflation-adjusted dollars) and is now $1.6 million ($1.3 million); the width of the distribution has narrowed slightly. The axis label for each bar on the histograms indicates the upper edge of the range of values assigned to that bin. The data for the 1990s include several centers that started and stopped during the last years of that decade. SOURCE: Division of Materials Research, National Science Foundation.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

BOX 3.1

Qualitative Tests of MRSEC Impact

A necessary exercise in assessing the impact of the Materials Research Science and Engineering Centers program (MRSEC program) is to look at the top-rated programs in materials research in the United States and to compare them with a list of institutions that have Materials Research Science and Engineering Centers (MRSECs). Because it is impossible to determine the causality between the existence of a MRSEC at an institution and the quality of that institution’s materials science and engineering program, the committee conducted a very cursory examination of several surveys, as discussed below. At best, the surveys exhibit a correlation.


Surveys by U.S. News and World Report


According to an annual survey conducted by U.S. News and World Report, the top five schools for undergraduate materials science and engineering schools in 2006 were the following:1

1.

 

Massachusetts Institute of Technology (MIT)

2.

 

University of Illinois at Urbana-Champaign

 

Northwestern University

3.

University of California at Berkeley

 

University of Michigan at Ann Arbor

Note that the last three schools tied for third place. Four of these schools have had MRSECs or Materials Research Laboratories (MRLs).

In 2007, according to U.S. News and World Report,2 the top four chemistry Ph.D. programs were at the California Institute of Technology, MIT, Stanford University, and the University of California at Berkeley. Three of these schools have MRSECs. Likewise, the top graduate physics programs were at MIT, Stanford University, and the California Institute of Technology. All three schools have MRSECs, although the Stanford MRSEC does not interact broadly with the physics department. Drilling down to the level of condensed-matter and materials physics, both of the top schools (Cornell University and Harvard University) have MRSECs closely connected with the physics departments; the University of Illinois at Urbana-Champaign has a Materials Research Laboratory that is now supported by the Department of Energy (DOE).


National Research Council Ph.D. Program Rankings


The National Research Council conducts a decadal survey of graduate programs. The most recent rankings are from 1995, with the next edition expected in 2008. When completed, the new rankings will nicely

  

1U.S. News and World Report, 2006. Available in limited form online at http://colleges.usnews.rankingsandreviews.com/usnews/2006/edu/college/rankings/rankindex_brief.php.

  

2U.S. News and World Report, 2007. Available in limited form online at http://colleges.usnews.rankingsandreviews.com/usnews/2007/edu/college/rankings/rankindex_brief.php.

The committee realized that a comparison (“control”) group would need to be defined in order to make substantive comparisons. For instance, it would be insufficient to observe, “Research conducted through the MRSEC program generally includes significant collaboration.” Rather, the committee sought to determine

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

bracket the first decade of the MRSEC program. In 1995, the top 10 graduate programs in materials science were as follows:3

  1. MIT

  2. Northwestern University

  3. Cornell University

  4. University of California at Berkeley

  5. University of Illinois at Urbana-Champaign

  6. Stanford University

  7. University of Massachusetts at Amherst

  8. University of California at Santa Barbara

  9. Pennsylvania State University

  10. University of Pennsylvania

Of these 10 schools, all but the University of California at Berkeley have a formal materials centers program dating back to the Interdisciplinary Laboratories of the 1960s and the Materials Research Laboratories of the 1970s and 1980s. The University of California at Berkeley has strong connections with the neighboring DOE Lawrence Berkeley National Laboratory as well as with Stanford’s Center on Polymer Interfaces and Macromolecular Assemblies.


National Doctoral Program Survey


In 2000, the National Association of Graduate and Professional Students published the results of a survey of over 32,000 participants.4 The survey ranked graduate programs on the basis of participants’ perception of the overall implementation of recommended best practices (admittedly nebulous). At the top of the list of materials science programs ranked by recommended practices were the following schools:

  1. MIT

  2. University of Massachusetts at Amherst

  3. Johns Hopkins University

  4. Pennsylvania State University

  5. Stanford University

  6. University of Delaware

  7. University of California at Berkeley

  8. University of Minnesota

Of these schools, all but the University of California at Berkeley and the University of Delaware have (had) MRSECs.

  

3National Research Council, Research Doctorate Programs in the United States: Continuity and Change, Washington, D.C.: National Academy Press, 1995.

  

4Available online at http://cresmet.asu.edu/nagps/about/index.php.

if the rate or nature of collaboration in the MRSEC program is different from the rate or nature of collaboration outside the program. A natural control group might therefore be the body of research enabled by the individual-investigator awards made through NSF’s Division of Materials Research. If a positive measurement

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

were obtained, one might then ask the importance of “group-based research” to the nation’s research enterprise.

This approach was complicated by the fact that research papers published in peer-reviewed journals do not, in general, uniquely attribute the research results to a single mechanism of support. No researcher finds his or her support entirely from a MRSEC, and both MRSEC and other contributions are influenced by participation in the MRSEC. Even at the level of an individual researcher’s activities, it is categorically impossible to separate out the uniquely “MRSEC-enabled” research. Site visits confirmed these impressions (see Box 3.2).

This caveat should be borne in mind when interpreting these analyses. Despite this intrinsic limitation, however, the committee designed and carried out several exercises, examining the activities enabled by the MRSEC program using several different techniques to “separate out” the MRSEC contributions and to construct “control” groups for comparison.

Finally, the committee emphasizes that its goal was not to evaluate the MRSEC program specifically, nor to recommend the continuation or termination of the program, but rather to describe and characterize its impact. Ideally, the committee would like to have answered a pointed question: If one had the opportunity to reinvest the annual budget of the MRSEC program purely on the grounds of its research impact, are there compelling examples of “what could not have happened otherwise?” Unfortunately, the inability to clearly separate what is “because of MRSEC” from “what is not” made it impossible to answer this question.

Moreover, any research, even by an individual researcher associated with a MRSEC, is a combination of activities supported both inside and outside the MRSEC. Thus, even if MRSECs have played a unique role in the research enterprise, such as in enabling the formulation of research projects that could not otherwise have been envisioned, there is no easy way to provide substantiation. Although the committee was unable to identify MRSEC-enabled research in “blind taste tests,” it successfully assessed the overall research quality in comparison with the research enabled by other mechanisms and elsewhere around the world. The basic questions to be answered are whether the research enabled by the MRSEC program is distinctive, if it is worthwhile and of high quality, and, finally, whether it is a good investment.

Many studies have tried to assess the quality of research programs in terms of objective criteria such as the citation numbers. A previous evaluation of the MRL program by the MITRE Corporation for the NSF concluded that there were no discernable trends in the quantity of publications or their citations when comparing MRLs with similarly funded programs. The committee’s exploration of the citation index produced similar conclusions. The committee found that identifying a set of comparable institutions was difficult, that the data would not be easy to obtain, and that the results would at best indicate the average output of the MRSECs and

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

BOX 3.2

Site Visits

The MRSEC Impact Assessment Committee conducted more than a dozen site visits at institutions that either have a MRSEC or a similar center-based research structure, that were contemplating a MRSEC application, or which had had a MRSEC that closed (see the section entitled “Site Visits” in Appendix F, “Data-Gathering Tools,” for a list of the institutions visited). These visits prompted candid conversations with researchers that provided valuable anecdotal information and firsthand impressions. The committee found this feedback very useful in its assessment of the MRSEC program. Below are some excerpts from conversations with faculty and staff.

  • From its site visits, the committee heard comments that, “Centers can only succeed if they help us integrate between disciplines,” and that “Without impetus from outside [the university], it is hard to initiate a center, despite whatever latent good will and intentions there are.” When asked to differentiate the MRSEC-style research center from departmental centers, some said, “Centers are intellectual foci of effort. They are at a larger scale than just one department with some of its faculty. You need to cut across more fields of research to really attack new problems and push forward; you need more than one or two departments.” When asked about their university’s perspective on centers, many university administration officials commented that they view centers with federal funding as having a higher degree of validity because they have received some external commitment and recognition.

  • When asked about canceling the MRSEC program, one university official opined, “I don’t think that the campus and state would take the initiative to invent such a center without the external incentive unless the affected topics were related to human health and wellness. Also, this campus is isolated geographically from the industrial community,” and would not be as able to engage industries in relationships pertaining to physical science projects. Others echoed these thoughts, saying that such centers are one of the only mechanisms, externally funded, that cut across disciplinary boundaries and the stovepipes of academic departments. Others commented that single-investigator awards are typically only 3 years, and the longevity of a center grant enables much more creativity, flexibility, and even security in trying out research ideas. A final comment suggested that the National Science Foundation fulfills a key national goal by providing support for basic research that is not directly connected to product commercialization (as opposed to state and local industry programs), and since centers are a key mechanism for supporting the basic-science enterprise, they should be continued.

their comparison group. In most areas of endeavor, it is not the average that leads to remarkable advances but, rather, remarkable discoveries that are large fluctuations from the norm.

While it was difficult to separate research uniquely enabled by the MRSEC program from research that was made possible by other means, the committee was clearer about causation. For instance, many of the more recently established NSF Nanoscale Science and Engineering Centers (NSECs) are located at institutions that have MRSECs; of the 10 active NSEC awards, 3 are at institutions without active MRSECs, and at least 1 more is in a research area very different from the

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

corresponding MRSEC. Do MRSECs enhance the probability for NSEC awards? Or does experience in the MRSEC competition simply add to an institution’s competitive edge? One could wonder about the potential for a “chicken-and-egg” problem at a strong institution that was awarded a MRSEC: which came first, the strong campus research effort or the center? In the committee’s judgment, the competitive selection process for MRSEC awards puts the burden on the pre-existing strength of the institutional research effort. While a MRSEC may enhance an institution’s materials research programs, it simply cannot bring them into being.

ANALYSIS OF SELECTED CONTRIBUTIONS FROM MATERIALS RESEARCH

The committee identified a seemingly promising exercise that ended up rather inconclusively. Based on personal judgment and discussion with colleagues, the committee constructed a list of selected important materials discoveries and inventions from 1960 to 2000 (see Figure 3.2 for the subfields in which these discoveries occurred). Because the list was subjective, the committee chose not to publish it here. The committee then identified where the research leading to each discovery had been done and, in particular, whether it had originated in the MRSEC program or its predecessor MRL program. The list contained very few items that occurred in the past decade and thus the discoveries on the list significantly predated the MRSEC program per se. While this list is admittedly subjective and does not purport to be definitive, it revealed that the number of discoveries attributable to U.S. universities is rather limited. Given the generally recognized quality of U.S. universities in materials research, it is surprising that only 5 of these 27 discoveries are attributable to U.S. universities. Three were attributable to MRSECs. The committee does not want to overstate the implications of this ad hoc analysis, but it at least suggests that MRSEC research is an important part of a U.S. university materials research portfolio (see Box 3.3).

The majority of the discoveries were undertaken by individuals or small groups of about two investigators. Many of the discoveries originated in the predominantly industrial research laboratories in the United States, which reflected the time period (1960-2000) considered. Many of these labs (AT&T, IBM, Xerox, GE, Exxon, and so on) have been greatly reduced or eliminated, raising important questions about whether MRSECs can compensate for these losses. One should note, however, the parallel between a scientific breakthrough and car racing—the car and the driver get all the credit for a win, but in truth a much larger team is needed to enable a victory.

A small fraction of these breakthroughs took place in universities with MRSECs or MRLs (see Figure 3.3). Although this may appear somewhat discouraging regarding the impact of MRSECs as primary sources for innovative materials, funding

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.2 Distribution of selected major materials research discoveries, by materials research subfield. A handful of the 27 discoveries identified by the committee did not fit into these categories and are labeled “Other.”

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

BOX 3.3

The MRSEC at the California Institute of Technology

One example of a successful MRSEC is at the California Institute of Technology (Caltech). A thumbnail sketch of Caltech’s Center for the Science and Engineering of Materials provides a view of the typical organization and breadth of activities of a MRSEC. The Caltech MRSEC focuses on several interdisciplinary areas.

The research program is organized into four interdisciplinary research groups (IRGs) and two seed projects. The IRGs are Biomolecular Materials, Ferroelectric Thin Films, Mesophotonics, and Bulk Metallic Glasses. The work in Biomolecular Materials explores the control of self-organization that can be achieved in polymers of absolutely defined comonomer sequence—genetically engineered artificial proteins—and the control of spatial arrangement and size that can be templated using surfactant nanostructures. The Ferroelectric Thin Films group aims to enable ultrahigh displacement microactuators based on high-strain ferroelectrics. The project on Mesophotonic Materials is motivated by advances in the synthesis and theoretical understanding of materials designed to manipulate light on scales at and below the wavelength of light, in order to move into the revolutionary domain of devices on scales of tens of nanometers.

The program on Bulk Metallic Glasses, which has been particularly effective in its industrial interactions, investigates the processing, microstructure, and mechanical behavior of bulk metallic glasses (BMGs) and their composites. The researchers are investigating the basic science and engineering that will enable new strategies to produce BMGs in which a crystalline phase is introduced to resist shear localization, creating a BMG composite with enhanced material properties. Some of this work has already reached the stage of commercial application in such products as cellular telephone cases and other electronic device packages. Future efforts, in conjunction with a number of industrial partners, involve applications in a wide variety of commercial, biomedical, and military applications.

levels must be considered. The total budget of the MRSEC program is about $50 million per year compared with about $2 billion per year spent for (basic and applied) materials research by the U.S. government, and the approximately $4 billion per year spent worldwide by governments on materials research. The contributions seem to be larger than might be expected simply from the funding ratio. The fraction of MRSEC dollars to total materials dollars is 0.05/4, or 1.25 percent. No statistical analysis of these fragmentary observations is possible; however, it is possible to say that there are discoveries of the highest significance occurring within the MRSEC program, as gauged by this subjective survey.

As can be seen later in this section, there is almost an orthogonality between the types of institutions responsible for “major discoveries” and “top cited papers,” the former originating in industrial laboratories and the latter in universities. The committee suggests that many of the more recent fundamental breakthroughs occur in academe, often with MRSEC-funded facilities, whereas materials discoveries more closely linked to commercial products were more naturally done in industrial settings. The trend may reflect the passing of the torch from the formerly powerful industrial labs to universities.

The field of organic/polymeric conductors was supported from its inception

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.3 Characteristics of the most significant discoveries in materials research. Where each breakthrough occurred. Note the predominance of U.S. industry. Number of senior researchers involved in each discovery. Note that most discoveries were made by teams of only one or two senior investigators.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

BOX 3.4

Progress on Conducting Polymers

Alan Heeger began his university career at the University of Pennsylvania (Penn) in 1961, just one year after Penn was established as one of the initial three materials Interdisciplinary Laboratories (IDLs) (see Table 2.2 in this report). His initial research was in metal-insulator transitions, particularly in one-dimensional systems. This early work was the genesis of Heeger’s work with Alan MacDiarmid leading to the development of plastic electronics. Alan Heeger was director of the Penn Materials Research Laboratory (MRL) when this work accelerated with the synthesis of polyacetylene published in 1977 by Shirakawa, MacDiarmid, Loius, Chiang, and Heeger (see Figure 3.4.1). Financial support from the MRL, along with major support from the Office of Naval Research, enabled these seminal discoveries that led to the award of the 2000 Nobel Prize in chemistry to Shirakawa, MacDiarmid, and Heeger. In 1982, Alan Heeger moved to the University of California at Santa Barbara, where soon after he successfully proposed a new Materials Research Group on conducting polymers. This became the nucleus for the founding of the University of California at Santa Barbara MRSEC in 1993.

FIGURE 3.4.1 Alan Heeger. Image of the 1977 Nobel Prize–winning paper.

in the MRL program (see Box 3.4). This highlight of the MRL/MRSEC program is on the list of selected major materials discoveries (see Box 3.5 for other examples of MRSEC research). Early contributions to the fundamental physics of quasi-one-dimensional conductors as new materials were produced in the MRLs, in U.S. universities, and in European and Japanese laboratories. A breakthrough in

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

BOX 3.5

Examples of MRSEC Research

Magnetic Tubules: Cellular Tracks Follow the Field at Pennsylvania State University MRSEC


Motor proteins deliver intracellular cargo to specific locations inside cells. These so-called kinesin motors take 8 nanometer (nm) steps along intracellular highways, called microtubules, 25 nm wide. This transport machinery can be reassembled outside the cell and used to transport nanoscale cargo for separations, sensors, assembly, and other biomechanical devices. However, to fully harness these biological motors outside the cell, a means is needed both to attach cargo and to lay down the tracks at the desired locations and orientations. Materials Research Science and Engineering Center (MRSEC) researchers at Pennsylvania State University are using magnetic fields to control the placement and transport of microtubules. In the reverse of a mobile engine on a stationary railroad track, the biomotor track (the microtubules) is actually mobile while the motors (kinesins) are bound upside-down to the surface, ready to push the microtubule along like a person body surfing at a rock concert. Magnetic nanoparticles of CoFe2O4 are attached to the microtubules as magnetic “handles.” By adjusting the ambient magnetic field, the microtubules can be reoriented, allowing them to be transported in any desired direction. Even weak magnets can direct the biomotor-driven transport of thousands of microtubules at once. Magnetically labeled microtubules also provide a new tool for investigating the role of microtubules and motors in cellular processes such as cell division, axonal transport, and flagellar motility.


High-Performance Transparent Inorganic-Organic Hybrid Thin-Film n-Type Transistors at Northwestern University MRSEC


Thin-film transistors, already indispensable in a number of portable electronics, would benefit from optical transparency and compatibility with flexible, lightweight plastics. Transistors with these qualities would be a major advance if they could be fabricated by a scalable, large-area process. Researchers at the Northwestern University MRSEC have adopted a hybrid approach in developing “invisible” thin-film transistors that heterogeneously integrate a transparent, inorganic semiconductor with a large carrier mobility and a nanoscopic, organic gate dielectric.


Bacterial Nanoreactors at University of Southern Mississippi MRSEC


The nanometer-scale polyhedral protein compartments (carboxysomes) found in many bacteria harbor an enzyme (RubisCO) that converts carbon dioxide to sugars, which in turn are used by the cell to synthesize other biomolecules. One of the carboxysome shell proteins (e-carbonic anhydrase) was found to catalyze the dehydration of bicarbonate and to direct the resulting CO2 toward the inside of the carboxysome, where it is efficiently metabolized by RubisCO. It is currently thought that the orientation of the carbonic anhydrase in the protein shell constitutes a chemical diode that makes the carboxysome shell directionally permeable to CO2 and allows it to function analogous to polymer film-immobilized catalysts. Work is currently under way to understand the self-assembly of carboxysome protein components, which ultimately may guide efforts to synthesize selectively permeable protein-based films for potential pharmaceutical or manufacturing applications.

the synthesis and characterization of polyacetylene turned the field around. This collaborative research was conducted at an MRL and involved strong interactions between the physics and chemistry departments at Pennsylvania State University and a group in Japan; it led to new materials, conducting polymers, and new con-

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

cepts: solitons, fractional charge, spin-charge separation. Further developments by several groups led to new technology, especially for organic optical displays; the formation of several companies in the United States and overseas; and the awarding of the 2000 Nobel Prize in chemistry to Alan J. Heeger, Alan G. MacDiarmid, and Hideki Shirakawa.

The committee experimented with a related exercise by trying to identify the founding papers in several topical areas of research. For instance, in the field of magnesium diboride research, the breakthrough paper is clearly the 2001 Nature article by Nagamatsu and colleagues (cited in the list below). Employing a scientific publication citation analysis tool (Scopus), the committee identified the top 20 most highly cited (subsequent) papers that cited the founding paper. Using institutional affiliation and some knowledge of the MRSEC IRG membership, the committee examined the role of MRSECs in this new set of “soon afterward” papers. The results were largely inconclusive, but interesting nonetheless.

  • MgB2: J. Nagamatsu, N. Nakagawa, T. Muranaka, Y. Zenitani, and J. Akimitsu, “Superconductivity at 39 K in Magnesium Diboride,” Nature 410 (6824): 63-64, 2001. Cited 1,804 times.

    • Of the top 20 most highly cited articles and reviews, none was from an institution with a MRSEC.

  • Spintronics: Y. Ohno, D.K. Young, B. Beschoten, F. Matsukura, H. Ohno, and D.D. Awschalom, “Electrical Spin Injection in a Ferromagnetic Semiconductor Heterostructure,” Nature 402 (6763): 790-792, 1999. Cited 867 times.

    • Of the top 20 most highly cited articles, 1 was from an institution with a MRSEC.

  • Magnetic semiconductors: T. Dietl and H. Ohno, “Zener Model Description of Ferromagnetism in Zinc-Blende Magnetic Semiconductors,” Science 287 (5455): 1019-1022, 2000. Cited 1,284 times.

    • Of the top 20 most highly cited articles, 2 were from institutions with MRSECs.

  • Novel oxides: P. Schiffer, A.P. Ramirez, W. Bao, and S.-W. Cheong, “Low Temperature Magnetoresistance and the Magnetic Phase Diagram of La1xCaxMnO3,” Physical Review Letters 75 (18): 3336-3339, 1995. Cited 1,061 times.

    • Of the top 20 most highly cited articles, 5 came from institutions with MRSECs.

  • Stripes in high-temperature superconductivity: J.M. Tranquada, B.J. Sternlieb, J.D. Axe, Y. Nakamura, and S. Uchida, “Evidence for Stripe Correlations of Spins and Holes in Copper Oxide Superconductors,” Nature 375 (6532): 561-563, 1995. Cited 1,266 times.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
  • Of the top 20 most highly cited articles, 6 were from institutions with MRSECs.

PUBLICATION CITATION ANALYSES

A study of highly cited papers was conducted to obtain a more objective overview of current and past research directions and impact in the materials community. A potential metric for examining the overall impact of published research results is to consider publication citations—the number of times subsequent papers refer to an earlier work; one variation on this approach was already discussed. A number of assumptions need to be made to imbue this technique with credible utility.2

The committee notes an important limitation of this analysis in advance: because the MRSEC program is only about one decade old, any research publications authored under its auspices are still relatively nascent in the field. That is, truly eminent articles generally take 10 to 15 years to demonstrate their impact on the field.3 Thus, the committee’s efforts to assess the research impact of the MRSEC program through a study of its publication citations is a bit premature. In its defense, the committee chose to compare the MRSEC program to just the past decade of materials research papers in order to include the same systematic error in the reference case. In order to avoid replicating the MITRE report described earlier and in order to keep its task tractable and focused on the MRSEC program, the committee chose not to analyze the full legacy of the IDLs and MRLs that led to the MRSECs. However, in so doing, the committee’s analysis could be interpreted to simply conclude that the MRSEC program is too young for impact assessment.

In a larger sense, the committee also sought to investigate some of the “urban myths” surrounding the MRSEC program. The committee made the most progress in addressing the question of whether MRSEC-enabled research results were empirically distinguishable in character and/or quality from other research. Box 3.6 shows some characteristic publications from the MRSECs.

Top 100 Most Highly Cited Papers in Materials Research

Using a scientific journal publication citation tool (Scopus), the top 100 most highly cited papers in materials research since about 1996 were identified. A break-

2

See, for instance, David Adam, “Citation Analysis: The Counting House,” Nature 415: 726-729 (2002) for a discussion of the intrinsic limitations of these techniques.

3

Using the Essential Science Indicators tool provided for public use online by the ISI Web of Knowledge, the 100 most highly cited papers of all time in the field of materials science were queried. Of these 100, more than 40 percent were published before 2000.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

BOX 3.6

Selected Research Results Enabled by the MRSEC Program

Despite the difficulty in empirically separating MRSEC-enabled research from research supported by other mechanisms, the MRSEC Impact Assessment Committee was able to examine a set of research papers self-reported by the MRSECs. Many of these had high citation indices; a very small subset are listed here as examples.

  • J. Needleman, “A Continuum Model for Void Nucleation by Inclusion Debonding,” Journal of Applied Mechanics-Transactions of the ASME 54 (3): 525-531 (1987). (Brown University MRL/ MRSEC, 461 citations)

  • J. Park, A.N. Pasupathy, J.I. Goldsmith, C. Chang, Y. Yaish, J.R. Petta, M. Rinkoski, J.P. Sethna, H.D. Abruña, P.L. McEuen, and D.C. Ralph, “Coulomb Blockade and the Kondo Effect in Single-Atom Transistors,” Nature 417 (6890): 722-725 (2002). (Cornell MRSEC, 337 citations)

  • C.S. Chen, M. Mrksich, S. Huang, G.M. Whitesides, and D.E. Ingber, “Geometric Control of Cell Life and Death,” Science 276: 1425 (1997). (Harvard MRSEC, 910 citations)

  • B.M. Discher, Y.-Y. Won, D.S. Ege, J.C.-M. Lee, F.S. Bates, D.E. Discher, and D.A. Hammer, “Polymersomes: Vesicles Made from Diblock Copolymers,” Science 284: 1143 (1999). (Minnesota MRSEC, 310 citations)

  • D.Y. Zhao, J.L. Feng, Q.S. Huo, N. Melosh, G.H. Fredrickson, B.F. Chmelka, and G.D. Stucky, “Triblock Copolymer Syntheses of Mesoporous Silica with Periodic 50 to 300 Angstrom Pores,” Science 279 (5350): 548-552 (1998). (Santa Barbara MRSEC, 1,489 citations)

  • A. Thess, R. Lee, P. Nikolaev, H.J. Dai, P. Petit, J. Robert, C.H. Xu, Y.H. Lee, S.G. Kim, A.G. Rinzler, D.T. Colbert, G.E. Scuseria, D. Tomanek, J.E. Fischer, and R.E. Smalley, “Crystalline Ropes of Metallic Carbon Nanotubes,” Science 273 (5274): 483-487 (1996). (Pennsylvania State University MRSEC, 1,898 citations)

  • Z.A. Peng and X. Peng, “Synthesis of High Quality Cadmium Chalcogenides Semiconductor Nanocrystals Using CdO as Precursor,” Journal of the American Chemical Society 123: 168 (2001). (Oklahoma/Alaska MRSEC, 333 citations)

down of these hundred papers in terms of subfields is shown as the light-colored bars in Figure 3.4. The affiliation information from the citations was examined to determine several characteristics for each paper: nationality, national laboratory/industry/university origins, and MRL/MRSEC connection. The MRL/MRSEC connection was evaluated by determining whether the institution had an operating MRSEC at the time of publication.4 The results are plotted in Figure 3.5. It is

4

The committee notes that the assumption employed here is a weak point in the exercise. Materials research papers arising from institutions with MRSECs do not necessarily come from MRSEC-enabled research. However, the committee could not find a better alternative. Even cross-checking the principal investigators with MRSEC faculty lists would not work because faculty conduct research under many different auspices and with many different funding sources.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.4 Grouping by subfield of the top 100 most-cited materials research papers (light bars). For comparison, the number of highly cited papers by subfield, for the set of “top 5” papers reported by the Materials Research Science and Engineering Centers, is included as the second bar (dark) in each category.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.5 Analysis of the affiliations listed by the authors of the overall top 100 most highly cited papers in materials research, 1996-2006. The total number of entries exceeds 100 because the committee could not clearly assign the origins for several papers. NOTE: MRL, Materials Research Laboratories; MRSEC, Materials Research Science and Engineering Centers.

noteworthy that institutions with an MRL/MRSEC program accounted for about 10 percent of the most highly cited papers worldwide and about 20 percent of those from the United States. These are considerably larger percentages than might be indicated by the relative funding levels mentioned earlier. This is the first indication that institutions with MRSECs are among the leaders in materials research. However, the task of directly associating research quality with the MRSEC program is complex, since it cannot be distinguished whether the best institutions are likely to succeed in the MRSEC competition or whether the MRSECs play a dominant role in the materials effort at these institutions. It is probable that both effects are present. Separating institutional publication impact caused by the MRSEC program as opposed to simple correlation with a MRSEC (or even as part of the reason for winning a MRSEC) is hard. The difficulty in assigning credit for these highly cited papers is that most authors report support from many sources.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

Portfolio of MRSEC Research Activities

At the committee’s request, many of the current MRSECs provided a list of their 5 most cited papers over the past 10 years.5 The distribution of MRSEC “top papers” by subfield was compared with the distribution of “top papers” for the entire materials field, thereby indirectly testing whether the MRSEC research portfolio matched the overall global materials research portfolio. A close examination of Figure 3.4 shows the result: the number of “top 100 papers” per subfield is plotted alongside the number of “top MRSEC” papers per subfield. The comparison indicates a strong correspondence between high-impact research done in the MRSEC program and the interests of the materials community as a whole.

The committee also independently examined the current MRSEC research portfolio. Although the MRSEC program is programmatically contained within the NSF Mathematical and Physical Sciences Directorate’s Division of Materials Research (i.e., separate from the NSF Engineering Directorate), the intended scope of the MRSEC program includes materials engineering. The list of research topics studied by the current suite of MRSECs has very limited intersection with the engineering side of materials (see Appendix C for a list of the current IRG research topics).

Average MRSEC Citation Impact

To gauge the “average impact” of each MRSEC, the average citations for the top 5 papers from each MRSEC were computed (see Figure 3.6). Each entry represents the average citation count for one MRSEC. The average number of citations per MRSEC ranged from 37 to 994 per highly cited paper. Note that for the 100 most cited papers in materials research from 1996 to 2006, the average number of citations per paper was 892. Thus, the “best” materials research papers are better (in terms of citations) than the “best” MRSEC papers. Although it would not be reasonable to expect the best papers to arise exclusively from MRSECs, it is noteworthy that some of the best MRSEC papers do rate among the best papers overall. As shown in Table 3.1, the average citation rate for MRSEC-related papers is about 13. This average number weights over many broad valleys in addition to high peaks; for instance, younger MRSECs with less-established research programs in new areas tend to have lower citation rates and therefore pull the average down. However, the average citation count per paper for all materials science papers

5

The committee used this “top 5” set in several key exercises; although self-reported by the MRSECs, it represented a list of more than 100 research results that MRSECs indicated playing a role in. The committee had no other reliable means of obtaining a set of MRSEC-enabled research papers beyond asking the MRSECs themselves.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.6 Distribution of average number of citations for the set of “top 5” papers reported by each currently active Materials Research Science and Engineering Center, 1994-2006. The lowest bin extends down to zero. The average of this histogram is about 350. The top 100 most-cited papers had an average of 892 citations per paper, with a median of 775.

TABLE 3.1 Impact of MRSECs Compared with the German System of the Max Planck Institutes

Research Institute

Publications 1995-1996 (No.)

Citations 1995-2006 (No.)

Citations per Publication (No.)

MRSECs

483

6,269

13.0

MPI-FKF

6,309

65,279

10.3

MPI-MF

3,316

31,349

9.5

MPI-POLY

3,455

48,162

13.9

MPI-MSP

1,781

19,427

10.9

NOTE: MRSECs, Materials Research Science and Engineering Centers; MPI-FKF, Max Planck Institute for Solid State Research, Stuttgart; MPI-MF, Max Planck Institute for Metals Research, Stuttgart; MPI-POLY, Max Planck Institute for Polymer Research, Mainz; MPI-MSP, Max Planck Institute for Microstructure Physics, Halle.

(368,111) is 4.48.6 The top MRSEC papers thus are much better than the average materials research papers.

The committee compares the top MRSEC papers to the average papers for two reasons. First, it supports the observation that MRSECs have contributed several important research advances to the field. Second, the committee found it difficult to extract independently average values for the set of all MRSEC papers—mainly

6

Similarly for physics, the average is 7.22 citations per for 834,162 papers; for chemistry the average is 8.24 citations per paper for 1,028,375 papers, according to Scopus.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

because MRSEC papers are not well labeled in the literature; the only known data set is the one provided by MRSECs themselves. The committee found that the average MRSEC paper performed similar to the average materials paper.

Comparison of Citation Impact for Max Planck Institutes and MRSECs

The committee contacted institutions in other countries to discuss techniques for assessing the quality of their research activities, particularly those activities that are center-based. While there was no consensus as to the value of different procedures, many of the institutions contacted took what they considered was the “easy way out” and used publication-citation indices. Bernhard Keimer at the Max Planck Institute in Stuttgart kindly offered the services of his library staff to compare citations for several Max Planck Institutes (MPIs) and the MRSEC program. The search picked out papers in which a MRSEC was explicitly listed in an author’s address field; thus, many papers were missed if the author had a different home department as his or her address. Nonetheless, the comparison, shown in Table 3.1, had some value.

From these data, the MRSECs compare favorably in citations per publication with the MPIs. The library scientists who compiled the data noted that there was no significant difference between the MRSECs and the MPIs in citations. MPIs are among the premier institutions for materials research in Europe. This cursory survey confirms the previous results of the MITRE report which suggested that citation index comparisons do not sharply distinguish between research results from MRSECs and those from elsewhere; in fact, the MRSECs seem to be just as successful as the German MPI research centers by this metric.

Analysis by Subfields Within the MRSEC Program

The committee also examined the finer structure of the materials research field to determine if MRSECs contributed distinctively in certain subfields—particularly the areas on which the IRGs focused. As part of a survey from each MRSEC, the committee requested a list of the “top 5 scientific questions currently being addressed.” Synthesizing those responses and the IRG descriptions, the committee developed the following list of 23 subfields of materials research that may show differential MRSEC impact. The committee does not believe that this list is definitive.

  1. Biomaterials

  2. Ceramics

  3. Composites

  4. Ferroelectrics

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
  1. Granular Material

  2. Interfaces

  3. Liquid Crystals

  4. Magnetic Materials

  5. Materials for Energy Storage

  6. Materials Growth

  7. Mesoscopics

  8. Mechanical Properties

  9. Nanomaterials

  10. Nanostructures

  11. Organic Semiconductors/Molecular Electronics

  12. Oxides

  13. Photonics/Optical Materials

  14. Polymers (including copolymers)

  15. Self-Assembly

  16. Spintronics

  17. Superconductivity

  18. Supramolecular Materials

  19. Transport Properties

It is clear that there is a great deal of overlap with the topics covered in the 100 most-cited materials papers and those in the most-cited papers from the MRSECs as shown in Figure 3.4. The committee chose to pursue the research-subfield impact hypothesis in two different ways: an objective analysis relying on publication citations and a subjective analysis using perceived standing from a panel of voting experts.

Using a scientific publication-citation analysis tool (Scopus), the committee identified the top 30 papers since 1995 in each subfield. Nearly 700 papers were selected for analysis. To get an idea of the level of activity in different subfields, the total number of citations for the top 10 papers in each subfield was tabulated (as shown in Figure 3.7). Comparison with Figure 3.4 indicates that there is substantial overlap of the most active areas with the corresponding efforts in the MRSEC program. Figure 3.8 shows a breakdown by country of origin, and Figure 3.9 shows the total.

Further analysis of the data set involved sorting the top-cited papers in terms of the type of institution where the research was performed. As can be seen in Figure 3.10, over the past 10 years it is the universities that have supplied the most highly cited research. This contrasts with the finding from the first look at significant new materials, which showed that they most often originated in industrial laboratories. This may result either from the higher value placed on publications in academe or from the decline of support at many industrial research laboratories.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.7 Total number of citations for the top 10 papers in each subfield.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.8 Top-cited papers in each of 22 subfields of materials research, by world region (United States, Europe, Asia), 1995-2006. The United States remains the largest single source (54 percent) of highly cited papers, followed by Europe (28 percent) and Asia (12 percent). The United States has the most top-cited papers in every subfield except mesoscopics, although Europe has similar numbers in interfaces, energy storage materials, polymers, and supramolecular systems. Top-cited papers from Asia seem to be concentrated most strongly in several areas, particularly the key areas of superconducting and magnetic materials. The integration over subfields is shown in Figure 3.9.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.9 Geographical region of origin for top-cited papers summed over the 23 subfields of materials research, 1995-2006: number of papers and percentage of total. Eighty-four percent of the papers had all authors from the same region.

FIGURE 3.10 Performers of top-cited materials research summed over the 23 subfields, 1995-2006: percentage by type of institution.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

One of the main objectives was to see whether this study of highly cited work could document that the MRSECs played a substantial role. Figure 3.11 shows the acknowledged funding sources for the most highly cited papers summed over the different subfields. The fraction associated directly with MRSECs is small, 4 percent of the total. Of course the fraction should be considered with respect to the total amount of research money that is provided by the different sources. (A more detailed discussion of the publications vis-à-vis the funding from different sponsors is presented later in this chapter.) However, another factor that must be considered is that most of the papers acknowledge more than one funding source. It is difficult to assign agency ownership to a discovery, a new material, or even just a publication.

FIGURE 3.11 Sources of funding for top-cited papers from U.S. institutions, summed over the 23 subfields of materials research: number of funding acknowledgments and percentage of total. Papers acknowledging Materials Research Science and Engineering Center (MRSEC) support are counted separately from those supported by other programs at the National Science Foundation (NSF). Seventy-seven percent of the papers acknowledged multiple sources of funding. DOE, Department of Energy; NIH, National Institutes of Health; DOD, Department of Defense.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

This analysis also found substantial evidence that MRSEC research is as collaborative as non-MRSEC research. The committee examined the set of MRSEC self-reported top-5 publications. The typical number of senior principal investigators was two per paper, and there were more with single senior authors than with three senior authors. Figure 3.12 compares the proportion of top-cited papers that have single and multiple authors among the different subfields of materials research. While there are some fields in which the top-cited papers are almost evenly divided between single and multiple authorship (i.e., nanomaterials and nanostructures, liquid crystals, organic and molecular systems, photonics and optics, and polymers and supramolecular materials), for the most part papers having multiple senior authors are the norm. Overall, these data indicate that 65 percent of all top-cited papers involve multiple senior authors; however, Figure 3.13 shows that the most likely collaboration is between pairs of senior authors, with vanishing incidence of collaborations with three or more senior authors. There is no evidence that top-cited papers by MRSEC investigators display a different trend, although a primary argument used to rationalize the MRSEC organization is that these larger-scale collaborations are only possible in centers that include a large number of researchers from different departments. Thus, value added by MRSECs in collaborative research is likely on the “input” side (conceptual collaboration in choosing and initiating research directions) rather than the “output” side (research results as measured by published papers).

It is important to note that not all these collaborations are within single institutions. Figure 3.10 showed the distribution of top-cited papers, grouped over all subfields, with respect to the institution of the authors. University-based researchers, including those at MRSEC hosting institutions, are the authors of the highest percentage of top-cited papers worldwide (69 percent), with the remaining 30 percent coming from industry and government laboratories. The committee notes that 44 percent of all top-cited research papers involved collaborations among multiple institutions and also multiple institution types (i.e., collaborations between universities and national laboratories, industry and university, and so on). The committee sought also to quantify the prevalence of international collaboration among these top-cited papers. It was found that collaboration remains largely confined to individual countries, with only 16 percent of the papers involving international collaboration.

In summary, the committee set out to establish the baseline publication-citation characteristics of the general materials research community in order to enable a comparison with the self-reported MRSEC publications. However, in so doing, the committee came to realize that distinguishing MRSEC-enabled research papers was much harder than imagined. As Figure 3.11 shows, identifying a “MRSEC” paper is a subjective assertion about “whose dollars” put the research task over the

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.12 Comparison of the number of top-cited papers by single or multiple senior principal investigators (PIs) among the different subfields of materials research. Sixty-five percent of the papers had multiple PIs.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.13 Proportion of top-cited papers with number of senior authors indicated, 1995-2006, summed over the 23 subfields. The key result is that more than half of the papers were authored by at least two senior principal investigators.

tipping point. The committee did not conduct a fully parallel analysis of the self-reported MRSEC papers as a result.

DEMOGRAPHICS OF RESEARCH PERFORMERS

A primary objective of the original MRL program that has continued into the MRSEC program is to provide a setting that stimulates and nurtures interdisciplinary collaborative materials research. The original MRLs were located at institutions that already had a substantial interdisciplinary materials effort. Figure 3.14 shows the mix of disciplines in the first MRLs. In the original 10 MRLs, about 26 percent

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.14 Reported disciplinary affiliations for participants in the Materials Research Laboratory program in the mid-1970s. SOURCE: MITRE Corporation, Evaluative Study of the Materials Research Laboratory Program, 1976.

of the researchers came from physics departments, with a similar number from materials science and engineering departments and chemistry departments. Similarly, about 23 percent of the participants were from other engineering departments (including electrical engineering). It is interesting to note from Figure 3.15 that this departmental mix is almost the same in current MRSECS, with the only meaningful change being a modest increase in the participation from physics departments. A generation after the establishment of the MRLs, there is no question that MRSEC research remains both broad and multidisciplinary, and perhaps one can make the argument that this intrinsic attribute of the MRL and MRSEC programs has led the trend in materials research more generally.

The committee finds, therefore, that on the metric of multidisciplinarity as measured by departmental affiliation on research papers, the MRSEC program performs similar to the overall materials research community. However, the committee did not examine the paper-by-paper distribution of departmental affiliations. And, as before, it could not find a clear way to measure what the degree of multidisciplinarity would have been in the absence of the MRSEC program.

Figure 3.16 shows the distribution of departments of authors of top-cited papers from the committee’s list of the most-cited papers by subfield. The distribution

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.15 Individual academic departments by discipline as a percentage of total departments participating in the Materials Research Laboratory and Materials Research Science and Engineering Centers programs. The most recent information, from 2004, is on the left-hand side of the figure. The order of departments from top to bottom on the chart is shown in the legend.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.16 Departmental affiliations of authors of top-cited materials papers, summed over the 23 subfields, 1995-2006: number of papers cited and percentage of total. NOTE: MSE, materials science and engineering.

is similar in many ways to the departmental mix in MRSECs, with almost identical representation of physics and chemistry in the two distributions and a somewhat weaker participation of materials science and certainly overall engineering in the author distribution. Assuming that the likelihood of a field producing a top-cited paper increases when there are more researchers in the field, these data suggest that the mix of disciplines represented in current MRSECs is similar to the mix found in the worldwide materials research community, which has become deeply interdisciplinary.

A modest longitudinal study was also performed to see how the division of top-cited papers among different types of U.S. and foreign research organizations has evolved over the period of time during which MRSECs and MRLs have been active. Here, the data were not broken out into subfields; instead, the 100 top-cited papers in materials research over each of the decades 1965-1974, 1975-1984, 1985-1994, and 1995-2006 were selected. This follows the initial survey in which the very general criterion “materials research” for the 100 most-cited papers was used. It is unlike the previous citation results, which were broken down into MRSEC subareas.

As shown in Figure 3.17, the percentage of top-cited papers is plotted for U.S.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.17 Sources of the top 100 most-cited materials papers as a percentage of total for four decades, 1965-2006. The order from top to bottom on the chart is shown in the legend. NOTE: MRL, Materials Research Laboratory; MRSEC, Materials Research Science and Engineering Center.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

universities, universities having a MRSEC or an MRL, national laboratories, and industrial laboratories, as well as the total foreign citations for each of the decades in Figure 3.17. (Similarly, the number and percentage of top-cited papers in each of the 23 subfields, since 1995, by region of origin, were shown in Figures 3.8 and 3.9.)

It is clear that the United States enjoyed a near-monopoly on top-cited materials papers in 1965-1974 but that this percentage has fallen steadily in subsequent decades to its current level of 54 percent as foreign governments invest in the creation of their own materials science and engineering (MSE) knowledge base.7 Considering that it may take a decade or more for citations to develop fully for pathbreaking work, this plot is a cause for concern. The percentage of top-cited work has remained roughly constant for U.S. national laboratories, industrial laboratories, and perhaps has even grown slightly for MRSEC-hosting universities over this 40-year period. Yet it is clear that the growth of top-cited papers from foreign institutions has largely been at the expense of top-cited papers from U.S. universities, which in the 1960s produced half of the top-cited work. Ensuring the strength of university-based materials research is crucial, not only because it is the single largest sector of materials researchers in the United States (see Figure 3.18), but also because this is where future generations of materials researchers—both domestic and foreign—will be trained.

THE LEADING GROUPS IN MATERIALS RESEARCH

To assess the perceived excellence of the programs in 23 different subfields of materials research that may show differential MRSEC impact, the committee undertook an informal survey of the opinions of experts. It was proposed initially that the experts be selected by choosing the senior authors of the top 10 most-cited papers from each of the 23 subfields; however, the committee decided that the list should be augmented by authors of highly cited papers who were not selected by the simple algorithm above. For the purposes of this exercise, “expert” is defined as one of the senior authors of one of the 10 most highly cited papers in each of the 23 subfields listed above. The experts were then contacted by e-mail with the following sample note:

Dear Dr. X,


We’re working on a National Research Council report on materials programs in the US. As part of the evaluation we thought it would be useful to find out where the best research is being done. To wit, we have identified a set of experts in materials related subfields and would like to solicit their opinions. We would therefore greatly

7

National Research Council, Globalization of Materials R&D: Time for a National Strategy, Washington, D.C.: The National Academies Press, 2005, p. 2.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.18 Affiliation of U.S. materials research community as estimated by demographics analysis of members of the Materials Research Society. The committee notes that the membership of the Materials Research Society is not broadly reflective of the overall composition of the materials research community, but it does have certain parallels to the university-based research community of Materials Research Science and Engineering Centers. (Courtesy of the Materials Research Society.)

appreciate your expert opinion of the top research labs (~10), world-wide, in the area of “granular materials.” Thank you for your help.


Sincerely,


NRC MRSEC Impact Assessment Committee

About 200 e-mail inquiries were sent, and 55 experts replied with lists. Several were experts in more than one field and provided several lists. It was not possible to meaningfully rank institutions in each subfield on the basis of these data. By combining the subfields, however, the committee found sufficient evidence from which to draw conclusions as to the reputation of different institutions in the overall area of materials research. The responding experts were widely distributed in foreign laboratories and universities and domestically in institutions with MRSECs and without them, as shown in Figure 3.19.

The votes were then tallied for each subfield. Almost all respondents sent “top 10 lists” with a note that the institutions were not in order of excellence. A vote was counted each time that an institution was mentioned on an expert’s list. The

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.19 Sources of “expert votes” in the survey of leading research groups in materials. The “United States” category does not include the votes from institutions with a Materials Research Science and Engineering Center (labeled “MRSEC” in the chart).

main finding of the exercise is shown in Figure 3.20. The institutions were sorted according to their status as domestic university or national laboratory/industrial laboratory, and European, Asian, and other (Canada, India, Israel). The U.S. universities are subdivided further as to whether or not there is a MRSEC on the university campus.

FIGURE 3.20 Location by general categories of the “leading research groups,” summed over the 23 subfields of materials research, according to votes by experts. Note that the data in this figure closely match the data in Figures 3.8 and 3.9.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

This survey of the most highly regarded research laboratories, across the subfields of materials, documents the leading role played by U.S. universities with MRSECs. As a group, U.S. universities with MRSECs are more identifiable with perceived excellence in materials research than any other grouping in the committee’s survey. Beyond the strong correlation of universities with MRSECs and perceived leadership, it is difficult to document whether this correlation is cause or effect. Some of the groups are MRSEC-supported while others are not. None of these world-leading groups is solely supported by a MRSEC. In at least one case, an expert specifically claimed that the institution’s MRSEC was not supporting the top-ranked group.

One might be tempted to contrast these results with those in the list of major discoveries described above in this chapter. Note, however, that the major discoveries were dominated by developments from 10 to 40 years ago, while this virtual-voting exercise was sensitive to contemporary impressions of perceived importance. Furthermore, the exercises were sensitive to different characteristics of research excellence: major discoveries versus overall high quality.

The distribution of leading groups was not uniform across the MRSEC program. Of course, there is some correlation between the number of years that an institution has had a MRSEC and its funding level and how well it does on this plot. But again, it is difficult to draw direct conclusions other than that MRSECs are situated at places that do excellent materials research.

RESEARCH IMPACT VERSUS FUNDING: QUALITY PER DOLLAR

Figure 2.2 in Chapter 2 showed the total federal funding for basic materials research between 1982 and 2002. The as-spent funding for materials research was almost constant in the 1980s, although the decade was followed by growth of about 35 percent between 1994 and 2000, in part reflecting the broadening of fields considered to be “materials research.”

Federal agencies support materials research at basic, applied, and developmental levels. When all of these expenditures are aggregated, the total exceeds $2 billion in FY 1995. The last published summaries of these expenditures were made for FY 1994 by the Materials and Technology Committee (MatTec), a subcommittee of the Federal Coordinating Council for Science, Engineering, and Technology reporting to the Office of Science and Technology Policy. It is important to note that virtually all major federal agencies supporting research are represented in this total. Figure 3.21 shows how the $2.124 billion total in 1994 was apportioned among the different agencies. NSF accounted for about 16% of this total, with about half (56%) of this total coming from DMR. (Note: subtracting facilities, $288 million for the Department of Energy [DOE] and $28 million for NSF, one gets a total of $1.793 million, with NSF at 15% and DOE at 34%.) The MRSEC percentage was

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.21 Expenditures by federal agency for basic, applied, and developmental materials research and engineering in millions of U.S. dollars and as percentage of total expenditure ($2,124 million) in FY 1994. The data do not include classified research or the construction and operating costs associated with facilities. SOURCE: Materials Technology Subcommittee, National Science and Technology Council, 1994 Annual Report, 1994.

about 25% of the total DMR expenditure, very similar to 2006 proportions. Altogether, MRSEC expenditures represent a very small fraction of the federal materials portfolio, amounting to about 2% of the total.

Of course, given the variety of activities funded by this portfolio and the different programmatic needs of the different agencies, there is no a priori reason to believe that the number of top-cited papers claimed by a given funding agency is proportional to its relative level of materials funding. The acknowledged sources of funding in the top-cited papers are shown in Figure 3.11. It is important to note that fully 77% of all papers acknowledged multiple sources of funding, implying that—increasingly—no one agency can take sole credit for funding any piece of work. There is no simple relationship between the level of federal funding and the percentage of top-cited papers enabled by this funding. For instance, DOE provides 43% of the support for all basic materials research and garnered 10% of the top-cited papers, while NSF provided 16% and was acknowledged in 16% of

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

the top-cited papers. These data are compiled in Figure 3.22, which compares the percentage of top-cited papers from 1996 to 2006 that acknowledge each agency with the percentage of the total federal budget for materials research spent by that agency.

While the overall monetary investment is very different, for the National Institutes of Health (NIH) and the Department of Defense (DOD) there is good agreement between the percentage of the top-cited papers and the percentage of the federal budget used to enable the research in these papers. The NSF represents relatively good value for investment, yielding top-cited research papers at almost twice the rate per dollar invested. The MRSEC program (2% of total materials investment, 5% of top-cited papers) is similar in “efficiency” to NSF overall (14.6% of total materials investment, 30% of top-cited papers). DOE has very high research expenditures but a relatively lower participation in top-cited papers. This is probably because the embedded cost of constructing and operating the DOE user

FIGURE 3.22 Comparison of number of top-cited materials papers to materials research expenditures for major federal agencies, 1996-2006. The solid line has a unit slope. The committee was not able to easily separate the university and national laboratory components of the DOE materials research budget; therefore, the DOE data point is not consistent with the university-research program budgets used for the other agencies. NOTE: NSF, National Science Foundation; MRSEC, Materials Research Science and Engineering Center; DOD, Department of Defense; DOE, Department of Energy; NIH, National Institutes of Health.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

facilities is not properly accounted for. The committee also notes that in selecting the data for the plot in Figure 3.22, the papers originating only from DOE laboratories were excluded in order to allow a comparison of DOE-supported university research with NSF-supported university research. Including the DOE national laboratories more than doubles the number of such papers. It can be concluded from Figure 3.22 that NSF research, including that carried out in MRSECs, is more likely to result in a top-cited publication than research funded by any other major agency. Within these statistics, there is little evidence that the MRSECs are more or less productive in this respect than any other NSF materials program.

SHARED EXPERIMENTAL FACILITIES

An often-cited key element of the MRSEC program is its explicit provision of shared experimental facilities (SEFs) at each center. The MRSEC program provides only limited explicit support for underwriting the capital costs of acquiring and maintaining a comprehensive instrument suite; rather, institutions must find other mechanisms for purchasing equipment (including the use of other NSF programs). MRSECs SEF funds, originating from budgets for IRGs, seeds, and facilities, are usually expended to cover operating costs of equipment and facilities such as maintenance, supplies, or portions of a salary for technical support staff (see Figures 3.23 and 3.24 for the distribution of MRSEC SEF budgets compared to each center’s budget). In 2004, DMR estimated that 12 percent of the MRSEC budgets was spent on capital equipment.

The research and training of students and postdoctoral associates in the MRSECs is completely dependent on the availability of SEFs with forefront capabilities. The MRSEC SEFs support a very broad range of materials research, which is essential to a broad community (including many supported by single-investigator grants) but is not altruistic, since the MRSECs could not carry out their own research without the user fees generated by these users.

As identified in the National Research Council (NRC) report Midsize Facilities: The Infrastructure for Materials Research,8 each materials research facility secures its capital and operating sources of support in a unique and highly individualized fashion. NSF MRSEC SEF support is often only one component of a complex array of funding mechanisms. Many MRSECs operate their SEF facilities with some user fees in order to recover some of the operating costs. In the larger MRSECs, the SEF user community is larger than the number of MRSEC students by at least a factor of 10. This large user base is necessary to pay SEF staff salaries that could not be sustained on the MRSEC budget alone. Another common feature was that

8

National Research Council, Midsize Facilities: The Infrastructure for Materials Research, Washington, D.C.: The National Academies Press, 2006, p. 62.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

FIGURE 3.23 Distribution of Shared Experimental Facilities budgets for most of the Materials Research Science and Engineering Centers in 2005-2006.

FIGURE 3.24 Correlation plot of total Materials Research Science and Engineering Center annual budget versus Shared Experimental Facilities budget. As expected, the correlation is positive but not linear.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

faculty and students who participated in the MRSEC would receive slightly discounted rates for using the instruments as compared with other users on campus. It is also important to note that in most cases, the instrumentation supported under the MRSEC SEF program element was part of a larger suite maintained by the institution.

In terms of impact, the committee believes that the shared facilities supported by MRSECs do have significant impact in the larger community, but the committee was not convinced that the MRSEC SEF support was dramatically more effective or leveraged more than any other instrumentation program. For instance, the committee learned from the Midsize Facilities report referred to above that operating costs for shared facilities (including the MRSEC program) are recovered about equally from federal grants, user fees, state awards, and institutional commitments. In examining the MRSEC annual reports, the committee observed a similar mix of reported sources of operating costs for the SEFs. However, the committee was unable to collect reliable data about sources of funds for the acquisition of capital equipment both inside and outside the MRSEC program. It is the committee’s view that MRSEC centers likely attract elevated levels of cost sharing from institutional leaders because they attract attention and provide explicit federal leveraging. The committee notes again that the specific impacts are probably diluted when viewing average trends. For instance, MRSEC participants at the University of Southern Mississippi credited the MRSEC with helping empower them to compete successfully for additional instrumentation awards from NSF and other agencies. The committee did not measure and compare the degrees of utilization of facilities inside and outside the MRSEC program and therefore cannot comment on the relative accessibility of the instrumentation to the broader community. The general perception seems to be, however, that MRSECs do allow wide-ranging access to their facilities.

The committee found that MRSECs invest in facilities at a rate comparable to DMR overall and that MRSECs provide about 20 percent of the DMR instrument portfolio. The committee also observed that—averaged over the past 10 years—institutions with MRSECs attracted “instrumentation for materials research” awards at roughly the same rate as institutions without MRSECs. The committee could not easily measure, however, whether institutions with MRSECs attracted a higher volume of instrumentation awards from sources outside NSF.

The committee collected information on the levels of MRSEC support that were directed toward shared experimental facilities from the annual reports of the MRSECs. The following observations were made.

  • In 2004, the average MRSEC budget spent on facilities was $276,000 (median $187,000) per year with a total reported investment of $6.6 million;

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

$6.6 million is about 13 percent of the annual $50 million MRSEC program budget.

  • In 2004, the portion of the DMR budget spent on equipment and instrumentation was $30 million (beyond that of the MRSEC program), or about 12 percent of the division’s full budget. In addition, DMR distributed about $5.7 million in equipment and instrumentation funds through the Instrumentation for Materials Research (IMR) program. Thus, in 2004, DMR invested about 18 percent of its annual budget (excluding the MRSECs) in equipment and instrumentation.

The committee observes then, that MRSECs invest in facilities and equipment at a rate similar to the overall DMR portfolio of investments. This analysis is extremely informal. It should also be noted that the committee did not compare the type of instruments bought through MRI and IMR awards and those secured through and for MRSECs. Another estimate suggested that MRSECs house about 20 percent of the overall federal investment at universities in million-dollar-class instrumentation for materials research. Also, it should be noted that SEFs in the materials area are not unique to MRSECs, but at institutions with larger MRSECs the SEFs are usually managed and operated by the MRSEC. If MRSECs did not do this, DMR would need to create some other strong facilities program to support materials research.

The MRSEC facilities budget also supports (at least in part) technical staff members, who train students and maintain the equipment. About $240,000 per year is spent on capital equipment. Estimating that half of the equipment purchased through the NSF instrumentation programs (DMR’s Instrumentation for Materials Research program or NSF’s agency-wide Major Research Instrumentation program) within DMR ends up in a MRSEC facility, another $5 million—or an average of about $200,000 per center—is added to this amount. Assuming a 10-year life for forefront materials characterization equipment, a center might thus afford a total inventory of equipment of about $4.4 million.

The committee also examined the potential correlation between MRSECs and instrumentation funding.

  • In the timeframe 1995-2006, the DMR IMR program awarded about $75 million of grants for the acquisition and development of instrumentation for materials research.

  • Of these awards, 30 percent (by dollar value) were made to institutions with MRSECs (that were active at the time).

  • MRSEC institutions received $402 million during this time frame, about 39 percent of the total $1.04 billion or so awarded by DMR to all institutions (excluding the $544 million for MRSEC funding).

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
  • Thus, the committee observes that institutions with MRSECs attract IMR awards roughly in proportion with their level of materials research activity (measured by DMR funding levels).

MRSECs are, however, taking a lead in working together on facilities rather than competing with one another. Led by the MRSEC at the University of California at Santa Barbara, the University of Southern Mississippi MRSEC, in collaboration with those at the University of Minnesota and the University of Massachusetts at Amherst, proposed to NSF to create a national facilities network. The award has recently been funded and will be used to encourage off-campus users to take advantage of the facilities; it also helps send students and faculty among the four sets of facilities at “internal” user rates.

The variations in actual capital spending equipment from one MRSEC to another are considerable because the availability of resources hinges on other features of the institution such as the development office, relationships with corporate sponsors, and so on. The recent National Research Council report on shared experimental facilities found that most SEFs that serve the large majority of the materials community have a $1 million to $50 million replacement capital cost, with an average of about $10 million.9 In fact, the U.S. investment in such facilities is currently well below the replacement level,10 estimated to be on the order of several billion dollars per year. At present, other sources of support for SEF equipment (typically, the universities themselves, or in some cases foundations) are not large enough to make up the difference in needed support. Thus, the average age of equipment in SEFs continues to increase, with many individual items being more than 20 to 25 years old.

FINDINGS AND RECOMMENDATIONS

Conclusion: Consistent with previous analyses, the committee found no simple, quantitative, objective measure to clearly differentiate the MRSEC research product from that of other mechanisms supporting materials science and engineering research.

The committee found the task of evaluating the impact of MRSEC research quite daunting, primarily because research papers published in peer-reviewed

9

National Research Council, Midsize Facilities: The Infrastructure for Materials Research, Washington, D.C.: The National Academies Press, 2006, p. 2.

10

National Research Council, Midsize Facilities: The Infrastructure for Materials Research, Washington, D.C.: The National Academies Press, 2006, p. 113.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

journals rarely attribute the results to a single support mechanism. Moreover, any research, even by an individual researcher associated with a MRSEC, is a combination of activities supported “inside” and “outside” the MRSEC. Thus, even if MRSECs have played a unique role in the research enterprise, such as in enabling the formulation of research projects that could not otherwise have been envisioned, there is no easy way to provide substantiation. It could be that the research enterprise has evolved over the past decade, leading to greater convergence and overlap between MRSECs and other research practices. Thus, it is not currently possible to distinguish the unique contributions of MRSECs.

General Finding: Sponsors of research are increasingly unable to claim “sole ownership” of research results; MRSECs are no exception.

Most research publications now acknowledge multiple sponsors. It is not possible to demonstrate that the MRSEC support yields leadership in discoveries, publications, or citations. In part this is because funding per MRSEC has decreased significantly in the past decade, so that each group requires multiple sponsors.

General Finding: Most highly cited publications contain one or two senior authors, indicating that the size of research collaboration is usually small.

Although the materials field is highly collaborative and the general belief is that the community benefits from interactions between local groups of many individual investigators in the same field, discoveries and publication records indicate that over 50 percent of the published papers are from individuals and groups of two.

Although the committee was unable to identify MRSEC-enabled research in “blind taste tests,” it successfully assessed the overall research quality in comparison with the research enabled by other mechanisms and elsewhere around the world. For instance, do published research results that acknowledge MRSEC resources achieve citation indices and other measures of impact comparable with research enabled by individual-investigator awards?

Conclusion: Overall, the MRSEC program produces excellent, frontier science of the same high standard as that supported by NSF through other mechanisms. In terms of quality, MRSEC research is at least on a par with that of other multiple-principal-investigator programs and individual grants in the United States and internationally, and is an important element of the overall mix for support of materials research, including support for big centers and single-investigator grants.

  • The outstanding discoveries, leading research groups, and most significant publications worldwide are associated with universities at which there are MRSECs.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
  • MRSECs are involved in the most active areas of materials research as established by their publication records compared with those of the entire field.

  • The MRSEC program has the same level of research collaboration as found in comparable national and international groups.

The committee studied a set of major breakthroughs in materials research over the past four decades. U.S. universities, and in particular MRSECs and their predecessors the MRLs, played a limited but pivotal role in a handful of these discoveries. The committee conducted several comprehensive analyses comparing citations of MRSEC-reported research publications and those of the broader research community. The distribution of MRSEC-reported “top-cited papers” across subfields of materials research was very similar to that of the top 100 most-cited papers. Affiliations of the top 100 research papers also showed a 10 percent contribution from institutions with MRSECs or MRLs. The committee also found that the top MRSEC papers were cited much more frequently than the average materials research paper but that the best-of-the-best materials research papers had significantly more citations. However, these papers generally predate the emergence of the MRSEC program. The committee also found that the MRSEC program has the same level of collaboration as that found in comparable national and international groups. To some extent this may be the ultimate success of the MRSEC program in having fostered this type of research at an early stage. Finally, the breakdown of departmental affiliations of MRSEC authors and those of the top-cited materials research papers were quite similar.

In two related exercises, the committee examined the global stature of MRSEC-related research groups. In comparison to the Max Planck Research Institutes of Germany, the MRSECs’ publication-citation rates were quite comparable. In a “virtual voting” exercise, the committee contacted researchers around the world in several different subfields and solicited their opinions about world-leading research teams. Research teams at institutions with MRSECs dominated the results.

Although many of these measures are of correlation and not causation, the committee came to believe that the research program enabled by MRSEC awards has been, in general, at least as effective as that enabled by other mechanisms.

Conclusion: The MRSEC program offers one of the principal opportunities in materials research to support shared experimental facilities (SEFs) that include not only equipment but also the personnel to provide training for students and to perform maintenance. Growing constraints on the per capita MRSEC budget have greatly diminished this ability, which is a concern for the infrastructure of materials research in general.

It should be noted that SEFs in the materials area are not unique to MRSECs,

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×

but at institutions with larger MRSECs the SEFs often are managed and operated by the MRSEC. If MRSECs did not do this, DMR would need to create some other strong facilities program to support materials research. A large user base is necessary to pay SEF staff salaries that cannot be supported solely by the MRSEC budget. The MRSEC SEFs support a very broad range of materials research (and sometimes other kinds), which is essential to a broad community (including many supported by single-investigator grants), but it is not just altruistic—the MRSECs could not carry out their own research without the user fees generated by these users. Shared facilities are an important resource for the overall community. For instance, individual investigators are unlikely to be able to afford to acquire and maintain a cutting-edge transition electron microscope, whereas a MRSEC SEF would be ideally suited to do so. Such an instrument sited at a MRSEC would be highly leveraged (because of institutional commitments to existing infrastructure and an established user community that would supply fees-for-use) and would greatly expand the opportunities available to the local research community. The committee encourages recent efforts by the centers and NSF to use modest supplemental grants to encourage and promote broader access to these facilities. These instruments are a core part of the value of the MRSEC program and can have enhanced national impact through improved communication and coordination.

As described in the beginning of this chapter, the committee concludes that the merit of the research enabled by the MRSEC program is comparable with the best of the materials research supported by other mechanisms. The committee notes, however, that it focused on measuring the impact of research results and that the ancillary benefits of MRSECs are not reflected by these metrics.

Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 71
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 72
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 73
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 74
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 75
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 76
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 77
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 78
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 79
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 80
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 81
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 82
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 83
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 84
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 85
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 86
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 87
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 88
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 89
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 90
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 91
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 92
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 93
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 94
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 95
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 96
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 97
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 98
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 99
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 100
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 101
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 102
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 103
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 104
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 105
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 106
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 107
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 108
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 109
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 110
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 111
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 112
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 113
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 114
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 115
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 116
Suggested Citation:"3 Assessment of Research and Facilities Impact." National Research Council. 2007. The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward. Washington, DC: The National Academies Press. doi: 10.17226/11966.
×
Page 117
Next: 4 Assessment of the Impact of MRSEC Education and Outreach »
The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward Get This Book
×
Buy Paperback | $70.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!