The National Science Foundation's Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward (2007)

In carrying out its impact assessment task, the committee first analyzed the issues broadly in several different categories: research, education and outreach, collaboration with other sectors, and other areas. After the following introductory remarks, an analysis of the impact of Materials Research Science and Engineering Centers’ (MRSECs’) research is presented in this chapter.

The Materials Research Science and Engineering Centers program (MRSEC program) was established by the National Science Foundation (NSF) in its Division of Materials Research (DMR) in 1994. As described in Chapter 2, the MRSEC program was born out of the decision to transform the Materials Research Laboratory (MRL) and Materials Research Group (MRG) programs to the current structure. The goal of this new initiative was to provide focused support for complex interdisciplinary materials research and education at the university level. To receive a MRSEC award, an institution must:

Awards granted under the program provide support for a 6-year period; during the last 2 years of this period there is an external review under recompetition requirements in the program’s language. After the original competition in 1994, additional competitions occurred in 1996, 1998, 2000, 2002, and 2005. At the inception of the program in 1994, 30 full proposals were submitted, and 11 awards were given to 9 universities. Owing to the phase difference in the transition between programs, 13 new MRSEC awards were granted 2 years later.

The program currently funds about 29 MRSECs (26 active MRSECs and 3 on phase-out funding), which split a total of about $51 million with a range of $1.0 million to $5.0 million per institution per year, as shown in Figure 3.1. The awards are fully competed every 6 years but are staggered on the basis of the date of first award. An institution that does not receive continued MRSEC funding after recompetition is provided with phase-out support. The total number of MRSECs funded since the program’s inception in 1994 is as follows:

A MRSEC provides a forum for researchers to come together and to share thoughts and ideas. Researchers participate because they realize the great advantages of working in an interdisciplinary team with exciting colleagues. The long-term nature of MRSEC support is welcomed because it allows researchers to pursue high-risk but potentially transformative ideas. Those ideas may lead to a new research direction for the MRSEC or may gain funding from other sources. MRSECs also provide a context for pursuing fundamental research that may not have immediately obvious payoffs but that is critical to future discovery. Students working within a MRSEC have a unique opportunity to learn from multiple mentors and to gain experience with techniques and ideas outside their own immediate field. Speaking to the committee, Harvard University MRSEC Director David Weitz emphasized these points by stating, “The most important products of the MRSEC are ideas (science, start-ups, etc.) and well-trained people.”

Evaluating MRSEC research is a daunting task. The committee considered several strategies, realizing that the MRSEC program contributes to the NSF mission in multiple ways even though “short-term research results” are usually considered the primary objective (see Box 3.1).

The committee realized that a comparison (“control”) group would need to be defined in order to make substantive comparisons. For instance, it would be insufficient to observe, “Research conducted through the MRSEC program generally includes significant collaboration.” Rather, the committee sought to determine

if the rate or nature of collaboration in the MRSEC program is different from the rate or nature of collaboration outside the program. A natural control group might therefore be the body of research enabled by the individual-investigator awards made through NSF’s Division of Materials Research. If a positive measurement

were obtained, one might then ask the importance of “group-based research” to the nation’s research enterprise.

This approach was complicated by the fact that research papers published in peer-reviewed journals do not, in general, uniquely attribute the research results to a single mechanism of support. No researcher finds his or her support entirely from a MRSEC, and both MRSEC and other contributions are influenced by participation in the MRSEC. Even at the level of an individual researcher’s activities, it is categorically impossible to separate out the uniquely “MRSEC-enabled” research. Site visits confirmed these impressions (see Box 3.2).

This caveat should be borne in mind when interpreting these analyses. Despite this intrinsic limitation, however, the committee designed and carried out several exercises, examining the activities enabled by the MRSEC program using several different techniques to “separate out” the MRSEC contributions and to construct “control” groups for comparison.

Finally, the committee emphasizes that its goal was not to evaluate the MRSEC program specifically, nor to recommend the continuation or termination of the program, but rather to describe and characterize its impact. Ideally, the committee would like to have answered a pointed question: If one had the opportunity to reinvest the annual budget of the MRSEC program purely on the grounds of its research impact, are there compelling examples of “what could not have happened otherwise?” Unfortunately, the inability to clearly separate what is “because of MRSEC” from “what is not” made it impossible to answer this question.

Moreover, any research, even by an individual researcher associated with a MRSEC, is a combination of activities supported both inside and outside the MRSEC. Thus, even if MRSECs have played a unique role in the research enterprise, such as in enabling the formulation of research projects that could not otherwise have been envisioned, there is no easy way to provide substantiation. Although the committee was unable to identify MRSEC-enabled research in “blind taste tests,” it successfully assessed the overall research quality in comparison with the research enabled by other mechanisms and elsewhere around the world. The basic questions to be answered are whether the research enabled by the MRSEC program is distinctive, if it is worthwhile and of high quality, and, finally, whether it is a good investment.

Many studies have tried to assess the quality of research programs in terms of objective criteria such as the citation numbers. A previous evaluation of the MRL program by the MITRE Corporation for the NSF concluded that there were no discernable trends in the quantity of publications or their citations when comparing MRLs with similarly funded programs. The committee’s exploration of the citation index produced similar conclusions. The committee found that identifying a set of comparable institutions was difficult, that the data would not be easy to obtain, and that the results would at best indicate the average output of the MRSECs and

their comparison group. In most areas of endeavor, it is not the average that leads to remarkable advances but, rather, remarkable discoveries that are large fluctuations from the norm.

While it was difficult to separate research uniquely enabled by the MRSEC program from research that was made possible by other means, the committee was clearer about causation. For instance, many of the more recently established NSF Nanoscale Science and Engineering Centers (NSECs) are located at institutions that have MRSECs; of the 10 active NSEC awards, 3 are at institutions without active MRSECs, and at least 1 more is in a research area very different from the

corresponding MRSEC. Do MRSECs enhance the probability for NSEC awards? Or does experience in the MRSEC competition simply add to an institution’s competitive edge? One could wonder about the potential for a “chicken-and-egg” problem at a strong institution that was awarded a MRSEC: which came first, the strong campus research effort or the center? In the committee’s judgment, the competitive selection process for MRSEC awards puts the burden on the pre-existing strength of the institutional research effort. While a MRSEC may enhance an institution’s materials research programs, it simply cannot bring them into being.

The committee identified a seemingly promising exercise that ended up rather inconclusively. Based on personal judgment and discussion with colleagues, the committee constructed a list of selected important materials discoveries and inventions from 1960 to 2000 (see Figure 3.2 for the subfields in which these discoveries occurred). Because the list was subjective, the committee chose not to publish it here. The committee then identified where the research leading to each discovery had been done and, in particular, whether it had originated in the MRSEC program or its predecessor MRL program. The list contained very few items that occurred in the past decade and thus the discoveries on the list significantly predated the MRSEC program per se. While this list is admittedly subjective and does not purport to be definitive, it revealed that the number of discoveries attributable to U.S. universities is rather limited. Given the generally recognized quality of U.S. universities in materials research, it is surprising that only 5 of these 27 discoveries are attributable to U.S. universities. Three were attributable to MRSECs. The committee does not want to overstate the implications of this ad hoc analysis, but it at least suggests that MRSEC research is an important part of a U.S. university materials research portfolio (see Box 3.3).

The majority of the discoveries were undertaken by individuals or small groups of about two investigators. Many of the discoveries originated in the predominantly industrial research laboratories in the United States, which reflected the time period (1960-2000) considered. Many of these labs (AT&T, IBM, Xerox, GE, Exxon, and so on) have been greatly reduced or eliminated, raising important questions about whether MRSECs can compensate for these losses. One should note, however, the parallel between a scientific breakthrough and car racing—the car and the driver get all the credit for a win, but in truth a much larger team is needed to enable a victory.

A small fraction of these breakthroughs took place in universities with MRSECs or MRLs (see Figure 3.3). Although this may appear somewhat discouraging regarding the impact of MRSECs as primary sources for innovative materials, funding

levels must be considered. The total budget of the MRSEC program is about $50 million per year compared with about $2 billion per year spent for (basic and applied) materials research by the U.S. government, and the approximately $4 billion per year spent worldwide by governments on materials research. The contributions seem to be larger than might be expected simply from the funding ratio. The fraction of MRSEC dollars to total materials dollars is 0.05/4, or 1.25 percent. No statistical analysis of these fragmentary observations is possible; however, it is possible to say that there are discoveries of the highest significance occurring within the MRSEC program, as gauged by this subjective survey.

As can be seen later in this section, there is almost an orthogonality between the types of institutions responsible for “major discoveries” and “top cited papers,” the former originating in industrial laboratories and the latter in universities. The committee suggests that many of the more recent fundamental breakthroughs occur in academe, often with MRSEC-funded facilities, whereas materials discoveries more closely linked to commercial products were more naturally done in industrial settings. The trend may reflect the passing of the torch from the formerly powerful industrial labs to universities.

The field of organic/polymeric conductors was supported from its inception

BOX 3.4 Progress on Conducting Polymers Alan Heeger began his university career at the University of Pennsylvania (Penn) in 1961, just one year after Penn was established as one of the initial three materials Interdisciplinary Laboratories (IDLs) (see Table 2.2 in this report). His initial research was in metal-insulator transitions, particularly in one-dimensional systems. This early work was the genesis of Heeger’s work with Alan MacDiarmid leading to the development of plastic electronics. Alan Heeger was director of the Penn Materials Research Laboratory (MRL) when this work accelerated with the synthesis of polyacetylene published in 1977 by Shirakawa, MacDiarmid, Loius, Chiang, and Heeger (see Figure 3.4.1). Financial support from the MRL, along with major support from the Office of Naval Research, enabled these seminal discoveries that led to the award of the 2000 Nobel Prize in chemistry to Shirakawa, MacDiarmid, and Heeger. In 1982, Alan Heeger moved to the University of California at Santa Barbara, where soon after he successfully proposed a new Materials Research Group on conducting polymers. This became the nucleus for the founding of the University of California at Santa Barbara MRSEC in 1993. FIGURE 3.4.1 Alan Heeger. Image of the 1977 Nobel Prize–winning paper.

in the MRL program (see Box 3.4). This highlight of the MRL/MRSEC program is on the list of selected major materials discoveries (see Box 3.5 for other examples of MRSEC research). Early contributions to the fundamental physics of quasi-one-dimensional conductors as new materials were produced in the MRLs, in U.S. universities, and in European and Japanese laboratories. A breakthrough in

the synthesis and characterization of polyacetylene turned the field around. This collaborative research was conducted at an MRL and involved strong interactions between the physics and chemistry departments at Pennsylvania State University and a group in Japan; it led to new materials, conducting polymers, and new con-

cepts: solitons, fractional charge, spin-charge separation. Further developments by several groups led to new technology, especially for organic optical displays; the formation of several companies in the United States and overseas; and the awarding of the 2000 Nobel Prize in chemistry to Alan J. Heeger, Alan G. MacDiarmid, and Hideki Shirakawa.

The committee experimented with a related exercise by trying to identify the founding papers in several topical areas of research. For instance, in the field of magnesium diboride research, the breakthrough paper is clearly the 2001 Nature article by Nagamatsu and colleagues (cited in the list below). Employing a scientific publication citation analysis tool (Scopus), the committee identified the top 20 most highly cited (subsequent) papers that cited the founding paper. Using institutional affiliation and some knowledge of the MRSEC IRG membership, the committee examined the role of MRSECs in this new set of “soon afterward” papers. The results were largely inconclusive, but interesting nonetheless.

• MgB₂: J. Nagamatsu, N. Nakagawa, T. Muranaka, Y. Zenitani, and J. Akimitsu, “Superconductivity at 39 K in Magnesium Diboride,” Nature 410 (6824): 63-64, 2001. Cited 1,804 times.

  • Of the top 20 most highly cited articles and reviews, none was from an institution with a MRSEC.

• Spintronics: Y. Ohno, D.K. Young, B. Beschoten, F. Matsukura, H. Ohno, and D.D. Awschalom, “Electrical Spin Injection in a Ferromagnetic Semiconductor Heterostructure,” Nature 402 (6763): 790-792, 1999. Cited 867 times.

  • Of the top 20 most highly cited articles, 1 was from an institution with a MRSEC.

• Magnetic semiconductors: T. Dietl and H. Ohno, “Zener Model Description of Ferromagnetism in Zinc-Blende Magnetic Semiconductors,” Science 287 (5455): 1019-1022, 2000. Cited 1,284 times.

  • Of the top 20 most highly cited articles, 2 were from institutions with MRSECs.

• Novel oxides: P. Schiffer, A.P. Ramirez, W. Bao, and S.-W. Cheong, “Low Temperature Magnetoresistance and the Magnetic Phase Diagram of La1xCaxMnO3,” Physical Review Letters 75 (18): 3336-3339, 1995. Cited 1,061 times.

  • Of the top 20 most highly cited articles, 5 came from institutions with MRSECs.

• Stripes in high-temperature superconductivity: J.M. Tranquada, B.J. Sternlieb, J.D. Axe, Y. Nakamura, and S. Uchida, “Evidence for Stripe Correlations of Spins and Holes in Copper Oxide Superconductors,” Nature 375 (6532): 561-563, 1995. Cited 1,266 times.

• Of the top 20 most highly cited articles, 6 were from institutions with MRSECs.

A study of highly cited papers was conducted to obtain a more objective overview of current and past research directions and impact in the materials community. A potential metric for examining the overall impact of published research results is to consider publication citations—the number of times subsequent papers refer to an earlier work; one variation on this approach was already discussed. A number of assumptions need to be made to imbue this technique with credible utility.²

The committee notes an important limitation of this analysis in advance: because the MRSEC program is only about one decade old, any research publications authored under its auspices are still relatively nascent in the field. That is, truly eminent articles generally take 10 to 15 years to demonstrate their impact on the field.³ Thus, the committee’s efforts to assess the research impact of the MRSEC program through a study of its publication citations is a bit premature. In its defense, the committee chose to compare the MRSEC program to just the past decade of materials research papers in order to include the same systematic error in the reference case. In order to avoid replicating the MITRE report described earlier and in order to keep its task tractable and focused on the MRSEC program, the committee chose not to analyze the full legacy of the IDLs and MRLs that led to the MRSECs. However, in so doing, the committee’s analysis could be interpreted to simply conclude that the MRSEC program is too young for impact assessment.

In a larger sense, the committee also sought to investigate some of the “urban myths” surrounding the MRSEC program. The committee made the most progress in addressing the question of whether MRSEC-enabled research results were empirically distinguishable in character and/or quality from other research. Box 3.6 shows some characteristic publications from the MRSECs.

Using a scientific journal publication citation tool (Scopus), the top 100 most highly cited papers in materials research since about 1996 were identified. A break-

down of these hundred papers in terms of subfields is shown as the light-colored bars in Figure 3.4. The affiliation information from the citations was examined to determine several characteristics for each paper: nationality, national laboratory/industry/university origins, and MRL/MRSEC connection. The MRL/MRSEC connection was evaluated by determining whether the institution had an operating MRSEC at the time of publication.⁴ The results are plotted in Figure 3.5. It is

FIGURE 3.5 Analysis of the affiliations listed by the authors of the overall top 100 most highly cited papers in materials research, 1996-2006. The total number of entries exceeds 100 because the committee could not clearly assign the origins for several papers. NOTE: MRL, Materials Research Laboratories; MRSEC, Materials Research Science and Engineering Centers.

noteworthy that institutions with an MRL/MRSEC program accounted for about 10 percent of the most highly cited papers worldwide and about 20 percent of those from the United States. These are considerably larger percentages than might be indicated by the relative funding levels mentioned earlier. This is the first indication that institutions with MRSECs are among the leaders in materials research. However, the task of directly associating research quality with the MRSEC program is complex, since it cannot be distinguished whether the best institutions are likely to succeed in the MRSEC competition or whether the MRSECs play a dominant role in the materials effort at these institutions. It is probable that both effects are present. Separating institutional publication impact caused by the MRSEC program as opposed to simple correlation with a MRSEC (or even as part of the reason for winning a MRSEC) is hard. The difficulty in assigning credit for these highly cited papers is that most authors report support from many sources.

At the committee’s request, many of the current MRSECs provided a list of their 5 most cited papers over the past 10 years.⁵ The distribution of MRSEC “top papers” by subfield was compared with the distribution of “top papers” for the entire materials field, thereby indirectly testing whether the MRSEC research portfolio matched the overall global materials research portfolio. A close examination of Figure 3.4 shows the result: the number of “top 100 papers” per subfield is plotted alongside the number of “top MRSEC” papers per subfield. The comparison indicates a strong correspondence between high-impact research done in the MRSEC program and the interests of the materials community as a whole.

The committee also independently examined the current MRSEC research portfolio. Although the MRSEC program is programmatically contained within the NSF Mathematical and Physical Sciences Directorate’s Division of Materials Research (i.e., separate from the NSF Engineering Directorate), the intended scope of the MRSEC program includes materials engineering. The list of research topics studied by the current suite of MRSECs has very limited intersection with the engineering side of materials (see Appendix C for a list of the current IRG research topics).

To gauge the “average impact” of each MRSEC, the average citations for the top 5 papers from each MRSEC were computed (see Figure 3.6). Each entry represents the average citation count for one MRSEC. The average number of citations per MRSEC ranged from 37 to 994 per highly cited paper. Note that for the 100 most cited papers in materials research from 1996 to 2006, the average number of citations per paper was 892. Thus, the “best” materials research papers are better (in terms of citations) than the “best” MRSEC papers. Although it would not be reasonable to expect the best papers to arise exclusively from MRSECs, it is noteworthy that some of the best MRSEC papers do rate among the best papers overall. As shown in Table 3.1, the average citation rate for MRSEC-related papers is about 13. This average number weights over many broad valleys in addition to high peaks; for instance, younger MRSECs with less-established research programs in new areas tend to have lower citation rates and therefore pull the average down. However, the average citation count per paper for all materials science papers

FIGURE 3.6 Distribution of average number of citations for the set of “top 5” papers reported by each currently active Materials Research Science and Engineering Center, 1994-2006. The lowest bin extends down to zero. The average of this histogram is about 350. The top 100 most-cited papers had an average of 892 citations per paper, with a median of 775.

(368,111) is 4.48.⁶ The top MRSEC papers thus are much better than the average materials research papers.

The committee compares the top MRSEC papers to the average papers for two reasons. First, it supports the observation that MRSECs have contributed several important research advances to the field. Second, the committee found it difficult to extract independently average values for the set of all MRSEC papers—mainly

because MRSEC papers are not well labeled in the literature; the only known data set is the one provided by MRSECs themselves. The committee found that the average MRSEC paper performed similar to the average materials paper.

The committee contacted institutions in other countries to discuss techniques for assessing the quality of their research activities, particularly those activities that are center-based. While there was no consensus as to the value of different procedures, many of the institutions contacted took what they considered was the “easy way out” and used publication-citation indices. Bernhard Keimer at the Max Planck Institute in Stuttgart kindly offered the services of his library staff to compare citations for several Max Planck Institutes (MPIs) and the MRSEC program. The search picked out papers in which a MRSEC was explicitly listed in an author’s address field; thus, many papers were missed if the author had a different home department as his or her address. Nonetheless, the comparison, shown in Table 3.1, had some value.

From these data, the MRSECs compare favorably in citations per publication with the MPIs. The library scientists who compiled the data noted that there was no significant difference between the MRSECs and the MPIs in citations. MPIs are among the premier institutions for materials research in Europe. This cursory survey confirms the previous results of the MITRE report which suggested that citation index comparisons do not sharply distinguish between research results from MRSECs and those from elsewhere; in fact, the MRSECs seem to be just as successful as the German MPI research centers by this metric.

The committee also examined the finer structure of the materials research field to determine if MRSECs contributed distinctively in certain subfields—particularly the areas on which the IRGs focused. As part of a survey from each MRSEC, the committee requested a list of the “top 5 scientific questions currently being addressed.” Synthesizing those responses and the IRG descriptions, the committee developed the following list of 23 subfields of materials research that may show differential MRSEC impact. The committee does not believe that this list is definitive.

1. Biomaterials

2. Ceramics

3. Composites

4. Ferroelectrics

5. Granular Material

6. Interfaces

7. Liquid Crystals

8. Magnetic Materials

9. Materials for Energy Storage

10. Materials Growth

11. Mesoscopics

12. Mechanical Properties

13. Nanomaterials

14. Nanostructures

15. Organic Semiconductors/Molecular Electronics

16. Oxides

17. Photonics/Optical Materials

18. Polymers (including copolymers)

19. Self-Assembly

20. Spintronics

21. Superconductivity

22. Supramolecular Materials

23. Transport Properties

It is clear that there is a great deal of overlap with the topics covered in the 100 most-cited materials papers and those in the most-cited papers from the MRSECs as shown in Figure 3.4. The committee chose to pursue the research-subfield impact hypothesis in two different ways: an objective analysis relying on publication citations and a subjective analysis using perceived standing from a panel of voting experts.

Using a scientific publication-citation analysis tool (Scopus), the committee identified the top 30 papers since 1995 in each subfield. Nearly 700 papers were selected for analysis. To get an idea of the level of activity in different subfields, the total number of citations for the top 10 papers in each subfield was tabulated (as shown in Figure 3.7). Comparison with Figure 3.4 indicates that there is substantial overlap of the most active areas with the corresponding efforts in the MRSEC program. Figure 3.8 shows a breakdown by country of origin, and Figure 3.9 shows the total.

Further analysis of the data set involved sorting the top-cited papers in terms of the type of institution where the research was performed. As can be seen in Figure 3.10, over the past 10 years it is the universities that have supplied the most highly cited research. This contrasts with the finding from the first look at significant new materials, which showed that they most often originated in industrial laboratories. This may result either from the higher value placed on publications in academe or from the decline of support at many industrial research laboratories.

One of the main objectives was to see whether this study of highly cited work could document that the MRSECs played a substantial role. Figure 3.11 shows the acknowledged funding sources for the most highly cited papers summed over the different subfields. The fraction associated directly with MRSECs is small, 4 percent of the total. Of course the fraction should be considered with respect to the total amount of research money that is provided by the different sources. (A more detailed discussion of the publications vis-à-vis the funding from different sponsors is presented later in this chapter.) However, another factor that must be considered is that most of the papers acknowledge more than one funding source. It is difficult to assign agency ownership to a discovery, a new material, or even just a publication.

FIGURE 3.11 Sources of funding for top-cited papers from U.S. institutions, summed over the 23 subfields of materials research: number of funding acknowledgments and percentage of total. Papers acknowledging Materials Research Science and Engineering Center (MRSEC) support are counted separately from those supported by other programs at the National Science Foundation (NSF). Seventy-seven percent of the papers acknowledged multiple sources of funding. DOE, Department of Energy; NIH, National Institutes of Health; DOD, Department of Defense.

This analysis also found substantial evidence that MRSEC research is as collaborative as non-MRSEC research. The committee examined the set of MRSEC self-reported top-5 publications. The typical number of senior principal investigators was two per paper, and there were more with single senior authors than with three senior authors. Figure 3.12 compares the proportion of top-cited papers that have single and multiple authors among the different subfields of materials research. While there are some fields in which the top-cited papers are almost evenly divided between single and multiple authorship (i.e., nanomaterials and nanostructures, liquid crystals, organic and molecular systems, photonics and optics, and polymers and supramolecular materials), for the most part papers having multiple senior authors are the norm. Overall, these data indicate that 65 percent of all top-cited papers involve multiple senior authors; however, Figure 3.13 shows that the most likely collaboration is between pairs of senior authors, with vanishing incidence of collaborations with three or more senior authors. There is no evidence that top-cited papers by MRSEC investigators display a different trend, although a primary argument used to rationalize the MRSEC organization is that these larger-scale collaborations are only possible in centers that include a large number of researchers from different departments. Thus, value added by MRSECs in collaborative research is likely on the “input” side (conceptual collaboration in choosing and initiating research directions) rather than the “output” side (research results as measured by published papers).

It is important to note that not all these collaborations are within single institutions. Figure 3.10 showed the distribution of top-cited papers, grouped over all subfields, with respect to the institution of the authors. University-based researchers, including those at MRSEC hosting institutions, are the authors of the highest percentage of top-cited papers worldwide (69 percent), with the remaining 30 percent coming from industry and government laboratories. The committee notes that 44 percent of all top-cited research papers involved collaborations among multiple institutions and also multiple institution types (i.e., collaborations between universities and national laboratories, industry and university, and so on). The committee sought also to quantify the prevalence of international collaboration among these top-cited papers. It was found that collaboration remains largely confined to individual countries, with only 16 percent of the papers involving international collaboration.

In summary, the committee set out to establish the baseline publication-citation characteristics of the general materials research community in order to enable a comparison with the self-reported MRSEC publications. However, in so doing, the committee came to realize that distinguishing MRSEC-enabled research papers was much harder than imagined. As Figure 3.11 shows, identifying a “MRSEC” paper is a subjective assertion about “whose dollars” put the research task over the

FIGURE 3.13 Proportion of top-cited papers with number of senior authors indicated, 1995-2006, summed over the 23 subfields. The key result is that more than half of the papers were authored by at least two senior principal investigators.

tipping point. The committee did not conduct a fully parallel analysis of the self-reported MRSEC papers as a result.

A primary objective of the original MRL program that has continued into the MRSEC program is to provide a setting that stimulates and nurtures interdisciplinary collaborative materials research. The original MRLs were located at institutions that already had a substantial interdisciplinary materials effort. Figure 3.14 shows the mix of disciplines in the first MRLs. In the original 10 MRLs, about 26 percent

FIGURE 3.14 Reported disciplinary affiliations for participants in the Materials Research Laboratory program in the mid-1970s. SOURCE: MITRE Corporation, Evaluative Study of the Materials Research Laboratory Program, 1976.

of the researchers came from physics departments, with a similar number from materials science and engineering departments and chemistry departments. Similarly, about 23 percent of the participants were from other engineering departments (including electrical engineering). It is interesting to note from Figure 3.15 that this departmental mix is almost the same in current MRSECS, with the only meaningful change being a modest increase in the participation from physics departments. A generation after the establishment of the MRLs, there is no question that MRSEC research remains both broad and multidisciplinary, and perhaps one can make the argument that this intrinsic attribute of the MRL and MRSEC programs has led the trend in materials research more generally.

The committee finds, therefore, that on the metric of multidisciplinarity as measured by departmental affiliation on research papers, the MRSEC program performs similar to the overall materials research community. However, the committee did not examine the paper-by-paper distribution of departmental affiliations. And, as before, it could not find a clear way to measure what the degree of multidisciplinarity would have been in the absence of the MRSEC program.

Figure 3.16 shows the distribution of departments of authors of top-cited papers from the committee’s list of the most-cited papers by subfield. The distribution

FIGURE 3.16 Departmental affiliations of authors of top-cited materials papers, summed over the 23 subfields, 1995-2006: number of papers cited and percentage of total. NOTE: MSE, materials science and engineering.

is similar in many ways to the departmental mix in MRSECs, with almost identical representation of physics and chemistry in the two distributions and a somewhat weaker participation of materials science and certainly overall engineering in the author distribution. Assuming that the likelihood of a field producing a top-cited paper increases when there are more researchers in the field, these data suggest that the mix of disciplines represented in current MRSECs is similar to the mix found in the worldwide materials research community, which has become deeply interdisciplinary.

A modest longitudinal study was also performed to see how the division of top-cited papers among different types of U.S. and foreign research organizations has evolved over the period of time during which MRSECs and MRLs have been active. Here, the data were not broken out into subfields; instead, the 100 top-cited papers in materials research over each of the decades 1965-1974, 1975-1984, 1985-1994, and 1995-2006 were selected. This follows the initial survey in which the very general criterion “materials research” for the 100 most-cited papers was used. It is unlike the previous citation results, which were broken down into MRSEC subareas.

As shown in Figure 3.17, the percentage of top-cited papers is plotted for U.S.

universities, universities having a MRSEC or an MRL, national laboratories, and industrial laboratories, as well as the total foreign citations for each of the decades in Figure 3.17. (Similarly, the number and percentage of top-cited papers in each of the 23 subfields, since 1995, by region of origin, were shown in Figures 3.8 and 3.9.)

It is clear that the United States enjoyed a near-monopoly on top-cited materials papers in 1965-1974 but that this percentage has fallen steadily in subsequent decades to its current level of 54 percent as foreign governments invest in the creation of their own materials science and engineering (MSE) knowledge base.⁷ Considering that it may take a decade or more for citations to develop fully for pathbreaking work, this plot is a cause for concern. The percentage of top-cited work has remained roughly constant for U.S. national laboratories, industrial laboratories, and perhaps has even grown slightly for MRSEC-hosting universities over this 40-year period. Yet it is clear that the growth of top-cited papers from foreign institutions has largely been at the expense of top-cited papers from U.S. universities, which in the 1960s produced half of the top-cited work. Ensuring the strength of university-based materials research is crucial, not only because it is the single largest sector of materials researchers in the United States (see Figure 3.18), but also because this is where future generations of materials researchers—both domestic and foreign—will be trained.

To assess the perceived excellence of the programs in 23 different subfields of materials research that may show differential MRSEC impact, the committee undertook an informal survey of the opinions of experts. It was proposed initially that the experts be selected by choosing the senior authors of the top 10 most-cited papers from each of the 23 subfields; however, the committee decided that the list should be augmented by authors of highly cited papers who were not selected by the simple algorithm above. For the purposes of this exercise, “expert” is defined as one of the senior authors of one of the 10 most highly cited papers in each of the 23 subfields listed above. The experts were then contacted by e-mail with the following sample note:

FIGURE 3.18 Affiliation of U.S. materials research community as estimated by demographics analysis of members of the Materials Research Society. The committee notes that the membership of the Materials Research Society is not broadly reflective of the overall composition of the materials research community, but it does have certain parallels to the university-based research community of Materials Research Science and Engineering Centers. (Courtesy of the Materials Research Society.)

About 200 e-mail inquiries were sent, and 55 experts replied with lists. Several were experts in more than one field and provided several lists. It was not possible to meaningfully rank institutions in each subfield on the basis of these data. By combining the subfields, however, the committee found sufficient evidence from which to draw conclusions as to the reputation of different institutions in the overall area of materials research. The responding experts were widely distributed in foreign laboratories and universities and domestically in institutions with MRSECs and without them, as shown in Figure 3.19.

The votes were then tallied for each subfield. Almost all respondents sent “top 10 lists” with a note that the institutions were not in order of excellence. A vote was counted each time that an institution was mentioned on an expert’s list. The

FIGURE 3.19 Sources of “expert votes” in the survey of leading research groups in materials. The “United States” category does not include the votes from institutions with a Materials Research Science and Engineering Center (labeled “MRSEC” in the chart).

main finding of the exercise is shown in Figure 3.20. The institutions were sorted according to their status as domestic university or national laboratory/industrial laboratory, and European, Asian, and other (Canada, India, Israel). The U.S. universities are subdivided further as to whether or not there is a MRSEC on the university campus.

FIGURE 3.20 Location by general categories of the “leading research groups,” summed over the 23 subfields of materials research, according to votes by experts. Note that the data in this figure closely match the data in Figures 3.8 and 3.9.

This survey of the most highly regarded research laboratories, across the subfields of materials, documents the leading role played by U.S. universities with MRSECs. As a group, U.S. universities with MRSECs are more identifiable with perceived excellence in materials research than any other grouping in the committee’s survey. Beyond the strong correlation of universities with MRSECs and perceived leadership, it is difficult to document whether this correlation is cause or effect. Some of the groups are MRSEC-supported while others are not. None of these world-leading groups is solely supported by a MRSEC. In at least one case, an expert specifically claimed that the institution’s MRSEC was not supporting the top-ranked group.

One might be tempted to contrast these results with those in the list of major discoveries described above in this chapter. Note, however, that the major discoveries were dominated by developments from 10 to 40 years ago, while this virtual-voting exercise was sensitive to contemporary impressions of perceived importance. Furthermore, the exercises were sensitive to different characteristics of research excellence: major discoveries versus overall high quality.

The distribution of leading groups was not uniform across the MRSEC program. Of course, there is some correlation between the number of years that an institution has had a MRSEC and its funding level and how well it does on this plot. But again, it is difficult to draw direct conclusions other than that MRSECs are situated at places that do excellent materials research.

Figure 2.2 in Chapter 2 showed the total federal funding for basic materials research between 1982 and 2002. The as-spent funding for materials research was almost constant in the 1980s, although the decade was followed by growth of about 35 percent between 1994 and 2000, in part reflecting the broadening of fields considered to be “materials research.”

Federal agencies support materials research at basic, applied, and developmental levels. When all of these expenditures are aggregated, the total exceeds $2 billion in FY 1995. The last published summaries of these expenditures were made for FY 1994 by the Materials and Technology Committee (MatTec), a subcommittee of the Federal Coordinating Council for Science, Engineering, and Technology reporting to the Office of Science and Technology Policy. It is important to note that virtually all major federal agencies supporting research are represented in this total. Figure 3.21 shows how the $2.124 billion total in 1994 was apportioned among the different agencies. NSF accounted for about 16% of this total, with about half (56%) of this total coming from DMR. (Note: subtracting facilities, $288 million for the Department of Energy [DOE] and $28 million for NSF, one gets a total of $1.793 million, with NSF at 15% and DOE at 34%.) The MRSEC percentage was

FIGURE 3.21 Expenditures by federal agency for basic, applied, and developmental materials research and engineering in millions of U.S. dollars and as percentage of total expenditure ($2,124 million) in FY 1994. The data do not include classified research or the construction and operating costs associated with facilities. SOURCE: Materials Technology Subcommittee, National Science and Technology Council, 1994 Annual Report, 1994.

about 25% of the total DMR expenditure, very similar to 2006 proportions. Altogether, MRSEC expenditures represent a very small fraction of the federal materials portfolio, amounting to about 2% of the total.

Of course, given the variety of activities funded by this portfolio and the different programmatic needs of the different agencies, there is no a priori reason to believe that the number of top-cited papers claimed by a given funding agency is proportional to its relative level of materials funding. The acknowledged sources of funding in the top-cited papers are shown in Figure 3.11. It is important to note that fully 77% of all papers acknowledged multiple sources of funding, implying that—increasingly—no one agency can take sole credit for funding any piece of work. There is no simple relationship between the level of federal funding and the percentage of top-cited papers enabled by this funding. For instance, DOE provides 43% of the support for all basic materials research and garnered 10% of the top-cited papers, while NSF provided 16% and was acknowledged in 16% of

the top-cited papers. These data are compiled in Figure 3.22, which compares the percentage of top-cited papers from 1996 to 2006 that acknowledge each agency with the percentage of the total federal budget for materials research spent by that agency.

While the overall monetary investment is very different, for the National Institutes of Health (NIH) and the Department of Defense (DOD) there is good agreement between the percentage of the top-cited papers and the percentage of the federal budget used to enable the research in these papers. The NSF represents relatively good value for investment, yielding top-cited research papers at almost twice the rate per dollar invested. The MRSEC program (2% of total materials investment, 5% of top-cited papers) is similar in “efficiency” to NSF overall (14.6% of total materials investment, 30% of top-cited papers). DOE has very high research expenditures but a relatively lower participation in top-cited papers. This is probably because the embedded cost of constructing and operating the DOE user

FIGURE 3.22 Comparison of number of top-cited materials papers to materials research expenditures for major federal agencies, 1996-2006. The solid line has a unit slope. The committee was not able to easily separate the university and national laboratory components of the DOE materials research budget; therefore, the DOE data point is not consistent with the university-research program budgets used for the other agencies. NOTE: NSF, National Science Foundation; MRSEC, Materials Research Science and Engineering Center; DOD, Department of Defense; DOE, Department of Energy; NIH, National Institutes of Health.

facilities is not properly accounted for. The committee also notes that in selecting the data for the plot in Figure 3.22, the papers originating only from DOE laboratories were excluded in order to allow a comparison of DOE-supported university research with NSF-supported university research. Including the DOE national laboratories more than doubles the number of such papers. It can be concluded from Figure 3.22 that NSF research, including that carried out in MRSECs, is more likely to result in a top-cited publication than research funded by any other major agency. Within these statistics, there is little evidence that the MRSECs are more or less productive in this respect than any other NSF materials program.

An often-cited key element of the MRSEC program is its explicit provision of shared experimental facilities (SEFs) at each center. The MRSEC program provides only limited explicit support for underwriting the capital costs of acquiring and maintaining a comprehensive instrument suite; rather, institutions must find other mechanisms for purchasing equipment (including the use of other NSF programs). MRSECs SEF funds, originating from budgets for IRGs, seeds, and facilities, are usually expended to cover operating costs of equipment and facilities such as maintenance, supplies, or portions of a salary for technical support staff (see Figures 3.23 and 3.24 for the distribution of MRSEC SEF budgets compared to each center’s budget). In 2004, DMR estimated that 12 percent of the MRSEC budgets was spent on capital equipment.

The research and training of students and postdoctoral associates in the MRSECs is completely dependent on the availability of SEFs with forefront capabilities. The MRSEC SEFs support a very broad range of materials research, which is essential to a broad community (including many supported by single-investigator grants) but is not altruistic, since the MRSECs could not carry out their own research without the user fees generated by these users.

As identified in the National Research Council (NRC) report Midsize Facilities: The Infrastructure for Materials Research,⁸ each materials research facility secures its capital and operating sources of support in a unique and highly individualized fashion. NSF MRSEC SEF support is often only one component of a complex array of funding mechanisms. Many MRSECs operate their SEF facilities with some user fees in order to recover some of the operating costs. In the larger MRSECs, the SEF user community is larger than the number of MRSEC students by at least a factor of 10. This large user base is necessary to pay SEF staff salaries that could not be sustained on the MRSEC budget alone. Another common feature was that

faculty and students who participated in the MRSEC would receive slightly discounted rates for using the instruments as compared with other users on campus. It is also important to note that in most cases, the instrumentation supported under the MRSEC SEF program element was part of a larger suite maintained by the institution.

In terms of impact, the committee believes that the shared facilities supported by MRSECs do have significant impact in the larger community, but the committee was not convinced that the MRSEC SEF support was dramatically more effective or leveraged more than any other instrumentation program. For instance, the committee learned from the Midsize Facilities report referred to above that operating costs for shared facilities (including the MRSEC program) are recovered about equally from federal grants, user fees, state awards, and institutional commitments. In examining the MRSEC annual reports, the committee observed a similar mix of reported sources of operating costs for the SEFs. However, the committee was unable to collect reliable data about sources of funds for the acquisition of capital equipment both inside and outside the MRSEC program. It is the committee’s view that MRSEC centers likely attract elevated levels of cost sharing from institutional leaders because they attract attention and provide explicit federal leveraging. The committee notes again that the specific impacts are probably diluted when viewing average trends. For instance, MRSEC participants at the University of Southern Mississippi credited the MRSEC with helping empower them to compete successfully for additional instrumentation awards from NSF and other agencies. The committee did not measure and compare the degrees of utilization of facilities inside and outside the MRSEC program and therefore cannot comment on the relative accessibility of the instrumentation to the broader community. The general perception seems to be, however, that MRSECs do allow wide-ranging access to their facilities.

The committee found that MRSECs invest in facilities at a rate comparable to DMR overall and that MRSECs provide about 20 percent of the DMR instrument portfolio. The committee also observed that—averaged over the past 10 years—institutions with MRSECs attracted “instrumentation for materials research” awards at roughly the same rate as institutions without MRSECs. The committee could not easily measure, however, whether institutions with MRSECs attracted a higher volume of instrumentation awards from sources outside NSF.

The committee collected information on the levels of MRSEC support that were directed toward shared experimental facilities from the annual reports of the MRSECs. The following observations were made.

• In 2004, the average MRSEC budget spent on facilities was $276,000 (median $187,000) per year with a total reported investment of $6.6 million;

$6.6 million is about 13 percent of the annual $50 million MRSEC program budget.

• In 2004, the portion of the DMR budget spent on equipment and instrumentation was $30 million (beyond that of the MRSEC program), or about 12 percent of the division’s full budget. In addition, DMR distributed about $5.7 million in equipment and instrumentation funds through the Instrumentation for Materials Research (IMR) program. Thus, in 2004, DMR invested about 18 percent of its annual budget (excluding the MRSECs) in equipment and instrumentation.

The committee observes then, that MRSECs invest in facilities and equipment at a rate similar to the overall DMR portfolio of investments. This analysis is extremely informal. It should also be noted that the committee did not compare the type of instruments bought through MRI and IMR awards and those secured through and for MRSECs. Another estimate suggested that MRSECs house about 20 percent of the overall federal investment at universities in million-dollar-class instrumentation for materials research. Also, it should be noted that SEFs in the materials area are not unique to MRSECs, but at institutions with larger MRSECs the SEFs are usually managed and operated by the MRSEC. If MRSECs did not do this, DMR would need to create some other strong facilities program to support materials research.

The MRSEC facilities budget also supports (at least in part) technical staff members, who train students and maintain the equipment. About $240,000 per year is spent on capital equipment. Estimating that half of the equipment purchased through the NSF instrumentation programs (DMR’s Instrumentation for Materials Research program or NSF’s agency-wide Major Research Instrumentation program) within DMR ends up in a MRSEC facility, another $5 million—or an average of about $200,000 per center—is added to this amount. Assuming a 10-year life for forefront materials characterization equipment, a center might thus afford a total inventory of equipment of about $4.4 million.

The committee also examined the potential correlation between MRSECs and instrumentation funding.

• In the timeframe 1995-2006, the DMR IMR program awarded about $75 million of grants for the acquisition and development of instrumentation for materials research.

• Of these awards, 30 percent (by dollar value) were made to institutions with MRSECs (that were active at the time).

• MRSEC institutions received $402 million during this time frame, about 39 percent of the total $1.04 billion or so awarded by DMR to all institutions (excluding the $544 million for MRSEC funding).

• Thus, the committee observes that institutions with MRSECs attract IMR awards roughly in proportion with their level of materials research activity (measured by DMR funding levels).

MRSECs are, however, taking a lead in working together on facilities rather than competing with one another. Led by the MRSEC at the University of California at Santa Barbara, the University of Southern Mississippi MRSEC, in collaboration with those at the University of Minnesota and the University of Massachusetts at Amherst, proposed to NSF to create a national facilities network. The award has recently been funded and will be used to encourage off-campus users to take advantage of the facilities; it also helps send students and faculty among the four sets of facilities at “internal” user rates.

The variations in actual capital spending equipment from one MRSEC to another are considerable because the availability of resources hinges on other features of the institution such as the development office, relationships with corporate sponsors, and so on. The recent National Research Council report on shared experimental facilities found that most SEFs that serve the large majority of the materials community have a $1 million to $50 million replacement capital cost, with an average of about $10 million.⁹ In fact, the U.S. investment in such facilities is currently well below the replacement level,¹⁰ estimated to be on the order of several billion dollars per year. At present, other sources of support for SEF equipment (typically, the universities themselves, or in some cases foundations) are not large enough to make up the difference in needed support. Thus, the average age of equipment in SEFs continues to increase, with many individual items being more than 20 to 25 years old.

The committee found the task of evaluating the impact of MRSEC research quite daunting, primarily because research papers published in peer-reviewed

journals rarely attribute the results to a single support mechanism. Moreover, any research, even by an individual researcher associated with a MRSEC, is a combination of activities supported “inside” and “outside” the MRSEC. Thus, even if MRSECs have played a unique role in the research enterprise, such as in enabling the formulation of research projects that could not otherwise have been envisioned, there is no easy way to provide substantiation. It could be that the research enterprise has evolved over the past decade, leading to greater convergence and overlap between MRSECs and other research practices. Thus, it is not currently possible to distinguish the unique contributions of MRSECs.

Most research publications now acknowledge multiple sponsors. It is not possible to demonstrate that the MRSEC support yields leadership in discoveries, publications, or citations. In part this is because funding per MRSEC has decreased significantly in the past decade, so that each group requires multiple sponsors.

Although the materials field is highly collaborative and the general belief is that the community benefits from interactions between local groups of many individual investigators in the same field, discoveries and publication records indicate that over 50 percent of the published papers are from individuals and groups of two.

Although the committee was unable to identify MRSEC-enabled research in “blind taste tests,” it successfully assessed the overall research quality in comparison with the research enabled by other mechanisms and elsewhere around the world. For instance, do published research results that acknowledge MRSEC resources achieve citation indices and other measures of impact comparable with research enabled by individual-investigator awards?

• The outstanding discoveries, leading research groups, and most significant publications worldwide are associated with universities at which there are MRSECs.

• MRSECs are involved in the most active areas of materials research as established by their publication records compared with those of the entire field.

• The MRSEC program has the same level of research collaboration as found in comparable national and international groups.

The committee studied a set of major breakthroughs in materials research over the past four decades. U.S. universities, and in particular MRSECs and their predecessors the MRLs, played a limited but pivotal role in a handful of these discoveries. The committee conducted several comprehensive analyses comparing citations of MRSEC-reported research publications and those of the broader research community. The distribution of MRSEC-reported “top-cited papers” across subfields of materials research was very similar to that of the top 100 most-cited papers. Affiliations of the top 100 research papers also showed a 10 percent contribution from institutions with MRSECs or MRLs. The committee also found that the top MRSEC papers were cited much more frequently than the average materials research paper but that the best-of-the-best materials research papers had significantly more citations. However, these papers generally predate the emergence of the MRSEC program. The committee also found that the MRSEC program has the same level of collaboration as that found in comparable national and international groups. To some extent this may be the ultimate success of the MRSEC program in having fostered this type of research at an early stage. Finally, the breakdown of departmental affiliations of MRSEC authors and those of the top-cited materials research papers were quite similar.

In two related exercises, the committee examined the global stature of MRSEC-related research groups. In comparison to the Max Planck Research Institutes of Germany, the MRSECs’ publication-citation rates were quite comparable. In a “virtual voting” exercise, the committee contacted researchers around the world in several different subfields and solicited their opinions about world-leading research teams. Research teams at institutions with MRSECs dominated the results.

Although many of these measures are of correlation and not causation, the committee came to believe that the research program enabled by MRSEC awards has been, in general, at least as effective as that enabled by other mechanisms.

It should be noted that SEFs in the materials area are not unique to MRSECs,

but at institutions with larger MRSECs the SEFs often are managed and operated by the MRSEC. If MRSECs did not do this, DMR would need to create some other strong facilities program to support materials research. A large user base is necessary to pay SEF staff salaries that cannot be supported solely by the MRSEC budget. The MRSEC SEFs support a very broad range of materials research (and sometimes other kinds), which is essential to a broad community (including many supported by single-investigator grants), but it is not just altruistic—the MRSECs could not carry out their own research without the user fees generated by these users. Shared facilities are an important resource for the overall community. For instance, individual investigators are unlikely to be able to afford to acquire and maintain a cutting-edge transition electron microscope, whereas a MRSEC SEF would be ideally suited to do so. Such an instrument sited at a MRSEC would be highly leveraged (because of institutional commitments to existing infrastructure and an established user community that would supply fees-for-use) and would greatly expand the opportunities available to the local research community. The committee encourages recent efforts by the centers and NSF to use modest supplemental grants to encourage and promote broader access to these facilities. These instruments are a core part of the value of the MRSEC program and can have enhanced national impact through improved communication and coordination.

As described in the beginning of this chapter, the committee concludes that the merit of the research enabled by the MRSEC program is comparable with the best of the materials research supported by other mechanisms. The committee notes, however, that it focused on measuring the impact of research results and that the ancillary benefits of MRSECs are not reflected by these metrics.