3
Assessment of Research and Facilities Impact

In carrying out its impact assessment task, the committee first analyzed the issues broadly in several different categories: research, education and outreach, collaboration with other sectors, and other areas. After the following introductory remarks, an analysis of the impact of Materials Research Science and Engineering Centers’ (MRSECs’) research is presented in this chapter.

INTRODUCTION

The Materials Research Science and Engineering Centers program (MRSEC program) was established by the National Science Foundation (NSF) in its Division of Materials Research (DMR) in 1994. As described in Chapter 2, the MRSEC program was born out of the decision to transform the Materials Research Laboratory (MRL) and Materials Research Group (MRG) programs to the current structure. The goal of this new initiative was to provide focused support for complex interdisciplinary materials research and education at the university level. To receive a MRSEC award, an institution must:

[demonstrate] outstanding research quality and intellectual breadth, provide support for research infrastructure and flexibility in responding to new opportunities, and strongly emphasize the integration of research and education. These centers foster active collaboration between universities and other sectors, including industry, and they constitute a national network of university-based centers in materials research. MRSECs address problems of a scope or complexity requiring the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward 3 Assessment of Research and Facilities Impact In carrying out its impact assessment task, the committee first analyzed the issues broadly in several different categories: research, education and outreach, collaboration with other sectors, and other areas. After the following introductory remarks, an analysis of the impact of Materials Research Science and Engineering Centers’ (MRSECs’) research is presented in this chapter. INTRODUCTION The Materials Research Science and Engineering Centers program (MRSEC program) was established by the National Science Foundation (NSF) in its Division of Materials Research (DMR) in 1994. As described in Chapter 2, the MRSEC program was born out of the decision to transform the Materials Research Laboratory (MRL) and Materials Research Group (MRG) programs to the current structure. The goal of this new initiative was to provide focused support for complex interdisciplinary materials research and education at the university level. To receive a MRSEC award, an institution must: [demonstrate] outstanding research quality and intellectual breadth, provide support for research infrastructure and flexibility in responding to new opportunities, and strongly emphasize the integration of research and education. These centers foster active collaboration between universities and other sectors, including industry, and they constitute a national network of university-based centers in materials research. MRSECs address problems of a scope or complexity requiring the

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward advantages of scale and interdisciplinary interaction provided by a campus-based research center.1 Awards granted under the program provide support for a 6-year period; during the last 2 years of this period there is an external review under recompetition requirements in the program’s language. After the original competition in 1994, additional competitions occurred in 1996, 1998, 2000, 2002, and 2005. At the inception of the program in 1994, 30 full proposals were submitted, and 11 awards were given to 9 universities. Owing to the phase difference in the transition between programs, 13 new MRSEC awards were granted 2 years later. The program currently funds about 29 MRSECs (26 active MRSECs and 3 on phase-out funding), which split a total of about $51 million with a range of $1.0 million to $5.0 million per institution per year, as shown in Figure 3.1. The awards are fully competed every 6 years but are staggered on the basis of the date of first award. An institution that does not receive continued MRSEC funding after recompetition is provided with phase-out support. The total number of MRSECs funded since the program’s inception in 1994 is as follows: Year: 1994 1996 1998 2000 2002 2004 No. 11 24 25 27 27 27 A MRSEC provides a forum for researchers to come together and to share thoughts and ideas. Researchers participate because they realize the great advantages of working in an interdisciplinary team with exciting colleagues. The long-term nature of MRSEC support is welcomed because it allows researchers to pursue high-risk but potentially transformative ideas. Those ideas may lead to a new research direction for the MRSEC or may gain funding from other sources. MRSECs also provide a context for pursuing fundamental research that may not have immediately obvious payoffs but that is critical to future discovery. Students working within a MRSEC have a unique opportunity to learn from multiple mentors and to gain experience with techniques and ideas outside their own immediate field. Speaking to the committee, Harvard University MRSEC Director David Weitz emphasized these points by stating, “The most important products of the MRSEC are ideas (science, start-ups, etc.) and well-trained people.” Evaluating MRSEC research is a daunting task. The committee considered several strategies, realizing that the MRSEC program contributes to the NSF mission in multiple ways even though “short-term research results” are usually considered the primary objective (see Box 3.1). 1 National Science Foundation, Program Solicitation for Materials Research Science and Engineering Centers, NSF 04-580, Washington, D.C., 2004.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward FIGURE 3.1 Distribution of annual Materials Research Science and Engineering Center budgets in the 1990s and in the current decade. The average (median) center budget in the late 1990s was $1.5 million ($1.2 million in inflation-adjusted dollars) and is now $1.6 million ($1.3 million); the width of the distribution has narrowed slightly. The axis label for each bar on the histograms indicates the upper edge of the range of values assigned to that bin. The data for the 1990s include several centers that started and stopped during the last years of that decade. SOURCE: Division of Materials Research, National Science Foundation.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward BOX 3.1 Qualitative Tests of MRSEC Impact A necessary exercise in assessing the impact of the Materials Research Science and Engineering Centers program (MRSEC program) is to look at the top-rated programs in materials research in the United States and to compare them with a list of institutions that have Materials Research Science and Engineering Centers (MRSECs). Because it is impossible to determine the causality between the existence of a MRSEC at an institution and the quality of that institution’s materials science and engineering program, the committee conducted a very cursory examination of several surveys, as discussed below. At best, the surveys exhibit a correlation. Surveys by U.S. News and World Report According to an annual survey conducted by U.S. News and World Report, the top five schools for undergraduate materials science and engineering schools in 2006 were the following:1 1.   Massachusetts Institute of Technology (MIT) 2.   University of Illinois at Urbana-Champaign   Northwestern University 3. University of California at Berkeley   University of Michigan at Ann Arbor Note that the last three schools tied for third place. Four of these schools have had MRSECs or Materials Research Laboratories (MRLs). In 2007, according to U.S. News and World Report,2 the top four chemistry Ph.D. programs were at the California Institute of Technology, MIT, Stanford University, and the University of California at Berkeley. Three of these schools have MRSECs. Likewise, the top graduate physics programs were at MIT, Stanford University, and the California Institute of Technology. All three schools have MRSECs, although the Stanford MRSEC does not interact broadly with the physics department. Drilling down to the level of condensed-matter and materials physics, both of the top schools (Cornell University and Harvard University) have MRSECs closely connected with the physics departments; the University of Illinois at Urbana-Champaign has a Materials Research Laboratory that is now supported by the Department of Energy (DOE). National Research Council Ph.D. Program Rankings The National Research Council conducts a decadal survey of graduate programs. The most recent rankings are from 1995, with the next edition expected in 2008. When completed, the new rankings will nicely    1U.S. News and World Report, 2006. Available in limited form online at http://colleges.usnews.rankingsandreviews.com/usnews/2006/edu/college/rankings/rankindex_brief.php.    2U.S. News and World Report, 2007. Available in limited form online at http://colleges.usnews.rankingsandreviews.com/usnews/2007/edu/college/rankings/rankindex_brief.php. The committee realized that a comparison (“control”) group would need to be defined in order to make substantive comparisons. For instance, it would be insufficient to observe, “Research conducted through the MRSEC program generally includes significant collaboration.” Rather, the committee sought to determine

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward bracket the first decade of the MRSEC program. In 1995, the top 10 graduate programs in materials science were as follows:3 MIT Northwestern University Cornell University University of California at Berkeley University of Illinois at Urbana-Champaign Stanford University University of Massachusetts at Amherst University of California at Santa Barbara Pennsylvania State University University of Pennsylvania Of these 10 schools, all but the University of California at Berkeley have a formal materials centers program dating back to the Interdisciplinary Laboratories of the 1960s and the Materials Research Laboratories of the 1970s and 1980s. The University of California at Berkeley has strong connections with the neighboring DOE Lawrence Berkeley National Laboratory as well as with Stanford’s Center on Polymer Interfaces and Macromolecular Assemblies. National Doctoral Program Survey In 2000, the National Association of Graduate and Professional Students published the results of a survey of over 32,000 participants.4 The survey ranked graduate programs on the basis of participants’ perception of the overall implementation of recommended best practices (admittedly nebulous). At the top of the list of materials science programs ranked by recommended practices were the following schools: MIT University of Massachusetts at Amherst Johns Hopkins University Pennsylvania State University Stanford University University of Delaware University of California at Berkeley University of Minnesota Of these schools, all but the University of California at Berkeley and the University of Delaware have (had) MRSECs.    3National Research Council, Research Doctorate Programs in the United States: Continuity and Change, Washington, D.C.: National Academy Press, 1995.    4Available online at http://cresmet.asu.edu/nagps/about/index.php. if the rate or nature of collaboration in the MRSEC program is different from the rate or nature of collaboration outside the program. A natural control group might therefore be the body of research enabled by the individual-investigator awards made through NSF’s Division of Materials Research. If a positive measurement

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward were obtained, one might then ask the importance of “group-based research” to the nation’s research enterprise. This approach was complicated by the fact that research papers published in peer-reviewed journals do not, in general, uniquely attribute the research results to a single mechanism of support. No researcher finds his or her support entirely from a MRSEC, and both MRSEC and other contributions are influenced by participation in the MRSEC. Even at the level of an individual researcher’s activities, it is categorically impossible to separate out the uniquely “MRSEC-enabled” research. Site visits confirmed these impressions (see Box 3.2). This caveat should be borne in mind when interpreting these analyses. Despite this intrinsic limitation, however, the committee designed and carried out several exercises, examining the activities enabled by the MRSEC program using several different techniques to “separate out” the MRSEC contributions and to construct “control” groups for comparison. Finally, the committee emphasizes that its goal was not to evaluate the MRSEC program specifically, nor to recommend the continuation or termination of the program, but rather to describe and characterize its impact. Ideally, the committee would like to have answered a pointed question: If one had the opportunity to reinvest the annual budget of the MRSEC program purely on the grounds of its research impact, are there compelling examples of “what could not have happened otherwise?” Unfortunately, the inability to clearly separate what is “because of MRSEC” from “what is not” made it impossible to answer this question. Moreover, any research, even by an individual researcher associated with a MRSEC, is a combination of activities supported both inside and outside the MRSEC. Thus, even if MRSECs have played a unique role in the research enterprise, such as in enabling the formulation of research projects that could not otherwise have been envisioned, there is no easy way to provide substantiation. Although the committee was unable to identify MRSEC-enabled research in “blind taste tests,” it successfully assessed the overall research quality in comparison with the research enabled by other mechanisms and elsewhere around the world. The basic questions to be answered are whether the research enabled by the MRSEC program is distinctive, if it is worthwhile and of high quality, and, finally, whether it is a good investment. Many studies have tried to assess the quality of research programs in terms of objective criteria such as the citation numbers. A previous evaluation of the MRL program by the MITRE Corporation for the NSF concluded that there were no discernable trends in the quantity of publications or their citations when comparing MRLs with similarly funded programs. The committee’s exploration of the citation index produced similar conclusions. The committee found that identifying a set of comparable institutions was difficult, that the data would not be easy to obtain, and that the results would at best indicate the average output of the MRSECs and

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward BOX 3.2 Site Visits The MRSEC Impact Assessment Committee conducted more than a dozen site visits at institutions that either have a MRSEC or a similar center-based research structure, that were contemplating a MRSEC application, or which had had a MRSEC that closed (see the section entitled “Site Visits” in Appendix F, “Data-Gathering Tools,” for a list of the institutions visited). These visits prompted candid conversations with researchers that provided valuable anecdotal information and firsthand impressions. The committee found this feedback very useful in its assessment of the MRSEC program. Below are some excerpts from conversations with faculty and staff. From its site visits, the committee heard comments that, “Centers can only succeed if they help us integrate between disciplines,” and that “Without impetus from outside [the university], it is hard to initiate a center, despite whatever latent good will and intentions there are.” When asked to differentiate the MRSEC-style research center from departmental centers, some said, “Centers are intellectual foci of effort. They are at a larger scale than just one department with some of its faculty. You need to cut across more fields of research to really attack new problems and push forward; you need more than one or two departments.” When asked about their university’s perspective on centers, many university administration officials commented that they view centers with federal funding as having a higher degree of validity because they have received some external commitment and recognition. When asked about canceling the MRSEC program, one university official opined, “I don’t think that the campus and state would take the initiative to invent such a center without the external incentive unless the affected topics were related to human health and wellness. Also, this campus is isolated geographically from the industrial community,” and would not be as able to engage industries in relationships pertaining to physical science projects. Others echoed these thoughts, saying that such centers are one of the only mechanisms, externally funded, that cut across disciplinary boundaries and the stovepipes of academic departments. Others commented that single-investigator awards are typically only 3 years, and the longevity of a center grant enables much more creativity, flexibility, and even security in trying out research ideas. A final comment suggested that the National Science Foundation fulfills a key national goal by providing support for basic research that is not directly connected to product commercialization (as opposed to state and local industry programs), and since centers are a key mechanism for supporting the basic-science enterprise, they should be continued. their comparison group. In most areas of endeavor, it is not the average that leads to remarkable advances but, rather, remarkable discoveries that are large fluctuations from the norm. While it was difficult to separate research uniquely enabled by the MRSEC program from research that was made possible by other means, the committee was clearer about causation. For instance, many of the more recently established NSF Nanoscale Science and Engineering Centers (NSECs) are located at institutions that have MRSECs; of the 10 active NSEC awards, 3 are at institutions without active MRSECs, and at least 1 more is in a research area very different from the

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward corresponding MRSEC. Do MRSECs enhance the probability for NSEC awards? Or does experience in the MRSEC competition simply add to an institution’s competitive edge? One could wonder about the potential for a “chicken-and-egg” problem at a strong institution that was awarded a MRSEC: which came first, the strong campus research effort or the center? In the committee’s judgment, the competitive selection process for MRSEC awards puts the burden on the pre-existing strength of the institutional research effort. While a MRSEC may enhance an institution’s materials research programs, it simply cannot bring them into being. ANALYSIS OF SELECTED CONTRIBUTIONS FROM MATERIALS RESEARCH The committee identified a seemingly promising exercise that ended up rather inconclusively. Based on personal judgment and discussion with colleagues, the committee constructed a list of selected important materials discoveries and inventions from 1960 to 2000 (see Figure 3.2 for the subfields in which these discoveries occurred). Because the list was subjective, the committee chose not to publish it here. The committee then identified where the research leading to each discovery had been done and, in particular, whether it had originated in the MRSEC program or its predecessor MRL program. The list contained very few items that occurred in the past decade and thus the discoveries on the list significantly predated the MRSEC program per se. While this list is admittedly subjective and does not purport to be definitive, it revealed that the number of discoveries attributable to U.S. universities is rather limited. Given the generally recognized quality of U.S. universities in materials research, it is surprising that only 5 of these 27 discoveries are attributable to U.S. universities. Three were attributable to MRSECs. The committee does not want to overstate the implications of this ad hoc analysis, but it at least suggests that MRSEC research is an important part of a U.S. university materials research portfolio (see Box 3.3). The majority of the discoveries were undertaken by individuals or small groups of about two investigators. Many of the discoveries originated in the predominantly industrial research laboratories in the United States, which reflected the time period (1960-2000) considered. Many of these labs (AT&T, IBM, Xerox, GE, Exxon, and so on) have been greatly reduced or eliminated, raising important questions about whether MRSECs can compensate for these losses. One should note, however, the parallel between a scientific breakthrough and car racing—the car and the driver get all the credit for a win, but in truth a much larger team is needed to enable a victory. A small fraction of these breakthroughs took place in universities with MRSECs or MRLs (see Figure 3.3). Although this may appear somewhat discouraging regarding the impact of MRSECs as primary sources for innovative materials, funding

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward FIGURE 3.2 Distribution of selected major materials research discoveries, by materials research subfield. A handful of the 27 discoveries identified by the committee did not fit into these categories and are labeled “Other.”

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward BOX 3.3 The MRSEC at the California Institute of Technology One example of a successful MRSEC is at the California Institute of Technology (Caltech). A thumbnail sketch of Caltech’s Center for the Science and Engineering of Materials provides a view of the typical organization and breadth of activities of a MRSEC. The Caltech MRSEC focuses on several interdisciplinary areas. The research program is organized into four interdisciplinary research groups (IRGs) and two seed projects. The IRGs are Biomolecular Materials, Ferroelectric Thin Films, Mesophotonics, and Bulk Metallic Glasses. The work in Biomolecular Materials explores the control of self-organization that can be achieved in polymers of absolutely defined comonomer sequence—genetically engineered artificial proteins—and the control of spatial arrangement and size that can be templated using surfactant nanostructures. The Ferroelectric Thin Films group aims to enable ultrahigh displacement microactuators based on high-strain ferroelectrics. The project on Mesophotonic Materials is motivated by advances in the synthesis and theoretical understanding of materials designed to manipulate light on scales at and below the wavelength of light, in order to move into the revolutionary domain of devices on scales of tens of nanometers. The program on Bulk Metallic Glasses, which has been particularly effective in its industrial interactions, investigates the processing, microstructure, and mechanical behavior of bulk metallic glasses (BMGs) and their composites. The researchers are investigating the basic science and engineering that will enable new strategies to produce BMGs in which a crystalline phase is introduced to resist shear localization, creating a BMG composite with enhanced material properties. Some of this work has already reached the stage of commercial application in such products as cellular telephone cases and other electronic device packages. Future efforts, in conjunction with a number of industrial partners, involve applications in a wide variety of commercial, biomedical, and military applications. levels must be considered. The total budget of the MRSEC program is about $50 million per year compared with about $2 billion per year spent for (basic and applied) materials research by the U.S. government, and the approximately $4 billion per year spent worldwide by governments on materials research. The contributions seem to be larger than might be expected simply from the funding ratio. The fraction of MRSEC dollars to total materials dollars is 0.05/4, or 1.25 percent. No statistical analysis of these fragmentary observations is possible; however, it is possible to say that there are discoveries of the highest significance occurring within the MRSEC program, as gauged by this subjective survey. As can be seen later in this section, there is almost an orthogonality between the types of institutions responsible for “major discoveries” and “top cited papers,” the former originating in industrial laboratories and the latter in universities. The committee suggests that many of the more recent fundamental breakthroughs occur in academe, often with MRSEC-funded facilities, whereas materials discoveries more closely linked to commercial products were more naturally done in industrial settings. The trend may reflect the passing of the torch from the formerly powerful industrial labs to universities. The field of organic/polymeric conductors was supported from its inception

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward FIGURE 3.3 Characteristics of the most significant discoveries in materials research. Where each breakthrough occurred. Note the predominance of U.S. industry. Number of senior researchers involved in each discovery. Note that most discoveries were made by teams of only one or two senior investigators.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward This survey of the most highly regarded research laboratories, across the subfields of materials, documents the leading role played by U.S. universities with MRSECs. As a group, U.S. universities with MRSECs are more identifiable with perceived excellence in materials research than any other grouping in the committee’s survey. Beyond the strong correlation of universities with MRSECs and perceived leadership, it is difficult to document whether this correlation is cause or effect. Some of the groups are MRSEC-supported while others are not. None of these world-leading groups is solely supported by a MRSEC. In at least one case, an expert specifically claimed that the institution’s MRSEC was not supporting the top-ranked group. One might be tempted to contrast these results with those in the list of major discoveries described above in this chapter. Note, however, that the major discoveries were dominated by developments from 10 to 40 years ago, while this virtual-voting exercise was sensitive to contemporary impressions of perceived importance. Furthermore, the exercises were sensitive to different characteristics of research excellence: major discoveries versus overall high quality. The distribution of leading groups was not uniform across the MRSEC program. Of course, there is some correlation between the number of years that an institution has had a MRSEC and its funding level and how well it does on this plot. But again, it is difficult to draw direct conclusions other than that MRSECs are situated at places that do excellent materials research. RESEARCH IMPACT VERSUS FUNDING: QUALITY PER DOLLAR Figure 2.2 in Chapter 2 showed the total federal funding for basic materials research between 1982 and 2002. The as-spent funding for materials research was almost constant in the 1980s, although the decade was followed by growth of about 35 percent between 1994 and 2000, in part reflecting the broadening of fields considered to be “materials research.” Federal agencies support materials research at basic, applied, and developmental levels. When all of these expenditures are aggregated, the total exceeds $2 billion in FY 1995. The last published summaries of these expenditures were made for FY 1994 by the Materials and Technology Committee (MatTec), a subcommittee of the Federal Coordinating Council for Science, Engineering, and Technology reporting to the Office of Science and Technology Policy. It is important to note that virtually all major federal agencies supporting research are represented in this total. Figure 3.21 shows how the $2.124 billion total in 1994 was apportioned among the different agencies. NSF accounted for about 16% of this total, with about half (56%) of this total coming from DMR. (Note: subtracting facilities, $288 million for the Department of Energy [DOE] and $28 million for NSF, one gets a total of $1.793 million, with NSF at 15% and DOE at 34%.) The MRSEC percentage was

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward FIGURE 3.21 Expenditures by federal agency for basic, applied, and developmental materials research and engineering in millions of U.S. dollars and as percentage of total expenditure ($2,124 million) in FY 1994. The data do not include classified research or the construction and operating costs associated with facilities. SOURCE: Materials Technology Subcommittee, National Science and Technology Council, 1994 Annual Report, 1994. about 25% of the total DMR expenditure, very similar to 2006 proportions. Altogether, MRSEC expenditures represent a very small fraction of the federal materials portfolio, amounting to about 2% of the total. Of course, given the variety of activities funded by this portfolio and the different programmatic needs of the different agencies, there is no a priori reason to believe that the number of top-cited papers claimed by a given funding agency is proportional to its relative level of materials funding. The acknowledged sources of funding in the top-cited papers are shown in Figure 3.11. It is important to note that fully 77% of all papers acknowledged multiple sources of funding, implying that—increasingly—no one agency can take sole credit for funding any piece of work. There is no simple relationship between the level of federal funding and the percentage of top-cited papers enabled by this funding. For instance, DOE provides 43% of the support for all basic materials research and garnered 10% of the top-cited papers, while NSF provided 16% and was acknowledged in 16% of

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward the top-cited papers. These data are compiled in Figure 3.22, which compares the percentage of top-cited papers from 1996 to 2006 that acknowledge each agency with the percentage of the total federal budget for materials research spent by that agency. While the overall monetary investment is very different, for the National Institutes of Health (NIH) and the Department of Defense (DOD) there is good agreement between the percentage of the top-cited papers and the percentage of the federal budget used to enable the research in these papers. The NSF represents relatively good value for investment, yielding top-cited research papers at almost twice the rate per dollar invested. The MRSEC program (2% of total materials investment, 5% of top-cited papers) is similar in “efficiency” to NSF overall (14.6% of total materials investment, 30% of top-cited papers). DOE has very high research expenditures but a relatively lower participation in top-cited papers. This is probably because the embedded cost of constructing and operating the DOE user FIGURE 3.22 Comparison of number of top-cited materials papers to materials research expenditures for major federal agencies, 1996-2006. The solid line has a unit slope. The committee was not able to easily separate the university and national laboratory components of the DOE materials research budget; therefore, the DOE data point is not consistent with the university-research program budgets used for the other agencies. NOTE: NSF, National Science Foundation; MRSEC, Materials Research Science and Engineering Center; DOD, Department of Defense; DOE, Department of Energy; NIH, National Institutes of Health.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward facilities is not properly accounted for. The committee also notes that in selecting the data for the plot in Figure 3.22, the papers originating only from DOE laboratories were excluded in order to allow a comparison of DOE-supported university research with NSF-supported university research. Including the DOE national laboratories more than doubles the number of such papers. It can be concluded from Figure 3.22 that NSF research, including that carried out in MRSECs, is more likely to result in a top-cited publication than research funded by any other major agency. Within these statistics, there is little evidence that the MRSECs are more or less productive in this respect than any other NSF materials program. SHARED EXPERIMENTAL FACILITIES An often-cited key element of the MRSEC program is its explicit provision of shared experimental facilities (SEFs) at each center. The MRSEC program provides only limited explicit support for underwriting the capital costs of acquiring and maintaining a comprehensive instrument suite; rather, institutions must find other mechanisms for purchasing equipment (including the use of other NSF programs). MRSECs SEF funds, originating from budgets for IRGs, seeds, and facilities, are usually expended to cover operating costs of equipment and facilities such as maintenance, supplies, or portions of a salary for technical support staff (see Figures 3.23 and 3.24 for the distribution of MRSEC SEF budgets compared to each center’s budget). In 2004, DMR estimated that 12 percent of the MRSEC budgets was spent on capital equipment. The research and training of students and postdoctoral associates in the MRSECs is completely dependent on the availability of SEFs with forefront capabilities. The MRSEC SEFs support a very broad range of materials research, which is essential to a broad community (including many supported by single-investigator grants) but is not altruistic, since the MRSECs could not carry out their own research without the user fees generated by these users. As identified in the National Research Council (NRC) report Midsize Facilities: The Infrastructure for Materials Research,8 each materials research facility secures its capital and operating sources of support in a unique and highly individualized fashion. NSF MRSEC SEF support is often only one component of a complex array of funding mechanisms. Many MRSECs operate their SEF facilities with some user fees in order to recover some of the operating costs. In the larger MRSECs, the SEF user community is larger than the number of MRSEC students by at least a factor of 10. This large user base is necessary to pay SEF staff salaries that could not be sustained on the MRSEC budget alone. Another common feature was that 8 National Research Council, Midsize Facilities: The Infrastructure for Materials Research, Washington, D.C.: The National Academies Press, 2006, p. 62.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward FIGURE 3.23 Distribution of Shared Experimental Facilities budgets for most of the Materials Research Science and Engineering Centers in 2005-2006. FIGURE 3.24 Correlation plot of total Materials Research Science and Engineering Center annual budget versus Shared Experimental Facilities budget. As expected, the correlation is positive but not linear.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward faculty and students who participated in the MRSEC would receive slightly discounted rates for using the instruments as compared with other users on campus. It is also important to note that in most cases, the instrumentation supported under the MRSEC SEF program element was part of a larger suite maintained by the institution. In terms of impact, the committee believes that the shared facilities supported by MRSECs do have significant impact in the larger community, but the committee was not convinced that the MRSEC SEF support was dramatically more effective or leveraged more than any other instrumentation program. For instance, the committee learned from the Midsize Facilities report referred to above that operating costs for shared facilities (including the MRSEC program) are recovered about equally from federal grants, user fees, state awards, and institutional commitments. In examining the MRSEC annual reports, the committee observed a similar mix of reported sources of operating costs for the SEFs. However, the committee was unable to collect reliable data about sources of funds for the acquisition of capital equipment both inside and outside the MRSEC program. It is the committee’s view that MRSEC centers likely attract elevated levels of cost sharing from institutional leaders because they attract attention and provide explicit federal leveraging. The committee notes again that the specific impacts are probably diluted when viewing average trends. For instance, MRSEC participants at the University of Southern Mississippi credited the MRSEC with helping empower them to compete successfully for additional instrumentation awards from NSF and other agencies. The committee did not measure and compare the degrees of utilization of facilities inside and outside the MRSEC program and therefore cannot comment on the relative accessibility of the instrumentation to the broader community. The general perception seems to be, however, that MRSECs do allow wide-ranging access to their facilities. The committee found that MRSECs invest in facilities at a rate comparable to DMR overall and that MRSECs provide about 20 percent of the DMR instrument portfolio. The committee also observed that—averaged over the past 10 years—institutions with MRSECs attracted “instrumentation for materials research” awards at roughly the same rate as institutions without MRSECs. The committee could not easily measure, however, whether institutions with MRSECs attracted a higher volume of instrumentation awards from sources outside NSF. The committee collected information on the levels of MRSEC support that were directed toward shared experimental facilities from the annual reports of the MRSECs. The following observations were made. In 2004, the average MRSEC budget spent on facilities was $276,000 (median $187,000) per year with a total reported investment of $6.6 million;

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward $6.6 million is about 13 percent of the annual $50 million MRSEC program budget. In 2004, the portion of the DMR budget spent on equipment and instrumentation was $30 million (beyond that of the MRSEC program), or about 12 percent of the division’s full budget. In addition, DMR distributed about $5.7 million in equipment and instrumentation funds through the Instrumentation for Materials Research (IMR) program. Thus, in 2004, DMR invested about 18 percent of its annual budget (excluding the MRSECs) in equipment and instrumentation. The committee observes then, that MRSECs invest in facilities and equipment at a rate similar to the overall DMR portfolio of investments. This analysis is extremely informal. It should also be noted that the committee did not compare the type of instruments bought through MRI and IMR awards and those secured through and for MRSECs. Another estimate suggested that MRSECs house about 20 percent of the overall federal investment at universities in million-dollar-class instrumentation for materials research. Also, it should be noted that SEFs in the materials area are not unique to MRSECs, but at institutions with larger MRSECs the SEFs are usually managed and operated by the MRSEC. If MRSECs did not do this, DMR would need to create some other strong facilities program to support materials research. The MRSEC facilities budget also supports (at least in part) technical staff members, who train students and maintain the equipment. About $240,000 per year is spent on capital equipment. Estimating that half of the equipment purchased through the NSF instrumentation programs (DMR’s Instrumentation for Materials Research program or NSF’s agency-wide Major Research Instrumentation program) within DMR ends up in a MRSEC facility, another $5 million—or an average of about $200,000 per center—is added to this amount. Assuming a 10-year life for forefront materials characterization equipment, a center might thus afford a total inventory of equipment of about $4.4 million. The committee also examined the potential correlation between MRSECs and instrumentation funding. In the timeframe 1995-2006, the DMR IMR program awarded about $75 million of grants for the acquisition and development of instrumentation for materials research. Of these awards, 30 percent (by dollar value) were made to institutions with MRSECs (that were active at the time). MRSEC institutions received $402 million during this time frame, about 39 percent of the total $1.04 billion or so awarded by DMR to all institutions (excluding the $544 million for MRSEC funding).

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward Thus, the committee observes that institutions with MRSECs attract IMR awards roughly in proportion with their level of materials research activity (measured by DMR funding levels). MRSECs are, however, taking a lead in working together on facilities rather than competing with one another. Led by the MRSEC at the University of California at Santa Barbara, the University of Southern Mississippi MRSEC, in collaboration with those at the University of Minnesota and the University of Massachusetts at Amherst, proposed to NSF to create a national facilities network. The award has recently been funded and will be used to encourage off-campus users to take advantage of the facilities; it also helps send students and faculty among the four sets of facilities at “internal” user rates. The variations in actual capital spending equipment from one MRSEC to another are considerable because the availability of resources hinges on other features of the institution such as the development office, relationships with corporate sponsors, and so on. The recent National Research Council report on shared experimental facilities found that most SEFs that serve the large majority of the materials community have a $1 million to $50 million replacement capital cost, with an average of about $10 million.9 In fact, the U.S. investment in such facilities is currently well below the replacement level,10 estimated to be on the order of several billion dollars per year. At present, other sources of support for SEF equipment (typically, the universities themselves, or in some cases foundations) are not large enough to make up the difference in needed support. Thus, the average age of equipment in SEFs continues to increase, with many individual items being more than 20 to 25 years old. FINDINGS AND RECOMMENDATIONS Conclusion: Consistent with previous analyses, the committee found no simple, quantitative, objective measure to clearly differentiate the MRSEC research product from that of other mechanisms supporting materials science and engineering research. The committee found the task of evaluating the impact of MRSEC research quite daunting, primarily because research papers published in peer-reviewed 9 National Research Council, Midsize Facilities: The Infrastructure for Materials Research, Washington, D.C.: The National Academies Press, 2006, p. 2. 10 National Research Council, Midsize Facilities: The Infrastructure for Materials Research, Washington, D.C.: The National Academies Press, 2006, p. 113.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward journals rarely attribute the results to a single support mechanism. Moreover, any research, even by an individual researcher associated with a MRSEC, is a combination of activities supported “inside” and “outside” the MRSEC. Thus, even if MRSECs have played a unique role in the research enterprise, such as in enabling the formulation of research projects that could not otherwise have been envisioned, there is no easy way to provide substantiation. It could be that the research enterprise has evolved over the past decade, leading to greater convergence and overlap between MRSECs and other research practices. Thus, it is not currently possible to distinguish the unique contributions of MRSECs. General Finding: Sponsors of research are increasingly unable to claim “sole ownership” of research results; MRSECs are no exception. Most research publications now acknowledge multiple sponsors. It is not possible to demonstrate that the MRSEC support yields leadership in discoveries, publications, or citations. In part this is because funding per MRSEC has decreased significantly in the past decade, so that each group requires multiple sponsors. General Finding: Most highly cited publications contain one or two senior authors, indicating that the size of research collaboration is usually small. Although the materials field is highly collaborative and the general belief is that the community benefits from interactions between local groups of many individual investigators in the same field, discoveries and publication records indicate that over 50 percent of the published papers are from individuals and groups of two. Although the committee was unable to identify MRSEC-enabled research in “blind taste tests,” it successfully assessed the overall research quality in comparison with the research enabled by other mechanisms and elsewhere around the world. For instance, do published research results that acknowledge MRSEC resources achieve citation indices and other measures of impact comparable with research enabled by individual-investigator awards? Conclusion: Overall, the MRSEC program produces excellent, frontier science of the same high standard as that supported by NSF through other mechanisms. In terms of quality, MRSEC research is at least on a par with that of other multiple-principal-investigator programs and individual grants in the United States and internationally, and is an important element of the overall mix for support of materials research, including support for big centers and single-investigator grants. The outstanding discoveries, leading research groups, and most significant publications worldwide are associated with universities at which there are MRSECs.

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward MRSECs are involved in the most active areas of materials research as established by their publication records compared with those of the entire field. The MRSEC program has the same level of research collaboration as found in comparable national and international groups. The committee studied a set of major breakthroughs in materials research over the past four decades. U.S. universities, and in particular MRSECs and their predecessors the MRLs, played a limited but pivotal role in a handful of these discoveries. The committee conducted several comprehensive analyses comparing citations of MRSEC-reported research publications and those of the broader research community. The distribution of MRSEC-reported “top-cited papers” across subfields of materials research was very similar to that of the top 100 most-cited papers. Affiliations of the top 100 research papers also showed a 10 percent contribution from institutions with MRSECs or MRLs. The committee also found that the top MRSEC papers were cited much more frequently than the average materials research paper but that the best-of-the-best materials research papers had significantly more citations. However, these papers generally predate the emergence of the MRSEC program. The committee also found that the MRSEC program has the same level of collaboration as that found in comparable national and international groups. To some extent this may be the ultimate success of the MRSEC program in having fostered this type of research at an early stage. Finally, the breakdown of departmental affiliations of MRSEC authors and those of the top-cited materials research papers were quite similar. In two related exercises, the committee examined the global stature of MRSEC-related research groups. In comparison to the Max Planck Research Institutes of Germany, the MRSECs’ publication-citation rates were quite comparable. In a “virtual voting” exercise, the committee contacted researchers around the world in several different subfields and solicited their opinions about world-leading research teams. Research teams at institutions with MRSECs dominated the results. Although many of these measures are of correlation and not causation, the committee came to believe that the research program enabled by MRSEC awards has been, in general, at least as effective as that enabled by other mechanisms. Conclusion: The MRSEC program offers one of the principal opportunities in materials research to support shared experimental facilities (SEFs) that include not only equipment but also the personnel to provide training for students and to perform maintenance. Growing constraints on the per capita MRSEC budget have greatly diminished this ability, which is a concern for the infrastructure of materials research in general. It should be noted that SEFs in the materials area are not unique to MRSECs,

OCR for page 71
The National Science Foundation’s Materials Research Science and Engineering Centers Program: Looking Back, Moving Forward but at institutions with larger MRSECs the SEFs often are managed and operated by the MRSEC. If MRSECs did not do this, DMR would need to create some other strong facilities program to support materials research. A large user base is necessary to pay SEF staff salaries that cannot be supported solely by the MRSEC budget. The MRSEC SEFs support a very broad range of materials research (and sometimes other kinds), which is essential to a broad community (including many supported by single-investigator grants), but it is not just altruistic—the MRSECs could not carry out their own research without the user fees generated by these users. Shared facilities are an important resource for the overall community. For instance, individual investigators are unlikely to be able to afford to acquire and maintain a cutting-edge transition electron microscope, whereas a MRSEC SEF would be ideally suited to do so. Such an instrument sited at a MRSEC would be highly leveraged (because of institutional commitments to existing infrastructure and an established user community that would supply fees-for-use) and would greatly expand the opportunities available to the local research community. The committee encourages recent efforts by the centers and NSF to use modest supplemental grants to encourage and promote broader access to these facilities. These instruments are a core part of the value of the MRSEC program and can have enhanced national impact through improved communication and coordination. As described in the beginning of this chapter, the committee concludes that the merit of the research enabled by the MRSEC program is comparable with the best of the materials research supported by other mechanisms. The committee notes, however, that it focused on measuring the impact of research results and that the ancillary benefits of MRSECs are not reflected by these metrics.