Major Award Decisionmaking at the National Science Foundation (1994)

ROBERT H. RUTFORD, Ph.D., the panel's Chairman, is President of the University of Texas at Dallas and Professor of Geosciences. He is also Chairman of the Polar Research Board of the National Research Council. Dr. Rutford holds the National Science Foundation's Distinguished Service Medal and the Antarctic Service Medal, and he was previously Director of its Division of Polar Programs. He is the U. S. Delegate to the Scientific Committee on Antarctic Research, a Fellow of the Geological Society of America, and a member of the Board of Trustees of Baylor Dental College, as well as a member of a variety of other community and professional boards and committees.

CLARENCE R. ALLEN, Ph.D., is Professor of Geology and Geophysics, Emeritus, California Institute of Technology. Dr. Allen has been President of the Seismological Society of America and the Geological Society of America. He is a member of the American Association for the Advancement of Science, the American Geophysical Union, the Geological Society of America, the National Academy of Sciences, and the National Academy of Engineering. Dr. Allen received the first G.K. Gilbert Award in Seismic Geology.

ALBERT A. BARBER, Ph.D., Special Assistant to the Chancellor, University of California, Los Angeles (UCLA), acts as the university liaison with federal agencies and higher education associations. He was formerly Vice Chancellor—Research Programs and Chairman and Professor of Zoology at UCLA. Dr. Barber chairs the Board of Directors of the National Association for Biomedical Research and is

a member of the American Association for the Advancement of Science, the American Physiological Society, the American Society of Biochemistry and Molecular Biology, Phi Beta Kappa, and Sigma Xi.

HARVEY BROOKS, Ph.D., is Gordon McKay Professor of Applied Physics, Emeritus, in the Division of Applied Sciences, and Benjamin Peirce Professor of Technology and Public Policy, Emeritus, at the John F. Kennedy School of Government, Harvard University. He is a former member of the President's Science Advisory Committee and the National Science Board and was President of the American Academy of Arts and Sciences. Recently he was a member of the Advisory Council of the Carnegie Commission on Science, Technology, and Government and served on four of its task forces. Dr. Brooks is a member of the National Academy of Sciences, the National Academy of Engineering, the Institute of Medicine, and the American Philosophical Society.

CHRISTOPHER COBURN is Director of Public Technology Programs, Battelle Memorial Institute. He is also staff Director of the Carnegie Commission on Science, Technology, and Government's Task Force on Science and Technology and the States. Formerly, he served as Executive Director of Ohio's Thomas Edison Program and Science and Technology Advisor to the Governor of Ohio. He founded and chaired the Science and Technology Council of the States.

SUSAN E. COZZENS, Ph.D., is Associate Professor in the Department of Science and Technology Studies at Rensselaer Polytechnic Institute and Director of Graduate Studies for the department. She was formerly a policy analyst in the Division of Policy Research and Analysis of the National Science Foundation. While at NSF she also served as Associate Executive Secretary of the Director's Advisory Committee on Merit Review and as a consultant in the review and reorganization of its program evaluation activities.

Dr. Cozzens is outgoing editor of Science, Technology and Human Values , the journal of the Society for Social Studies of Science.

FRANK D. DRAKE, Ph.D., Professor of Astronomy and Astrophysics, University of California, Santa Cruz, previously served as the university's Acting Associate Vice Chancellor, University Advancement, and Dean, Natural Sciences Division. Dr. Drake is a former Director of the National Astronomy & Ionosphere Center, which includes the Arecibo observatory. He has chaired the U.S. National Committee for the International Astronomical Union and the Division of Planetary Sciences of the American Astronomical Society, and has been the Chairman of the Board on Physics and Astronomy of the National Research Council. He is a member of the National Academy of Sciences and the American Academy of Arts and Sciences.

DONALD S. FREDRICKSON, M.D., is President of D.S. Fredrickson, Inc., an international consulting firm, and a part-time Scholar of the National Library of Medicine, engaged in historical research on the support of biomedical research. Dr. Fredrickson was formerly Director of the National Institutes of Health and President of the Institute of Medicine. More recently, he served on the White House Science Council and as President and Chief Executive Officer of the Howard Hughes Medical Institute. Dr. Fredrickson is a member of the Institute of Medicine, the National Academy of Sciences, the American Philosophical Society, and the American Academy of Arts and Sciences.

FREDRICK S. HUMPHRIES, Ph.D., has been President of Florida A&M University since 1985. He was previously President of Tennessee State University. Dr. Humphries currently serves on the Commission of the Future of the South, the Science and Technology Advisory Committee of NAFEO, the White House Science and Technology Advisory Committee, and the State Board of Education Advisory Committee on the Education of Blacks in Florida, which he

chairs. He holds the Meritorious and Distinguished Achievement in Education Award, Nashville Chapter, and the Distinguished Service to the Advancement of Education for Black Americans Award, among others.

ANITA K. JONES, Ph.D. (NOTE: Dr. Jones resigned from the panel on May 31, 1993, to become Director of Defense Engineering, Department of Defense, and did not participate in drafting the report after that date.) Until going to the Department of Defense, Dr. Jones was Professor and Chair of the Department of Computer Science, University of Virginia, and Editor-in-Chief of Transactions on Computer Systems , a quarterly journal. Previously she founded and served as Vice-President of Tartan Laboratories, Inc. Dr. Jones has been a trustee of the MITRE Corporation, member of the Air Force Science Advisory Board, the Lincoln Laboratory Advisory Board, and the Defense Science Board. She has participated as the chair or member of numerous program committees for computer science conferences and has served as an officer in several professional organizations.

LARRY K. MONTEITH, Ph.D., Chancellor of North Carolina State University, also served as Interim Chancellor, Dean of Engineering and Head of the Department of Electrical Engineering. Prior to his appointments at North Carolina State, Dr. Monteith was head of the Materials and Devices Laboratory at the Solid State Laboratory of the Research Triangle Institute and a member of the Technical Staff of AT&T Bell Laboratories.

DOUGLAS D. OSHEROFF, Ph.D., is Professor of Physics, Stanford University. He was previously the Head of the Solid State and Low Temperature Physics Research Department at AT&T Bell Laboratories. Dr. Osheroff holds the Walter J. Gores Award for Excellence in Teaching and the MacArthur Prize Fellow Award, among others. He is a member of several professional associations, including the American Academy of Arts and Sciences, the American

Physical Society, and the National Academy of Sciences, and also serves as Secretary of the International Union of Pure and Applied Physics Commission on Low Temperature Physics.

JUDITH A. RAMALEY, Ph.D., President and Professor of Biology at Portland State University, has held faculty and administrative positions at Indiana University, the University of Nebraska, the State University of New York at Albany, and the University of Kansas at Lawrence. She was Chair of the Academic Affairs Council of the National Association of State Universities and Land Grant Colleges, and Chair of the Commission of Women in Higher Education of the American Council on Education. She is a charter member of the Advisory Committee for the Biological Sciences of the National Science Foundation and also serves on a variety of professional boards, committees, and associations.

LYLE H. SCHWARTZ, Ph.D., was appointed to his current position as Director of the Materials Science and Engineering Laboratory of the National Institute of Standards and Technology in October 1984. Previously he served as Professor and then as Director of the University Materials Research Center at Northwestern University. Dr. Schwartz chaired the panel on international competition and cooperation of the Materials Science and Engineering Study of the National Research Council, and chairs the intergovernmental Committee on Materials Science and Engineering (COMAT). In 1990 he received the Presidential Rank Award of Meritorious Executive for outstanding government service.

This appendix includes (1) a typology of major awards by mechanism (center, facility, etc.); (2) an estimate of how the appropriation for the Research and Related Activities budget of the National Science Foundation (NSF) is distributed by mechanism; and (3) an overview of how the major awards are distributed among the NSF research directorates.

NSF funds about 60 centers large enough to fall into the major award category. The rationale for supporting research centers at universities is to focus on complex scientific and engineering problems that need more expensive facilities and equipment, longer-term support, and larger-scale (usually interdisciplinary) attention than grants to individual investigators or small groups of researchers. It should be noted that the individual investigators associated with centers may and often apply for and receive support from standard NSF research grants and from other agencies for part of their work. Similarly, many of the facilities exist mostly if not completely to provide access to expensive instruments to individuals and small groups who could not otherwise afford to have them.

Materials Research Laboratories. The first centers program supported by NSF, the Materials Research Laboratories (MRLs) set

the pattern. The MRLs were originally established as ''Interdisciplinary Laboratories'' by the Advanced Research Projects Agency in the early 1960s to foster sustained interdisciplinary research on materials using costly, sophisticated equipment (Sproull, 1987). When the centers were transferred to NSF in 1972, the emphasis on interdisciplinary work was increased (Schwartz, 1987). As NSF (1973) put it at the time, "scientific excellence is viewed as a necessary but no longer sufficient condition to qualify for MRL core support." The majority of funding was expected to go to "coherent multi-investigator projects in major thrust areas requiring the expertise of two or more materials-related disciplines," and the MRL proposals also had to meet additional criteria, including effectiveness of local management, extent of support by university administration, level of interdepartmental cooperation, amount of education and training, and fit of proposed research areas within the overall program (NSF, 1973:3). Today there are 9 MRLs, including 8 of the original 12 (several others have entered and left the program since 1972).

Engineering Research Centers. In the 1980s, NSF launched two large center programs for engineering research and science and technology research. The 18 Engineering Research Centers (ERCs) are campus-based interdisciplinary research centers focused on problems related to national economic competitiveness. The program's goals, design, and proposal review process were based on advice from the National Academy of Engineering (NAE, 1983, 1984). As with the MRLs, there are important criteria in addition to the technical excellence of the research proposed. These additional criteria include the contribution of the center type of organization to sustained interdisciplinary research on relevant problems, the degree of cost sharing by state government and industry in order to promote relevance of the research to eventual industrial users, and the impact on education and training. The review process prior to award was elaborate. Each proposal was sent out by mail for review to at least six experts, followed by a panel meeting to identify proposals worthy of a site visit. After the site visits, the review panel met again to

choose the best proposals. The NSF staff then recommended awards to the top-ranked proposals as far as the funding went, and the National Science Board (NSB) reviewed and approved them. The first six ERCs were funded in FY 1985, five more in FY 1986, three in FY 1987, three in 1989, and four in FY 1990; three of these were terminated after their first five-year award.

Science and Technology Research Centers. The Science and Technology Center (STC) program grew out of a presidential initiative to foster basic research in areas of potential significance to national economic competitiveness. The first 11 STCs were funded in FY 1988, and 14 more were started in FY 1990. As with the ERCs, the proposal review process was elaborate, involving mail reviews, site visits, and panels to winnow down the numbers to a small group of finalists. As in the other center programs, factors other than the technical quality of the proposed research per se were important.

Other Major Research Centers. The MRL, ERC, and STC programs account for 52 of the 59 major centers subject to NSB approval. The other seven include the Earthquake Engineering Research Center (EERC) (1986), three biological research centers (1988), a plant science center (1988), the National Center for Geographic Information and Analysis (1989), and one of the 50 Industry/University Cooperative Research Centers (1992) (the other 49 are too small to require NSB review).

Centers represent a class of major awards that grew rapidly in the mid-to late 1980s. They constitute the majority of awards that the NSB must review and approve each year because they are supposed to undergo full merit reviews every three years in order to receive a new five-year award.

The oldest and largest facilities supported by NSF are managed by consortia of the institutions most involved in the relevant field of research. The costs of operating these facilities account for a large share of the funding for major awards. They include

• National Center for Atmospheric Research, Boulder, Colorado ($50 million annually), managed by University Consortium for Atmospheric Research;

• National Optical Astronomy Observatory (NOAO), three sites in Arizona, New Mexico, and Chile ($29 million a year), managed by Associated Universities for Research in Astronomy;

• National Radio Astronomy Observatory, Green Bank, West Virginia, and New Mexico ($27 million a year), managed by Associated Universities, Inc.;

• academic fleet, stationed at a number of universities ($50 million a year), their use managed by the University National Oceanographic Laboratory System, an association of institutions operating the ships for NSF and representatives of the academic oceanographic research community;

• Ocean Drilling Program ($36 million a year), managed by Joint Oceanographic Institutions, Inc.;

• Global Seismic Network and a portable seismic array for fine-detail local studies of the earth's crust ($7 million a year for operations), managed by Incorporated Research Institutions for Seismology; and

• Laser Interferometer Gravitational Wave Observatory (LIGO), Louisiana and Washington ($212 million to construct and an estimated $12 million a year to operate), to be built and managed by a collaboration between the California Institute of Technology and the Massachusetts Institute of Technology and secondary involvement by several other universities with interests in gravitational research.

A distinctive feature of the awards for these facilities is that they are not solicited competitively. The awards are based on the assumption that the facilities are national resources for the use of the entire research community, developed in conjunction with and managed by that community. Competition occurs when individual researchers submit proposals to use the facilities, which are evaluated by a selection committee including outside experts that is administered by the facility manager.

The projects must apply to renew their awards every three to five years. The renewal proposals are subject to the merit review process, just as standard investigator-initiated proposals are, usually involving mail reviews and site visits for input into the decision on whether or not to continue the activity. Proposals for upgrading or expanding facilities, such as NOAO's 8-meter GEMINI telescope, are handled as a separate award and reviewed separately, but still not using open competition.

Over the years the sharpest controversies over award decisions stem from these cases. The EERC and the National High Magnetic Field Laboratory (NHMFL) cases have been mentioned above. More recent cases include the National Nanofabrication Users Facility. As noted, the controversy surrounding the NHMFL stemmed from NSF's decision to make the award to the proposal ranked second by outside peer reviewers. The competing proposal ranked higher by peer reviewers came from the university that had had the most advanced high magnetic field facility in the world for many years and had been supported by NSF since 1972.

In the NHMFL and other single-facility cases, NSF faces the inherently difficult situation of choosing between a proposal from a long-established program or facility and a highly promising proposal from a place that has not had such a program or facility. These cases also place a high premium on procedural fairness, although they may

reduce NSF's flexibility in reaching its program goals. For example, NSF opened itself to criticism in the EERC case for being inconsistent in applying criteria and procedures to competing proposers and thus appearing to prejudge the decision before the review process was complete.¹ Clear procedural fairness and consistency is also important because it encourages competition, especially in cases where there is an existing facility.

The Supercomputer Center facility awards are an intermediate case. They were highly desirable for the universities that received them. NSF's task of choosing among proposers who varied in their background experience and approaches while meeting several program goals was simplified because it was able to choose four winners, not one.

NSF supports a number of other expensive facilities located at and operated by universities, including the Cornell Electron Storage Ring, one of five national high-energy physics accelerators, and two of the seven accelerators available nationally for nuclear physics (the other accelerators are supported by the Department of Energy [DOE]). NSF also supports several university-based synchrotron light sources for materials research, research institutes for pure and applied mathematics and for theoretical physics, and the National Astronomy and Ionosphere Center operated by Cornell University in Arecibo, Puerto Rico.

As research programs and supporting facilities become more expensive, international cooperation and financial participation become more desirable (and, with computer networks, more feasible). The Ocean Drilling Program (ODP), for example, has long had

substantial foreign participation. Six international partners (France, the Federal Republic of Germany, Japan, the United Kingdom, the European Science Foundation representing 12 smaller countries, and a consortium representing Canada and Australia) contribute $2.75 million a year each toward drilling costs and associated laboratories, core repositories, data banks, and engineering development activities. Each country separately funds the costs of the substantive research conducted by its scientists.

International participation creates a more complicated review process, because each member nation has to be consulted about the proposal and any changes in it. Such coordination issues were among the reasons that NSF decided to go ahead with the construction of two LIGO sites within the United States rather than wait to see if other countries would participate in an international effort involving additional sites around the world.

Another international project is the new 8-meter GEMINI telescope facility planned for Hawaii and Chile. Congress mandated the 50 percent foreign financial participation in the $178 million project that was only recently secured. Arrangements will now have to be worked out for joint reviews by the countries supporting the project.

Centers constituted 3 percent of the R&RA budget in FY 1982, 7 percent in FY 1990, and 8 percent ($145 million) in FY 1993. They would form a larger share of the budget if the initial plans for the ERC and STC programs had not been scaled back (NSF originally projected that they would be no more than 10 percent of the R&RA budget if it were doubled in five years, as proposed in FY 1987).

Facilities are also growing as a share of the NSF budget, from 15 percent in FY 1982, to 18 percent in FY 1990, and 22 percent ($404 million) in FY 1993. More recently, this category has grown with the start of several major capital construction projects (e.g., LIGO [$212 million in NSF funding], the GEMINI telescopes [$79 million], and NHMFL [$60 million]).²

NSF considers the majority (more than 70 percent) of the R&RA budget to be "disciplinary research." Most of the disciplinary research consists of individual research support, but it does include a few major awards. Among these are several large group projects in physics using DOE-funded national particle accelerator facilities, a long-term panel survey of income dynamics, and the ODP.

The overall balance among funding mechanisms varies by directorate (see Table B-1). Centers form a major part of the NSF engineering program (23 percent). Facilities are concentrated in the geosciences (35 percent), physical sciences (35 percent), and especially computer sciences and engineering (43 percent).

Most major awards are planned to complement traditional small research projects, for example, by providing them with access to expensive and/or unique facilities or facilitating the formation of a

critical mass of researchers with skills needed to address an important research problem.

Astronomy is an example of a field in which most individual researchers are dependent on large-scale facilities to conduct their work. Two-thirds of the funding for astronomy goes to the national observatories, and part of the remaining one-third goes to university-based telescopes. Many individual investigator grants support the researchers while they are using the national facilities.

A little more than half the funding for materials science goes to individual investigators and small groups. The rest goes to MRLs and the national facilities (synchrotrons, high magnetic field laboratory, etc.) and to a fast-growing activity, collaborative materials research groups. Individual grants help underwrite the costs of researchers using the national facilities.

In the atmospheric sciences, 52 percent of the funding goes to research project grants; the rest goes to the National Center for Atmospheric Research and other facilities. The ocean sciences are also very dependent on facilities; 52 percent goes to research project grants and the rest to the ODP, the academic fleet, and other facilities.

According to the Program Announcement for ERCs issued in 1988 for FY 1990 awards, the criteria used in the evaluation of proposals were based on the guidelines for merit review in the National Science Foundation (NSF, 1992d) document Grants for Research and Education in Science and Engineering and on the key features of an ERC. They were

1. research merit and potential impact on U.S. competitiveness;

2. strength and impact of educational programs;

3. industrial/other user participation and knowledge/technology transfer;

4. leadership and performance competence;

5. institutional environment and support;

6. effect on the infrastructure of engineering.

STC proposals went through a three-stage review process. In phase one, mail and panel reviewers were asked to evaluate proposals based on the selection criteria below. In phase two, the most promising proposals underwent a site visit. In phase three, a special review panel was convened to recommend awards by considering ''the relative merit of the proposals using the criteria listed below, the balance of awards among scientific fields, and the combined ability

of the centers to meet the objectives of the STC program, as well as to enhance the Nation's economic competitiveness'':

1a. intrinsic merit of the research;

1b. research performance competence;

2. effect of the center on the infrastructure of science and engineering;

3. rationale for the center;

4. utility or relevance of the research;

5. institutional support and management plan.

In addition to the basic four criteria described in Grants for Research and Education in Science and Engineering (NRC, 1992d), the following criteria were "taken into consideration in rating the proposals":

• the relevance of the center to the NSF role in earthquake hazard mitigation;

• relevance of the selected research center problem area to earthquake hazard mitigation;

• demonstrated capability to manage, direct, and focus research center activities to establish a coordinated and directed effort in the problem area;

• detailed statements of objectives, goals, and mission of the research center and the methodology for achievement;

• management plan and methodology to allow center activities to be directed, coordinated, and focused;

• plan and methodology for incorporating earthquake researchers from other institutions, industry, and government into the research center's activities;

• plan and methodology for integrating the education of engineers into the research center to provide highly trained professionals in earthquake engineering; and

• plan and methodology for effective and accelerated technology transfer of research results to the end users and the subsequent solution of the relevant problem area.

According to the solicitation announcement, proposals for the NHMFL were to be evaluated by using a two-stage review process. In stage one, mail and panel reviews were used to identify the most meritorious proposals. In stage two, the most meritorious proposals were visited by a site visit team of experts. In this case, the top proposals were also reviewed by the NSF Materials Research Advisory Committee. The following criteria were to be used in the selection process:

1a. intrinsic merit of the research;

1b. research performance competence;

2. effect of NHMFL on the infrastructure of science and engineering:

a. the form, appropriateness, effectiveness, and strength of scientific and technical connections and exchanges with other sections and groups;

b. the quality and appropriateness of the educational and training components;

3. utility or relevance of the research;

4. management plan;

5. institutional and other sector support.

The four general criteria described in Grants for Research and Education in Science and Engineering (NSF, 1992d) were employed. along with and the degree to which each proposal contained the following:

• a clear identification of basic research problems in geographic analysis and geographic information systems that will be the focus of the center's activities. Such problems must be of common concern to scholars in a number of academic disciplines;

• specific plans for productive, multidisciplinary cooperation among faculty, students, and GIS [geographic information systems] practitioners on topics of mutual interest and concern;

• programs that will help alleviate the serious shortage of personnel trained in geographic information systems and geographic analysis in the public, private, and academic sectors;

• plans for acting as a clearinghouse and conduit for information regarding the existence, characteristics, and availability of geographic data bases, domestically and internationally;

• measures designed to maintain and enhance the international competitiveness of the United States with respect to geographic analysis and geographic information systems;

• significant commitments of institutional funds and a plan for obtaining support from external sources in the forms of funds, equipment, and personnel that ensure the involvement of practitioners in the center's research and instructional programs; and

• a management plan for the center that assures broad and continued participation in center oversight by scholars and practitioners from throughout the nation's GIS community.

Criteria for evaluation of MRL proposals were drawn from the guidelines for merit review in Grants for Research and Education in Science and Engineering (NSF, 1992d) and the following key features of an MRL:

1. Research thrust areas

   • Intrinsic merit of the research

   • Research performance competence

   • Degree of interconnection

1. The MRL as a whole

   • Institutional setting and rationale for the MRL

   • Central facilities

   • Seed funding

   • Effect of the MRL on the infrastructure of science and engineering

   • Institutional support and management plan

1. IRIS (Incorporated Research Institutions for Seismology)

2. Engineering Research Centers

3. National Nanofabrication Facility

4. Earthquake Engineering Research Center

5. Ocean Drilling Program

6. National High Magnetic Field Laboratory

7. GEMINI 8-Meter Telescopes

8. Science and Technology Centers

9. Supercomputer Centers

10. LIGO (Laser Interferometer Gravitational Wave Observatory)