Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 13
Major Award Decisionmaking at the National Science Foundation 1 Major Awards at NSF This report originated in a request by congressional conferees on appropriations for the National Science Foundation (NSF) that the National Academy of Sciences (NAS) study the criteria weighed by the NSF in making major award decisions and assess the roles in the merit review decisionmaking process of outside scientists and executive agency staff. The study was assigned to the Committee on Science, Engineering, and Public Policy (COSEPUP) of the NAS, which has the task of addressing important questions that cut across all areas of science and engineering, such as peer review of research proposals. COSEPUP agreed to study NSF's system of decisionmaking for major awards, with the understanding that "major" awards were those subject to review by the National Science Board (NSB) because of size (at least $1.5 million a year or $6 million over five years) and funded from NSF's appropriation for Research and Related Activities (R&RA). The report does not address decisionmaking on awards for projects funded from the appropriations for Education and Human Resources or the U.S. Antarctic Program, although the recommendations may also be adaptable to those projects. The NSB has and generally uses the authority to review and approve all major awards. Currently NSB reviews between 30 and 50 decisions a year, mostly involving large research centers and research facilities. NSB-approved awards constitute about 30 percent of NSF's R&RA budget of $2.0 billion in FY 1994. Authority for the other 8,000 or so awards made annually, which average less than $90,000 a year, is delegated to the director of NSF (although the
OCR for page 14
Major Award Decisionmaking at the National Science Foundation awards must be made within program areas that have been approved by the NSB). COSEPUP also obtained NSF's agreement that the report would not comment on the substantive merits or "correctness" of any particular award decision, although the immediate impetus for the congressional request resulted from a specific case (i.e., the decision to award the National High Magnetic Field Laboratory to a university that was not the first choice of outside peer review panels). Evaluating the merit of even one specific decision would require a full-scale peer review process parallel to the original one, an effort beyond the time and resources available for the study. COSEPUP concluded that it was more useful and appropriate to evaluate NSF's capacity to make wise decisions in the future on awards (i.e., to see if its review policies, procedures, structure, and resources are conducive to good decisionmaking). Even the best merit review decisionmaking process for major awards cannot guarantee a perfect result; the projects are too complex and the knowledge is too imperfect for that. Nor will an excellent merit review process always prevent complaints from applicants who are denied a major award, but the basis for the decision should be understandable to fair-minded observers. If the process is fair and understandable, NSF will also remain free of the pork barrel pressures that have affected other science appropriations. COSEPUP, with the approval of the president of the NAS, appointed a panel to carry out the study. The panel consisted of 14 experts in physics, astronomy, geosciences, engineering, biology, and social science research; science policy and peer review; organization and management of federal research agencies, academic institutions, and large research projects; and federal grant and contract administration. The panel was also constituted to have members from institutions and regions that have received relatively few NSF awards, as well as from those that have received many over the years. An effort was made to appoint members who were not direct participants in programs resulting from major awards. A few members, however, were from institutions involved in major projects supported by NSF; they provided a grantee perspective on NSF award decisionmaking.
OCR for page 15
Major Award Decisionmaking at the National Science Foundation The panel studied NSF’s policies and procedures governing major awards; consulted with past NSF directors, current staff, and NSB members; and examined 10 case studies of past major awards for research centers and facilities (Appendix E). It studied the series of cycles that a major award undergoes—the initial decision to launch a major project; the planning and implementation of the review process; the decisionmaking leading to the award; and subsequent decisions to renew, recompete, or terminate a project at the end of each award period—focusing on the roles of peer reviewers, staff, outside advisory groups, and the NSB in merit review. The panel also looked at the extent of involvement in, and understanding of, the process by communities outside NSF. OVERVIEW OF MAJOR AWARDS The mission of the NSF is to foster the growth of new knowledge through a balanced program of investments in high-quality science and engineering research projects, education and training programs, and related research infrastructure. To help carry out this mission, NSF plans and makes some major awards for national-scale research facilities, multidisciplinary research centers, and large-scale organized research programs. Projects supported by major awards vary widely in terms of function, size, uniqueness, size of user community, and sponsorship. Functionally, most are either facilities or interdisciplinary research centers.1 Operationally, they range in size from the threshold level of $1.5 million in annual funding to $50 million a year for the National Center for Atmospheric Research (NCAR). They also 1 Other types of activities include a large-scale longitudinal social and economic survey (the Panel Study of Income Dynamics run by the University of Michigan); some large university-based disciplinary research groups (e.g., nuclear physics research with electrons, photons, and antiprotons at the University of Illinois); and several institutes for mathematics and theoretical physics.
OCR for page 16
Major Award Decisionmaking at the National Science Foundation include multiyear construction projects in a range of sizes up to more than $200 million for the Laser Interferometer Gravitational Wave Observatory (LIGO). Many of the major awards are part of multiaward programs. There are, for example, 25 Science and Technology Research Centers (STCs), 18 Engineering Research Centers (ERCs), and nine Materials Research Laboratories (MRLs) located at universities around the country. There are four NSF-supported Supercomputer Center facilities. Some major awards are for one-of-a-kind national facilities based at and operated by one university or operated in a separate location by a consortium of universities. Competitions for single university-based facilities have occasioned the most controversy because the stakes are high and there can be only one winner. Examples include the National Nanofabrication Users Facility (Cornell University), Earthquake Engineering Research Center (State University of New York at Buffalo), National High Magnetic Field Laboratory (Florida State University), and the National Astronomy and Ionosphere Center (operated by Cornell at Arecibo, Puerto Rico). The roles of careful planning, clarity about criteria and their weightings, choice of appropriate reviewers, prior understanding of the review process that will be used, and justification of the final decision are especially important in these cases. Some of the major NSF-supported projects serve a large number of users and are accessible to any researcher with a suitable project. The Supercomputer Centers, for example, serve all fields of science and are readily accessible by NSFNET to researchers across the country. Most facilities, however, are set up to serve a particular field or subfield.2 The global seismic detector network operated by 2 Many of the facilities have in-house research staff who work part-time on their own research projects that do not go through the same merit review process an outside researcher undergoes to obtain an NSF award for a project using the facility. This opportunity to access unique facilities enables the facilities to recruit high-quality staff who can better assist outside researchers in using the facilities more effectively as well as develop state-of-the-art instrumen
OCR for page 17
Major Award Decisionmaking at the National Science Foundation the Incorporated Research Institutions for Seismology (IRIS) can be used remotely by earth scientists. LIGO is a specialized facility for gravitational physics research. Some of the largest awards go to consortia of research institutions. The University Consortium for Atmospheric Research (UCAR), which operates the National Center for Atmospheric Research, is a consortium of more than 50 universities active in atmospheric research. The national optical telescopes at Kitt Peak, Arizona, and Cerro Tololo, Chile, are managed by the Associated Universities for Research in Astronomy (AURA). AURA has 17 university members. IRIS, which is constructing and managing the global network of seismometers, has more than 80 member universities and colleges. The Ocean Drilling Program has seven international partners representing 18 nations that are also members of an international scientific organization, the Joint Oceanographic Institutions for Deep Earth Sampling, which provides scientific advice and direction to the program. A number of ERCs and STCs, although based at one university, involve networks of universities. Most major awards are solicited (i.e., there is an open competition for which proposals are formally requested). Not all are competed, however. Some of the largest and most expensive facilities have been developed jointly with the particular group of research institutions whose researchers would be likely to make the most use of them. NSF invites the group to incorporate and encourages it to submit proposals to build and manage the project. This was the approach taken with NCAR (UCAR) and the Kitt Peak National Observatory (AURA). IRIS is a more recent example of an unsolicited proposal. The universities involved in research on the earth's crust were asked to submit a proposal to build and operate the global seismic network and tation and useful data sets, but it reduces the time available to outside researchers. The question of appropriate balance between in-house research and outside users to achieve the most overall progress in a field is one that cannot be determined here but must be resolved during the planning and management of each major project.
OCR for page 18
Major Award Decisionmaking at the National Science Foundation manage the data. Although invited, it was reviewed as a traditional unsolicited proposal (i.e., no special criteria or procedures were specified, the four basic review criteria were used, and there were no competing proposals). The same approach was also taken with LIGO. The leading university groups involved in gravitational research decided with NSF's encouragement to submit a joint proposal to build and manage the facility. Proposals to provide the two sites for LIGO were, however, formally solicited in an open competition. These projects also vary in the expectations and conditions for renewing or reopening them to competition. The traditional approach has been an open-ended one in which awards are expected to be renewed as long as the grantee's performance is satisfactory. UCAR has managed NCAR, and AURA has operated Kitt Peak since 1960 under a series of renewed awards that were not openly competed. A similar approach was taken with the Francis Bitter National Magnet Laboratory at the Massachusetts Institute of Technology (MIT), which was inherited from the Air Force in 1970); and the National Nanofabrication Users Facility at Cornell after it was established through a competitive solicitation in 1977. NSF eventually decided to open those awards to competition. NSF has taken a different approach in the programs of campus-based research centers. The 12 original MRLs that NSF inherited from the Advanced Research Projects Agency in the Department of Defense began to compete with new proposals for renewal funding, and as a result, some lost NSF support. ERCs and STCs receive five-year awards, but undergo a full merit review during the third year of each award and either receive a new five-year award or are phased out at the end of the current award period. After 11 years, a center is supposed to be on its own or must reapply de novo in competition with other new proposals.
OCR for page 19
Major Award Decisionmaking at the National Science Foundation MAJOR AWARDS AND MERIT REVIEW All awards, large and small, are made in response to proposals that are reviewed for merit by outside peer experts, NSF staff, and in some cases, scientific advisory committees or other federal officials with relevant expertise. This merit review system is designed to ensure that appropriate criteria are used to evaluate proposals and identify the proposal with the best promise of achieving the goals of the project. Merit review, when used for major awards, often involves more diverse criteria and a more complicated process (ultimately involving the NSB) than when it is used for the typical small research project grant. In a given fiscal year, NSF supports about 18,000 research awards. Annually about half are new awards chosen from among about 24,000 proposals (including renewals). The remainder are continuing awards in their second or later year. Most of the 18,000 awards are grants to individual researchers or small research groups. Some are very large awards for research centers, facilities, and other major projects. Although fewer than 100 major awards of $1.5 million or more are made each year, they account for about 30 percent of annual R&RA budget expenditures.3 Since awards for research centers and facilities tend to be larger and awarded for longer periods than average, they constitute a greater mortgage on future R&RA funding than individual research grants.4 Major awards constitute an even larger share of the budgets of certain NSF 3 In contrast, the mean annual size of all awards is less than $90,000. In fact, because most are two-year grants to individual university-based researchers working with a few graduate students and postdoctoral fellows, the median annual award size was about $50,000 in FY 1990 (compared with $51,000 in 1988) (NSF Executive Information System). 4 For example, in June 1991, NSF projected commitments in FY 1992 and beyond of $1.1 billion, of which $478 million was committed to center and facility awards (NSB, 1991a:C-5); thus, these awards accounted for less than 30 percent of current spending but 43 percent of funding committed in the future.
OCR for page 20
Major Award Decisionmaking at the National Science Foundation directorates with major infrastructural support, for example, Computer and Information Science and Engineering (48 percent of current funding and 63 percent of commitments) and Geosciences (31 percent of current funding and 53 percent of commitments). Major awards are also significant because they usually support activities that promise to have a major influence on the conduct of research in their field—facilities such as state-of-the-art telescopes, supercomputers, a global network of seismic detectors, and research centers that promote interdisciplinary or application-oriented research or both. Unlike the traditional small research project grants to individual researchers and small research groups, major awards may also invite geographic competition and ''pork barrel'' pressures from Congress because of the economic benefits they may bring to state and regional economies. These situations offer NSF an opportunity to leverage cost sharing from private industry and state governments. They may also provide opportunities for international scientific cooperation and cost sharing. From the beginning, NSF has tried to ensure the quality of the research that it supports by openly inviting researchers to compete for funding through proposals. NSF has also promoted quality through an evaluation process that includes outside peer review of proposals by active researchers best able to apply the selection criteria. Peer review is a key part of the merit review process, because it increases the objectivity and impartiality of proposal evaluation and helps to ensure the quality of the activities supported by NSF. It helps give awards decisions credibility with the scientific community, Congress, and the public. The peer review aspects of merit review maximize the role of technical considerations in making awards and help shield the decisionmaking process against internal agency bureaucratic interests and outside political pressures. This competitive peer review mechanism was later extended to large-scale research facilities and centers even though the award criteria for such projects tend to be more numerous and varied. Despite the greater complexity and higher stakes involved in major award decisions, NSF has successfully made many without controversy. It has also maintained its
OCR for page 21
Major Award Decisionmaking at the National Science Foundation autonomy to choose projects to support and proposals to fund without political interference.5 The criteria, procedures, and management practices associated with decisionmaking for major awards must be clear, effective, fair, and accountable. Some recent decisions by NSB have raised questions about the review process, selection criteria, choice of reviewers, staff discretion, or the NSB's role. For example, one impetus behind this study was NSF's 1990 decision, approved by the NSB, to award a five-year, $60 million grant to build and operate a National High Magnetic Field Laboratory to a consortium headed by Florida State University, even though peer review groups—the site visit panel and divisional advisory committee—had recommended a proposal by MIT. The NSF-NSB decision in 1990 to award $212 million to build and operate LIGO has been criticized because it may squeeze the budgets for other projects if NSF's funding does not grow as planned. That decision also drew criticism because of questions about the engineering feasibility of achieving the levels of sensitivity needed to detect gravitational waves. Even if there were no controversial award decisions, a review of NSF's capacity to make wise decisions on major awards would be prudent and timely. The number of major awards and their share of the NSF budget have been increasing in response to several trends. Research facilities and instruments are increasing in capability and cost. They, in turn, foster larger-scale and more interdisciplinary research efforts so that areas traditionally dominated by individual investigator research are becoming dependent on expensive facilities, coordinated research programs, or both. The budgetary impact of the growth of research center programs and the increasing costs of research facilities has been offset by the steady real growth in the NSF budget since 1986, but such growth is not guaranteed. 5 NSF is one of the few agencies that has not been subjected to academic pork barreling (e.g., it has not had funding for facilities at particular universities or research institutions earmarked in its congressional appropriations).
OCR for page 22
Major Award Decisionmaking at the National Science Foundation MAJOR AWARDS AND THE NSB The focus of this report is the set of awards for research and related activities subject to NSB approval because of their size (more than $1.5 million a year or more than $6.0 million over the award period, which may not be longer than five years) or importance. In FY 1992, for example, the NSB reviewed 49 awards—32 for research-related activities and 17 for education-related projects (listed in Appendix C). Over the five-year period FY 1988 through FY 1992, NSB reviewed 157 awards—120 for research, 35 for education, and 2 for logistical support in Antarctica (Appendix C). The 120 research awards involved 91 discrete projects (some projects received multiple awards during the five-year period). The NSB also reviews and approves "project development plans" (for most major facilities) and solicitation announcements (for major awards that have been formally solicited). It approves new programs, which usually consist of individual investigator grants but may also include new programs of research centers. This report addresses the steps that occur prior to the review of proposals (Chapter 2) because they determine the goals of a major award, which in turn should shape the review process and criteria (Chapter 3). The better planned a major project is, and the better planned the review process is, the more likely is the award decision to be understood and supported in the affected communities. The NSB has at least three opportunities to review new large-scale activities as they move through the major award process (Table 1.1).6 First, in the case of physical facilities, the NSB approves early in the process a project development plan that establishes the need and technical feasibility of the facility. In the case of research centers, the NSB receives a short document describing the new program in conjunction with a draft of the solicitation notice. 6 Although the NSB does not approve smaller grants individually, it does approve the programs through which such grants are made. In October 1989, for example, NSB approved a new Program for Arctic Social Science.
OCR for page 23
Major Award Decisionmaking at the National Science Foundation TABLE 1.1: Decision Points for NSB Involvement in Major Awards Phase NSB Action Planning Project Development Plan Approval for Research Facility Awards Since 1979, NSB has required its review and approval of "project development plans" for "big science" project involving large-scale commitment of funds; capital facilities and major equipment; multiyear duration; and continuing expenditures for maintenance, replacement, operating costs, and research. General Program Approval for Center and Other Nonfacility Awards NSF cannot accept proposals unless the program within which the award will be made has been reviewed and approved by NSB. Design Approval of Solicitations for Major Awards NSB reviews and approves Requests for Proposals and other solicitations in which the resulting awards are expected to require NSB approval. The solicitations document the specific procedures and criteria that will be used to decide on awards. (Solicitations may be issued before NSB approval in exceptional circumstances as long as NSB is advised at its next meeting.) Award Approval of Proposed Major Award Recipients NSB reviews and approves decisions to make awards of at least $1.5 million a year or $6.0 million over the award period (up to five years). (NSB may grant a waiver for routine or continuing awards that pose no significant problems or policy issues.)
OCR for page 28
Major Award Decisionmaking at the National Science Foundation • Reviews and evaluates reports and publications submitted by awardees • Makes site visits and consults with awardees • Initiates revisions of project budgets and project descriptions when necessary, and gives guidance and management oversight to an extent appropriate and authorized by the conditions of the award • Provides for dissemination of research accomplishments or other results from awards • Reviews complete awards, including technical reports, summaries, and journal publications Consultation and Liaison • Advises prospective awardees and institutional representatives about NSF objectives, policies, and practices or refers them to DGA • Serves as primary consultant within NSF on technical matters in his or her area of competence • On request, coordinates and advises on aspects of his or her program that involve other facets of society such as national resources, technological assessment, and social and cultural organization • Exchanges program information with other agencies and institutions • Represents NSF at professional meetings and seminars Administration • Formulates plans, supervises program staff, and assigns and reviews work • Prepares reports • Fulfills internal budget and operating requirements • Performs staff work for, and participates in, program review and evaluation activities • Recommends new, and improves existing, procedures for program management procedures NOTE: Many of the functions listed are not the final responsibility of the program director but involve section heads, divisions directors, assistant directors, and other parts of NSF.
OCR for page 29
Major Award Decisionmaking at the National Science Foundation has promoted this practice as a way to bring in program directors with current knowledge of pioneering research and to prevent the agency from becoming too bureaucratic or out of touch. On the other hand, rotating staff makes continuity of policies and procedures more difficult, which may cause inconsistent treatment of the kind of longer-term activities that tend to be supported by major awards. Two trends could have a negative impact on the ability of NSF program directors to manage the merit review process for all awards, including major awards. First, the number of proposals has increased much faster than the number of staff. Second, travel funds required by program staff (for site visits by program directors to oversee the implementation of major awards, and for participation in scientific conferences and other professional activities that keep them abreast of current research developments and familiar with productive researchers) have not increased. The number of fully reviewed proposals has gone up 70 percent over the past 10 years. NSF's budget grew 76 percent in constant dollars from FY 1983 to FY 1993. The number of staff, however, decreased slightly during that period, from 1,213 to 1,192. As a result, the percentage of the NSF budget devoted to administration has declined from more than 6 in 1983 to less than 4 in 1993. NSF has coped with the growing workload by adopting new information technologies and simplifying proposal processing procedures. In 1984 an internal staff task force recommended a number of measures to speed up processing (NSF, 1984b). In 1990 a task force on merit review recommended additional measures to simplify forms and streamline procedures (NSF, 1990c). NSF also has moved to reduce the proposal burden on the research community by reducing the number of pages allowed in a proposal and increasing the average award length to three years (NSF, 1992d). Despite these steps, NSB member James Powell (representing the NSB chairman) brought up the imbalance between funding and staffing levels as the one concern he could "single out from all the others that the Science Board has" at hearings on NSF's FY 1993 appropriation (U.S. Congress, 1992:5). Powell said the staff was overworked because of the large increase in the number of proposals
OCR for page 30
Major Award Decisionmaking at the National Science Foundation reviewed and funded. He also brought up the greater extent of active oversight called for by ERCs and other new types of complex activities undertaken by NSF in recent years. At the same hearing, NSF director Walter Massey testified that NSF efforts to cope with growth by investing in computers, electronic mail, and other new technologies had reached their limit, and he asked for an increase in staffing (U.S. Congress, 1992:6). In addition to automation, NSF has coped with the large increase in number of proposals by increasing the number of professionals within its static staff size. Nevertheless, the average number of proposals per program director has been increasing. Program directors for major awards may handle only a few awards. The large national facilities for astronomy and atmospheric research, for example, which come up for noncompetitive renewal every five years, have several program directors managing their awards. Other program directors handle a small portfolio of center awards that go through a full proposal review every three years. For example, ERCs and Industry/University Cooperative Research Centers are administered by a separate Division of Engineering Education and Centers in the Directorate for Engineering. Supercomputer Centers are under a separate division of the Directorate for Computer and Information Science and Engineering. Some program directors have a mixed portfolio: they oversee a large number of individual investigator proposals and one or more centers or facilities. Up to now, concerns about excessive workload have focused on programs with large numbers of individual investigator proposals. Nevertheless, if NSF increases its efforts to plan major awards, adopts more elaborate review procedures, or expects greater oversight of ongoing awards as part of the renewal process, the panel believes that the number of staff needed to administer merit review for these awards should be increased.
OCR for page 31
Major Award Decisionmaking at the National Science Foundation OVERALL CONCLUSIONS The panel concluded that merit review has generally served well to ensure fairness, effectiveness, and efficiency in decisionmaking on research projects over the years, but for major awards the system needs some adjustments to accommodate evolving conditions and special features of costly large-scale, long-term projects. NSF has successfully made many highly visible and important awards with relatively few controversies. The merit review system has been the major reason for the high quality of the activities selected for support, and it has served to discourage the use of inappropriate or parochial considerations in the allocation of NSF's research funding. Merit review is not perfect, but no clearly superior method of selecting research and research-related projects for support has been discovered after many years of experience here and abroad. The panel observed that merit review procedures and criteria governing major awards have been adapted from those originally used for awards to individual investigators. The scientific community perceives the criteria and procedures for merit review as difficult to understand and sometimes inconsistent when criteria in addition to strict technical merit are employed, which is typically the case in large-scale projects. The panel encountered cases in which NSF's objectives and criteria—or their relative importance—seemed to be applied inconsistently during the review process: the combined expertise of the reviewers did not match the ostensible criteria; the relative roles of staff, reviewers, and advisory groups were unclear; and the rationale for the final decision was not clearly stated. To sustain confidence in NSF's review process, such problems need to be systematically reduced for large awards, both in fact and in the perception of the research community, Congress, and other interested groups. NSF should recognize that major projects are different enough from individual investigator awards—in nature (larger scale and longer term), degree of impact on a research area, and visibility at the local level—that review and selection procedures must be followed
OCR for page 32
Major Award Decisionmaking at the National Science Foundation more consistently and decisions must be justified publicly. These differences are becoming even greater. Major projects are growing larger (often involving one-of-a-kind national undertakings); each directly affects the NSF budget for its immediate field of science. Some are large enough to affect the allocation of NSF's research budget across fields if annual budget increases are smaller than anticipated. Not only are large projects costly in current dollars; they also reduce future flexibility within their fields to respond to new research opportunities. Most ideas for major projects and facilities arise from the accumulated findings and insights of individual investigator-initiated research projects. It is important for the future health and vitality of a field that the resources for large multiyear projects are not so great as to reduce the ''seed corn'' for future major projects. Large projects aimed at realizing current new opportunities always introduce a degree of rigidity in planning and programming that reduces overall flexibility to take advantage of future developments. It is important, therefore, that decisions to engage in large projects and their planning and implementation be closely scrutinized and fully justified. Although controversial decisions have been relatively rare, they have revealed problems in NSB and NSF policies and procedures that could be avoided. When such problems occur or are believed to occur, they undermine the confidence in the merit review system of the research community, research institutions that compete or hope to compete for major awards in a fair process, and Congress. So far, the success of the merit review system has helped insulate NSF and NSB decisionmaking on major awards from congressional intervention. If confidence in the system is not maintained, the temptation for research institutions to try to have Congress preempt NSF decisionmaking will increase, and to the extent that legislative involvement replaces merit review with political considerations in project selection, the quality of the nation's research system may be negatively affected. The panel recommends a number of changes to strengthen or improve the planning, review and selection, and subsequent renewal
OCR for page 33
Major Award Decisionmaking at the National Science Foundation of major awards. Detailed recommendations are discussed in subsequent chapters, but the key points follow: Clear Rules of the Game The "rules of the game" (i.e., the criteria, procedures, and roles of participants in the merit review process) must be absolutely clear in advance. In some cases, the criteria or requirements needed to meet them have not been clear or were seemingly redefined during the review process. Although too much detail in specifying criteria might limit the flexibility to respond to innovative proposals, we concluded that to increase procedural fairness, NSB and NSF should be more precise about the criteria and review process to be used. In particular, the primary technical criteria as distinct from the other criteria to be considered in the merit review process should be identified in advance in each case. The panel recommends stronger planning efforts that would help contribute to clearer criteria (Recommendation 1). The panel also recommends that NSF concentrate more effort in designing a better—understood review process for each major award (Recommendation 8). Primacy of Technical Merit Technical merit must be the primary consideration in making awards. The panel strongly supports the primacy of technical merit in the selection of major projects (Recommendation 3), and it endorses the use of a two-phase review process that would clearly indicate the
OCR for page 34
Major Award Decisionmaking at the National Science Foundation ranking of projects on technical merit before other merit factors are considered (see next section). Technical merit must be paramount to maximize the likelihood that the project will achieve its substantive research goals. Other criteria of merit should also be given due consideration in selecting the overall winner or winners, but any project receiving an award should rank among the very highest in technical quality. That should be made clear to all reviewers and decisionmakers, along with a sense of the nature and relative priority of each of the criteria. NSF and NSB must be clearer in each case about the relative priority of the various criteria used, especially of the technical relative to the nontechnical criteria. Otherwise, the weightings of criteria are implicit and can shift continually at the discretion of individual reviewers and program staff. Appropriate Roles of Peer Reviewers and Staff The review process must be structured so that the roles of peer reviewers and staff in evaluating and recommending proposals are clearly understood, and trade-offs among technical and other criteria are clearly explained, at each subsequent level of decisionmaking. Currently, the summary rating and ranking of proposals by staff at various decision points does not always distinguish peer review from staff judgments. Although staff should make their best case for a recommended decision, the NSF director and the NSB should always know the results of the peer review. The two-phase review process, properly documented, would make it easier to implement this objective (Recommendation 6). This two-phase process would facilitate the preparation of a summary document that explains the rationale for the decision, including the treatment of peer review results and the trade-offs made between
OCR for page 35
Major Award Decisionmaking at the National Science Foundation technical and nontechnical criteria in reaching the final decision (see next section). Public Documentation of Decisionmaking There should be a public document explaining the results of the review and the rationale for the final decision by the NSB. NSB minutes rarely record the basis for a major award decision, and no public document of explanation for the final decision is prepared or disseminated. The lack of such documentation leads to public confusion and controversy that could be avoided. The panel recommends a short, carefully prepared memorandum that summarizes the results of each stage of the merit review process and outlines the rationale for choosing a winning proposal (Recommendation 9). Such memoranda would increase public understanding of major award decisions and therefore enhance public confidence in the system that produces them. More Stringent Setting of Priorities Decisions to solicit proposals for very large major awards should take into account their impact on NSF's overall program as well as on the particular research field involved, and they should be contingent on the realization of expected funds and technological progress. Careful front-end planning, combined with broad consultation with affected research communities and constant evaluation of priorities at each decision point, must be a part of the process of soliciting and reviewing proposals for a very large major award. Solicitations for awards that have serious long-range budget implications must be based on a broader range of considerations than in the
OCR for page 36
Major Award Decisionmaking at the National Science Foundation past. The priority within a given field should be clearly established and compared with the overall priorities of NSF across fields. After initial approval of a large project, contingency plans for possible unfavorable program or budget developments should be made for each project and updated annually. The potential impact on NSF priorities if there are unrealized budgetary expectations or unexpected technological problems or opportunities should be carefully reviewed at each decision point. In this way, NSF and NSB would avoid letting a series of small decisions in the development of a major project result in a project that no longer matches the agency's overall program priorities or budget. The panel calls for stronger planning efforts, including contingency plans for lower funding levels than expected (Recommendation 1), based in part on a broader range of input from research communities affected directly and indirectly by a major project (Recommendation 2). NSB should also put more emphasis on its long-range planning and priority-setting activities (Recommendation 7), and should periodically reconsider the contribution of every project to agency priorities as part of a systematic renewal process (Recommendation 10). In making the recommendations to carry out these goals—better planning, more explicit processes and criteria, more standardized procedures, and more public justification of award decisions when made—the panel does not mean to imply that NSF and NSB are not making any attempts to meet them. Elaborate planning procedures exist (described in Chapter 2), but we believe they could be improved in certain ways to ensure that the research community is well informed and reasonably supportive, and that greater attention is paid to worst-case budgetary and technical scenarios. The project development plan requirement that exists for large-scale facility construction should be extended to all major award projects. NSF has recently augmented its procedures for planning the review process prior to soliciting proposals (Chapter 4); the next step is to ensure that the key features of the review plan are published explicitly in the proposal solicitation document. A distinctive multistep review process has evolved for some types of major awards
OCR for page 37
Major Award Decisionmaking at the National Science Foundation (described in Chapter 3); the benefits of this decisionmaking approach could be usefully applied in all major awards. In recent years, NSF has begun to include requirements for recompetition with explicit timetables in some major awards (Chapter 5); this approach could also be used more generally. The greatest change in current policies and practices would come in the area of explicit justification of award decisions after final approval. In following the traditions of peer review of individual investigator research projects, the results of peer reviews and rationale for the award decision have not been made public (Chapter 4). We believe that such public explanation is justified by the scientific importance, budgetary size, and public visibility of major awards and would increase support for the decisionmaking process. The recommendations in this report are offered to strengthen and protect the use of merit review in government research decisionmaking because, despite its shortcomings, it is the best way known to ensure quality. We do not think it is possible, however, to devise a perfect review process. There will always be losing applicants for a key award who would not be happy even if a new system were devised weekly in response to complaints. If our recommendations are adopted, we believe that the major award review system will still have adequate flexibility to respond to the variation and unpredictability of the real world. The panel understands that its recommendations cannot guarantee a perfect result or prevent individuals and institutions who are denied awards from complaining about the system. This is especially true of awards for large, one-of-a-kind national facilities that must satisfy many expectations. We believe that the changes recommended in this report will result in a fairer and more understandable process; will increase confidence in, and support of, the system by fair-minded participants and interested groups; and will help forestall outside pressure to fund projects on a nonmerit basis.
OCR for page 38
Major Award Decisionmaking at the National Science Foundation This page in the original is blank.
Representative terms from entire chapter: