Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation (2004)

MAJOR AWARD

DECISIONMAKING AT THE NATIONAL SCIENCE FOUNDATION

Panel on NSF Decisionmaking for Major Awards

Committee on Science, Engineering, and Public Policy

National Academy of Sciences

National Academy of Engineering

Institute of Medicine

NATIONAL ACADEMY PRESS

Washington, D.C. 1994

This report assesses and makes recommendations to strengthen the merit review system used by the National Science Foundation (NSF) to make major awards to support important research facilities, centers, and other large-scale research-related activities. The purpose of the recommendations is to ensure that the most meritorious projects are chosen for support, that the selection process is fair in practice and perception, and that the results in each case are clearly and publicly explained. In this way, the effectiveness and accountability of the major award process will be increased, and the confidence of the research community, Congress, and the public in the system will be enhanced.

The United States has built the most successful research system in the world. The use of peer review to identify the best ideas for support has been a major ingredient in this success. Peer review-based procedures such as those in use at NSF, the National Institutes of Health, and other federal research agencies remain the best procedures known for ensuring the technical excellence of research projects that receive public support. Today, the nation is facing serious international economic competition, which extends to scientific and engineering research. To maintain our world class research enterprise, we will have to be more careful than ever to choose wisely the projects that receive support. The difference between an excellent proposal and one that is merely above average is critical in this effort. The merit review system must be maintained and strengthened to perform the function of choosing the best research for public support.

During the past decade, NSF has established Engineering Research Centers, Supercomputer Centers, Science and Technology Centers, and other large research centers and facilities. A few awards were controversial, and called into question NSF policies and procedures for making large award decisions. Some of those involving the location of one-of-a-kind national facilities have generated the sharpest questions about selection procedures. Decisions by the National Science Board (NSB) and the NSF to devote substantial resources to some new center programs and very expensive facilities have also raised questions about the adequacy of their planning procedures. The congressional conference report on FY 1991 appropriations for NSF requested a National Academy of Sciences (NAS) study of the criteria weighed in making major awards and an assessment of the roles of outside experts and agency staff in the merit review decisionmaking process at NSF. The NAS agreed to undertake the project because of the importance of merit review for making major research awards. The study was assigned to the Committee on Science, Engineering, and Public Policy (COSEPUP), which is chartered by the NAS, the

National Academy of Engineering, and the Institute of Medicine to address important questions that cut across all areas of science and engineering.

COSEPUP, with the approval of the president of the NAS, appointed a panel with a broad range of expertise to carry out the study (Appendix A). The panel studied NSF’s policies and procedures governing major awards, defined as those awards for research and related activities that are subject to approval by the NSB because of their cost. Members of the panel consulted with past NSF directors, current officials, and NSB members, and examined in detail 10 case studies of major awards for research centers and facilities (listed in Appendix E). The NSB reviews between 30 and 50 decisions a year on major awards, which account for about 30 percent of NSF’s Research and Related Activities budget of $2.0 billion in FY 1994 (recent awards are listed in Appendix C).

The panel carefully examined the cycles that each major award goes through. These included the processes leading to the initial decision to announce a major project; the planning and implementation of the merit review process; the decisionmaking leading to the award; and subsequent decisions to renew, recompete, or terminate a project at appropriate intervals. The panel focused on the roles of expert peer reviewers, staff, outside advisory groups, and NSB in the merit review process, and on the public explanation of the process, and its outcomes.

In addition to examining NSF policies and procedures, and the organization and resources it has to carry them out, the panel focused on the role and capacity of NSB in discharging its legal authority for design of the review process and for approving each major award. At each stage, NSB has an opportunity to approve, cancel, or postpone further action.

The panel concluded that merit review has generally served well to ensure fairness, effectiveness, and efficiency in decisionmaking on research projects over the years, but for major awards the system needs some changes to accommodate evolving conditions and special features of costly large-scale, long-term projects. NSF has successfully made many highly visible and important awards with relatively few controversies. The merit review system has been the major reason for the high quality of the activities selected for support, and it has served to discourage the use of inappropriate or parochial considerations in the allocation of NSF’s research finding. Merit review is not perfect, but no clearly superior method of selecting research and research-related projects for support has been discovered after many years of experience here and abroad.

Although controversial decisions have been relatively rare, they have revealed problems in NSB and NSF policies and procedures that could be avoided. When such problems occur or are believed to occur, they under-

mine the confidence in the merit review system of the research community, research institutions that compete or hope to compete for major awards in a fair process, and Congress. So far, the success of the merit review system has helped insulate NSF and NSB decisionmaking on major awards from congressional intervention. If confidence in the system is not maintained, the temptation for research institutions to try to have Congress preempt NSF decision making will increase, and to the extent that legislative involvement replaces merit review with political considerations in project selection, the quality of the nation’s research system may be negatively affected.

The panel recommends a number of changes to strengthen or improve the planning, review and selection, and subsequent renewal of major awards. Detailed recommendations are contained in various chapters of the report, but the key points follow:

The “rules of the game” (i.e., the criteria, procedures, and roles of participants in the merit review process) must be absolutely clear in advance.

In some cases, the criteria or requirements needed to meet them have not been clear or were seemingly redefined during the review process. Although too much detail in specifying criteria might limit the flexibility to respond to innovative proposals, we concluded that to increase procedural fairness, NSB and NSF should be more precise about the criteria and review process to be used. In particular, the primary technical criteria as distinct from other criteria to be considered in the merit review process should be identified in advance in each case.

The panel recommends stronger planning efforts that would help contribute to clearer criteria (Recommendation 1). The panel also recommends that NSF concentrate more effort in designing a better-understood review process for each major award (Recommendation 8).

Technical merit must be the primary consideration in making awards.

The panel strongly supports the primacy of technical merit in the selection of major projects (Recommendation 3), and it endorses the use of a two-phase review process that would clearly indicate the ranking of projects on technical merit before other merit factors are considered (see next section).

Technical merit must be paramount to maximize the likelihood that the project will achieve its substantive research goals. Other criteria of merit should also be given due consideration in selecting the overall winner or winners, but any project receiving an award should rank among the very highest in technical quality. That should be made clear to all reviewers and decisionmakers, along with a sense of the nature and relative priority of each of the criteria.

NSF and NSB must be clearer in each case about the relative priority of the various criteria used, especially of the technical relative to the nontechnical criteria. Otherwise, the weightings of criteria are implicit and can shift continually at the discretion of individual reviewers and program staff.

The review process must be structured so that the roles of peer reviewers and staff in evaluating and recommending proposals are clearly understood, and trade-offs among technical and other criteria are clearly explained, at each subsequent level of decision making.

Currently, the summary rating and ranking of proposals by staff at various decision points does not always distinguish peer review from staff judgments. Although staff should make their best case for a recommended decision, the NSF director and the NSB should always know the results of the peer review.

The two-phase review process, properly documented, would make it easier to implement this objective (Recommendation 6). This two-phase process would facilitate the preparation of a summary document that explains the rationale for the decision, including the treatment of peer review results and the trade-offs made between technical and nontechnical criteria in reaching the final decision (see next section).

There should be a public document explaining the results of the review and the rationale for the final decision by the NSB.

NSB minutes rarely record the basis for a major award decision, and no public document of explanation for the final decision is prepared or disseminated. The lack of such documentation leads to public confusion and controversy that could be avoided.

The panel recommends a short, carefully prepared memorandum that summarizes the results of each stage of the merit review process and

outlines the rationale for choosing a winning proposal (Recommendation 9). Such memoranda would increase public understanding of major award decisions and therefore enhance public confidence in the system that produces them.

Decisions to solicit proposals for very large major awards should take into account their impact on NSF’s overall program as well as on the particular research field involved, and they should be contingent on the realization of expected funds and technological progress.

Careful front-end planning, combined with broad consultation with affected research communities and constant evaluation of priorities at each decision point, must be a part of the process of soliciting and reviewing proposals for a very large major award. Solicitations for awards that have serious long-range budget implications must be based on a broader range of considerations than in the past. The priority within a given field should be clearly established and compared with the overall priorities of NSF across fields. After initial approval of a large project, contingency plans for possible unfavorable program or budget developments should be made for each project and updated annually. The potential impact on NSF priorities if there are unrealized budgetary expectations or unexpected technological problems or opportunities should be carefully reviewed at each decision point. In this way, NSF and NSB would avoid letting a series of small decisions in the development of a major project result in a project that no longer matches the agency’s overall program priorities or budget.

The panel calls for stronger planning efforts, including contingency plans for lower funding levels than expected (Recommendation l), based in part on a broader range of input from research communities affected directly and indirectly by a major project (Recommendation 2). NSB should also put more emphasis on its long-range planning and priority-setting activities (Recommendation 7) and should periodically reconsider the contribution of every project to agency priorities as part of a more systematic project renewal process (Recommendation 10).

The panel understands that its recommendations cannot guarantee a perfect result or prevent individuals and institutions who are denied awards from complaining about the system. This is especially true of awards for large, one-of-a-kind national facilities that must satisfy many expectations. We believe that the changes recommended in this report will result in a fairer and more understandable process and will increase confidence in, and support by, fair-minded participants and interested groups.

Recommendation 1: Justification for Major Project Awards

The NSB should ensure that the large-scale research-related projects that result in major awards are well justified and planned—that is, each is (a) scientifically justified, (b) technically feasible, (c) designed to enhance other activities already in place to achieve the proposed project’s goals, (d) of high national priority, and (e) the subject of careful contingency planning.

Recommendation 2: Involvement and Support of the Research Community in Planning

The NSB and NSF should make stronger efforts to see that the basis for initiating large-scale activities is well explained, understood, and accepted to the extent possible by affected research communities. NSB and NSF should take steps to ensure broader consultation with relevant communities beyond those benefiting directly from a major project award, including educational, governmental, and industrial organizations and institutions.

Recommendation 3: Primacy of Technical Merit Criteria

The NSB and NSF should continue to make technical excellence the primary criterion in evaluating the merit of proposals for major awards. To ensure that research funding is used most effectively, no major award should ever be made to a project that is not of very high technical merit. Additional criteria should be used only to choose the best overall proposal from among those whose technical merit is among the most highly rated.

Recommendation 4: Human Resource Development and Equal Opportunity as a Criterion

The contribution of every major award proposal to overall human resource development should be emphasized. The number of students to be involved—and the inclusion of minorities and women at all levels, from students to senior investigators and project managers-are important components of human resource development and equal opportunity. They should receive more explicit attention in the review process.

Recommendation 5: Cost Sharing as a Criterion

Cost sharing should be used only to demonstrate commitment to the project’s goals and never simply to extend NSF funds. Where cost sharing is required, NSF should spell out its expectations in the solicitation announcement. The amount of credit for cost sharing for purposes of

evaluating proposals should be stated clearly and capped at a reasonable level. Due account should be taken of the likelihood that cost-sharing commitments will in fact be met in the out years.

Recommendation 6: A Two-Phase Merit Review Process

For major awards, the peer review part of the merit review process should be conducted in two phases. The first phase would be a strictly technical review; to help assure the primacy of technical merit, only those proposals judged to be technically superior would be forwarded to the second phase for any further consideration. In the second phase, the additional merit criteria would be weighed and balanced with the technical criteria by a more broadly constituted group of reviewers. This second-phase panel would recommend the proposal (or proposals) best meeting the full set of criteria. If the proposal judged to have the highest merit overall is not the one ranked highest in the first phase of review for technical merit, the second-phase panel must explain its recommendation fully. If the top-ranked technical proposal is subsequently not recommended by NSF staff, the chair of the first-phase panel or another member of that panel should present the case for it at the NSB level.

Recommendation 7: Reorienting the NSB Workload

NSB should manage its proposal review workload to ensure that adequate time is left for its most important activities of broad policy direction, long-range planning, and program oversight. That could be accomplished by using its exemption authority more frequently, by raising the delegation threshold, or both.

Recommendation 8: Planning the Review Process and Criteria

NSF and NSB should further strengthen their effort to implement a review process for each major award that (a) imposes a reasonable schedule, (b) identifies the appropriate selection criteria and their relative priority, (c) uses the two-phase review process, (d) selects appropriate reviewers to address each criterion at each stage, and (e) is assisted by a central office for review of major projects that ensures quality and consistency based on extensive experience with such complex project reviews.

Recommendation 9: More and Better Public Documentation of Award Decisions

The review and award process should be fully documented and the results made more accessible than is standard or necessary for traditional individual investigator proposals. This process includes such documentation as site visit and panel reports, and the staff-prepared

director’s memorandum to the NSB summarizing the review results and recommending the awards. In particular, as recommended above, any decision to pass over the proposal rated highest technically (Phase 1) or to recommend a proposal other than the one selected in Phase 2 of the merit review process must be fully explained, and relevant documents should be publicly available.

Recommendation 10: More Recompetitions

The initial planning of every major award should specify the conditions for renewing, recompeting, or terminating the project. As a general rule, each project (or perhaps, in the case of large national facilities, its management) should be openly recompeted within a time period appropriate to the nature of the activity. Such periodic recompetitions should be preceded by an assessment of whether such an activity, however successful, is still needed or whether the funds would be better used in research areas of higher priority or for other mechanisms (e,g., grants to individual investigators instead of a research center, or a program of university instrumentation awards in place of a central national facility).

This page intentionally left blank.