Major Award Decisionmaking at the National Science Foundation (1994)

Projects receiving major awards are usually long-term activities supported by multiyear continuing grants or cooperative agreements. These activities require continuing oversight by the National Science Foundation (NSF) and its programmatic advisory committees. Periodic decisions must also be made on whether or not to renew support when the current grant or cooperative agreement expires, which occurs at least every five years. The ongoing evaluation process typically involves annual reports from the project, visiting committees, and site visit evaluations by NSF staff and advisors. This may lead to conditions being placed in the grant or contract renewal that require programmatic or managerial changes. This ongoing evaluation process occasionally leads to terminations or opening of a renewal award to competition.

Several internal and external postaudit processes are in place to ensure that NSF policies and procedures for handling proposals and making awards, are followed. These processes include in their scope major awards, although they are not treated separately or specially. For example, major awards are reviewed as part of the committee-of-visitors evaluation of each NSF program that takes place on a three-year cycle. Also, the director makes an annual report to the National Science Board (NSB) on the performance of the peer review process. The Office of the Inspector General audits a sample of awards each year to determine the level of compliance with official policies and procedures. From time to time, the General Accounting Office audits a particular decision (e.g., the Earthquake Engineering Research

Center [GAO, 1987] or the home basing of oceanographic research vessels [GAO, 1982]).

Many of the major projects, involving most of the major award funding, come up for noncompetitive renewals at the end of each grant or contract period (usually three to five years) and are expected to be continued if their performance has been satisfactory. These include both the large national facilities managed by consortia of research institutions and the national user facilities managed by individual universities (e.g., the National Optical Astronomy Observatories at Kitt Peak, Arizona, and near Cierra Tololo, Chile; National Radio Astronomy Observatory at Green Bank, West Virginia; National Astronomy and Ionosphere Center's radio/radar telescope at Arecibo, Puerto Rico; National Center for Atmospheric Research; the Incorporated Research Institutions for Seismology's (IRIS) global seismometer network; oceanographic centers, vessels, and other facilities; the Cornell Electron Storage Ring and various nuclear physics facilities; and Supercomputer Centers. In addition, the following new facilities under construction will be managed on an open-ended basis (no sunset period specified): the GEMINI telescopes; the new Green Bank radio telescope; the National High Magnetic Field Laboratory (NHMFL); and the Laser Interferometer Gravitational Wave Observatory (LIGO).

Decisions must be made periodically on what to do when the current grant or cooperative agreement for a major project expires. At the end of the specified grant period, NSF should make several decisions:

1. Is this still a worthwhile activity or has it become obsolete in the face of scientific and technological advances?

In the 1970s, for example, NSF withdrew support from a number of on-campus nuclear reactors and accelerators, converting some of the projects to research groups using national nuclear and high-energy physics facilities. In the 1980s, NSF superseded its

support of campus computer centers with regional supercomputer centers that can be accessed remotely by all researchers. As supercomputers become cheap enough for every campus to afford, supercomputer centers may in turn become obsolete.

To determine whether an activity is still worthwhile, NSF must evaluate program relevance and priority. This aspect of major project decisionmaking is not well developed (it is a continuation of the planning process described in Chapter 2, which our recommendations in that chapter would strengthen and make more explicit), compared to NSF's more elaborate procedures for deciding whether or not a particular facility or center is worth continuing or should be competed again.

2. If the activity is still deemed worthwhile, is the current grantee doing a good job that merits continuation or should the award be opened to competition?

This question is addressed primarily by the usual proposal review policies and procedures of the NSF and NSB. This process works reasonably well, although we make recommendations in Chapters 3 and 4 to strengthen it. Since most projects can show evidence of being productive and still worthwhile, they can justify renewal. The NSF and NSB should have procedures in place to ensure, therefore, that a facility, center, or other major project is being operated by the best possible grantee.

Recently, NSF has begun to emphasize such procedures in the conditions it sets for renewing or recompeting awards. The Engineering Research Center (ERC) awards, for example, have been made subject to elaborate renewal procedures every several years and have an absolute sunset provision of 11 years, after which a center is on its own or must recompete on an equal basis with new proposers. Science and Technology Center's (STCs) are subject to a similar procedure.

Sunset provisions have been built into other awards recently. The last time the Ocean Drilling Program (ODP) was renewed, for

example, the NSF and NSB reviewed and approved a 10-year plan for the period beginning in FY 1994; approved a five-year contract using the current drill ship; called for review and renegotiation of the contract after 1998 to accommodate a new or additional drill ship if needed; and stated NSF's "intent to terminate the ODP by the end of FY 2003" (NSB, 1992).

Unique national user facilities operated at and by a particular institution constitute a class of major awards that has been especially difficult to compete and recompete. They may provide a great benefit to the host institution and community (or loss if another place wins the recompetition), involve sunk costs in the existing facility that may be lost if it is terminated, and pose significant transition costs for the national user community while the new facility gets off the ground. Nevertheless, NSF has begun to recompete some of the university-based national facilities previously considered open-ended, including the National Nanofabrication Users Facility and the NHMFL. It is planning to recompete the Arecibo radar/radio telescope facility operated by Cornell.

In 1986 an NSF staff task force studied policies and procedures for terminating programs and major projects. They documented examples of successful terminations but also identified factors that tend to impede termination (e.g., the inherent interest of the peer review system, advisory committees, and program officers in identifying expansion areas and new opportunities; the development of constituencies for established activities and consequent loss of interest in other options or modes of program activity). The task force recommended a comprehensive but flexible system of sunset review of all major activities and the articulation of termination plans and contingencies as part of the initial planning (cited in NSB, 1988a:32).

In 1988 an NSB committee on centers and individual investigator awards addressed program termination as a possible way to ensure funding new people and activities even with steady or declining budgets. The committee's guidelines for orderly termination of projects no longer needed were based on competitive proposal review

to ensure that the most meritorious projects were supported (NSB, 1988a).

In 1991 the NSB considered adopting a time limit on all continuing activities (NSB, 1991b:2). Although the sense of the meeting was that major projects should be reconsidered every 10 years, NSB stopped short of setting a specific number: "Automatically recompeting a major center or research facility every five years does not appear reasonable; however, renewal proposals undergo a rigorous peer review, typically including a site visit. Continuity of support is needed for at least 10 years, provided the facility or center is performing in a satisfactory manner." NSB also decided that "when recommendations for major projects are presented to the Director's Action Review Board for waiver or explicit consideration, a statement regarding plans for the end of the grant period will be included. Further, changes in the renewal plan can be made as appropriate; however, any deviation from the renewal plan will be brought to the attention of the appropriate NSB committee." Currently there is no strict sunset requirement, although there is now a sense that major projects should be reconsidered at preset intervals and at least every 10 years.

Finally, as described in Chapter 3, the director recently approved a new program initiation protocol that requires among other things that the expected duration of a new program be specified (NSF, 1993a). The protocol also calls for a monitoring and evaluation plan to determine when an activity should be discontinued because it is no longer effective or needed.

The panel believes that periodic competition results in the highest-quality proposals and that grantees will perform better knowing that eventually they will have to defend their stewardship of a major center or facility. This benefit justifies the extra costs of a periodic competition. This recognition should be built into the solicitation of major awards whenever possible. This might extend to the management of large national research facilities, although it may be very difficult to find a competitor when the project is managed by a national consortium of universities engaged in the area of research, such as the University Consortium for Atmospheric Research (UCAR) and the Associated Universities for Research in Astronomy (AURA), or a facility is located at a single university and cannot be moved, such as the Cornell Electron Storage Ring.

In awards for unique national facilities that can be competed, the grantee should be required to agree from the beginning to cooperate fully in a transition to a new operator if the original operator should lose a recompetition. For its part, NSF should develop and NSB should adopt uniform guidelines for orderly transitions so that facility operators know what to expect should they lose a recompetition some day.

Since an existing facility tends to have a natural advantage over a proposal on paper—the existing facility is operational, has staff and a track record, and sunk costs have been paid already—NSF should make every effort to create a level playing field. In establishing or relocating major facilities, however, it is very difficult accurately to estimate transition costs and the loss of momentum in ongoing

activities. This is especially true in rapidly moving fields, which are often the ones most likely to need new investments and institutions. The problem of underestimating these costs must be balanced against tilting the competition toward the incumbent by overestimating transition costs. This means that in recompetitions, the impacts on users and transition costs should be fully considered and included in the budget plan, especially when it will take time for a new facility to become operational. An agreement also should be worked out with the Office of Management and Budget (OMB) and the appropriations committees as to how to handle budgeted transition costs if the incumbent grantee wins the recompetition.

The problem is that if the transition costs go to the successful grantee, the user community and NSF benefit from renewing the incumbent, which may bias the competition. On the other hand, if realistic transition costs are not taken into account, the bias will be against the incumbent. Given that the existing center or facility has a natural advantage, the purpose of a recompetition should be to seek better proposals that would justify the extra costs of relocation.

The adequacy of support should also be carefully assessed as part of the periodic review of the continuing programmatic need for a major project, which we recommend should take place before considering whether or not to renew or recompete any award. There is a natural tendency in launching new centers programs, for example, to respond to budget pressures by funding each center at lower levels than expected in order to get as many centers started as possible. This happened with the ERC and STC programs. In programs with multiple centers or regional facilities, therefore, NSF should periodically assess the adequacy of funding of each center. Despite pressures to keep every center going, NSF should make adequate funding of each center it sponsors a priority even if it has to cut the overall number of centers.

Maintenance and upgrading are also often the first to go when funding is tight. In the case of large-scale physical facilities, therefore, NSF should take care to ensure adequate maintenance and upgrading. Even with significant real budget growth in the late 1980s, existing facilities did not always receive adequate funding for

maintenance and incremental upgrades. The usability of the Very Large Array, for example, had seriously declined. In response to this and similar problems with other astronomy facilities, the Astronomy and Astrophysics Survey Committee of the National Research Council made ''restoring the infrastructure'' of existing equipment its highest priority (NRC, 1991).