Major Award Decisionmaking at the National Science Foundation (1994)

Awards for major projects pose some issues that are different from, and more complicated than, those involved in traditional research project grants to individuals and small groups. Major awards are much more expensive and longer term than individual research grants. Because of their size and duration, they promise to have significant impacts on the way science is funded and conducted in their field of research. Since the budget of the National Science Foundation (NSF) cannot meet all the needs of the many fields of science and engineering, each major award entails a critical initial investment decision and a subsequent long-term "mortgage" on NSF funds. If these issues are addressed fully in the planning phases, even before proposals are solicited, many problems in the review and award phase of a major project can be avoided or corrected.

In assessing NSF's planning of large projects requiring major awards, the panel identified certain essential features of an ideal planning process. Planning for major research projects should be an integral part of overall planning and setting of priorities at NSF. Priorities should be based primarily on research opportunities. Major projects should be weighed against individual research projects and other modes of research support in deciding on the best overall program. Plans for major projects should be extensively reviewed and widely discussed with advisory committees and the research

community before funding is approved. Funding decisions should be based on the best estimate of operating and maintenance costs over the expected lifetime of the project, as well as initial setup or construction costs. The conditions under which the funding award would be renewed, recompeted, or terminated should be considered in the planning process and clarified up front, in case the project did not go as planned or the anticipated budget resources were not realized.

NSF has a number of arrangements for obtaining advice from, and consulting with, the science and engineering research communities about its programs and projects both large and small. Taken together, they form a continuous, decentralized, and open planning process that may be “driven by a scientific breakthrough, the availability of a new technology, national or international concerns, or simply the existence of a new idea” (NSF, 1990a:3). NSF also participates in the annual federal budget process, which involves top-level decisionmaking across research fields and agencies. Although the budget process is an annual exercise, the resulting decisions may have much longer-term implications.

1. Advisory Committees: NSF has had 96 formally chartered advisory committees with nearly 6,500 members from the science, engineering, and education communities. Those committees include standing panels that programs in some directorates use to review proposals and, until recently, programmatic advisory committees to many of the program divisions.

   Most divisional advisory committees represented a particular discipline or scientific field, although some were more broadly based to advise on interdisciplinary programs. They met once or twice a year in open meetings. NSF consulted the advisory committees in the development of its program plans and priorities, and involved them

in assessments of current activities. Occasionally, they reviewed proposals for major awards (the Materials Research Advisory Committee, for example, reviewed the National High Magnetic Field Laboratory [NHMFL] proposals).

In response to the recent executive order directing each federal agency to cut advisory committees by a third, NSF is eliminating 34 of its advisory committees. That is being accomplished by eliminating division-level committees and establishing advisory committees to each of the six research directorates¹. This drastic reduction in programmatic advisory committees will allow NSF to retain most of the proposal review panels, although some of them may be consolidated at the directorate level. It will also affect the nature of advisory input on program issues from the scientific and engineering communities, and will reduce interactions with the disciplines, although it may promote interdisciplinary planning.

2. Quarterly Reviews: The NSF director and senior staff meet with each directorate four times a year to review overall program activities, budgets, and management. Division (now directorate) advisory committee members and other outside experts are usually invited to participate. The round of reviews held each spring provides part of the input into the long-term planning session of the National Science Board (NSB) in June. According to NSF, these sessions "often provide the first airing of concepts which later emerge as scientific initiatives or new programs," including those destined to be major awards (NSF, 1990a:5).

3. Ad Hoc Task Groups: NSF may form an ad hoc advisory group or interdirectorate staff working group to assess new program ideas, whether they involve large or small projects, and to consider now they might be implemented. NSF's first large projects—the radio telescopes at Greenbank, West Virginia, and optical telescopes

at Kitt Peak, Arizona—grew out of discussions at scientific conferences funded in the early 1950s by NSF (England, 1982:280–281). In more recent examples, an NSF-appointed ad hoc advisory panel proposed and developed the specifications for a next-generation national high magnetic field laboratory, which was awarded in 1990 (NSF, 1988a). Another NSF-appointed panel on large-scale computing in science and engineering called for increased supercomputer access by academic researchers (NSF, 1982), and a staff working group developed an action plan that included what became the supercomputer centers program in 1984 (NSF, 1983).

4. Professional Societies: Representatives of scientific and engineering societies and associations participate regularly in advisory committee and NSB meetings. The NSF director and other senior staff meet with officials of such groups and with other groups of higher education officials, industrial and federal laboratory directors, and foundation heads. NSF staff also stay informed about scientific developments through participation in activities of professional societies.

5. National Research Council: NSF often asks the National Research Council or one of its governing academies (National Academy of Sciences, National Academy of Engineering, Institute of Medicine) for scientific, policy, and programmatic advice. These reports can have a major influence on program development and decisions on priorities. National Academy of Engineering reports helped shape the Engineering Research Centers program, for example. Other examples include the "decade studies" of astronomy and physics that have been influential in setting priorities among major NSF-supported projects in those fields, particularly in establishing the categorization of activities among which priorities are set.

In addition to the external advisory mechanisms listed above, the NSB, which has statutory responsibility for setting NSF policies and approving its programs (and for reviewing and approving the major individual awards addressed in this report), has members who

represent many of the scientific and engineering fields supported by NSF. The NSB assesses current and planned NSF activities through its standing committees on Programs and Plans and on Education and Human Resources—and through special task forces, committees, and commissions. Finally, one-third of the professional program staff of NSF consists of visiting scientists and engineers on one-to three-year assignments, who bring another source of knowledge about current research opportunities and needs in their field. Rotation also helps bring in new ideas and perspectives.

The ongoing decentralized activities described above result in many interesting suggestions, some of them for major large-scale projects. In some cases, these activities yield broad and clear consensus on priorities among major projects in a field. For example, the decade studies have helped forge consensus on needs and priorities in astronomy. Another example is the traditionally ''small-science'' earth sciences community. After a series of planning discussions over several years, that community agreed that a large-scale coordinated program had become necessary to make progress. The result was the Continental Lithosphere program, which included small-grant research, coordinated field projects, continental drilling projects, and a global network of seismographs.

Not all ideas that turn into major awards originate from the bottom up. The Engineering Research Centers initiative, for example, came out of discussions between NSF officials and the Science Advisor to the President. It was seen as a response to the declining economic competitiveness of the United States as well as a way to meet an engineering research and education need; that initiative helped justify a doubling of the NSF budget at the same time that it tried to help reform academic engineering education by introducing more experience in interdisciplinary work and team efforts.

Although NSF's internal planning and programming are continuous and long-term, it is funded only one year at a time. Decisions about which existing activities will be continued, expanded, scaled back, or terminated and which new initiatives will be funded for the first time in the coming fiscal year have to be made with some uncertainty about the current year's budget and greater uncertainty about future budget levels (the FY 1993 budget process is described in Box 2-1). Out-year adjustments in the budget category for traditional individual investigator grants are relatively easy to make because the grants average 2.5 years in length; thus, about 40 percent of that part of the NSF budget becomes available for reprogramming each year. Major projects require longer-term funding commitments that cannot be reduced as easily if the NSF budget is cut or fails to grow as much as anticipated in future years. As a result, increased funding for individual investigator grants tends to be squeezed out first unless there is careful contingency planning.

In addition to the long-range planning exercises and annual budget process described above, the NSB has special review and approval procedures for most if not all activities expected to result in major awards. According to policies adopted by the NSB in January 1979, the NSF directorates are supposed to submit for approval by the director and the NSB "project development plans" for "big science" initiatives (NSB, 1979). Big science projects are defined by NSB as those that have certain characteristics:

• large-scale commitment of financial resources;

• investment of capital in facilities and major equipment;

• duration of several years or more; and

• continuing expenditures for maintenance, replacement, operating costs, and research.

According to the NSB, project development plans for big science projects are supposed to document the following:

• scientific need;

• views of the appropriate advisory group concerning the priority of the project; its effect on the balance and concentration of “big science versus little science” within the field, under varying resource assumptions (including essentially level budgets); and the opportunities that would be forgone by undertaking or not undertaking it;

• estimates of all initial and out-year costs;

• principal management, procurement, and legal considerations;

• origin and periodicity of management and fiscal reports, and timing and other considerations for evaluating the project;

• identification of principal phases or milestones; and

• arrangements to update plans at least annually.

All such big science projects proposed for a given budget year are to be reviewed at the same time so that NSB can set priorities (based on prior review by, and recommendations of, the director). Once approved, project development plans are supposed to be updated annually and revised whenever a significant shift in the terms or funding of the project is being considered. The approved plan is also the basis for dealing with the Office of Management and Budget (OMB), Office of Science and Technology Policy (OSTP), Congress, and other external organizations and groups.

According to the NSB policy document, the size of a project must be considered both in absolute (cost and complexity) and in relative (the share of resources of a particular field and of NSF's overall budget) terms. The policy does not set a limit on the share of a research program's budget that can go to large-scale projects, because the appropriate balance varies across fields and is subject to periodic review by advisory groups, staff, and the NSB. If funding is too high to be accommodated within the appropriate disciplinary budget, the director and NSB must decide whether or not the project can be accommodated within anticipated NSF budget levels or must become a special item justified above and beyond anticipated NSF budgets.

In 1983, the NSB planning and budget committees examined NSF's capital facilities planning and budgeting procedures (NSB, 1983c:9). That review concluded that NSB's 1979 policies on initiating large-scale projects and its regular long-range planning procedures were adequate. That stance began to change in 1988, when a 10-year projection indicated a need for $1.6 billion in capital to repair and replace aging NSF-supported research facilities and enable them to take advantage of new research opportunities (NSB, 1988b). That amount was well beyond the level that could be met even by the planned doubling of the NSF budget. The projection indicated a need to treat major capital items of $10 million or more

differently in order to improve planning, decide on priorities, decide on funding alternatives, and justify them to OMB and Congress.

By FY 1990, NSF's support for major research facilities, (e.g., astronomical observatories, supercomputing centers, high-energy and nuclear physics facilities, oceanographic research ships, and atmospheric research facilities) totaled almost $400 million, 25 percent of its appropriation for Research and Related Activities. About one-quarter of that was for capital costs. Three-quarters was for operations, including research. NSF estimated that support for facilities would increase to about $620 million by FY 1995 (NSF, 1990b:1).

As a result of these trends, NSF adopted a new approach to budgeting for capital facilities in the Mathematical and Physical Sciences (MPS) Directorate (NSF, 1990b:6):

The success of the capital facilities subaccount in MPS remains to be seen. It has been used to justify and gain support for very large facilities each involving more than $100 million over its lifetime (including non-NSF funding). NSF was able to start the major projects it had proposed, including the 8-meter telescopes, NHMFL, and LIGO. NSF also may succeed in establishing the account as a

revolving fund with Congress, enabling the agency to fund new projects as approved ones are completed and turned over to the research programs for operating support.

One risk of putting the capital projects in a separate account is their increased visibility to Congress. Congressional committees may direct that projects in the subaccount be fully funded even when NSF does not receive its full budget request. So far, that has not happened. In FY 1992, when it did not receive its full request for Research and Related Activities (R&RA), NSF was able to revise its operating plan by reducing the funding for ongoing construction projects from $51.5 million to $37.9 million. In FY 1993, however, when NSF received about the same budget for R&RA as in 1992, it was directed at first to protect funding for LIGO even though it wanted to scale back this activity. NSF had to negotiate with the appropriations committees to scale back LIGO funding in FY 1993 so that it did not squeeze funding for other research. Thus, whereas NSF may gain the opportunity to justify expensive new facilities projects, it also may lose the flexibility to adjust expenditures within a research field in the face of reduced funding, especially when Congress does not follow NSF's capital budget plans.

The NSF planning process is very decentralized, continuous, and open. It works well to encourage new ideas. The process works best within individual fields or disciplines (e.g., physics, astronomy, biology)—but not across them. The budget process also imposes a very short-term planning horizon, because NSF receives annual, not multiyear, funding.

Because the planning process produces more good ideas than can be funded in any given year, the NSF director—with the advice of the assistant directors—makes the key internal decisions on priorities, subject to NSB oversight and concurrence. Initiatives and priorities do not always come from the bottom up; higher-level

governmental oversight groups also affect priority decisionmaking (e.g., the Science Advisor to the President, OMB, the Federal Coordinating Council for Science, Engineering, and Technology, and congressional committees).

Although a large share of NSF's resources goes into building, maintaining, and updating large-scale facilities and instruments, capital budgeting is currently done within fields if at all. Most capital projects are in two directorates, MPS and Geosciences. MPS recently established a budget subaccount for "Major Research Equipment" that includes LIGO, the GEMINI telescopes, and the National High Magnetic Field Laboratory.

Project development plans are usually prepared for projects involving the construction of facilities (e.g., Continental Lithosphere Program (which included the Incorporated Research Institutions for Seismology [IRIS]), 1984; LIGO, 1984; NSFNET, 1987; NHMFL, 1988; 8-meter GEMINI telescopes, 1991). Project development plans have not been prepared for centers or center programs (e.g., Earthquake Engineering Research Center, the Engineering Research Centers [ERCs], or the Science and Technology Centers [STCs]). In those cases, NSB approved the next stage, the proposed solicitation document. Project development plans also are not required for ongoing campus-based research facilities primarily intended for local use (e.g., nuclear physics facilities at the University of Illinois, Michigan State, and Indiana University).

The decision to construct a major research facility, launch a program of interdisciplinary research centers, or support a large-scale coordinated research program has a strong and lasting impact on the way research is carried out in a scientific or engineering field. Also, the large cost of major projects imposes opportunity costs—the funds committed to them, usually for long periods, cannot be used for other, perhaps more productive, but less visible, activities. These opportunity costs are felt strongly within the affected research area, and in this era of very constrained federal budgets, they may affect other areas as well.

The case studies provide illustrations of these effects. The decision to establish a national Earthquake Engineering Research Center had a major impact on the mode of earthquake engineering research supported by NSF, because it shifted one-third of the approximately $15 million a year in NSF's earthquake hazard mitigation program from individual investigator grants to the center mechanism. LIGO, the new $200 million instrument for the specific research area of gravitational physics, is a relatively costly project that may have an impact on the funding of other fields as well as physics if NSF's research budget does not increase as much as planned.

The NSB recognized the critical importance of the initial decision to undertake a major project and since 1979 has required submission of a formal project development plan for the construction of large research facilities. This requirement has not been followed in all major award cases, however, although the need for it today is greater than ever before, for several reasons:

• Scientific opportunities are growing faster than NSF resources, which means that costly new initiatives need to be subjected to special scrutiny to ensure cost-effectiveness in comparison with alternatives such as a program of small grants.

• The scale of research has increased in size and cost, bringing big science into additional research fields, which means that the interrelationships among modes of research within a field and with closely related fields should be carefully worked out to ensure balance.

• The more needed and better planned the project is, the easier it will be to solicit good proposals, develop appropriate review criteria and procedures, and identify the best proposal to carry out the goals of the project.

The panel concluded that diligent use of the project development plan mechanism for all major awards, not just for facility construction projects involving construction of facilities, would help ensure that

such projects are well justified scientifically, of high priority, well designed and technically feasible, and complementary to the rest of NSF's activities in that and related fields. Wider use of project development plans would also ensure that appropriate analyses of options under different budgetary and technological scenarios have been conducted.

The NSB should ensure that the large-scale research-related projects that result in major awards are well justified and planned—that is, each is (a) scientifically justified, (b) technically feasible, (c) designed to enhance other activities already in place to achieve the proposed project's goals, (d) of high national priority, and (e) the subject of careful contingency planning.

These factors should be fully considered before proposals are invited and issues of appropriate review procedures and criteria are addressed:

1. Scientific Justification: A decision to undertake a major project must promise important scientific contributions, whether directly through support of large-scale or interdisciplinary research that could not be done otherwise or indirectly by providing access to state-of-the-art facilities for individual researchers or research groups.

2. Technical Feasibility: Major projects must be technically ''doable'' as well as scientifically desirable. In the case of state-of-the-art facilities, research and development on new instruments should be advanced enough to justify full-scale deployment. In the case of ERCs and STCs, there should be evidence that they will be able to produce research that will be useful economically.

3. Complementarity: Major projects should be part of, and should enhance, a balanced program of mechanisms for supporting productive science and engineering research and education in a field, which work together to maximize scientific progress in that field.

There is no a priori way to determine what share of NSF's resources should go to individual investigator and small-group research projects; university-based, regional, or national facilities; centers; or large research groups—the balance varies by field and within fields over time. Major projects reduce NSF's overall programmatic and budgetary flexibility. The burden of proof, therefore, should be on a potential major award to demonstrate that it might add more value than an equally expensive program of individual projects or other, more flexible research mechanisms. Also, NSF should take steps to ensure that alternative views are heard in the final decisionmaking. That might involve building in an advocacy process and would help ensure that the best case is made, for example, for an equivalent program of individual investigator or small-group grants, or for instrumentation grants to many or several colleges and universities rather than for the establishment of a single national facility.

4. Programmatic Priority: In NSF's open and continuous planning process, more good ideas come up than can be funded. NSF leadership and the NSB must make decisions about allocating the approximately 40 percent of NSF's budget base that is available each year (the rest is committed to multiyear projects), as well as any overall budget increases or decreases. These decisions are difficult because they inevitably favor one research area over another. The more distant the areas are from each other, the less relevant are purely technical considerations in making the appropriate choices.

   According to NSB policy, all large projects subject to the requirement for a project development plan are supposed to be reviewed at the same time so NSB may establish priorities among them. This procedure takes place annually at the June NSB meeting, which is devoted to long-range planning, but the full set of project development plans is not always available to provide detailed information on all factors that must be taken into account in setting priorities (as intended by the project development plan policy).

5. Contingency Planning: The broader implications and long-term budgetary impact of major awards should be carefully considered, including a worst-case analysis, and weighed against the project's need and priority, as well as its risk and potential payoffs.

Large projects tend to achieve tremendous momentum, scientifically and politically. If the NSF research budget fails to grow or shrinks in real terms, large projects may squeeze out other less visible, but perhaps equally valuable, programs. This analysis should be ongoing, with specific trigger points identified when a project should be postponed, scaled back, stopped, or stretched out in time.

NSF and NSB currently address all these planning criteria in one way or another and to some degree. The panel would like to see them addressed more systematically by adoption of the format for project development plans for all major awards (not just those involving construction). In some ways, the criteria in Recommendation 1 go beyond those currently required in a project development plan, especially in seeking a broader context for decisions and contingency plans. A major project not only should be justified as part of an overall plan within its field of research, but also should be considered by NSB along with all other major awards vying for funding during that budget cycle and in longer-range plans. The consequences of smaller-than-expected appropriations should be very seriously considered, and contingency plans made and communicated to the relevant public, including congressional committees, as early as possible. The contingency plans should include explicit trigger points for reconsideration of projects that develop technical or budget problems (or opportunities that justify additional investments).

The additional emphasis on national priorities and contingency planning requires consultation with a wider range of research interests than before. Increased competition for limited resources means that unanimity in a particular field (even if it can be achieved) is not a sufficient condition for going ahead with a project. A broader perspective on research priorities is required than before. The NSB itself should place greater emphasis on its responsibilities for planning and setting priorities (see Recommendation 7) and should look for a wider base of consultation early in the development of major projects.

Technical feasibility is another requirement highlighted here. In the 1979 NSB policy on project development plans, it appeared as a "probability of success" subfactor under the "scientific need" requirement. Facilities and centers pose the issue of technical

feasibility more sharply than do individual investigator grants. Before a telescope, accelerator, or other facility can help researchers, it must be built to very exacting specifications. Meeting the specifications to do forefront research involves expertise and judgment about engineering feasibility and accuracy of cost estimates.

This expertise and judgment is different from that necessary to determine the scientific promise of individual investigator grants. For example, an issue in the LIGO project, in addition to its affordability at a time of slowly-growing budgets, was its technical readiness. Achieving the goals of the project involves a degree of instrument sensitivity several orders of magnitude beyond that previously reached. Cost estimates had already doubled several times during the planning of the project. Technical uncertainties and increasing cost made it an especially risky decision.

Center programs pose a different technical feasibility issue. In most cases they are created to produce research that promises to be economically beneficial. Determining whether their research areas are economically relevant involves different expertise than that required to evaluate whether the research will also be of high quality.

These initial planning criteria are focused on a critical decision—whether or not to initiate a major project in the first place. They are also important considerations in reviewing the eventual proposals, and the review process and reviewers should be selected accordingly (although the emphasis in selecting the winning proposal shifts to technical merit). The criteria and process for selecting proposals are addressed in the next chapter.

The NSB and NSF should make stronger efforts to see that the basis for initiating large-scale activities is well explained, understood, and accepted to the extent possible by affected research communities. NSB and NSF should take steps to ensure broader consultation with relevant communities beyond those benefiting directly from a major project award, including educational, governmental, and industrial organizations and institutions.

Greater understanding of, and support for, the goals of a major project by the research community makes it more likely that high-quality proposals will be submitted, that external peer reviewers and advisory groups will understand and use the criteria appropriately, and that the final award decision will be understood and accepted, although NSB and NSF should not always wait for unanimity in the research community before proceeding. Even in the latter case, NSF should make every effort to inform the relevant affected communities about its plans.

The panel found that the "need" for a major project has not always been understood or accepted by the relevant research community. Among the case studies, for example, more groundwork was laid for the Engineering Research Centers program than for the Science and Technology Centers program. Both were initiated at higher levels, but NSF went through several planning exercises with the engineering community before deciding to approve the ERC program and solicit proposals. These exercises included the NSB (NSB, 1983a, 1983b) and early consultation with the engineering research community (NAE, 1983). Although a project development plan was not submitted per se, the draft solicitation was based on the recommendations of a committee of the National Academy of Engineering (1984). It was submitted to the NSB with a three-page concept paper from the staff.

The STC program was launched more hurriedly in January 1987 in response to a presidential speech. The only outside input was a National Academy of Sciences committee report in June 1987 on how to design the program and the process for soliciting proposals (NAS, 1987). The NSB reviewed and approved the draft solicitation announcement at its August 1987 meeting. Although the response was huge (323 proposals), there were more confusion about and lack of support for the concept than there had been for the ERC program. In this case the problem of gaining consensus in the research community was made more difficult by the breadth of areas covered by the program; virtually every field supported by NSF was eligible to apply.

Another case—the global seismology network of IRIS—illustrates the benefits of early involvement of the research community in developing a major project. In this case, NSF worked

with the relevant research community for several years in developing a balanced program for the solid-earth sciences that encompassed (1) small-project support for individual investigators and small groups; (2) large multi-investigator, interdisciplinary projects; and (3) major facilities and instrumentation for crustal drilling and seismology. The program was presented to NSB in a project development plan that included all aspects of the program—small-grant support as well as large facilities that come to NSB later for approval of individual awards. The program was implemented smoothly, and awards for several large-scale facilities were made without controversy, although the earth sciences hitherto had been primarily a small-science enterprise.

The research community is not homogeneous; it consist of many specialized, mostly discipline-based groups that have different needs and priorities. Depending on the project, it may be difficult to consult with, and gain the support of, every affected research community. Attempts to broaden the range of groups consulted also makes consensus building more difficult. The panel nevertheless concluded that it is highly desirable to involve and seek the support of the research community as much and as early as possible in the process of deciding to support a major project. Such early and continuous involvement can help ensure that such projects are scientifically justified and truly complementary to other activities in a field. Such involvement also assists in determining the priority of the project within a field. It can help ensure that the solicitation is well designed, that the external review is carried out with a better understanding of what is required, and that the final award decision is better understood and supported.

Sometimes, as in the IRIS case, NSF may be able to seek the consensus of the research community affected in deciding on new or revised programs, especially those involving major awards for facilities, centers, or other large-scale and long-term activities. NSF can and often has played a leadership role by sponsoring such consensus-building activities from time to time in each research field.

That does not mean, however, that NSF must always wait for consensus to develop in a scientific field before developing new projects and programs. The federal budget process responds to initiatives from many sources, and budget opportunities sometimes

arise on short notice. In other cases, conditions seem ripe for a new field to develop in which only a few, probably younger, researchers are engaged. In these and other cases, NSF has a leadership role to play. NSF and NSB should retain an ability to initiate and fund some activities of high risk, or innovation in science will suffer. Even when they take a leadership role, however, it is important to bring along as wide a community as possible through active consensus-building efforts.

Regardless of where the idea for a major project originates, NSF should develop and communicate a well-though-out rationale for the initiative to the affected scientific community and should involve it as much as possible in planning the project. The project development plan procedure calls for NSF to document the scientific need for the project and the views of appropriate advisory groups on its priority; its effect on the balance of research mechanisms in the field; and the opportunity costs of undertaking it.

Major projects almost always have broader effects than individual investigator awards—on colleges and universities, industry, and local economies—and the NSB should ensure that plans are communicated to these constituencies and that their views are given an opportunity to be heard and seriously considered even if they are not followed in the end. This kind of consultation is especially necessary when NSF undertakes projects aimed at applications and national goals or those involving nonfederal cost sharing, as in the ERC and STC programs.

Generally, NSB should be very wary of approving projects in which community input has not been considered and documented. That does not mean NSF must wait for universal consensus within a scientific field before moving ahead; indeed, initiative by NSF leadership and NSB may be appropriate. However, if a new initiative is undertaken without substantial consensus, the rationale for the project and reasons for going against criticism should be explained clearly in public documentation (see Recommendation 9).