Major Award Decisionmaking at the National Science Foundation (1994)

This chapter addresses the three final aspects of the proposal review and award phase for major projects: the role and procedures of the National Science Board (NSB), the requirements for designing the review process and issuing the solicitation announcement, and documentation of the basis for the final decision.

If awards are very large (at least $6 million over five years or $1.5 million a year) or if they pose new programmatic or policy issues, they are subject to review and approval by the NSB. After a proposal is recommended for funding by the program director, an NSB decision package is prepared and put through a three-stage internal review process that begins about six to seven weeks before the NSB meeting (NSF, 1993b). First, a directorate review board reviews the NSB package for merit, adequacy of merit review, and completeness. Second, a new body—the Administrative Review Group (ARG)—reviews the NSB package for compliance with the administrative policies and procedures of the National Science Foundation (NSF). ARG members include the executive secretaries of the NSB committees on Programs and Plans (CPP) and on Education and Human Resources (EHR), and the NSB executive officer. They also include representatives of the Office of General Counsel (OGC), the Division of Grants and Agreements, and the Budget Division.

After ARG review and sign-off, the assistant director for the originating directorate approves the award and forwards the NSB package to a third-level review by the Director's Action Review Board (DARB) before final sign-off by the director about two weeks before the NSB meeting. DARB conducts a final review of the proposed award on behalf of the director and discusses issues likely to be raised by the NSB. DARB is chaired by the deputy director and includes several assistant directors and other staff chosen on an ad hoc basis by the deputy director.

DARB plays a very important gatekeeper role in the review process. It stands between the directorate that wants to make an award and the director. DARB recommends to the director whether or not (1) the review plan and timetable for each major project solicitation are adequate and (2) the case for the award decision has been made and documented adequately in the materials forwarded to NSB. If DARB asks hard questions about procedural and substantive issues, the burden on NSB to review every award decision is lessened, which allows it to devote more attention to long-range planning and program balance issues.

After it has cleared DARB, NSB receives the decision package, which includes

• a director's memorandum that summarizes information and issues related to the proposed award;

• the program officer's recommendation;

• the summary budget;

• a list of reviewers and review analysis (Form 7); and

• verbatim peer reviews.

The director's memorandum, which is prepared by the originating program staff, is supposed to include "objectives, alternatives, potential policy implications, precedents involved, and any other factors that could be considered nonroutine" (NSF, 1992c:III-2). It should also include a short statement of responses to the major

concerns raised by reviewers and an analysis of immediate and long-term budget implications.

Other awards "of unusual sensitivity" must must be submitted to the NSB for approval. Those involve

• special policy issues;

• the establishment of new centers, institutes, or facilities;

• potential for rapid growth in funding or special budgetary initiatives;

• research community or political sensitivity;

• a previous expression of NSB concern; and

• any other awards selected by the director or assistant directors.

NSB conducts its work through committees. Program planning and major award reviews for research and related activities are assigned to CPP (the same functions for education activities are performed by EHR). NSB met an average of seven times a year in 1990–1992, up from six a year in 1986–1989. CPP meets several times to review recommended awards during each NSB meeting, which usually last two or three days. The June meetings of NSB are reserved for long-range planning, although the heavy CPP review schedule required it to approve specific awards during the June 1992 planning meeting (Appendix B lists the awards approved at each meeting from 1986 to the present).

CPP also reviews and recommends approval to the full NSB of all proposals to start new research programs (e.g., a program of arctic social science research was approved in October 1989), project development plans for proposed new or revised projects (e.g., National Nanofabrication Users Facility in November 1992), and proposed solicitation announcements.

CPP, and NSB as a whole, must divide their attention among long-range planning, broad program oversight, and specific award approvals. The number of awards subject to NSB approval has been increasing in recent years, causing workload problems.

In FY 1992, for example, 49 of the approximately 9,000 awards made by NSF went to NSB for approval, totaling nearly $1 billion (up from 37 for $520 million in FY 1991 and 28 for $530 million in FY 1990). Of the 49, 31 were for research centers, facilities, and other large research projects (the other 18 were education projects).¹

As the number of projects coming to it has increased in the last several years, NSB has become concerned about having to spend most of its time reacting to specific project awards. It has discussed various ways to reduce the workload, including

• raising the delegation thresholds of $1.5 million a year or $6 million total;

• concentrating on approving research center programs rather than individual project awards made under the programs, or approving the first round of individual center awards but subsequently reviewing only the program as a whole (letting the staff decide on renewal awards after full merit review); and

• greater use of the authority to waive reviews for routine types of decisions, as is already done with renewals of materials research laboratories and some smaller astronomy and physics facilities (again, letting the staff decide on renewal awards after full peer review).

The NSB has endorsed the last option and has begun to make greater use of waivers during the past year.

NSB should manage its proposal review workload to ensure that adequate time is left for its most important activities of broad policy direction, long-range planning, and program oversight. That could be accomplished by using its exemption authority more frequently, or by raising the delegation threshold, or both.

The panel found that CPP spends much of its time responding to recommendations for specific awards. That leaves less time for it to examine overall program direction and balance or to assess the continued justification for major facilities and programs. Moreover, the number of proposals that CPP and NSB must review for awards is increasing. The panel concluded that this trend is impinging on NSB's most important functions—broad oversight and policy direction.

NSB should devote its energies first to planning (see Recommendation 1), second to overseeing the design of solicitations, and third to reviewing specific decisions on very large or significant awards.

The greatest increase in workload has come from the creation and expansion of the centers programs in the mid-1980s, in which individual centers receive full merit reviews every three years. The NSB could concentrate on evaluating the centers as programs, perhaps after reviewing the initial round of individual awards in a particular centers program, rather than on revisiting each center every three years. That would reduce the proposal review workload considerably. For example, 25 of the 31 research awards approved by the NSB in FY 1992 were renewals of individual centers in three programs (Engineering Research Centers [ERCs], Science and Technology Centers [STCs], and Materials Research Laboratories [MRLs]).

The NSB also could raise the delegation thresholds to concentrate on the very largest projects. Just adjusting the thresholds for inflation, since they were originally calculated in 1983 (but not adopted until 1986), would raise them to about $2 million a year or $8 million over five years.

NSF has had special procedural requirements for planning and conducting reviews of "large special projects" in its Proposal and Award Manual since 1987 (NSF, 1992a:1–7). These requirements, which must be approved by the assistant director and DARB, apply to competitions involving one or a few awards of $1 million a year or more that result from special program announcements or program solicitations.

NSB also must review and approve a summary document describing a proposed solicitation announcement, which states the goals of the project, prescribes the format for proposals, and describes the procedures and criteria that will be used to evaluate the proposal. The development and approval of solicitations are discussed below.

According to NSF review planning requirements, a plan and schedule for all significant review events must be approved by the cognizant assistant director and DARB.² The events to be planned

and scheduled include peer review deadlines, selection of panel members, panel meetings, site visits, DARB action, and NSB approvals. The plan and schedule are to be updated as necessary, and updates are to be approved according to the same procedures used for the initial plan.

Review formats are supposed to be structured according to the announced selection criteria for that specific competition and approved in advance by the cognizant assistant director. This is meant to ensure that reviewer comments address the criteria, that all criteria are covered, and that only the announced criteria are used.

Site visitors should have similar instructions, and the site visit reports should be structured so that criteria commented on in one report are also addressed in the reports on visits to other sites. If new items are added as a result of later site visits, appropriate addenda should be attached to earlier reports.

The summary review by program staff is supposed to be structured similarly to the review formats, (i.e., organized around the announced selection criteria). At every stage, the documentation of declinations should be comparable in level of detail to awards and proposals still in competition. Recommendations for awards are not supposed to be made until the review of all proposals has been completed. Finally, the review and selection process should be described in sufficient detail to show that NSF requirements were followed, including notes on telephone or other electronic communications with reviewers and proposers.

The timing and content of these procedural requirements indicate that they were responses to criticisms by the General Accounting Office (GAO) of the competition for the Earthquake Engineering Research Center (EERC). GAO found problems and inconsistencies in the way the reviews of the two leading proposals were handled,

including reviewer comments that were not linked to the stated criteria and a preliminary decision to fund one proposal before the review of the other proposal was complete (GAO, 1987: Ch.3).

In practice, the directorates have fulfilled the 1987 requirement for formal planning of the review of large special projects in various ways. In any case, the requirement is procedural rather than substantive, focusing on the timetable for reviews and site visits rather than on consideration and justification of criteria, reviewer qualifications, or types of review mechanisms to be used. Any higher-level review and justification of the review process to be used for a major award takes place in the context of preparing the solicitation document (e.g., Request for Proposal, program announcement, project solicitation), which should be approved by NSB.

NSF recently adopted new internal guidelines for initiating new projects and programs—the ''Design, Review, and Management Protocol'' (NSF, 1993a). The new protocol calls for

• a demonstrated need and explicit goals toward which progress can be measured;

• a set of clear policies to guide the review process;

• adequate budget and staff to manage the proposal review process;

• management plan for funds and personnel that specifies responsibilities, with input from all participating units;

• closer coordination among administrative units involved in the grant-making process—the offices of grants and contracts, financial management, and information systems;

• criteria for measuring progress and an assessment plan for monitoring the impact of the activity;

• mechanisms for full and timely communication with the constituent research community and NSF staff; and

• the ability to be modified or improved in light of new information on progress, or discontinued if and when the goals are reached.

The initiating unit is directed to prepare a "management package" for approval at least at the assistant director level that includes, among other items, a review management plan outlining the process, criteria, deadlines, and administrative responsibilities to be employed in the competition. If the activity is subject to NSB review and approval, the management package is now expected to be an integral part of the background material supporting the decision. Also, program announcements, solicitations, "Dear Colleague" letters, and other external communications to the research communities that go through NSF's internal clearance process must be supported by an approved management package.

In the 1970s, NSB began to require that it approve all formal announcements inviting proposals in which it was expected to decide on the eventual award (NSF, 1977). Among the other case studies, project announcements soliciting competitive proposals were approved by the NSB for the National Nanofabrication Users Facility (1977), ERCs (1984), Supercomputer Centers (1984), EERC (1985), and STCs (1987). In a related action, the NSB asked to review the site selection process and criteria for the two Laser Interferometer Gravitational Wave Observatory (LIGO) sites in October 1990.³

The project solicitations for the National High Magnetic Field Laboratory (NHMFL) and the recent recompetition of the National Nanofabrication Users Facility were not, however, formally reviewed by NSB.

Solicitation documents, although generated within the various directorates, have a fairly common format. The first "program announcement" for the ERC program, for example, began with a

description of the goal of the program as a whole, the defining characteristics of ERCs, and expected features of the centers (NSF, 1984a). These were based on a report of the National Academy of Engineering on guidelines for ERCs (NAE, 1983). The ERC program announcement also included information on who could submit proposals, deadlines, and expected award size and duration.

The announcement cited the four basic criteria used to evaluate all proposals (i.e., research performance competence; intrinsic merit of the research; utility or relevance of the research; and effect of the research on the infrastructure of science and engineering). It went on to say that "within these general criteria," consideration would be given to certain features in evaluating how well the proposed center might meet the objectives of the ERC program. These included such items as the importance of the research area addressed by the center; impact of the center on engineering education; industrial participation; cross-disciplinary nature of the center; and management plan.

ERC program announcements were modified each year. By 1989 the program announcement prescribed a more elaborate proposal format and a revised list of review criteria that subsumed the four basic criteria and included additional ones relevant to a university-based research center (NSF, 1988b:5): (1) research merit and potential impact on U.S. competitiveness; (2) strength and impact of educational programs; (3) industrial/other user participation and knowledge/technology transfer; (4) leadership and performance competence; (5) institutional environment and support; and (6) effect on the infrastructure of engineering.⁴

The proposal solicitation for the NHMFL, issued by the NSF Division of Materials Research in late 1989, described a two-stage review process. In the first stage, proposals would be evaluated by mail and/or panel reviews. In the second stage, the institutions identified as having the most meritorious proposals in the first stage would be visited by a team of experts. "On the basis of the recommendations of this site visit team, the most highly meritorious proposal will be selected . . . and transmitted to the National Science Board for its review and approval" (NSF, 1989:3). The criteria included the four general criteria (with each explained in terms of the NHMFL) and two additional criteria⁵:

• the likely effectiveness of proposed management plans, and

• the level and nature of institutional and other sector commitments.

The program solicitation for STCs was based in large part on a report of a National Academy of Sciences (NAS, 1987) panel. It outlined a more elaborate two-tier review process:

The criteria for selecting STCs were similar to those for ERCs and other center programs (e.g., intrinsic merit, competence, utility or relevance, and infrastructure impact, plus appropriateness of the center approach, management plan, educational effects, and private sector linkages and knowledge transfer arrangements), and as with the ERCs, no indication was given of their ranking.

Other project solicitations examined by the panel had a similar format: a page or two describing the project goals, and sections on who may apply, responsibilities of the principal investigator, deadlines, proposal format and content, evaluation criteria, and the award size and instrument. Most referred to the four general NSF criteria and also listed additional ones. Only a few indicated the priority of the criteria.⁶ Most described the review process in a short general paragraph that gave maximum discretion to NSF: proposals will be evaluated by a combination of peer review, panel review, and site visits; proposers may be asked for additional information; and proposals may be rejected any time after the initial peer review. The solicitations for recent multidisciplinary ERC and STC programs have announced more elaborate multistage review processes, but the

priority of criteria at each stage has not been articulated, except in the unissued 1992 version of the ERCs solicitation announcement (NSF, 1992c).

NSF and NSB should further strengthen their effort to implement a review process for each major award that (a) imposes a reasonable schedule, (b) identifies the appropriate selection criteria and their relative priority, (c) uses the two-phase review process, (d) selects appropriate reviewers to address each criterion at each stage, and (e) is assisted by a central office for review of major projects that ensures quality and consistency based on extensive experience with such complex project reviews.

1. Reasonable Schedule: The deadlines for the proposal and review process should leave adequate time for proposers to prepare proposals and for reviewers to do their jobs, and should be specified in the solicitation.

2. Criteria and Their Priority: NSF should make explicit the criteria and their priority in advance of the solicitation. The primary technical criteria that will be used in the first phase and the other criteria to be taken into account in the second phase should be identified at this time. Explicitness would also make it easier to identify appropriate reviewers to address the criteria at each stage.

3. Two-Phase Review Process: The details of the review process to be used for this award should be determined and specified in the solicitation document: how the first-phase technical review and second-phase overall review will be structured and scheduled.

4. Appropriate Reviewers: The reviewers with relevant expertise should be identified for each phase of the review process, and plans should be made to carry over some of the first-phase technical reviewers to the second-phase panel to participate in identifying the best proposal overall (and to ensure that technical quality remains the primary criterion).

5. Learning from Experience: NSF should have a central review office for major awards—an "institutional memory" that ensures consistency and learning from experience in designing and managing the review processes for major awards.

The panel found that the proposal review and selection process has been flexible and varied. This might be appropriate for individual grants, but it can be too inconsistent and easily misunderstood for major awards. The review phase is sometimes confusing and not always well understood by proposers or other observers. The design and implementation of the proposal review process for major projects should be done with special care, should follow directly from the stronger and more detailed project development planning process recommended in Chapter 2, and should be communicated fully to the research community and public.

NSF has procedures for designing an appropriate review process for new programs and projects, including major project awards, and these have recently been strengthened. The new protocol adds requirements for ensuring adequate resources to conduct the review process and for considering explicitly the conditions for discontinuing the project (see Chapter 5).

The goal of the review planning exercise is to ensure that the criteria, procedures, types of reviewers, use of site visits and review panels, involvement of the directorate advisory committee, and other aspects of the review are well thought out and appropriate. The panel strongly endorses this approach and urges NSF to implement the new policies fully. They should be closely linked to the processes for

obtaining NSB approval of the project development plan and of the program announcement or solicitation that follows the plan.

A carefully prepared solicitation based on a thorough plan for the review could go far to head off subsequent misunderstandings and conflicts when an award is made. As described in Chapter 3, the project development plan for the NHMFL explicitly called for a 50 percent match of the then-estimated project cost of $105 million, not counting capital costs of construction or renovation of a building. The solicitation announcement only said that the facility was intended to be "heavily cost-shared" between NSF and other federal, state, or private sources. NSF gave the award to the proposal that had the 50 percent match (including $58 million from the state government). Recently, NSF decided to recompete the National Nanofabrication Users Facility that a university had operated with NSF support since it won the original competition in 1978. Although a national user conference had called for additional facilities, NSF decided to recompete the existing facility in 1992; thus, the solicitation called for proposals to manage a facility. NSF received several strong proposals and recommended two to the NSB for approval (including the incumbent). At this time, NSB heard from other institutions that they had decided not to apply because they thought the odds of succeeding against an incumbent were too low, but they probably would have applied if the solicitation had said that more than one facility would be awarded. As a result, NSB decided to have another competition in which the solicitation said that NSF intended to make more than one award.

There is no guarantee that discussion of the solicitation announcements by NSB would have avoided these situations, Nevertheless the requirement for NSB review and approval at the solicitation stage would give that body, with its broad outside perspective, an opportunity to identify potential issues and misunderstandings.

According to NSF's current procedures, at the time of the decision to recommend funding or to decline the proposal, the proposal folder should contain

• the review plan required for large special projects;

• the request-for-review letter that refers to the selection criteria and their relative importance;

• the ad hoc mail reviews;

• advisory committee/review panel reviews and summary;

• site-visit reports;

• responses of the principal investigator to review comments or program questions; and

• correspondence, memoranda, or diary notes relating to the recommendation.

The program officer then completes the "Review Record" (NSF Form 7), which accompanies every recommendation for a new award, renewal, or declination or any other action subject to peer review. The form was developed in response to the curriculum development controversies in 1975. The purpose of the form was to document who the peer reviewers were and how they rated the proposal, and to ensure that the program officer responded to the concerns of each reviewer, especially those not consistent with the program officer's award decision.⁷

The first part of Form 7 lists the name, sex, department or field, and institutional affiliation of everyone asked to review the proposal, and indicates the summary rating (letter or number) given

by the reviewer. If external review was required but there were fewer than three mail reviewers or panelists, the program officer must justify making a recommendation on the basis of one or two reviews.

If a review panel assigned an overall rating, it is recorded on the form.⁸

The program officer gives a summary rating of the reviewers' comments on previous NSF-supported work (and may indicate his or her rating if different from the reviewers').

The second part of Form 7 is the "review analysis," in which the program officer justifies the recommendation. If the recommendation is negative, "excellent" review ratings must be explained. If the recommendation is favorable, the program director must explain any ''fair" or "poor'' ratings or significant negative comments by reviewers.

After the program officer signs the completed Form 7 and prepares an abstract, a summary budget, and an administrative processing form, the proposal jacket is forwarded for review and approval via the section head (if there is one) to the division director. The division director is usually the final sign-off authority for the traditional small grant. The section head and division director are supposed to determine that the number and quality of external peer reviews were adequate, that significant peer review comments contrary to the recommendation have been dealt with adequately, that the rationale for the recommendation is reasonable, and that proper administrative procedures have been followed.

In the case of major awards, these determinations are made by ARG and DARB (as explained earlier). After reviewing the decision package going to CPP and EHR and to NSB—which contains the proposal, peer reviews, site visit and panel reports, and program

director's recommendation—DARB reviews the memorandum prepared by the directorate for the director, recommending that NSB approve the award. The director's memorandum is normally a few pages that "summarize strategic information and issues on the proposed action, including objectives, alternatives, potential policy implications, precedents involved, and any other factors that could be considered nonroutine" (NSF, 1992a:III-2). It should have a short statement responding to any "major concerns" raised by reviewers, a summary of budget totals, the percentage of the program or division budget involved in the award, and the out-year budgetary implications.

The director's memorandum is the only public document laying out the basis and rationale for a major award. Currently, the review process for major awards is modeled closely on the peer review process for individual research projects, in which confidentiality is deemed necessary. Accordingly, the peer reviews and the site visit and panel reports are confidential. In most cases, the director's memorandum also discusses only the winning proposal because unsuccessful proposals are deemed confidential. Except in the case of the NHMFL award, in which the top two proposals were explicitly compared, the basis for the award is never comparative, which limits NSB and public understanding of the decision.

Because of their size and importance in a field, major project awards are more significant and more public than small grant proposals submitted by individuals and small groups. They also usually involve multiple criteria and complicated choices whose basis is harder to understand than purely technical merit in making individual research projects grants.

As noted in Chapter 3, the panel found that the criteria and procedures are not always well understood by participants, including proposers and peer reviewers, or by interested observers. They are sometimes not sure of the relative importance of the various criteria or how to fulfill them, and they do not always understand how the criteria are eventually integrated by staff in making the final decision. Certain stages of the review process are not well documented, at least publicly. This is an additional source of misunderstanding of the basis for final decisions.

The panel concluded that it would be very beneficial for NSF staff to be more explicit and open in explaining and documenting recommendations to NSB and for NSB to be more explicit in documenting the basis for its approvals.

The most appropriate vehicle to explicate major award decisionmaking is the director's memorandum recommending a proposal for funding to NSB. This memorandum should be written with public dissemination and understanding in mind.

After making its decision, NSB also should issue a statement of its reasoning for approving the award, because it may amend or differ from the rationale contained in the director's memorandum. These documents would make the decisionmaking more understandable in some cases if there were an analysis of the strengths and weaknesses of the approved proposal, compared to those of the runner up or other leading proposals. This is especially necessary if a proposal other

than the one ranked highest in the first-phase technical review is recommended for funding. The comparison would follow more easily if the earlier recommendation were adopted that there be an advocate for the top technical proposal in such cases in the final decisionmaking by NSB.

We are not recommending that the names of peer reviewers, or other information that might dampen reviewer candidness, be made public. The director's memorandum is supposed to summarize reviewer comments; we believe that the documentation of the NSB award decision and its reasoning can be done well without disclosing the identity of individual peer reviewers or review panel members.

To ensure wide access to the documentation, NSF could use Internet to make the documentation for major award decisions readily available on-line through its Science and Technology Information Systems, as it already does abstracts of winning award proposals. NSF also could make major award policies and procedures more available electronically. NSF recently made the Grant Policy Manual available electronically and updates it periodically. The Grant Policy Manual is intended primarily to provide grant administration requirements to those who have received small grant awards (it does not, for example, mention the NSB role in reviewing large awards). NSF's (1993c) internal Proposal and Award Manual contains the full set of merit review and award policies and procedures for major awards, and the relevant sections should be made available electronically and updated regularly.