Major Award Decisionmaking at the National Science Foundation (1994)

The underlying theme of this chapter is that although each major project is different at some level, common characteristics of major awards distinguish them from individual research grants. These characteristics justify special review policies and procedures that are more explicit and consistent across the National Science Foundation (NSF) than is necessary for small research grants. Major projects:

• represent a substantial investment of NSF resources in a research area, with greater political costs and financial liabilities in the case of failure;

• are more complex than traditional individual investigator or small-group research projects and involve more varied criteria, some less exclusively scientific and technical than those of individual investigator proposals; and

• promise a substantial impact not only on their research community but also on the institutions and localities in which they are located.

This chapter begins by providing background on the history of merit review (and the role of peer review within it) at NSF. It then addresses and makes recommendations concerning two parts of the proposal review and award process for major projects: the review criteria used to select the most qualified proposals, and the procedures to ensure that the review criteria are appropriately applied and documented.

The policies and procedures for handling major awards today have evolved from, and closely resemble, those used for grants for small research projects. External peer review has been used from the beginning at NSF to evaluate all research and education proposals. The first set of research grants was approved by the National Science Board (NSB) and awarded in 1952 after the following process: "an independent evaluation of each of fifty-eight proposals by at least three expert reviewers; a two-day meeting of a screening panel of eleven consultants expert in the fields represented; a meeting of the newly-formed Divisional Committee regarding general policy and program within the field of the Division; and consideration by the full staff of the Director" (England, 1982:166, quoting NSB minutes of February 1, 1952).

Peer review of research proposals for technical merit by outside scientific and technical experts has been so important at NSF that, for years, the shorthand title for the award process was simply "peer review" (NSF, 1975:4). Criteria that augment technical quality and competence have become more prominent over time, however. These criteria include immediate practical relevance, and the development of science and engineering capacity in all groups and regions of the country. In 1985–1986, NSF (1986a) had an outside advisory committee study the proposal review process. As a result of the committee's report, the NSF director adopted the term "merit review" to emphasize the greater role of expanded criteria, especially for "center-based activities, research groups, and shared facilities" (NSF, 1986b: D-2).

NSF's current policy is that all formal proposals for grant funding are subject to peer review by appropriate experts external to the agency, with minor exceptions listed in a policy document approved by NSB (e.g., small travel grants, the Small Grants for Exploratory Research program). In addition, for special cases in which peer review is "impracticable or too costly," the NSF director

may waive it (NSF, 1993c:I-4).¹ No such waivers have been requested in recent years (Massey, 1991, 1992).²

Peer reviews are advisory; that is, external experts are used by full-time NSF program officers to help them make decisions on proposal funding. However, the final decision can be based on a range of considerations. Decisions are affirmed or reversed at higher administrative levels, depending on the size of the project and other criteria. If the projects are large enough, or raise new programmatic or policy issues, they must be reviewed and approved by the NSB.

According to the NSF Proposal and Award Manual (NSF, 1993c:I-3), it is agency policy to give every research proposal "proper consideration in accordance with established criteria approved by the NSB." The NSF Grant Policy Manual, which is published for the use of proposers, says that the review and evaluation criteria are "to be applied to all research proposals in a balanced and judicious

manner, in accordance with the objectives and content of each proposal'' (NSF, 1989:II-6).³

The criteria are as follows:

1. Competent performance of the research relates to the capability of the investigator(s), the technical soundness of the proposed approach, the adequacy of the institutional resources available, and the proposer's recent research performance.⁴

2. Intrinsic merit of the research is used to assess the likelihood that it will lead to new discoveries or fundamental advances within its field of science and engineering, or will have substantial impact on progress in that field or in other scientific and engineering fields.

3. Utility or relevance of the research is used to assess the likelihood that it can contribute to the achievement of a goal extrinsic or in addition to that of the research field itself, and thereby serve as the basis for new or improved technology or assist in the solution of societal problems.

4. Effect of the research on the infrastructure of science and engineering relates to its potential to contribute to better understanding or improvement of the quality, distribution, or effectiveness of the nation's scientific and engineering research, education, and human resources base.

NSF policies are not very specific about the relative priorities and weightings these different criteria should have. The Proposal and Award Manual (NSF, 1993c:I-3) provides the following discussion:

The merit selection criteria have evolved over the years. At one time there were separate criteria for large-scale facilities and centers (see Box 3-1).

NSF procedures permit special treatment of major award projects. Criteria in addition to the four basic ones may be applied to specific programs. They are supposed to be listed in program announcements and solicitations; specified in the cover letters sent to peer reviewers; and used by site visitors, panels and NSF staff in summarizing the basis for their recommendations. In addition, NSF still has "special guidelines for organization and research administration" for large projects, which were based on the 1974 criteria for creating or changing institutional structures:

1. Criterion of Need: If a new administrative structure is proposed, evidence should be provided that it is needed to address scientific problems in a manner or on a scale not possible with existing structures.

2. Criterion of Long-Range Potential: Evidence should be provided of a mission with such potential for high scientific productivity over an extended time period, that a significant number of excellent scientists are willing to commit their careers to it.

These criteria, which are descended from the longer list of criteria adopted by NSB in 1967, are not now part of any formal planning or review procedures although they are in the Proposal and Award Manual (NSF, 1993c:I-4). They were not used in the 10 case studies, or at least they are not mentioned in the documentation (although the criteria in some cases, for example, for Engineering Research Centers (ERCs) and Science and Technology Centers

(STCs), did cite the advantages of the center type of organization).

Major awards almost always involve criteria in addition to the basic four. They are listed in the solicitation announcements (see Appendix D for recent examples of selection criteria).⁵ The additional criteria can make decisionmaking more complex and difficult. The criteria may include, for example, managerial capacity, technology transfer, human resource development, participation of underrepre-

sented groups, and/or nonfederal cost sharing. The relative weighting of these criteria is rarely specified in the solicitation document and is indicated only generally in the letter to reviewers. The problem of priority weighting is further complicated by uncertainty about when in the review process each of the various criteria should be addressed. It also may not be clear which reviewers were selected to apply which criteria, making it difficult to know how to weigh the review comments.

The panel found that for major awards the review criteria and their relative importance were not always well understood by participants, including proposers and peer reviewers, or by knowledgeable observers. Peer reviewers often confine themselves to those criteria they are most qualified to judge, usually technical merit and capability of the proposer, and it is not always clear who is assessing the other criteria or how they are eventually integrated by staff in the final decision. The panel concluded that NSF should (1) state carefully and clearly the criteria that will be used in the review process, and (2) be explicit about the relative importance of the review criteria .

The panel does not advocate the use of strict numerical weightings, which is the approach taken, for example, in procurement contracts. Choosing the best research-related proposal necessarily involves a large degree of expert judgment and some degree of uncertainty; thus, decisionmaking about awards in these cases cannot be reduced to a quantitative algorithm. Nevertheless, without any guidance at all, reviewers are implicitly allowed to derive their own weights for the various criteria when arriving at their overall rating; NSF officials and NSB members also must use their own judgment in evaluating proposals against the criteria. We believe that it is possible and desirable to describe the relative importance of the criteria specified for a major award project to help ensure that reviewers give highest consideration to proposals best able to meet the research goals of the program.

The NSB and NSF should continue to make technical excellence the primary criterion in evaluating the merit of proposals for major awards. To ensure that research funding is used most effectively, no major award should ever be made to a project that is not of very high technical merit. Additional criteria should be used only to choose the best overall proposal from among those whose technical merit is among the most highly rated.

Research results are the main goal of a major award, although major awards may also promote the realization of other meritorious goals. Technical merit is the likelihood that a proposed project will achieve the projected research objectives, whatever they may be. Thus, technical merit has been and should remain the primary standard met by all NSF awards, including major awards, because it best assures effective use of the funding devoted to federally supported research. It is especially important in major awards because there is less opportunity for trial and error than in a program of small awards. Emphasizing technical excellence also gives merit review added credibility that should help forestall academic earmarking or other political interference in award decisionmaking at NSF.

For smaller awards to individual investigators, technical merit criteria have to do with the originality of the project, soundness of the methodology, and qualifications of the researcher to carry out the proposed research. In major awards, technical merit is more complex because the factors involved in successful implementation of a large-scale, long-term project are more extensive. For example, these factors may include such considerations as the managerial capacity of the principal investigator or investigators, and their institution or institutions, to implement and operate a major research facility or center; physical site characteristics if construction is involved; or the involvement of industry and technology transfer plans if an applied research effort is involved.

Major awards usually involve criteria in addition to the technical merit of the projects and the capacity of the proposers to carry them out. These additional merit criteria include (but are not limited to) human resource development potential, participation of women and

minorities, geographical and institutional distribution and capacity building, and/or relevance to national socioeconomic goals. Peer reviewers selected for their technical expertise may not feel qualified to apply the nontechnical criteria, although they may comment on them. Because the additional considerations must be taken fully into account by NSF officials in recommending the best overall proposal, reviewers should be appointed specifically to assess each of them and provide input to the process.

This recommendation would be more easily implemented in conjunction with the two-phase review process recommended later in this report, which provides a mechanism for systematically ensuring that both high technical merit and the other criteria are taken into account in reaching a final decision on the overall merit of major award proposals.

The precise dividing line between technical and nontechnical merit criteria cannot always be determined a priori; it is a decision contingent on the nature of each project's objectives. The definition of technical criteria must be determined as part of the design of the review process for each project (recommended later).

The technical merit criteria should measure a proposed project's likelihood of achieving its primary scientific or engineering research objectives. These would be different for different types of projects. For example, a proposal to construct and operate a national research facility with state-of-the-art instrumentation would have to show not only that top scientists were involved and first-rate research was likely in the in-house program, but also that the engineering requirements and management capacity made the project feasible. A proposal for an interdisciplinary research center would have to show promise of "centerness," that is, that the center would be organized and managed in such a way that the research done would contribute more than an equally expensive program of individual grants.

In each case, what is most important is that a clear decision be made about which criteria will be considered technical (e.g., integral to the project's research achievements) and which are secondary (e.g., desirable in addition to research results). Other criteria may be ones that qualify proposals for consideration in the first place (e.g., a cost-sharing requirement or, in the case of a facility, the absence of physical problems such as seismic activity or light pollution). The

criteria, their classification as technical or nontechnical, and their relative importance should be stated clearly in the solicitation announcement.

The contribution of every major award proposal to overall human resource development should be emphasized. The number of students to be involved—and the inclusion of minorities and women at all levels, from students to senior investigators and project managers—are important components of human resource development and equal opportunity. They should receive more explicit attention in the review process.

Research opportunities for minorities, women, and other underrepresented groups constitute a criterion second only to scientific and technical merit. Long-term scientific progress depends not only on today's research but also on tomorrow's well-trained and experienced researchers and research leaders. The research enterprise is also a major source of trained personnel for industry—an important mechanism for the transfer of knowledge from research to product development. Any major investment of NSF's resources should be assessed for its potential impact on the training and education of scientists and engineers, especially minorities and women who are currently underrepresented in the nation's technical work force.

At the same time, the women and minority investigators already in science are underrepresented in research leadership positions, particularly in the major awards category. In the case studies it examined, the panel did not find a single female principal investigator, and its seem likely that minority investigators were equally rare. Given the gradual shift in NSF resources toward this category of award, special attention should be devoted to opening the door of opportunity for leadership on major award projects at least as wide as it has been opened for women and minorities to be principal investigators on individual investigator grants.

Accordingly, all proposals for major awards should document the expected number of students who will be involved and the plans for participation of minorities and women at all levels. Once the primary criterion of technical quality is met, proposals that emphasize human resource development and equal opportunity features should be given strong consideration.

Cost sharing should be used only to demonstrate commitment to the project's goals and never simply to extend NSF funds. Where cost sharing is required, NSF should spell out its expectations in the solicitation announcement. The amount of credit for cost sharing for purposes of evaluating proposals should be stated clearly and capped at a reasonable level. Due account should be taken of the likelihood that cost-sharing commitments will in fact be met in the out years.

The level of nonfederal cost sharing in major awards has been increasing for a decade. In some cases, the lack of clarity about the matching requirement has caused problems in the review process.

Nonfederal cost sharing may be a legitimate criterion for a major award, but in accord with NSB policy, it should be required only for programmatic reasons (e.g., to ensure industrial relevance of the proposed work, to strengthen its linkages to an interdisciplinary research center, or to show that a university is committed to running a high-quality national facility). Open-ended cost-sharing requirements may place an unintended and unfair burden on universities and state governments, and interfere with their ability to order their own research support priorities.

In some cases, cost-sharing requirements may not be achievable by potential proposers in states where legislatures are not in session, for example, because they only meet biennially. Most importantly, excessive cost-sharing requirements can severely limit the scope and quality of the proposals received.

Concerned with the rapid increase in cost sharing, an NSB committee studied the issue in 1986 (NSB, 1986). The committee's

conclusion was adopted by the NSB and reaffirmed after a staff update of the study in 1989 (NSB, 1989):

The panel found that unclear cost-sharing requirements had caused problems in the review of several major awards. In the case of the Earthquake Engineering Research Center, for example, a 50-percent match requirement deterred some potential proposers from applying because they could not obtain $5 million a year from their states (GAO, 1987:31). The lack of clarity about the nature, source, and timing of the match also caused problems for several that did apply. The match for one proposal came very late in the process because the state legislature had not met, and NSF ''conditionally'' recommended that the award be made to another proposal before the review of the first had been completed (GAO, 1987:32).

In the case of the National High Magnetic Field Laboratory, the project development plan included a 50-percent matching requirement. When the solicitation was drafted later, it said only that the facility would be "heavily cost-shared" (NSF, 1989). The winning proposal was able to obtain $58 million over five years in matching funds from the state government that, with some university funds, was a little more than the 50 percent match called for in the project development plan. This substantial sum was more than any other applicant could secure and appeared to many in the research community to have an undue influence on the award decision.

Although the panel's recommendation is current NSB policy, it is reaffirmed here because the temptation to leverage other funds to make NSF dollars go further is very strong. The level of nonfederal cost sharing in NSF awards has been increasing over the past decade. That increase occurred in part to ensure local commitment or the relevance of interdisciplinary research to industry, but there comes a point at which the burden on local institutions causes local problems. Accordingly, NSF and NSB must ensure that the cost sharing is

justified on a case-by-case basis and does not impose an excessive net burden on universities and states, or unduly distort their priorities or the priorities of NSF. They also should take into account the sources and quality of the cost-sharing commitment as well as its size.

The review part of the award process usually commences when NSF releases the solicitation document. (Some major awards are not competitively solicited, and the review begins when the proposal is received.) The review phase culminates in NSB's review and approval of the award. It includes the peer review of proposals, the decision by staff to recommend funding of a particular proposal or proposals, endorsement by the NSF director, and review and ratification (or disapproval) by NSB.

Once the criteria are developed for choosing among project proposals, NSF must design a process to ensure that the criteria are applied consistently and appropriately. In a merit review system, this means that individuals with the expertise to assess each criterion must be appointed and deployed as needed for that specific solicitation (e.g., ad hoc reviews by individual experts, perhaps site visits, and one or more panel reviews). Finally, these judgments must be weighed and combined in an overall ranking of proposals for decisionmaking at higher levels within NSF. In major award cases, the NSF director recommends a decision and presents it to NSB for approval. As noted above, technical excellence in carrying out the research-related objectives of the project should be paramount in choosing among proposals, but other criteria may and usually should be taken into account in choosing among top-rated proposals.

The NSF program director decides on a review process for each proposal that will enable him or her to make a sound, well-documented recommendation. Several "tools of the trade" are in common use at NSF, often in combination. They must include some form of external peer review, such as the following:

1. Ad Hoc Mail Review: The NSF program director sends the proposal to at least three (usually four to eight) experts on specific aspects of each proposal, who are asked to submit written reviews to the program officer.

2. Panel Review: In addition to or in place of mail review, a panel of experts may be convened to evaluate and rank proposals. In some programs, especially biology and earth sciences, standing advisory committees meet two or three times a year to consider proposals. In other programs, such as ocean sciences, ad hoc panels are formed to review each round of proposals.

3. Site Visit: For large projects, competitive solicitations for centers and facilities, or particularly difficult or unusual proposals, mail and panel reviews may be supplemented with site visits by teams of outside experts and NSF staff. NSB members may be site visitors as well. Site visits are generally also used for decisions on renewing projects.

The program director evaluates each proposal and may make a site visit, invite reviews by colleagues at NSF with appropriate expertise, or consult with other federal agency officials.

Program directors are required to select a set of reviewers that, in the aggregate, can provide the information needed to make a recommendation in accordance with the selection criteria.

Selection of Mail Reviewers. The virtue of mail review is that each proposal may be reviewed by the experts most knowledgeable about the subfield involved. Ideally, reviewers as a group will be able to evaluate a proposal with respect to all of the selection criteria. Therefore, the balance desired among criteria should affect the selection of reviewers. In selecting reviewers, program directors are also asked to take into account any additional criteria stated in program announcements and solicitations.

The award manual lists the "optimum" criteria for selecting reviewers (NSF, 1993c:I-5). For all proposals, some reviewers should be experts in the research subfields involved so they can evaluate the proposals on the criteria of competence, intrinsic merit, and utility. If the proposals involve substantial size or complexity, broad disciplinary or multidisciplinary content, or potential applications to significant national problems, some reviewers should have a broader or more generalized knowledge of the research subfields involved. Some reviewers are supposed to have a broad knowledge of the infrastructure of the national science and engineering enterprise to evaluate proposals for their impact on scientific and engineering education and human resource development, distribution of resources to institutions and geographical areas, and other social goals. Finally, reviewers as a group should reflect a balance among various characteristics such as geography, type of institution, and underrepresented groups.

Selection of Panel Members. The advantage of panels is the opportunity they provide for face-to-face interaction among reviewers and between reviewers and the program director. This interaction allows a more detailed discussion of each proposal and how it relates to the overall program. On the other hand, a relatively small group reviewing a variety of proposals is less likely to be as knowledgeable about every subfield involved as mail reviewers can. That is especially true of a standing advisory group. The NSF award manual notes that "it is important, nevertheless, that such groups be structured to provide broad representation and many views on matters under the advisory group's purview" (NSF, 1993c:I-6). The manual offers some general considerations that should be taken into account in achieving balance in advisory groups. They include

• individual qualifications;

• fields of expertise;

• public impact—some panel members should represent regions, organizations, or segments of the public directly affected by issues under consideration;

• academic and nonacademic impact—members should represent small, medium, and large institutions, as well as public and private institutions; representatives from outside the academic community are also desirable in most instances;

• underrepresented groups;

• age distribution; and

• geographic balance.

Careful selection of a balanced set of reviewers is especially important for major awards, not only because they are more visible but also because they are more complex. It is critical that the reviewers are—and are perceived to be—qualified, unbiased, and balanced as a group. NSF was criticized in the case of the Earthquake Engineering Research Center, for example, for having too few earthquake researchers and too few Westerners on the review panel (GAO, 1987).⁶

Merit review at NSF has been criticized many times as an "old-boy network" that concentrates awards in a relatively small number of institutions in a few states. Various policies and procedures have been added over the years to ensure that the system is open and fair. NSF program directors are not supposed to use anyone as a reviewer who (1) would be directly involved in the project as a collaborator or consultant, (2) is from the same institution as the applicant, or (3) has a family relationship with the applicant.

The program director is also required to document the existence of interests, affiliations, and relationships that might affect a reviewer's evaluation of a proposal. Reviewers are explicitly asked to describe relationships that may be or appear to be a conflict of interest. The program officer is required to indicate in the file how such conflicts or potential biases were handled in making the award decision.⁷ Reviews that contain intemperate personal attacks or other indications of bias are not supposed to be used in the selection process. Other policies and procedures to ensure fairness are described in Box 3-2.

Among the key actors in the decision process are NSF program directors. They make initial recommendations for the review format and specific reviewers to be used for each proposal or competitive set of proposals, which division directors and assistant directors review and approve. The program directors recommend whether or not to fund a proposal, as well as the amount and duration of support. They base the recommendation on the comments of reviewers, but they

may take other considerations into account, such as the balance among research topics or the amount available in the program budget.

In the case of individual investigator grants, recommendations by program directors can be approved at the division level. If the proposal is large enough, as in the case of major awards, the award decision is reviewed by the appropriate assistant director and at the NSF director's level before being sent for approval to NSB with a memorandum from the director.

Although external reviewers are asked to evaluate a proposal's impact on geographic and institutional balance and on participation of women and minorities, the program director and other NSF staff are often the only ones in a position to make these judgments in the case of a program of small grants. Facing proposals of nearly equal merit,

the program director may give greater consideration to scientific program balance and distributional factors (geographic, institutional, race, sex, etc.). This is especially true in small grant programs as they near the end of their available funds, when program directors may achieve greater balance among secondary criteria by selectively choosing slightly lower-rated proposals.⁸

In major award cases, however, secondary criteria cannot be balanced by letting NSF staff select some lower-rated proposals at the margin, after funding the best proposals technically. For major projects, only a few awards, or often just one, is made. This means that all the criteria, even though they are more numerous and complex than for small grants, must be met in one or a few proposals. Given the importance and complexity of major awards, the role of outside peer evaluation in weighing and balancing the various criteria, both technical and nontechnical, is especially important throughout the process. The recommendations in this report thus aim to create a merit review system in which outside peer review not only plays its usual major role in evaluating technical excellence but also plays the major role in evaluating secondary criteria and identifying the most meritorious proposal based on all criteria.

The panel found that the review processes for major awards vary greatly. Some involve elaborate and well-thought-out procedures for applying different types of criteria and expertise at different levels

and stages of the review. Some are simply handled as investigator-initiated proposals.

At one time, NSB had criteria and procedures for large-scale facilities and centers that were different from those for the small-project grants. Over time, however, the procedures for the latter came to be applied to all awards, regardless of their nature, size, or complexity. The panel concluded that the major awards are different enough from the typical small grant, and alike enough because of their size and impact, to warrant standard procedures of their own. The key is to structure the review process so that the various types of criteria are applied by appropriately qualified reviewers.

For major awards, the peer review part of the merit review process should be conducted in two phases. The first phase would be a strictly technical review; to help assure the primacy of technical merit, only those proposals judged to be technically superior would be forwarded to the second phase for any further consideration. In the second phase, the additional merit criteria would be weighed and balanced with the technical criteria by a more broadly constituted group of reviewers. This second-phase panel would recommend the proposal (or proposals) best meeting the full set of criteria. If the proposal judged to have the highest merit overall is not the one ranked highest in the first phase of review for technical merit, the second-phase panel must explain its recommendation fully. If the top-ranked technical proposal is subsequently not recommended by NSF staff, the chair of the first-phase panel or another member of that panel should present the case for it at the NSB level.

This two-phase methodology, which NSF and NSB have already employed successfully for some major award competitions, will help ensure that three important goals of the review process are met.

First, NSB will approve awards only to projects that meet the highest technical criteria and thus have the greatest promise of achieving their research-related objectives. At the same time, however, it will ensure that other important but secondary merit criteria are evaluated by qualified reviewers and appropriately weighed in the final ranking. Third, the process will better clarify the basis for the final decision and the respective input of peer reviewers and agency staff, which in turn will make the decisionmaking process more acceptable and accountable to both the research community and the broader public to which NSF must answer.

Procedures. If a large number of proposals is expected, NSF should use preproposals to narrow the field. The technical merit of the preproposals should be peer reviewed, either through the use of mail reviews or perhaps through a meeting of a Phase 1 review panel.

Phase 1 is the technical review by relevant experts. It can be done in stages or steps, including mail reviews, site visits, and panels or subpanels. In the final step of Phase 1, a panel of technically qualified reviewers evaluates all the proposals for technical merit (with input from the mail reviews, site visits, etc.), sorting them into no more than four or five categories of excellence, ranging from most qualified to unqualified. The panel should strive to produce a priority ranking of the proposals if at all possible. If the ranking by quality is reasonably clear, the panel should forward the proposals in rank order. If there are a number of excellent proposals, each with its strengths and weaknesses, the panel may forward a group of top-ranked proposals judged to be essentially equivalent in technical quality. In this case, the panel should comment carefully on the strengths and weaknesses of each proposal.

In research center competitions where there are numerous proposals, it is relatively easy to forward a set of top-rated proposals to the second phase, explicitly ranked in order of technical merit. In fact, NSF already uses a form of the two-phase merit review process for the Engineering Research Centers, Science and Technology Centers, and Supercomputer Centers. The more difficult cases are

competitions in which there are only a few proposals. We expect on the basis of the case studies, however, that there will usually, if not always, be at least several strong candidates in the second phase of a major award competition.

Phase 2 is a review of all other merit criteria (e.g., human resource development and equal opportunity, geographic or institutional distribution, technological relevance, industrial involvement, outreach and technology transfer activities). To complete this phase, the subgroup of proposals ranked ''excellent'' or "most qualified" in Phase 1 panel is considered by a new, Phase 2 panel of reviewers. This new panel should be qualified to evaluate proposals on both the additional criteria and the technical criteria identified for that project (although it should not redo the technical ranking). Additional review steps could occur during this phase, such as special subpanels to consider certain criteria (e.g., business people and economists to evaluate proposals for their potential contributions to economic competitiveness), additional site visits, or presentations by finalists.

The Phase 2 panel would recommend the proposal (proposals if there are multiple awards) that best meet the full set of criteria. If the recommendation is for a proposal that was not the most highly ranked technically in Phase 1, the Phase 2 panel would have to explain explicitly the reasons for its action. In that case, or if NSF staff decides to recommend a proposal to NSB that was not ranked highest by the Phase 1 review panel, NSB or at least its Committee on Programs and Plans (CCP) should hear a presentation in favor of the top-ranked technical proposal from the Phase 1 panel chair or another advocate from that panel. He or she should present to CPP the case for the best Phase 1 proposal and help CPP understand the research-related benefits that might be forgone if another proposal were chosen. Currently, CPP-NSB hears only the case in support of the proposal recommended by NSF and thus cannot make a decision on the basis of a comparative analysis of the top few proposals. We believe that NSB would be in a better position to make a more informed decision if it more clearly understood the trade-offs among merit criteria involved in choosing the winning proposal and that this

understanding is particularly important when the successful proposal is not the one with the highest technical merit according to the external peer reviewers.

Reviewer Roles. Reviewers in Phase 1 should be chosen so that, as a group they will be able to evaluate and rank the proposals according to technical merit (as measured by the technical criteria specified in the solicitation). In addition to providing an overall summary rating based on technical criteria, Phase 1 reviewers should be encouraged to comment, if they wish, on the other merit review criteria. In this case, they should give two ratings, one based on purely technical criteria and the other indicating how they might change their overall ratings in light of additional nontechnical criteria (at the same time, indicating their competence to apply the other criteria).

So far, the process is not very different from that employed in many cases, except that the procedure recommended here calls for the use of a panel in every case to integrate the reviews rather than rely on mail reviewers and staff alone. The panel is also charged with deriving an overall ranking of proposals according to technical merit only, without taking other criteria into account.

A new panel is required in Phase 2, although mail reviewers, subpanels, and site visitors may also be used. The Phase 2 panel should be constituted to apply the full set of merit criteria, technical and nontechnical, because this group has the task of identifying the best overall proposal. It should be noted that the eventual winning proposal has already had to achieve a high threshold of technical excellence; thus, no award would go to a project of low technical quality just because it ranked high on other merits. A winning proposal must rank high in both areas, technical and nontechnical.

At least some Phase 2 reviewers should be carried over from the first-phase panel. This continuity is helpful because the primary and secondary criteria usually interact in ways that need to be evaluated before reaching a final ranking. For example, one of two proposals basically equal in scientific promise might better involve students

because of the way the research is organized or where it is done. That would affect the rating positively when education is an important secondary criterion.

Phase 2 panels should be asked to provide a careful qualitative discussion of how they balanced the nontechnical criteria against the technical ones. We believe that in the typical case the discussion will revolve around whether proposals ranked second or third on the technical merit scale are the most meritorious overall and should be funded. In some cases, several proposals may have substantially equal technical merit. In these cases, important secondary criteria may play a tiebreaker role. Even here, however, each proposal will have its individual technical strengths and weaknesses, and it will be helpful to have carryover members from the Phase 1 panel familiar with the proposals as participants in the discussion of the balancing of technical and other merits. As already mentioned in Recommendation 3, if the Phase 2 panel decides to bypass the proposal or proposals ranked highest technically, it should provide a careful written explanation of the justification for doing so.