3
Awarding Major Projects: Criteria and Review Procedures

The underlying theme of this chapter is that although each major project is different at some level, common characteristics of major awards distinguish them from individual research grants. These characteristics justify special review policies and procedures that are more explicit and consistent across the National Science Foundation (NSF) than is necessary for small research grants. Major projects:

  • represent a substantial investment of NSF resources in a research area, with greater political costs and financial liabilities in the case of failure;

  • are more complex than traditional individual investigator or small-group research projects and involve more varied criteria, some less exclusively scientific and technical than those of individual investigator proposals; and

  • promise a substantial impact not only on their research community but also on the institutions and localities in which they are located.

This chapter begins by providing background on the history of merit review (and the role of peer review within it) at NSF. It then addresses and makes recommendations concerning two parts of the proposal review and award process for major projects: the review criteria used to select the most qualified proposals, and the procedures to ensure that the review criteria are appropriately applied and documented.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation 3 Awarding Major Projects: Criteria and Review Procedures The underlying theme of this chapter is that although each major project is different at some level, common characteristics of major awards distinguish them from individual research grants. These characteristics justify special review policies and procedures that are more explicit and consistent across the National Science Foundation (NSF) than is necessary for small research grants. Major projects: represent a substantial investment of NSF resources in a research area, with greater political costs and financial liabilities in the case of failure; are more complex than traditional individual investigator or small-group research projects and involve more varied criteria, some less exclusively scientific and technical than those of individual investigator proposals; and promise a substantial impact not only on their research community but also on the institutions and localities in which they are located. This chapter begins by providing background on the history of merit review (and the role of peer review within it) at NSF. It then addresses and makes recommendations concerning two parts of the proposal review and award process for major projects: the review criteria used to select the most qualified proposals, and the procedures to ensure that the review criteria are appropriately applied and documented.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation BACKGROUND: THE MERIT REVIEW PROCESS AT NSF The policies and procedures for handling major awards today have evolved from, and closely resemble, those used for grants for small research projects. External peer review has been used from the beginning at NSF to evaluate all research and education proposals. The first set of research grants was approved by the National Science Board (NSB) and awarded in 1952 after the following process: "an independent evaluation of each of fifty-eight proposals by at least three expert reviewers; a two-day meeting of a screening panel of eleven consultants expert in the fields represented; a meeting of the newly-formed Divisional Committee regarding general policy and program within the field of the Division; and consideration by the full staff of the Director" (England, 1982:166, quoting NSB minutes of February 1, 1952). Peer review of research proposals for technical merit by outside scientific and technical experts has been so important at NSF that, for years, the shorthand title for the award process was simply "peer review" (NSF, 1975:4). Criteria that augment technical quality and competence have become more prominent over time, however. These criteria include immediate practical relevance, and the development of science and engineering capacity in all groups and regions of the country. In 1985–1986, NSF (1986a) had an outside advisory committee study the proposal review process. As a result of the committee's report, the NSF director adopted the term "merit review" to emphasize the greater role of expanded criteria, especially for "center-based activities, research groups, and shared facilities" (NSF, 1986b: D-2). NSF's current policy is that all formal proposals for grant funding are subject to peer review by appropriate experts external to the agency, with minor exceptions listed in a policy document approved by NSB (e.g., small travel grants, the Small Grants for Exploratory Research program). In addition, for special cases in which peer review is "impracticable or too costly," the NSF director

OCR for page 61
Major Award Decisionmaking at the National Science Foundation may waive it (NSF, 1993c:I-4).1 No such waivers have been requested in recent years (Massey, 1991, 1992).2 Peer reviews are advisory; that is, external experts are used by full-time NSF program officers to help them make decisions on proposal funding. However, the final decision can be based on a range of considerations. Decisions are affirmed or reversed at higher administrative levels, depending on the size of the project and other criteria. If the projects are large enough, or raise new programmatic or policy issues, they must be reviewed and approved by the NSB. Current Review Criteria According to the NSF Proposal and Award Manual (NSF, 1993c:I-3), it is agency policy to give every research proposal "proper consideration in accordance with established criteria approved by the NSB." The NSF Grant Policy Manual, which is published for the use of proposers, says that the review and evaluation criteria are "to be applied to all research proposals in a balanced and judicious 1   External review is not required for proposals submitted in response to formal solicitations governed by the Federal Acquisition Regulations review, that is, in response to Requests for Proposals for contracts. Almost all major research awards, however, are cooperative agreements (a type of grant) rather than contracts. 2   Some major awards have been exempted in the past. In FY 1985, for example, peer review was waived for two planning proposals for multi-institutional consortia to manage large earth sciences projects, because each of the consortia included virtually all the institutions involved and thus made it impossible to find reviewers without conflicts of interest. Instead, the projects—IRIS (Incorporated Research Institutions for Seismology) and DOSECC (Deep Observation and Sampling of the Earth's Continental Crust)—were discussed extensively by the advisory committee for earth sciences and the NSB committee on programs and plans (Bloch, 1986).

OCR for page 61
Major Award Decisionmaking at the National Science Foundation manner, in accordance with the objectives and content of each proposal'' (NSF, 1989:II-6).3 The criteria are as follows: Competent performance of the research relates to the capability of the investigator(s), the technical soundness of the proposed approach, the adequacy of the institutional resources available, and the proposer's recent research performance.4 Intrinsic merit of the research is used to assess the likelihood that it will lead to new discoveries or fundamental advances within its field of science and engineering, or will have substantial impact on progress in that field or in other scientific and engineering fields. Utility or relevance of the research is used to assess the likelihood that it can contribute to the achievement of a goal extrinsic or in addition to that of the research field itself, and thereby serve as the basis for new or improved technology or assist in the solution of societal problems. Effect of the research on the infrastructure of science and engineering relates to its potential to contribute to better understanding or improvement of the quality, distribution, or effectiveness of the nation's scientific and engineering research, education, and human resources base. 3   NSF recently made the Grant Policy Manual available electronically on Internet and plans to update it periodically. 4   The phrase, ''and the proposer's recent research performance," which appears in the NSF (1993c) Proposal and Award Manual, is an addition to the criteria adopted by the NSB in 1981. The criteria listed on the back of the evaluation form sent to external reviewers (NSF Form 1) contains a similar addition: "Please include comments on the proposer's recent research performance."

OCR for page 61
Major Award Decisionmaking at the National Science Foundation NSF policies are not very specific about the relative priorities and weightings these different criteria should have. The Proposal and Award Manual (NSF, 1993c:I-3) provides the following discussion: Criteria 1, 2, and 3 constitute an integral set that are applied in a balanced way to all research proposals in accordance with the objectives and content of each proposal. Criterion 1, performance competence, is essential to the evaluation of the quality of every research proposal. The relative weight given criteria 2 and 3 depends on the nature of the proposed research. Criterion 2, intrinsic merit, is emphasized in the evaluation of basic research proposals, while criterion 3, utility or relevance, is emphasized in the evaluation of applied research proposals. Criterion 3 also relates to major goal oriented activities that the Foundation carries out such as those directed at improving the knowledge base underlying science and technology policy, furthering international cooperation in science and engineering, and addressing areas of national need. Criterion 4, effect on the infrastructure of science and engineering, permits the evaluation of research proposals in terms of their potential for improving the scientific and engineering enterprise and its educational activities in ways other than those encompassed by the first three criteria. Included under this criterion are questions relating to women, minorities, and the handicapped; the distribution of resources with respect to institutions and geographical area; stimulation of quality activities in important but underdeveloped fields; and the utilization of interdisciplinary approaches to research in appropriate areas.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation The merit selection criteria have evolved over the years. At one time there were separate criteria for large-scale facilities and centers (see Box 3-1). Review and Selection Criteria for Major Project Awards NSF procedures permit special treatment of major award projects. Criteria in addition to the four basic ones may be applied to specific programs. They are supposed to be listed in program announcements and solicitations; specified in the cover letters sent to peer reviewers; and used by site visitors, panels and NSF staff in summarizing the basis for their recommendations. In addition, NSF still has "special guidelines for organization and research administration" for large projects, which were based on the 1974 criteria for creating or changing institutional structures: Criterion of Need: If a new administrative structure is proposed, evidence should be provided that it is needed to address scientific problems in a manner or on a scale not possible with existing structures. Criterion of Long-Range Potential: Evidence should be provided of a mission with such potential for high scientific productivity over an extended time period, that a significant number of excellent scientists are willing to commit their careers to it. These criteria, which are descended from the longer list of criteria adopted by NSB in 1967, are not now part of any formal planning or review procedures although they are in the Proposal and Award Manual (NSF, 1993c:I-4). They were not used in the 10 case studies, or at least they are not mentioned in the documentation (although the criteria in some cases, for example, for Engineering Research Centers (ERCs) and Science and Technology Centers

OCR for page 61
Major Award Decisionmaking at the National Science Foundation BOX 3-1: Evolution of Merit Review Criteria The criteria used to review research proposals have been elaborated and added to over the years, but the primary emphasis has remained on the promise of the proposed research and the competence of the researcher. The first formal statement of selection criteria was approved by NSB in 1967. That statement reaffirmed NSF's practice of managing its basic research program "in such a way as to permit the development of science along lines dictated by the internal needs of science itself" (NSB, 1967:3). To achieve this goal, NSF looked to fund unsolicited proposals from university-based researchers who met certain criteria: promise of significant scientific results; past record of performance of the investigators; potential scientific impact of the proposed work; degree of novelty, originality, or uniqueness involved; educational value of the proposed research in terms of effects on students; and relevance of the proposed work to potential applications. NSB also recognized the place of national facilities and noneducational research institutions in a "balanced" science program. Examples at that time included the "national research centers" for radio and optical astronomy and atmospheric research, with their very large and expensive facilities and equipment, and large-scale research programs requiring logistical support and coordination among otherwise independent researchers. To determine how to support such large-scale activities, NSF had to make three types of decisions: whether to create a new research institution, augment a facility already in existence, or phase out or transfer programs from a facility that was no longer appropriate. NSB (1967:6) adopted a separate set of criteria for these decisions (although they are more relevant to deciding whether or not to initiate a major project than to deciding among competing proposals). Does the laboratory meet a real scientific need and an opportunity to attack important problems in a way, or on a scale, not otherwise feasible or promising? Is there a broad mission which is sufficiently specific to offer a continuing challenge to the laboratory with consequent assurance of high scientific productivity over an extended period? Have the requirements for continued evolution of capabilities and facilities been given adequate consideration in preliminary planning? Is there and will there continue to be a significant number of first-class scientists (as judged by their peers) who believe deeply in the proposed program and are willing to stake their personal scientific

OCR for page 61
Major Award Decisionmaking at the National Science Foundation reputations on its success, including direct involvement in the program on both a full-time and a long-term basis? Are there convincing arguments that the program objectives can better be achieved through the organization of a new program at a national center than through existing academic or other research institutions? To what degree would the new capability under consideration be unique on a national basis? Will the center or its programs strengthen or detract from related work in the universities? Will the center provide new research opportunities for academic and other scientists? Is there assurance that user scientists will be accepted into the facility primarily on the basis of the scientific merit of their projects? What contributions will the work of the laboratory make to the training of future scientists and/or technologist, including the training of future potential faculty members and industrial investigators as well as students generally? Will the laboratory foster transfer of new basic research techniques into technology and into other areas of science? What impact is the work of the laboratory likely to have on other areas of science? To what degree may tangible social benefits ultimately emerge from the work of the laboratory? The ultimate social benefits of the work of the laboratory are extremely difficult to foresee; hence, significant fundamental research programs should not be rejected because of inability to apply this criterion in a meaningful manner. By the same token, proposals for major programs, which argue their cause on the basis of intrinsically dubious forecasts of social benefits, require the most careful evaluation. In 1974, NSB revised and elaborated the selection criteria it had developed for individual investigator proposals and made them applicable to all proposals, from small research projects to large-scale national facilities. The criteria were grouped in four categories relating to competent performance, intrinsic scientific merit, utility or relevance, and future scientific potential of the nation.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation Only a subset of the planning-type criteria developed in 1967 for the national research facilities and large-scale coordinated research programs was carried forward in the 1974 formulation of the criteria to be used for proposals to "create or modify institutional and organizational structures." They were criteria of need (evidence of real scientific need; an opportunity to address important problems in a way or on a scale not otherwise possible) and criteria of long-range potential (well-defined mission with prospects of high scientific productivity over an extended period; evidence that a significant number of first-class scientists will stake their careers on it; evidence that the new organizational entity will strengthen rather than displace related work in educational institutions). The current review criteria for all proposals were approved by the NSB in 1981. The four basic criteria (described in the text) correspond to the four categories for individual and small-group research projects first identified in the 1974 guidelines. Provision was also made for additional specific criteria to be listed in the program announcement or solicitation for a specific project (NSF, 1992a:I-3). (STCs), did cite the advantages of the center type of organization). Major awards almost always involve criteria in addition to the basic four. They are listed in the solicitation announcements (see Appendix D for recent examples of selection criteria).5 The additional criteria can make decisionmaking more complex and difficult. The criteria may include, for example, managerial capacity, technology transfer, human resource development, participation of underrepre- 5   Additional criteria are not necessarily confined to large facilities and centers that are the subject of major awards. For example, proposals for materials research groups (MRGs) are evaluated according to the four basic criteria plus two key features of an MRG (NSF, 1992b): (5) need for MRG funding mode ("an assessment of the need for a group approach in order to make significant advances, or evidence that advances in the topic cannot be made via the individual investigator approach") and (6) degree of interconnection ("an assessment that the research plan constitutes an integrated and collaborative effort").

OCR for page 61
Major Award Decisionmaking at the National Science Foundation sented groups, and/or nonfederal cost sharing. The relative weighting of these criteria is rarely specified in the solicitation document and is indicated only generally in the letter to reviewers. The problem of priority weighting is further complicated by uncertainty about when in the review process each of the various criteria should be addressed. It also may not be clear which reviewers were selected to apply which criteria, making it difficult to know how to weigh the review comments. Findings and Recommendations on Criteria The panel found that for major awards the review criteria and their relative importance were not always well understood by participants, including proposers and peer reviewers, or by knowledgeable observers. Peer reviewers often confine themselves to those criteria they are most qualified to judge, usually technical merit and capability of the proposer, and it is not always clear who is assessing the other criteria or how they are eventually integrated by staff in the final decision. The panel concluded that NSF should (1) state carefully and clearly the criteria that will be used in the review process, and (2) be explicit about the relative importance of the review criteria . The panel does not advocate the use of strict numerical weightings, which is the approach taken, for example, in procurement contracts. Choosing the best research-related proposal necessarily involves a large degree of expert judgment and some degree of uncertainty; thus, decisionmaking about awards in these cases cannot be reduced to a quantitative algorithm. Nevertheless, without any guidance at all, reviewers are implicitly allowed to derive their own weights for the various criteria when arriving at their overall rating; NSF officials and NSB members also must use their own judgment in evaluating proposals against the criteria. We believe that it is possible and desirable to describe the relative importance of the criteria specified for a major award project to help ensure that reviewers give highest consideration to proposals best able to meet the research goals of the program.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation Recommendation 3: Primacy of Technical Merit Criteria The NSB and NSF should continue to make technical excellence the primary criterion in evaluating the merit of proposals for major awards. To ensure that research funding is used most effectively, no major award should ever be made to a project that is not of very high technical merit. Additional criteria should be used only to choose the best overall proposal from among those whose technical merit is among the most highly rated. Research results are the main goal of a major award, although major awards may also promote the realization of other meritorious goals. Technical merit is the likelihood that a proposed project will achieve the projected research objectives, whatever they may be. Thus, technical merit has been and should remain the primary standard met by all NSF awards, including major awards, because it best assures effective use of the funding devoted to federally supported research. It is especially important in major awards because there is less opportunity for trial and error than in a program of small awards. Emphasizing technical excellence also gives merit review added credibility that should help forestall academic earmarking or other political interference in award decisionmaking at NSF. For smaller awards to individual investigators, technical merit criteria have to do with the originality of the project, soundness of the methodology, and qualifications of the researcher to carry out the proposed research. In major awards, technical merit is more complex because the factors involved in successful implementation of a large-scale, long-term project are more extensive. For example, these factors may include such considerations as the managerial capacity of the principal investigator or investigators, and their institution or institutions, to implement and operate a major research facility or center; physical site characteristics if construction is involved; or the involvement of industry and technology transfer plans if an applied research effort is involved. Major awards usually involve criteria in addition to the technical merit of the projects and the capacity of the proposers to carry them out. These additional merit criteria include (but are not limited to) human resource development potential, participation of women and

OCR for page 61
Major Award Decisionmaking at the National Science Foundation Selection of Mail Reviewers. The virtue of mail review is that each proposal may be reviewed by the experts most knowledgeable about the subfield involved. Ideally, reviewers as a group will be able to evaluate a proposal with respect to all of the selection criteria. Therefore, the balance desired among criteria should affect the selection of reviewers. In selecting reviewers, program directors are also asked to take into account any additional criteria stated in program announcements and solicitations. The award manual lists the "optimum" criteria for selecting reviewers (NSF, 1993c:I-5). For all proposals, some reviewers should be experts in the research subfields involved so they can evaluate the proposals on the criteria of competence, intrinsic merit, and utility. If the proposals involve substantial size or complexity, broad disciplinary or multidisciplinary content, or potential applications to significant national problems, some reviewers should have a broader or more generalized knowledge of the research subfields involved. Some reviewers are supposed to have a broad knowledge of the infrastructure of the national science and engineering enterprise to evaluate proposals for their impact on scientific and engineering education and human resource development, distribution of resources to institutions and geographical areas, and other social goals. Finally, reviewers as a group should reflect a balance among various characteristics such as geography, type of institution, and underrepresented groups. Selection of Panel Members. The advantage of panels is the opportunity they provide for face-to-face interaction among reviewers and between reviewers and the program director. This interaction allows a more detailed discussion of each proposal and how it relates to the overall program. On the other hand, a relatively small group reviewing a variety of proposals is less likely to be as knowledgeable about every subfield involved as mail reviewers can. That is especially true of a standing advisory group. The NSF award manual notes that "it is important, nevertheless, that such groups be structured to provide broad representation and many views on matters under the advisory group's purview" (NSF, 1993c:I-6). The manual offers some general considerations that should be taken into account in achieving balance in advisory groups. They include

OCR for page 61
Major Award Decisionmaking at the National Science Foundation individual qualifications; fields of expertise; public impact—some panel members should represent regions, organizations, or segments of the public directly affected by issues under consideration; academic and nonacademic impact—members should represent small, medium, and large institutions, as well as public and private institutions; representatives from outside the academic community are also desirable in most instances; underrepresented groups; age distribution; and geographic balance. Careful selection of a balanced set of reviewers is especially important for major awards, not only because they are more visible but also because they are more complex. It is critical that the reviewers are—and are perceived to be—qualified, unbiased, and balanced as a group. NSF was criticized in the case of the Earthquake Engineering Research Center, for example, for having too few earthquake researchers and too few Westerners on the review panel (GAO, 1987).6 6   The General Accounting Office (GAO) audit report on the decision concluded that the composition of the panel had met NSF's requirement that there be reviewers with both special and general knowledge of the scientific subfields involved in the proposal. Although only one of the seven panelists was a recognized earthquake engineer, three others had some previous experience in earthquake engineering. Most were chosen for their experience in managing large research efforts at a university or in industry. GAO also had the technical sections of the leading proposals reviewed by four nationally known earthquake engineering experts; according to GAO, their assessments of each proposal's strengths and weaknesses concurred with those of the NSF-appointed panel (although their comments were more numerous and more detailed) (GAO, 1987:20).

OCR for page 61
Major Award Decisionmaking at the National Science Foundation Policies and Procedures for Dealing with Bias and Conflict of Interest Merit review at NSF has been criticized many times as an "old-boy network" that concentrates awards in a relatively small number of institutions in a few states. Various policies and procedures have been added over the years to ensure that the system is open and fair. NSF program directors are not supposed to use anyone as a reviewer who (1) would be directly involved in the project as a collaborator or consultant, (2) is from the same institution as the applicant, or (3) has a family relationship with the applicant. The program director is also required to document the existence of interests, affiliations, and relationships that might affect a reviewer's evaluation of a proposal. Reviewers are explicitly asked to describe relationships that may be or appear to be a conflict of interest. The program officer is required to indicate in the file how such conflicts or potential biases were handled in making the award decision.7 Reviews that contain intemperate personal attacks or other indications of bias are not supposed to be used in the selection process. Other policies and procedures to ensure fairness are described in Box 3-2. Award Decisionmaking Among the key actors in the decision process are NSF program directors. They make initial recommendations for the review format and specific reviewers to be used for each proposal or competitive set of proposals, which division directors and assistant directors review and approve. The program directors recommend whether or not to fund a proposal, as well as the amount and duration of support. They base the recommendation on the comments of reviewers, but they 7   NSF also has special procedures for dealing with proposals from individuals who are prospective, current, or former employees, NSB members, or Intergovernmental Personnel Act assignees at NSF.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation Box 3-2: Other NSF Policies and Procedures to Ensure Fairness in Merit Review Since the mid-1970s, NSF has taken a series of steps to open up the entire award decisionmaking process, including peer review. The intent of these changes was spelled out in 1977 (NSB, 1977:iii): . . . the peer review process should continue to preserve the traditional benefits of peer evaluation of intrinsic scientific merit. At the same time, it is essential that the research community perceive the peer review process to be fair, and equitable as well as accessible to all qualified persons—both as research applicants and as reviewers. Preserving the traditional benefits of peer review has meant maintaining the anonymity of reviewers, because NSF believes reviewers will not be candid if applicants know who they are. NSF has, however, acted to achieve the following: Provide applicants with (a) verbatim mail review remarks without identifying the reviewers, (b) the panel summary if a panel was used, and (c) site visit reports if a site visit was made.1 Let applicants suggest the names of individuals who should not review their proposal. Give every applicant a description of the contextual factors considered in making a decision and, in the case of declinations, an explanation of the basis for the decision (since 1990). Contextual factors are those beyond reviewer comments and ratings that staff take into account when making the award decision. Such factors include the total amount of funds available to the program for new and renewal proposals, the number of proposals the program expects to review that year and the approximate percentage that will be funded, the need to maintain balance among research subfields, and the availability of funding from other agencies. Allow the principal investigator (PI) of a declined proposal to ask the program officer or division director for additional information; if the PI still believes the decision was not fair or reasonable, he or she may write to the assistant director requesting a formal reconsideration. If the declination is upheld, the PI's institution may appeal to the NSF's deputy

OCR for page 61
Major Award Decisionmaking at the National Science Foundation director for a second reconsideration.2 Maintain ongoing monitoring and analyses of the peer review system. The director of NSF has submitted an annual report on peer review to NSB since 1977, with basic statistics on the number of reviews, approval rates, and appeals, as well as a listing of special waivers from peer review. The NSF Office of Inspector General audits a random sample of proposal jackets each year to see how well NSF is following its own policies and procedures. External peer reviews of program operations by Committees of Visitors evaluate each program every three years by reviewing a sample of proposal jackets. 1 The actual peer reviews have been included in the proposed award packages going to NSB since March 1976 (Windus, 1984:17). 2 In 1990, the grounds for reconsideration were expanded from procedural due process to include substance (NSF, 1990d). Requests for reconsiderations are few, therefore, and reversals are rare. In FY 1991, for example, there were 39 first-level and three second-level reconsideration actions. NSF's original decision was upheld in all but one case (Massey, 1992). Most PIs prefer to revise their proposals or submit them somewhere else rather than appeal. may take other considerations into account, such as the balance among research topics or the amount available in the program budget. In the case of individual investigator grants, recommendations by program directors can be approved at the division level. If the proposal is large enough, as in the case of major awards, the award decision is reviewed by the appropriate assistant director and at the NSF director's level before being sent for approval to NSB with a memorandum from the director. Although external reviewers are asked to evaluate a proposal's impact on geographic and institutional balance and on participation of women and minorities, the program director and other NSF staff are often the only ones in a position to make these judgments in the case of a program of small grants. Facing proposals of nearly equal merit,

OCR for page 61
Major Award Decisionmaking at the National Science Foundation the program director may give greater consideration to scientific program balance and distributional factors (geographic, institutional, race, sex, etc.). This is especially true in small grant programs as they near the end of their available funds, when program directors may achieve greater balance among secondary criteria by selectively choosing slightly lower-rated proposals.8 In major award cases, however, secondary criteria cannot be balanced by letting NSF staff select some lower-rated proposals at the margin, after funding the best proposals technically. For major projects, only a few awards, or often just one, is made. This means that all the criteria, even though they are more numerous and complex than for small grants, must be met in one or a few proposals. Given the importance and complexity of major awards, the role of outside peer evaluation in weighing and balancing the various criteria, both technical and nontechnical, is especially important throughout the process. The recommendations in this report thus aim to create a merit review system in which outside peer review not only plays its usual major role in evaluating technical excellence but also plays the major role in evaluating secondary criteria and identifying the most meritorious proposal based on all criteria. Findings and Recommendations on Review Procedures The panel found that the review processes for major awards vary greatly. Some involve elaborate and well-thought-out procedures for applying different types of criteria and expertise at different levels 8   In practice, proposals are rarely put in strict rank order because the proposal evaluation process is too imprecise; rather they are informally grouped. Even so, NSF does not usually have the budget to fund all the proposals in the top or excellent group. According to NSF officials, that means that using additional criteria to choose among them at the margin does not lower the average quality of the research supported.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation and stages of the review. Some are simply handled as investigator-initiated proposals. At one time, NSB had criteria and procedures for large-scale facilities and centers that were different from those for the small-project grants. Over time, however, the procedures for the latter came to be applied to all awards, regardless of their nature, size, or complexity. The panel concluded that the major awards are different enough from the typical small grant, and alike enough because of their size and impact, to warrant standard procedures of their own. The key is to structure the review process so that the various types of criteria are applied by appropriately qualified reviewers. Recommendation 6: A Two-Phase Merit Review Process For major awards, the peer review part of the merit review process should be conducted in two phases. The first phase would be a strictly technical review; to help assure the primacy of technical merit, only those proposals judged to be technically superior would be forwarded to the second phase for any further consideration. In the second phase, the additional merit criteria would be weighed and balanced with the technical criteria by a more broadly constituted group of reviewers. This second-phase panel would recommend the proposal (or proposals) best meeting the full set of criteria. If the proposal judged to have the highest merit overall is not the one ranked highest in the first phase of review for technical merit, the second-phase panel must explain its recommendation fully. If the top-ranked technical proposal is subsequently not recommended by NSF staff, the chair of the first-phase panel or another member of that panel should present the case for it at the NSB level. This two-phase methodology, which NSF and NSB have already employed successfully for some major award competitions, will help ensure that three important goals of the review process are met.

OCR for page 61
Major Award Decisionmaking at the National Science Foundation First, NSB will approve awards only to projects that meet the highest technical criteria and thus have the greatest promise of achieving their research-related objectives. At the same time, however, it will ensure that other important but secondary merit criteria are evaluated by qualified reviewers and appropriately weighed in the final ranking. Third, the process will better clarify the basis for the final decision and the respective input of peer reviewers and agency staff, which in turn will make the decisionmaking process more acceptable and accountable to both the research community and the broader public to which NSF must answer. Procedures. If a large number of proposals is expected, NSF should use preproposals to narrow the field. The technical merit of the preproposals should be peer reviewed, either through the use of mail reviews or perhaps through a meeting of a Phase 1 review panel. Phase 1 is the technical review by relevant experts. It can be done in stages or steps, including mail reviews, site visits, and panels or subpanels. In the final step of Phase 1, a panel of technically qualified reviewers evaluates all the proposals for technical merit (with input from the mail reviews, site visits, etc.), sorting them into no more than four or five categories of excellence, ranging from most qualified to unqualified. The panel should strive to produce a priority ranking of the proposals if at all possible. If the ranking by quality is reasonably clear, the panel should forward the proposals in rank order. If there are a number of excellent proposals, each with its strengths and weaknesses, the panel may forward a group of top-ranked proposals judged to be essentially equivalent in technical quality. In this case, the panel should comment carefully on the strengths and weaknesses of each proposal. In research center competitions where there are numerous proposals, it is relatively easy to forward a set of top-rated proposals to the second phase, explicitly ranked in order of technical merit. In fact, NSF already uses a form of the two-phase merit review process for the Engineering Research Centers, Science and Technology Centers, and Supercomputer Centers. The more difficult cases are

OCR for page 61
Major Award Decisionmaking at the National Science Foundation competitions in which there are only a few proposals. We expect on the basis of the case studies, however, that there will usually, if not always, be at least several strong candidates in the second phase of a major award competition. Phase 2 is a review of all other merit criteria (e.g., human resource development and equal opportunity, geographic or institutional distribution, technological relevance, industrial involvement, outreach and technology transfer activities). To complete this phase, the subgroup of proposals ranked ''excellent'' or "most qualified" in Phase 1 panel is considered by a new, Phase 2 panel of reviewers. This new panel should be qualified to evaluate proposals on both the additional criteria and the technical criteria identified for that project (although it should not redo the technical ranking). Additional review steps could occur during this phase, such as special subpanels to consider certain criteria (e.g., business people and economists to evaluate proposals for their potential contributions to economic competitiveness), additional site visits, or presentations by finalists. The Phase 2 panel would recommend the proposal (proposals if there are multiple awards) that best meet the full set of criteria. If the recommendation is for a proposal that was not the most highly ranked technically in Phase 1, the Phase 2 panel would have to explain explicitly the reasons for its action. In that case, or if NSF staff decides to recommend a proposal to NSB that was not ranked highest by the Phase 1 review panel, NSB or at least its Committee on Programs and Plans (CCP) should hear a presentation in favor of the top-ranked technical proposal from the Phase 1 panel chair or another advocate from that panel. He or she should present to CPP the case for the best Phase 1 proposal and help CPP understand the research-related benefits that might be forgone if another proposal were chosen. Currently, CPP-NSB hears only the case in support of the proposal recommended by NSF and thus cannot make a decision on the basis of a comparative analysis of the top few proposals. We believe that NSB would be in a better position to make a more informed decision if it more clearly understood the trade-offs among merit criteria involved in choosing the winning proposal and that this

OCR for page 61
Major Award Decisionmaking at the National Science Foundation understanding is particularly important when the successful proposal is not the one with the highest technical merit according to the external peer reviewers. Reviewer Roles. Reviewers in Phase 1 should be chosen so that, as a group they will be able to evaluate and rank the proposals according to technical merit (as measured by the technical criteria specified in the solicitation). In addition to providing an overall summary rating based on technical criteria, Phase 1 reviewers should be encouraged to comment, if they wish, on the other merit review criteria. In this case, they should give two ratings, one based on purely technical criteria and the other indicating how they might change their overall ratings in light of additional nontechnical criteria (at the same time, indicating their competence to apply the other criteria). So far, the process is not very different from that employed in many cases, except that the procedure recommended here calls for the use of a panel in every case to integrate the reviews rather than rely on mail reviewers and staff alone. The panel is also charged with deriving an overall ranking of proposals according to technical merit only, without taking other criteria into account. A new panel is required in Phase 2, although mail reviewers, subpanels, and site visitors may also be used. The Phase 2 panel should be constituted to apply the full set of merit criteria, technical and nontechnical, because this group has the task of identifying the best overall proposal. It should be noted that the eventual winning proposal has already had to achieve a high threshold of technical excellence; thus, no award would go to a project of low technical quality just because it ranked high on other merits. A winning proposal must rank high in both areas, technical and nontechnical. At least some Phase 2 reviewers should be carried over from the first-phase panel. This continuity is helpful because the primary and secondary criteria usually interact in ways that need to be evaluated before reaching a final ranking. For example, one of two proposals basically equal in scientific promise might better involve students

OCR for page 61
Major Award Decisionmaking at the National Science Foundation because of the way the research is organized or where it is done. That would affect the rating positively when education is an important secondary criterion. Phase 2 panels should be asked to provide a careful qualitative discussion of how they balanced the nontechnical criteria against the technical ones. We believe that in the typical case the discussion will revolve around whether proposals ranked second or third on the technical merit scale are the most meritorious overall and should be funded. In some cases, several proposals may have substantially equal technical merit. In these cases, important secondary criteria may play a tiebreaker role. Even here, however, each proposal will have its individual technical strengths and weaknesses, and it will be helpful to have carryover members from the Phase 1 panel familiar with the proposals as participants in the discussion of the balancing of technical and other merits. As already mentioned in Recommendation 3, if the Phase 2 panel decides to bypass the proposal or proposals ranked highest technically, it should provide a careful written explanation of the justification for doing so.