Appendix E
Examples of Criteria Used to Prioritize or Select Research Projects

This appendix provides excerpts from documents explaining project prioritization and selection criteria used by NSF, other agencies, and other countries. The majority of the text of this appendix consists of quoted material. The sources for these quotes are printed in italics, and explanatory material is enclosed in brackets.

NATIONAL SCIENCE FOUNDATION (1)

[Criteria for selecting projects for MREFC support, from Rita Colwell’s testimony before the House Committee on Science, Subcommittee on Research on September 6, 2001]

  • Intellectual merit

  • Broader impacts

  • Importance to science and engineering

  • Cost-benefit and risk analysis

  • Readiness to implement and manage

  • Appropriateness for NSF

  • Balance across fields and disciplines

  • Synergy with other large facilities supported by NSF, other agencies, and other nations



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Appendix E Examples of Criteria Used to Prioritize or Select Research Projects This appendix provides excerpts from documents explaining project prioritization and selection criteria used by NSF, other agencies, and other countries. The majority of the text of this appendix consists of quoted material. The sources for these quotes are printed in italics, and explanatory material is enclosed in brackets. NATIONAL SCIENCE FOUNDATION (1) [Criteria for selecting projects for MREFC support, from Rita Colwell’s testimony before the House Committee on Science, Subcommittee on Research on September 6, 2001] Intellectual merit Broader impacts Importance to science and engineering Cost-benefit and risk analysis Readiness to implement and manage Appropriateness for NSF Balance across fields and disciplines Synergy with other large facilities supported by NSF, other agencies, and other nations

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation NATIONAL SCIENCE FOUNDATION (2) [Criteria for selection of projects for MREFC support, from Facilities Management and Oversight Guide, November 8, 2002] Exceptional S&E [Science and Engineering] opportunity to enable frontier research and education Transformational regarding S&E impact High priority within relevant S&E communities Accessible to appropriately broad user community Partnership possibilities exploited Technical feasibility and risks addressed thoroughly High state of readiness NATIONAL SCIENCE FOUNDATION (3) [Criteria for selection of projects for future MREFC support, from Answers Provided by NSF to Questions from the House Science Committee Hearing on February 13, 2003] Significance of the opportunity to enable frontier research and education Degree of support within relevant S&E communities Readiness of project, in terms of feasibility, engineering and cost-effectiveness, interagency and international partnerships, and management NATIONAL SCIENCE FOUNDATION (4) [Criteria for placing highly rated projects in priority order, from Answers Provided by NSF to Questions from the House Science Committee Hearing on February 13, 2003] How “transformative” is the project? Will it change the way research is conducted or change fundamental S&E concepts/research frontiers? How great are the benefits of the project? How many researchers, educators and students will it enable? Does it broadly serve many disciplines? How pressing is the need? Is there a window of opportunity? Are there interagency and international commitments that must be met? Additionally, the director must weigh the impact of a proposed facility on the balance between scientific fields, the importance of the project with respect to national priorities, and possible societal benefits.

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation NATIONAL SCIENCE BOARD [From NSB Resolution 02-191 adopted on November 21, 2002, Setting Priorities for Major Research Facilities] Once construction for an approved and prioritized project commences, highest priority is given to moving that project forward through multiple years of construction in a cost-effective way, as determined by sound engineering and as long as progress is appropriate. It is most cost-effective to complete initiated projects in a timely way, rather than to commence new projects at the cost of stretching out in-progress construction. New candidate projects will be considered from the point of view of broadly serving the many disciplines supported by NSF. Multiple projects for a single discipline, or for closely related disciplines, will be ordered based on a judgment of the contribution that they will make toward the advancement of research in those related fields. Community judgment on this matter is considered. Projects will be authorized close to the time that funding requests are expected to be made. International and interagency commitments are considered in setting priorities among projects. The above are guidelines. Each facility consideration involves many complex issues. The Board will consider all relevant matters, and could deviate from these guidelines, given sound reasons to do so. HOUSE SCIENCE COMMITTEE [Priority setting, from H.R. 4664, Section 14A.3, the authorization bill for doubling NSF’s budget] Scientific merit Broad societal need and probable impact Consideration of the results of formal prioritization efforts by the scientific community Readiness of plans for construction and operation The applicant’s management and administrative capacity of large research facilities International and interagency commitments The order in which projects were approved by the [National Science] Board for inclusion in a future budget request

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation COSEPUP REPORT ON EVALUATING FEDERAL RESEARCH (1999) [Criteria for performance evaluation from Evaluating Federal Research Programs: Research and the Government Performance and Results Act, National Academies Press, 1999] For basic research programs, agencies should measure quality, relevance, and leadership. What is the quality of the research program—for example, how good is the proposed research work compared with other work being conducted in the field? Is the proposed research focused on subjects most relevant to the agency mission? Is the proposed research at the forefront of scientific and technological knowledge? OECD FORUM ON MEGASCIENCE POLICY ISSUES [Criteria for assigning priorities for major national research facilities, derived from Australian Science and Technology Council Reports by W. J. McG. Tegart, 1992] Benefits to Science and Technology Scientific objectives and their significance Does the proposal develop an area of scientific or engineering research of great importance and which is at the leading edge of international research? What are the key scientific questions that can be answered by having access to the proposed national facility? Why are the answers to the questions significant for the national science and technology? Will the proposed national facilities be made available to outside researchers subject to independent peer reviews? Established need Is the case for the proposed national facilities in terms of current national priorities? Does the proposal involve a major source of expenditure on a piece or pieces of physical equipment of a scale that it could be developed incrementally or funded by an institution or consortium of institutions without serious disruption to other commitments of equal or higher priority? Is there a community of outstanding scientists and technologists committed to the success of the proposed national facility?

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Unique characteristics Are there characteristics of the proposed facility that are uniquely appropriate for the nation? Degree of impact What impact will the proposed national facility have on fostering interdisciplinary research? Will the proposed national facility provide new opportunities for doctoral and post-doctoral training in research? Will the proposed facility contribute to public pride and the prestige of the nation’s science and technology? International characteristics Will the proposed national facility encourage international scientific collaboration by attracting researchers from overseas to spend time in the nation? Could the proposed national facility be located with advantage overseas in partnership with one or more other countries? Would the proposed national facility attract international partners? Benefits to the Nation Industry objectives and their significance Will the construction of the proposed national facility provide a technological stimulus to national industry? Will the proposed national facility provide unique services of benefit to national industry? Could the proposed national facility lead to better linkages between academic and research institutions and industry? Will the research output from the proposed national facility foster the development of new national enterprises? What contribution will the proposed national facility make to enhancing the skills base and training of national technology? Social objectives and their significance Is the proposed national facility of high national priority for the advancement of knowledge, economic growth, health, welfare, or national security? Does the proposed national facility contribute to a better understanding and management of our environment? Will the proposed national facility lead an improved understanding and appreciation by the national community of the accomplishments of science and technology? International standing Will the proposed national facility project and enhance the nation’s image as a technologically advanced nation? Will the nation’s position in international negotiations be strengthened as a result of the proposed national facility?

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation NIH STRATEGY FOR SETTING RESEARCH PRIORITIES [From NIH Director Harold Varmus’s testimony before the House Appropriations Subcommittee on June 10, 1997] Exploit recent discoveries, such as the isolation of new genes for human diseases Encourage studies of diseases that have been relatively neglected, poorly controlled, or recently made more accessible to scientific study Strengthen research technologies, such as computer science, imaging devices, neuroscience, or gene mapping, applicable to a broad range of disciplines and diseases. THE NATIONAL RESEARCH PRIORITIES TASKFORCE [Discussion excerpted from a report commissioned by the Australian government, Developing National Research Priorities, 2002] Ultimately, for national priorities to be worthwhile they must have three important characteristics: An increased research effort must be capable of delivering a measurable and significant positive impact on the objective underlying the priority Australia must be able to build the capacity needed to achieve that impact Australia must be able to capture the benefits of that research (either through its commercialization or application). Based on this, the government has developed three criteria for assessing nominated priorities. These criteria will be used by the expert advisory committee in developing the short-list of priorities for government consideration. The scope for Commonwealth Government investment in research to have a measurable and significant positive impact, by: achieving a ‘critical mass’ through specific support and/or coordination and collaboration at the national level addressing uniquely Australian needs and challenges arising from our geography, climate, bioresources, economy, way of life and/or culture. The scope for Australia to build quickly the capacity to achieve that impact, taking into account: existing expertise, experience and technological capacities or whether such capacities can be reasonably acquired

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation the availability, quality and scale of necessary research infrastructure the strategic research priorities of other nations and the potential benefits of international collaborations the overall magnitude of the investment required to make an impact. The scope for Australia to capture the benefits of the research, through the potential of the research to: achieve commercially or socially relevant outcomes over the cycle of the priorities regime enhance significantly Australia’s overall innovation capacity by the broadening of the knowledge base, and fostering acquisition of skills and understanding of emerging ‘hot’ sciences. DECISION CRITERIA FOR EVALUATING AGRICULTURAL RESEARCH PRIORITIES [Report from Northeastern Regional Association of State Agricultural Experiment Station Directors, 1999] Selecting the topics for future research investments in agricultural science requires a rationale that is defensible, and a process that is, for the public sector, completely transparent. This document sets out a system of points-to-consider that could make up a new approach to setting criteria for decision making on resource allocations. The fundamental goal for setting agreed-to-decision criteria for allocating public sector agricultural research should be to gain the largest possible returns on research investment. This then requires consideration of four factors: There must be a congruence between the dimensions of the intended topic and the allocation of research resources. Larger impacts can be expected by investing in topics that already have a large base in agriculture, forestry or rural development (e.g., wheat, hardwoods, community services), rather than trying to start from a smaller base (e.g., edible amaranth) or a regionally distributed environmental issue rather than local or state. This requires that some congruence analysis be done. The allocation of resources needs to directly reflect the needs of the intended stakeholders and customers. This requires us to be listening to our customers. Judgments are needed on what is feasible to accomplish through agricultural research, and this needs to be grounded in the best possible science. This in turn mandates some evaluation of the scientific potential of proposed research approaches by knowledgeable scientists.

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Projections of expected benefits are needed to permit informed choices of alternative allocations. These must be done with a set of assumptions that are understood by the participants and the claimants to the system, and are broadly agreed upon. Ex ante estimations need to go beyond economic consequences to the non-economic benefits and consequences of technologies (i.e., social, environmental). Assigning premium or discount coefficients to economic projections can do this. SYSTEMATIC RESEARCH PRIORITY CRITERIA [From UK Systematics Forum on Priorities in Systematic (Biology) Research and Training, 1995] Scientific excellence. The scientific excellence of the proposed study, the individual or team who will undertake the work and the collections available for study. Relevance. The relevance of the study to a significant and clearly identified user community. The relevance may be scientific, cultural, historical or economic and will vary in relation to the differing interests of different audiences or user communities. Enhancing scientific coverage. Filling gaps in knowledge about a group of organisms and making information available. Gaps in knowledge are not a sufficient criterion for establishing high priority, unless criterion 2 also applies and the results are of immediate relevance. Scale of relevance. Levels of priority may differ at different geographical levels (global, regional, national or local). Urgency. The level of priority given to a proposed study may reflect degree of urgency of the work determined in relation to the endemicity of the [proposing] group. Feasibility. The scientific feasibility of completing the proposed task in the timescale and with the resources proposed. UK DEPARTMENT OF HEALTH [Prioritization of the Policy Research Programme in collaboration with the R&D Directorate] Ministerial priority and relevance to the goals, aims and objectives of the Department of Health Size and importance of the problem to be addressed in terms of actual or potential burden of disease and social condition Well-defined plans for introducing research results into current policy activity or the formulation of future policy

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Timeliness Feasibility of research Likely return on the investment in research Appropriateness and availability of other research budgets GERMAN SCIENCE COUNCIL [From the German Wissenschaftsrat’s Large-scale Facilities for Basic Scientific Research, Working Group Report, 2003] Theses on the significance of large-scale facilities for basic scientific research: Scientific and technical developments for industry and society often start with the findings of basic research. The opening-up or development of totally new areas of research is closely related to the availability of specific new facilities. Large-scale facilities should stem from a broad initiative of scientific users with equal rights. It is important to make very sure that future generations of scientists receive adequate, forward-looking training using the large-scale facilities. With large-scale facilities on the scale with which we are concerned here, there must be long-term scientific visions, and the prerequisites for technical innovations must be given. Including foreign partners in project preparation at an early stage, letting them participate in decision making and be responsible for their own scientific or technological contributions are prerequisites. In order to avoid the duplication of research infrastructures, which would be detrimental to the effective capacity usage of large-scale facilities, there should be no comparable rival projects at the national or European level that are already in the realization phase. Large-scale facilities must be centrally incorporated into the strategic planning and research program of the institution(s) in charge and must be a core element of the spectrum of tasks. In the field of apparatus and equipment, which includes large-scale facilities, cooperation between places of further education and major research establishments should be a matter of course. Access to the facilities must be kept open for scientists from universities in a systematic and pragmatic way, so as to allow them to carry out research projects. Criteria and specific considerations: Expert Panel The probability of fundamentally new insights or the possibilities of decisive scientific advances which could only be achieved with the large-scale facility.

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation The large-scale facility’s technical feasibility and the degree of technical innovation. The scientific and technical competence of the institutions involved. The already existing or anticipated acceptance of the (potential) users from immediately relevant and from neighboring areas of expertise. The fulfillment of various objectives of importance for research (transfer, international perspectives, promoting young scientists). Policy Assessment Panel Scientific potential of the research program. Fulfillment of science and technology policy goals as formulated in the theses. Degree of maturity of the technical concept and, linked to it, the possible timeframe for implementing the individual facilities. The context of further national and international scientific development of the research fields they belong to and assess their interaction with other disciplines. [See Appendix D for additional information on this process.] UK OFFICE OF SCIENCE AND TECHNOLOGY [From UK Office of Science and Technology’s Large Facilities Strategic Roadmap Report, 2003] [The UK uses a “roadmap” portfolio approach to select projects for solicitation and a Gateway screening process that requires projects to meet certain conditions before moving forward. The concept behind the Gateway process is that all large capital investments should be managed as discrete projects and should be subject to review and independent scrutiny at all key stages in their life cycle. The Gateway Process is based on well-proven techniques that lead to more effective delivery of benefits together with more predictable costs and outcomes. The process considers the project at critical points in its development. These critical points are identified as Gateways. There are six Gateways during the lifecycle of a project, four before contract award and two looking at service implementation and confirmation of the operational benefits. The Process emphasizes early review for maximum added value.] The four-stage Gateway process: Strategic assessment. Initial assessment of the strength of the science requirement for potential projects. Scientific peer review must be a key element of this first review.

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Importance [depth] of science knowledge to be delivered by project. Breadth of science knowledge that will benefit from investment. Match with international positioning of UK science. Strength of opportunity for training [links to number of users]. Contribution to/from UK technology/industry base. Opportunity for spin-off and exploitation. Business justification. Justification and robustness of the business case. Procurement strategy. Confirm procurement strategy, project plan, etc. Readiness for service. Confirm contract decisions and let contracts. [See Appendix D for additional information on this process.]

OCR for page 183
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation This page intentionally left blank.