National Academies Press: OpenBook

Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation (2004)

Chapter: Appendix E: Examples of Criteria Used to Prioritize or Select

« Previous: Appendix D: Approval Processes in Other Agencies and Other Countries
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

Appendix E
Examples of Criteria Used to Prioritize or Select Research Projects

This appendix provides excerpts from documents explaining project prioritization and selection criteria used by NSF, other agencies, and other countries. The majority of the text of this appendix consists of quoted material. The sources for these quotes are printed in italics, and explanatory material is enclosed in brackets.

NATIONAL SCIENCE FOUNDATION (1)

[Criteria for selecting projects for MREFC support, from Rita Colwell’s testimony before the House Committee on Science, Subcommittee on Research on September 6, 2001]

  • Intellectual merit

  • Broader impacts

  • Importance to science and engineering

  • Cost-benefit and risk analysis

  • Readiness to implement and manage

  • Appropriateness for NSF

  • Balance across fields and disciplines

  • Synergy with other large facilities supported by NSF, other agencies, and other nations

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

NATIONAL SCIENCE FOUNDATION (2)

[Criteria for selection of projects for MREFC support, from Facilities Management and Oversight Guide, November 8, 2002]

  • Exceptional S&E [Science and Engineering] opportunity to enable frontier research and education

  • Transformational regarding S&E impact

  • High priority within relevant S&E communities

  • Accessible to appropriately broad user community

  • Partnership possibilities exploited

  • Technical feasibility and risks addressed thoroughly

  • High state of readiness

NATIONAL SCIENCE FOUNDATION (3)

[Criteria for selection of projects for future MREFC support, from Answers Provided by NSF to Questions from the House Science Committee Hearing on February 13, 2003]

  • Significance of the opportunity to enable frontier research and education

  • Degree of support within relevant S&E communities

  • Readiness of project, in terms of feasibility, engineering and cost-effectiveness, interagency and international partnerships, and management

NATIONAL SCIENCE FOUNDATION (4)

[Criteria for placing highly rated projects in priority order, from Answers Provided by NSF to Questions from the House Science Committee Hearing on February 13, 2003]

  • How “transformative” is the project? Will it change the way research is conducted or change fundamental S&E concepts/research frontiers?

  • How great are the benefits of the project? How many researchers, educators and students will it enable? Does it broadly serve many disciplines?

  • How pressing is the need? Is there a window of opportunity? Are there interagency and international commitments that must be met?

  • Additionally, the director must weigh the impact of a proposed facility on the balance between scientific fields, the importance of the project with respect to national priorities, and possible societal benefits.

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

NATIONAL SCIENCE BOARD

[From NSB Resolution 02-191 adopted on November 21, 2002, Setting Priorities for Major Research Facilities]

  • Once construction for an approved and prioritized project commences, highest priority is given to moving that project forward through multiple years of construction in a cost-effective way, as determined by sound engineering and as long as progress is appropriate. It is most cost-effective to complete initiated projects in a timely way, rather than to commence new projects at the cost of stretching out in-progress construction.

  • New candidate projects will be considered from the point of view of broadly serving the many disciplines supported by NSF.

  • Multiple projects for a single discipline, or for closely related disciplines, will be ordered based on a judgment of the contribution that they will make toward the advancement of research in those related fields. Community judgment on this matter is considered.

  • Projects will be authorized close to the time that funding requests are expected to be made.

  • International and interagency commitments are considered in setting priorities among projects.

  • The above are guidelines. Each facility consideration involves many complex issues. The Board will consider all relevant matters, and could deviate from these guidelines, given sound reasons to do so.

HOUSE SCIENCE COMMITTEE

[Priority setting, from H.R. 4664, Section 14A.3, the authorization bill for doubling NSF’s budget]

  • Scientific merit

  • Broad societal need and probable impact

  • Consideration of the results of formal prioritization efforts by the scientific community

  • Readiness of plans for construction and operation

  • The applicant’s management and administrative capacity of large research facilities

  • International and interagency commitments

  • The order in which projects were approved by the [National Science] Board for inclusion in a future budget request

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

COSEPUP REPORT ON EVALUATING FEDERAL RESEARCH (1999)

[Criteria for performance evaluation from Evaluating Federal Research Programs: Research and the Government Performance and Results Act, National Academies Press, 1999]

For basic research programs, agencies should measure quality, relevance, and leadership.

  • What is the quality of the research program—for example, how good is the proposed research work compared with other work being conducted in the field?

  • Is the proposed research focused on subjects most relevant to the agency mission?

  • Is the proposed research at the forefront of scientific and technological knowledge?

OECD FORUM ON MEGASCIENCE POLICY ISSUES

[Criteria for assigning priorities for major national research facilities, derived from Australian Science and Technology Council Reports by W. J. McG. Tegart, 1992]

Benefits to Science and Technology

Scientific objectives and their significance

  • Does the proposal develop an area of scientific or engineering research of great importance and which is at the leading edge of international research?

  • What are the key scientific questions that can be answered by having access to the proposed national facility?

  • Why are the answers to the questions significant for the national science and technology?

  • Will the proposed national facilities be made available to outside researchers subject to independent peer reviews? Established need

  • Is the case for the proposed national facilities in terms of current national priorities?

  • Does the proposal involve a major source of expenditure on a piece or pieces of physical equipment of a scale that it could be developed incrementally or funded by an institution or consortium of institutions without serious disruption to other commitments of equal or higher priority?

  • Is there a community of outstanding scientists and technologists committed to the success of the proposed national facility?

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

Unique characteristics

  • Are there characteristics of the proposed facility that are uniquely appropriate for the nation?

Degree of impact

  • What impact will the proposed national facility have on fostering interdisciplinary research?

  • Will the proposed national facility provide new opportunities for doctoral and post-doctoral training in research?

  • Will the proposed facility contribute to public pride and the prestige of the nation’s science and technology?

International characteristics

  • Will the proposed national facility encourage international scientific collaboration by attracting researchers from overseas to spend time in the nation?

  • Could the proposed national facility be located with advantage overseas in partnership with one or more other countries?

  • Would the proposed national facility attract international partners?

Benefits to the Nation

Industry objectives and their significance

  • Will the construction of the proposed national facility provide a technological stimulus to national industry?

  • Will the proposed national facility provide unique services of benefit to national industry?

  • Could the proposed national facility lead to better linkages between academic and research institutions and industry?

  • Will the research output from the proposed national facility foster the development of new national enterprises?

  • What contribution will the proposed national facility make to enhancing the skills base and training of national technology?

Social objectives and their significance

  • Is the proposed national facility of high national priority for the advancement of knowledge, economic growth, health, welfare, or national security?

  • Does the proposed national facility contribute to a better understanding and management of our environment?

  • Will the proposed national facility lead an improved understanding and appreciation by the national community of the accomplishments of science and technology?

International standing

  • Will the proposed national facility project and enhance the nation’s image as a technologically advanced nation?

  • Will the nation’s position in international negotiations be strengthened as a result of the proposed national facility?

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

NIH STRATEGY FOR SETTING RESEARCH PRIORITIES

[From NIH Director Harold Varmus’s testimony before the House Appropriations Subcommittee on June 10, 1997]

  • Exploit recent discoveries, such as the isolation of new genes for human diseases

  • Encourage studies of diseases that have been relatively neglected, poorly controlled, or recently made more accessible to scientific study

  • Strengthen research technologies, such as computer science, imaging devices, neuroscience, or gene mapping, applicable to a broad range of disciplines and diseases.

THE NATIONAL RESEARCH PRIORITIES TASKFORCE

[Discussion excerpted from a report commissioned by the Australian government, Developing National Research Priorities, 2002]

Ultimately, for national priorities to be worthwhile they must have three important characteristics:

  • An increased research effort must be capable of delivering a measurable and significant positive impact on the objective underlying the priority

  • Australia must be able to build the capacity needed to achieve that impact

  • Australia must be able to capture the benefits of that research (either through its commercialization or application).

Based on this, the government has developed three criteria for assessing nominated priorities. These criteria will be used by the expert advisory committee in developing the short-list of priorities for government consideration.

  • The scope for Commonwealth Government investment in research to have a measurable and significant positive impact, by:

    • achieving a ‘critical mass’ through specific support and/or coordination and collaboration at the national level

    • addressing uniquely Australian needs and challenges arising from our geography, climate, bioresources, economy, way of life and/or culture.

    • The scope for Australia to build quickly the capacity to achieve that impact, taking into account:

    • existing expertise, experience and technological capacities or whether such capacities can be reasonably acquired

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
  • the availability, quality and scale of necessary research infrastructure

  • the strategic research priorities of other nations and the potential benefits of international collaborations

  • the overall magnitude of the investment required to make an impact.

  • The scope for Australia to capture the benefits of the research, through the potential of the research to:

    • achieve commercially or socially relevant outcomes over the cycle of the priorities regime

    • enhance significantly Australia’s overall innovation capacity by the broadening of the knowledge base, and fostering acquisition of skills and understanding of emerging ‘hot’ sciences.

DECISION CRITERIA FOR EVALUATING AGRICULTURAL RESEARCH PRIORITIES

[Report from Northeastern Regional Association of State Agricultural Experiment Station Directors, 1999]

Selecting the topics for future research investments in agricultural science requires a rationale that is defensible, and a process that is, for the public sector, completely transparent. This document sets out a system of points-to-consider that could make up a new approach to setting criteria for decision making on resource allocations. The fundamental goal for setting agreed-to-decision criteria for allocating public sector agricultural research should be to gain the largest possible returns on research investment. This then requires consideration of four factors:

  • There must be a congruence between the dimensions of the intended topic and the allocation of research resources. Larger impacts can be expected by investing in topics that already have a large base in agriculture, forestry or rural development (e.g., wheat, hardwoods, community services), rather than trying to start from a smaller base (e.g., edible amaranth) or a regionally distributed environmental issue rather than local or state. This requires that some congruence analysis be done.

  • The allocation of resources needs to directly reflect the needs of the intended stakeholders and customers. This requires us to be listening to our customers.

  • Judgments are needed on what is feasible to accomplish through agricultural research, and this needs to be grounded in the best possible science. This in turn mandates some evaluation of the scientific potential of proposed research approaches by knowledgeable scientists.

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
  • Projections of expected benefits are needed to permit informed choices of alternative allocations. These must be done with a set of assumptions that are understood by the participants and the claimants to the system, and are broadly agreed upon. Ex ante estimations need to go beyond economic consequences to the non-economic benefits and consequences of technologies (i.e., social, environmental). Assigning premium or discount coefficients to economic projections can do this.

SYSTEMATIC RESEARCH PRIORITY CRITERIA

[From UK Systematics Forum on Priorities in Systematic (Biology) Research and Training, 1995]

  • Scientific excellence. The scientific excellence of the proposed study, the individual or team who will undertake the work and the collections available for study.

  • Relevance. The relevance of the study to a significant and clearly identified user community. The relevance may be scientific, cultural, historical or economic and will vary in relation to the differing interests of different audiences or user communities.

  • Enhancing scientific coverage. Filling gaps in knowledge about a group of organisms and making information available. Gaps in knowledge are not a sufficient criterion for establishing high priority, unless criterion 2 also applies and the results are of immediate relevance.

  • Scale of relevance. Levels of priority may differ at different geographical levels (global, regional, national or local).

  • Urgency. The level of priority given to a proposed study may reflect degree of urgency of the work determined in relation to the endemicity of the [proposing] group.

  • Feasibility. The scientific feasibility of completing the proposed task in the timescale and with the resources proposed.

UK DEPARTMENT OF HEALTH

[Prioritization of the Policy Research Programme in collaboration with the R&D Directorate]

  • Ministerial priority and relevance to the goals, aims and objectives of the Department of Health

  • Size and importance of the problem to be addressed in terms of actual or potential burden of disease and social condition

  • Well-defined plans for introducing research results into current policy activity or the formulation of future policy

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
  • Timeliness

  • Feasibility of research

  • Likely return on the investment in research

  • Appropriateness and availability of other research budgets

GERMAN SCIENCE COUNCIL

[From the German Wissenschaftsrat’s Large-scale Facilities for Basic Scientific Research, Working Group Report, 2003]

Theses on the significance of large-scale facilities for basic scientific research:

  • Scientific and technical developments for industry and society often start with the findings of basic research.

  • The opening-up or development of totally new areas of research is closely related to the availability of specific new facilities.

  • Large-scale facilities should stem from a broad initiative of scientific users with equal rights.

  • It is important to make very sure that future generations of scientists receive adequate, forward-looking training using the large-scale facilities.

  • With large-scale facilities on the scale with which we are concerned here, there must be long-term scientific visions, and the prerequisites for technical innovations must be given.

  • Including foreign partners in project preparation at an early stage, letting them participate in decision making and be responsible for their own scientific or technological contributions are prerequisites.

  • In order to avoid the duplication of research infrastructures, which would be detrimental to the effective capacity usage of large-scale facilities, there should be no comparable rival projects at the national or European level that are already in the realization phase.

  • Large-scale facilities must be centrally incorporated into the strategic planning and research program of the institution(s) in charge and must be a core element of the spectrum of tasks.

  • In the field of apparatus and equipment, which includes large-scale facilities, cooperation between places of further education and major research establishments should be a matter of course. Access to the facilities must be kept open for scientists from universities in a systematic and pragmatic way, so as to allow them to carry out research projects.

Criteria and specific considerations:

Expert Panel

  • The probability of fundamentally new insights or the possibilities of decisive scientific advances which could only be achieved with the large-scale facility.

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
  • The large-scale facility’s technical feasibility and the degree of technical innovation.

  • The scientific and technical competence of the institutions involved.

  • The already existing or anticipated acceptance of the (potential) users from immediately relevant and from neighboring areas of expertise.

  • The fulfillment of various objectives of importance for research (transfer, international perspectives, promoting young scientists).

Policy Assessment Panel

  • Scientific potential of the research program.

  • Fulfillment of science and technology policy goals as formulated in the theses.

  • Degree of maturity of the technical concept and, linked to it, the possible timeframe for implementing the individual facilities.

  • The context of further national and international scientific development of the research fields they belong to and assess their interaction with other disciplines.

[See Appendix D for additional information on this process.]

UK OFFICE OF SCIENCE AND TECHNOLOGY

[From UK Office of Science and Technology’s Large Facilities Strategic Roadmap Report, 2003]

[The UK uses a “roadmap” portfolio approach to select projects for solicitation and a Gateway screening process that requires projects to meet certain conditions before moving forward. The concept behind the Gateway process is that all large capital investments should be managed as discrete projects and should be subject to review and independent scrutiny at all key stages in their life cycle. The Gateway Process is based on well-proven techniques that lead to more effective delivery of benefits together with more predictable costs and outcomes. The process considers the project at critical points in its development. These critical points are identified as Gateways. There are six Gateways during the lifecycle of a project, four before contract award and two looking at service implementation and confirmation of the operational benefits. The Process emphasizes early review for maximum added value.]

The four-stage Gateway process:

  • Strategic assessment. Initial assessment of the strength of the science requirement for potential projects. Scientific peer review must be a key element of this first review.

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
  • Importance [depth] of science knowledge to be delivered by project.

  • Breadth of science knowledge that will benefit from investment.

  • Match with international positioning of UK science.

  • Strength of opportunity for training [links to number of users].

  • Contribution to/from UK technology/industry base.

  • Opportunity for spin-off and exploitation.

  • Business justification. Justification and robustness of the business case.

  • Procurement strategy. Confirm procurement strategy, project plan, etc.

  • Readiness for service. Confirm contract decisions and let contracts.

[See Appendix D for additional information on this process.]

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

This page intentionally left blank.

Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 183
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 184
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 185
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 186
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 187
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 188
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 189
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 190
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 191
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 192
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 193
Suggested Citation:"Appendix E: Examples of Criteria Used to Prioritize or Select." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 194
Next: Appendix F: NSF Background Materials »
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Get This Book
×
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 1995, the National Science Foundation (NSF) created a special account to fund large (several tens of millions of dollars) research facilities. Over the years, these facilities have come to represent an increasingly prominent part of the nation's R&D portfolio. Recently concern has intensified about the way NSF is selecting projects for this account. In 2003, six U.S. Senators including the chair and ranking member of the Senate Subcommittee on VA, HUD, and Independent Agencies Appropriations expressed these concerns in a letter to the NRC asking it to "review the current prioritization process and report to us on how it can be improved." This report presents a series of recommendations on how NSF can improve its priority setting process for large research facilities. While noting that NSF has improved this process, the report states that further strengthening is needed if NSF is to meet future demands for such projects.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!