National Academies Press: OpenBook
« Previous: 3 Concerns About National Science Foundation's Current Priority-Setting Process
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

Recommendations

To address the concerns regarding NSF’s current process for identifying, approving, constructing, and managing large research facility projects, the committee offers the following recommendations:

1. The National Science Board should oversee a process whereby the National Science Foundation produces a roadmap for large research facility projects that it is considering for construction over the next 10-20 years.

Broad inputs from the scientific community must form the basis for the roadmap.

The roadmap should take into consideration the need for continued funding of existing projects and should provide a set of well-defined potential new project starts for the near term (0-10 years). These projects should be ranked against other projects expected to be funded in a given year and according to where they are positioned in time on the roadmap. Projects further out in time (10-20 years) will necessarily be less well defined and ranked qualitatively, to yield a vision of the future rather than a precise funding agenda, as is the case for the earlier years.

Different categories of overlapping criteria, described briefly in the bullets below, need to be used as one moves from comparing projects within a field to comparing projects in a directorate or in the entire NSF. At each level, the criteria used in the previous level must continue to be considered.

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
  • Within a field (as defined by NSF division) or interdisciplinary area: scientific and technical criteria, such as scientific breakthrough potential and technological readiness.

  • Across a set of related fields: agency strategic criteria, such as balance across fields and opportunities to serve researchers in several disciplines.

  • Across all fields: national criteria that assess relative need—such as which projects maintain US leadership in key scientific and engineering fields or enable the greatest numbers of researchers, educators, and students.

See box below for a more in-depth discussion of the proposed criteria.

Criteria for Developing Large Facilities Roadmaps and Budgets

Overlapping categories of criteria should guide the preparation of the large facilities roadmap and NSF’s annual budget submissions. As shown in Figure 1, scientific and technical quality must be at the core of these criteria. Because these are large facility projects, they must have the potential to have a major impact on the science involved; otherwise, they should not reach the next step.

The rankings show what we would expect to happen first within a field, then within a directorate of NSF, and then across NSF. The criteria from earlier stages must continue to be used as the ranking proceeds from one stage to the next.

  • First Ranking: Scientific and Technical Criteria Assessed by Researchers in a Field or Interdisciplinary Area

    • Which projects have the most scientific merit, potential, and opportunities within a field or interdisciplinary area?

    • Which projects are the most technologically ready?

    • Are the scientific credentials of the proposers of the highest rank?

    • Are the project-management capabilities of the proposal team of the highest quality?

  • Second Ranking: Agency Strategic Criteria Assessed Across Related Fields by Using the Advice of Directorate Advisory Committees

    • Which projects will have the greatest impact on scientific advances in this set of related fields taking into account the importance of balance among fields for NSF’s portfolio management in the nation’s interest?

    • Which projects include opportunities to serve the needs of researchers from multiple disciplines or the ability to facilitate interdisciplinary research?

    • Which projects have major commitments from other agencies or countries that should be considered?

    • Which projects have the greatest potential for education and workforce development?

    • Which projects have the most readiness for further development and construction?

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

A key constraint that must be imposed in the final stages of development is that the roadmap must reflect a reasonable projection of the large research facility budget over the next 2 decades. The roadmap is not a guarantee of funding but rather a plan for the development of NSF’s large research facility program.

Clearly, no one can project budgets out 20 years. However, one can expect to have rough estimates of the cost of a project that would allow NSF to plan for the future and provide guidance for future planning and design seed money. Thus, to create a credible roadmap, NSF would construct a tentative budget that might look something like the schematic in Figure 2. It would probably not be published with explicit budget

  • Third Ranking: National Criteria Assessed Across All Fields by the National Science Board

    • Which projects are in new and emerging fields that have the most potential to be transformative? Which projects have the most potential to change how research is conducted or to expand fundamental science and engineering frontiers?

    • Which projects have the greatest potential for maintaining US leadership in key science and engineering fields?

    • Which projects produce the greatest benefits in numbers of researchers, educators, and students enabled?

    • Which projects most need to be undertaken in the near term? Which ones have the most current windows of opportunity, pressing needs, and international or interagency commitments that must be met?

    • Which projects will have the greatest impact on current national priorities and needs?

    • Which projects have the greatest degree of community support?

    • Which projects will have the greatest impact on scientific advances across fields taking into account the importance of balance among fields for NSF’s portfolio management in the nation’s interest?

Ranking projects across disciplines is inherently not an exact science; nevertheless, these criteria, as illustrated by the questions, provide a framework for a discussion of why one project is accorded a higher priority than another and a mechanism for the discussion to be as objective as possible in ranking projects across fields.

Within the ranking categories, the questions might change as governmentwide initiatives and unexpected occurrences shift priorities. Similarly, at times, some questions might have greater weight than others in the judgment of the NSB. The key element is for the questions and weighting to be identified before the ranking process begins and for a clear rationalization to be provided when proposed large research facility projects are ranked.

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

 



Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

FIGURE 1 Projects are compared and ranked at different steps of the priority-setting process; at each step, additional categories of criteria should be applied. The innermost category of scientific and technical criteria remains at the core of the entire process; it is first applied to projects within the same field. As projects develop and are compared across divisions within the directorate, agency strategic criteria must also be considered. Finally, projects must be compared across all fields at NSF, and national criteria must be considered for final selection and ranking. The overlapping ellipses in this diagram emphasize the “transcend and include” aspect of the model: at each step, the criteria used at the previous step must continue to be considered.

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

 



Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

FIGURE 2 An example of a 20-year budget roadmap for large facility projects. This figure is intended to illustrate some of the temporal and fiscal considerations that should inform the NSF roadmapping process.

The MREFC funding amounts depicted for existing (already funded or planned) individual projects through FY 2008 reflect the FY 2004 budget submission. The MREFC funding levels shown for existing projects beyond FY 2008 are taken from the individual funding profiles of each project.

Through FY 2007, the budget envelope shown in this figure reflects the amount of MREFC funding indicated in the 2002 NSF Doubling Act. After this doubling, an annual growth rate of 3 percent was assumed.

The first 10 years depicted in this figure demonstrate the detailed, project-specific budget information that the near-term roadmap should provide. Further into the future, when plans are less definite, the figure indicates, roughly, the amount of funding available in a given year for all new starts and for all current MREFC account projects without providing detailed information about individual projects.

The particulars of the long-term projections will necessarily change over time, but such a 20-year plan will provide a realistic vision of possibilities for future new starts. While including this figure, the committee recognizes that it may be challenging to publish a long-term budget and believes that there are other ways to represent this information (see, for example, the DOE roadmap report).

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

numbers, inasmuch as the numbers for projects further out on the roadmap will have to be approximate, and it is not good policy to preempt the president and Congress on budgets far out in the future. But that does not preclude using budget estimates as guidance.

NSF and the NSB should engage the scientific and engineering research community more widely in generating new ideas and initial proposals for large research facility projects and in providing comments directly throughout the approval process. They should make every effort to hear directly from the proposers of the projects.

The NSB has a particularly important role to play as the independent voice of the research community in ensuring that NSF’s process is transparent and responds to the community and reforms proposed to the process.

In developing proposals for large research facilities, the potential for interagency and international collaboration should be explored at an earlier stage with idea generation, and the ranking process should take into account the plans of other agencies and countries.

In some research fields, particularly fields that overlap or fall between disciplines, NSF staff should more actively foster the planning and assessment of large research facility projects. A broad and balanced portfolio of projects—disciplinary and interdisciplinary—needs to emerge from the research community.

The research community should identify and rank projects at an early stage at the discipline and related-discipline level. Through activities funded by NSF, the research community should be involved in ranking project ideas within individual fields. In some fields—such as astronomy, astrophysics, and high-energy physics—this planning already occurs. NSF should rank proposed projects across the fields encompassed in each of its directorates by using the advice of its directorate advisory committees. Ranking of projects above the directorate level should be proposed to the NSF director by a panel of top-level NSF officials. The NSF director will then propose the roadmap to the NSB, which by law has approval authority over all large facility projects.

Setting priorities across fields of science is a complex undertaking, but the committee believes that this responsibility belongs squarely with the senior leadership of NSF working with the NSB. At the same time, NSF and the NSB should make every effort to solicit input from the scientific community in producing the roadmap, for example, by placing a draft of the roadmap on NSF’s web site for public comment. All comments on the roadmap and NSF’s responses to the comments could also be placed on the web site. The resulting NSF roadmap should be widely distributed to the research community.

The final approval of the roadmap must lie with the NSB. It should work closely with the NSF staff and director to ensure the highest quality

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

of roadmap. The inclusion of a project on the 20-year roadmap would not necessarily guarantee funding in the president’s annual budget requests, in that priorities can change. However, the near-term projects should be closely reflected in the budget; if not, there should be an explanation. Also, depending on the state of affairs when NSF carries out the roadmapping process, one possible outcome is that the projects currently back-logged may not be considered to be the top priority at the time the roadmapping is conducted.

In the course of its work, the committee received a number of recommendations that it consider as models for the roadmapping processes. One example is that used by the high-energy physics community under the direction of the Department of Energy. Another is the roadmap developed by the astronomy community under the aegis of the National Aeronautics and Space Administration. We have adopted that idea, but we also hasten to point out that the process will have to be modified to mesh with NSF’s distinctive culture, budgets, and missions and, particularly, to encourage the involvement of scientists in fields that have traditionally not organized themselves in this fashion. Success will depend not only on NSF’s efforts but also on the good will of the members of the research community asked to think across disciplinary lines in the course of advising NSF on the creation of the roadmap.

2. The National Science Foundation, with the approval of the National Science Board, should base its annual MREFC budget submission to Congress on the roadmap. The annual budget submission should include the proposed yearly expenditures over the next 5 years for committed projects and for projects that will start in that period. It should supply a rank ordering of the proposed new starts and should include the rationale behind the proposed budget, the project ranking, and any differences between the budget submission and the roadmap.

The committee emphasizes that the final determination and approval of rankings across disciplines must be the responsibility of the NSF senior leadership subject to final approval by the NSB.

The observations and rationale used to rank one large facility project idea over another for inclusion in NSF’s annual budget requests should be clearly and publicly described so that policy makers and researchers understand the motivation for the decisions. NSF’s FY 2004 budget request and its followup letter to Congress are initial steps in that direction, but NSF should expand its discussion to respond to the need. The rationale should be prepared every year and should accompany, rather than follow, NSF’s annual budget requests.

Inevitably, the budgetary framework assumed in developing the

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

roadmap will not reflect actual annual budgets, so deviations from the roadmap necessary to prepare annual budget requests should be based on the above criteria, and the rationale for the decisions (or for deviation from the criteria) should be provided in the budget. Figure 3 summarizes this process by depicting the sequence of selection and ranking steps through which a large facility project must pass before being included in the president’s budget request as a candidate for MREFC funding.

3. To ensure that a large research facility project selected for funding is executed properly, on schedule, and within its budget, the National Science Foundation should enhance project preapproval planning and budgeting to develop a clear understanding of the project’s “technical definition” (also called “scope of work”) and the “implementation plan” needed to carry out the work.

Once a project is funded, there should be provision for a disciplined periodic independent review of the project’s progress relative to the original plan by a committee that includes internal and external engineering and construction experts and scientific experts and that will monitor the project’s status and provide its evaluation to the NSB and NSF.

After the construction phase, a committee with a different external and internal membership that includes scientists and people with expertise in managing large facilities should monitor facility operations annually (or as needed).

Finally, NSF has created a new position—Deputy Director, Large Facility Projects in NSF’s Office of Budget, Finance, and Award Management—to oversee the construction of these projects. Given the new nature and importance of this position, it should be reviewed by a committee of internal and external experts to evaluate its operation and effectiveness within a 2-year period.

If a project is highly ranked at the directorate level and is a candidate for a proposed start over the following 5 years, conceptual or preliminary engineering and plans for carrying out the project, in addition to the information described by the criteria, should be prepared before the NSB approves it for inclusion in the roadmap. The conceptual plans developed as a result of this process should describe how the project will be constructed and managed and should describe any constraints on its scope and implementation, such as funding, technology, development, or siting.

The conceptual plans should include the “technical definition” to provide sufficient information on the scope of the project. In complex projects, minor changes can evolve as more detail is developed, but there should be no major changes from this point out, without a thorough

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

analysis that weighs the potential advantage of such changes against the impact on the project’s cost and schedule and a review of that analysis by the National Science Board before initiation.

The “implementation plan” describes how the construction project will be accomplished. It includes such items as organization for implementation, management, contracting, scheduling, and budgeting. In addition, there should be plans and provisions for effective cost and schedule control, approval for changes, monitoring of progress, periodic review, and so on, all the way to commissioning and turning over the facility to the ultimate “owner and operator.”

As mentioned above, the roadmap should be the guide to funding project planning, engineering, and design activities to prepare a project proposal for MREFC funding. NSF has frequently been criticized for not supplying funding for this phase of a project.

During the construction of a large research facility, review committees should consist primarily of experts in constructing new and unique facilities and experts in the scientific and technical subjects that the project entails.

While the facility is in operation, the review committees should include people who have experience in managing large research facilities and, again, experts in the scientific and technical fields that use the facility. The value of informed, independent, and objective review in managing project construction and operations cannot be overemphasized.

The role of the deputy for large facility projects is to manage this process and bring all the various constituencies together so that the project happens on time, within budget, and with satisfactory performance. As a result, the person needs adequate and experienced project construction and management staff, access to qualified consultants and contractors, and the institutional authority to oversee the design engineering, construction, and operation phases adequately. Each project or program will have dedicated leadership, but it is this deputy who has principal responsibility to support the undertakings and for oversight and management. Because this deputy plays such a critical role, the office should be reviewed within 2 years to ensure that it is adequately staffed and providing the appropriate level of project oversight and leadership.

4. To ensure that potential international and interagency collaborations and ideas are discussed at the earliest possible stages, the Office of Science and Technology Policy in the Executive Office of the President should have a substantial early role in coordinating roadmaps across agencies and with other countries.

As noted in the remarks about the first recommendation, early discussions regarding potential interagency and international coordination

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

 



Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

FIGURE 3 Depiction of the sequence of selection and ranking steps through which a large facility project must pass before being included in the president’s budget request as a candidate for MREFC funding. Each gateway represents a phase in the priority-setting process of selection and ranking with oversight by a specific body. The category of criteria identified at each gateway is detailed in the box on page 22. Note that international and interagency cooperation considerations are explicit items in both the agency strategic criteria and national criteria categories and therefore are important throughout the entire process.

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×

and collaboration can reduce the financial costs of a project to participating agencies and countries and can enhance its prospects of success.

5. Given the congressional emphasis in the most recent National Science Foundation reauthorization bill and the importance of the MREFC account to the research community and to the health of the US research enterprise, the NSF leadership and the NSB must give careful attention to the implementation of reforms in the MREFC account.

The NSF Authorization Act (HR 4664), which authorizes appropriations for NSF for fiscal years 2003-2007, states the following in Section 14(e):

(1) STUDY—Not later than 3 months after the date of the enactment of this Act, the Director shall enter into an arrangement with the National Academy of Sciences to perform a study on setting priorities for a diverse array of disciplinary and interdisciplinary Foundation-sponsored large research facility projects.

(2) TRANSMITTAL TO CONGRESS—Not later than 15 months after the date of enactment of this Act, the Director shall transmit to the Committee on Science and the Committee on Appropriations of the House of Representatives and to the Committee on Commerce, Science, and Transportation, the Committee on Health, Education, Labor, and Pensions, and the Committee on Appropriations of the Senate, the study conducted by the National Academy of Sciences together with the Foundation’s reaction to the study authorized under paragraph (1).

The committee suggests a six-step process (described in the next section) to implement the recommendations that we hope NSF and NSB will adopt. The committee believes that NSF and NSB reactions to the present report would best be illustrated by a memorandum that would describe the degree to which the report’s recommendations and proposed implementation plan are adopted and discuss the extent to which there is agreement or disagreement with the report’s recommendations. In addition, NSF and the NSB should describe what actions they plan to undertake to set priorities among NSF’s large research facility projects. This report should be provided not only to Congress but to the entire research community for public comment.

Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 21
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 22
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 23
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 24
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 25
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 26
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 27
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 28
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 29
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 30
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 31
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 32
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 33
Suggested Citation:"4 Recommendations." National Research Council. 2004. Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/10895.
×
Page 34
Next: 5 Implementing the Recommendations »
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Get This Book
×
Buy Paperback | $69.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 1995, the National Science Foundation (NSF) created a special account to fund large (several tens of millions of dollars) research facilities. Over the years, these facilities have come to represent an increasingly prominent part of the nation's R&D portfolio. Recently concern has intensified about the way NSF is selecting projects for this account. In 2003, six U.S. Senators including the chair and ranking member of the Senate Subcommittee on VA, HUD, and Independent Agencies Appropriations expressed these concerns in a letter to the NRC asking it to "review the current prioritization process and report to us on how it can be improved." This report presents a series of recommendations on how NSF can improve its priority setting process for large research facilities. While noting that NSF has improved this process, the report states that further strengthening is needed if NSF is to meet future demands for such projects.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!