Executive Summary
Large facilities play a more prominent role in science and engineering research today than they have played in the past. In FY 1995, the National Science Foundation (NSF) created the Major Research Equipment and Facilities Construction (MREFC) account to support the “acquisition, construction, commissioning, and upgrading of major research equipment, facilities, and other such capital assets” that cost more than several tens of millions of dollars.
Although such large facility projects represent less than 4 percent of the total NSF budget, they are highly visible because of their large per-project budget, their potential to shape the course of future research, the economic benefits they bring to particular regions, and the prominence of the facilities in an increasing number of research fields.
A number of concerns have been expressed by policy makers and researchers about the process used to rank large research facility projects for funding. First, the ability of new projects to be considered for approval at the National Science Board (NSB) level has stalled in the face of a backlog of approved but unfunded projects. Second, the rationale and criteria used to select projects and set priorities among projects for MREFC funding have not been clearly and publicly articulated. Third, there is a lack of funding for disciplines to conduct idea-generating and project-ranking activities and, once ideas have some level of approval, a lack of funding for conceptual development, planning, engineering, and design—information needed when judging whether a project is ready for funding in light of its ranking and for preparing a project for funding if it is
selected. Those concerns have eroded confidence among policy makers and the research community that large research facility projects are being ranked on the basis of their potential returns to science, technology, and society.
To address the concerns regarding NSF’s process for identifying, approving, constructing, and managing large research facility projects, the committee makes the following recommendations:
1. The National Science Board should oversee a process whereby the National Science Foundation produces a roadmap for large research facility projects that it is considering for construction over the next 10-20 years.
Broad inputs from the scientific community must form the basis for the roadmap.
The roadmap should take into consideration the need for continued funding of existing projects and should provide a set of well-defined potential new project starts for the near term (0-10 years). These projects should be ranked against other projects expected to be funded in a given year and according to where they are positioned in time on the roadmap. Projects further out in time (10-20 years) will necessarily be less well defined and ranked qualitatively to yield a vision of the future rather than a precise funding agenda, as is the case for the earlier years.
Different categories of overlapping criteria, described briefly in the bullets below, need to be used as one moves from comparing projects within a field to comparing projects in a directorate or in the entire NSF. At each level, the criteria used in the previous level must continue to be considered.
-
Within a field (as defined by NSF division) or interdisciplinary area: scientific and technical criteria, such as scientific breakthrough potential and technological readiness.
-
Across a set of related fields: agency strategic criteria, such as balance across fields and opportunities to serve researchers in several disciplines.
-
Across all fields: national criteria that assess relative need—such as which projects maintain US leadership in key scientific and engineering fields or enable the greatest numbers of researchers, educators, and students.
See box on page 4 for a more in-depth discussion of the proposed criteria.
A key constraint that must be imposed in the final stages of development is that the roadmap must reflect a reasonable projection of the
large research facility budget over the next 2 decades. The roadmap is not a guarantee of funding but rather a plan for the development of NSF’s large research facility program.
2. The National Science Foundation, with the approval of the National Science Board, should base its annual MREFC budget submission to Congress on the roadmap. The annual budget submission should include the proposed yearly expenditures over the next 5 years for committed projects and for projects that will start in that period. It should supply a rank ordering of the proposed new starts and should include the rationale behind the proposed budget, the project ranking, and any differences between the budget submission and the roadmap.
The committee emphasizes that the final determination and approval of rankings across disciplines must be the responsibility of the NSF senior leadership subject to final approval by the NSB.
3. To ensure that a large research facility project selected for funding is executed properly, on schedule, and within its budget, the National Science Foundation should enhance project preapproval planning and budgeting to develop a clear understanding of the project’s “technical definition” (also called “scope of work”) and the “implementation plan” needed to carry out the work.
Once a project is funded, there should be provision for a disciplined periodic independent review of the project’s progress relative to the original plan by a committee that includes internal and external engineering and construction experts and scientific experts and that will monitor the project’s status and provide its evaluation to the NSB and NSF.
After the construction phase, a committee with a different external and internal membership that includes scientists and people with expertise in managing large facilities should monitor facility operations annually (or as needed).
Finally, NSF has created a new position—Deputy Director, Large Facility Projects in NSF’s Office of Budget, Finance, and Award Management—to oversee the construction of these projects. Given the new nature and importance of this position, it should be reviewed by a committee of internal and external experts to evaluate its operation and effectiveness within a 2-year period. (See page 17 for a description of this position.)
4. To ensure that potential international and interagency collaborations and ideas are discussed at the earliest possible stages, the Office of Science and Technology Policy in the Executive Office of the President
|
Criteria for Developing Large Facilities Roadmaps and Budgets Overlapping categories of criteria should guide the preparation of the large facilities roadmap and NSF’s annual budget submissions. Scientific and technical quality must be at the core of these criteria. Because these are large facility projects, they must have the potential to have a major impact on the science involved; otherwise, they should not reach the next step. The rankings show what we would expect to happen first within a field, then within a directorate of NSF, and then across NSF. The criteria from earlier stages must continue to be used as the ranking proceeds from one stage to the next.
|
should have a substantial early role in coordinating roadmaps across agencies and with other countries.
5. Given the congressional emphasis in the most recent National Science Foundation reauthorization bill and the importance of the MREFC account to the research community and the health of the US research enterprise, the NSF leadership and the NSB must give careful attention to the implementation of reforms in the MREFC account.
Ranking projects across disciplines is inherently not an exact science; nevertheless, these criteria, as illustrated by the questions, provide a framework for a discussion of why one project is accorded a higher priority than another and a mechanism for the discussion to be as objective as possible in ranking projects across fields. Within the ranking categories, the questions might change as governmentwide initiatives and unexpected occurrences shift priorities. Similarly, at times, some questions might have greater weight than others in the judgment of the NSB. The key element is for the questions and weighting to be identified before the ranking process begins and for a clear rationalization to be provided when proposed large research facility projects are ranked. |
The committee has outlined a six-step process to implement these recommendations.
Large research facility projects will continue to constitute a vital component of NSF’s science and technology portfolio by enabling researchers to examine previously inaccessible phenomena and answer previously intractable questions. NSF has strengthened the priority-setting process for these facilities in recent years, partly in response to reports from Congress and other organizations.
NSF now has an opportunity to strengthen the program further by incorporating the preparation of a roadmap into its planning process and by involving the research community more fully in the generation and ranking of ideas for large research facilities.
Making choices among competing proposals from different scientific fields will never be easy, but the recommendations and detailed steps described here can help NSF to excel in this critical part of its mission.
