Recommendations
To address the concerns regarding NSF’s current process for identifying, approving, constructing, and managing large research facility projects, the committee offers the following recommendations:
1. The National Science Board should oversee a process whereby the National Science Foundation produces a roadmap for large research facility projects that it is considering for construction over the next 10-20 years.
Broad inputs from the scientific community must form the basis for the roadmap.
The roadmap should take into consideration the need for continued funding of existing projects and should provide a set of well-defined potential new project starts for the near term (0-10 years). These projects should be ranked against other projects expected to be funded in a given year and according to where they are positioned in time on the roadmap. Projects further out in time (10-20 years) will necessarily be less well defined and ranked qualitatively, to yield a vision of the future rather than a precise funding agenda, as is the case for the earlier years.
Different categories of overlapping criteria, described briefly in the bullets below, need to be used as one moves from comparing projects within a field to comparing projects in a directorate or in the entire NSF. At each level, the criteria used in the previous level must continue to be considered.
-
Within a field (as defined by NSF division) or interdisciplinary area: scientific and technical criteria, such as scientific breakthrough potential and technological readiness.
-
Across a set of related fields: agency strategic criteria, such as balance across fields and opportunities to serve researchers in several disciplines.
-
Across all fields: national criteria that assess relative need—such as which projects maintain US leadership in key scientific and engineering fields or enable the greatest numbers of researchers, educators, and students.
See box below for a more in-depth discussion of the proposed criteria.
Criteria for Developing Large Facilities Roadmaps and Budgets Overlapping categories of criteria should guide the preparation of the large facilities roadmap and NSF’s annual budget submissions. As shown in Figure 1, scientific and technical quality must be at the core of these criteria. Because these are large facility projects, they must have the potential to have a major impact on the science involved; otherwise, they should not reach the next step. The rankings show what we would expect to happen first within a field, then within a directorate of NSF, and then across NSF. The criteria from earlier stages must continue to be used as the ranking proceeds from one stage to the next.
|
A key constraint that must be imposed in the final stages of development is that the roadmap must reflect a reasonable projection of the large research facility budget over the next 2 decades. The roadmap is not a guarantee of funding but rather a plan for the development of NSF’s large research facility program.
Clearly, no one can project budgets out 20 years. However, one can expect to have rough estimates of the cost of a project that would allow NSF to plan for the future and provide guidance for future planning and design seed money. Thus, to create a credible roadmap, NSF would construct a tentative budget that might look something like the schematic in Figure 2. It would probably not be published with explicit budget
Ranking projects across disciplines is inherently not an exact science; nevertheless, these criteria, as illustrated by the questions, provide a framework for a discussion of why one project is accorded a higher priority than another and a mechanism for the discussion to be as objective as possible in ranking projects across fields. Within the ranking categories, the questions might change as governmentwide initiatives and unexpected occurrences shift priorities. Similarly, at times, some questions might have greater weight than others in the judgment of the NSB. The key element is for the questions and weighting to be identified before the ranking process begins and for a clear rationalization to be provided when proposed large research facility projects are ranked. |
numbers, inasmuch as the numbers for projects further out on the roadmap will have to be approximate, and it is not good policy to preempt the president and Congress on budgets far out in the future. But that does not preclude using budget estimates as guidance.
NSF and the NSB should engage the scientific and engineering research community more widely in generating new ideas and initial proposals for large research facility projects and in providing comments directly throughout the approval process. They should make every effort to hear directly from the proposers of the projects.
The NSB has a particularly important role to play as the independent voice of the research community in ensuring that NSF’s process is transparent and responds to the community and reforms proposed to the process.
In developing proposals for large research facilities, the potential for interagency and international collaboration should be explored at an earlier stage with idea generation, and the ranking process should take into account the plans of other agencies and countries.
In some research fields, particularly fields that overlap or fall between disciplines, NSF staff should more actively foster the planning and assessment of large research facility projects. A broad and balanced portfolio of projects—disciplinary and interdisciplinary—needs to emerge from the research community.
The research community should identify and rank projects at an early stage at the discipline and related-discipline level. Through activities funded by NSF, the research community should be involved in ranking project ideas within individual fields. In some fields—such as astronomy, astrophysics, and high-energy physics—this planning already occurs. NSF should rank proposed projects across the fields encompassed in each of its directorates by using the advice of its directorate advisory committees. Ranking of projects above the directorate level should be proposed to the NSF director by a panel of top-level NSF officials. The NSF director will then propose the roadmap to the NSB, which by law has approval authority over all large facility projects.
Setting priorities across fields of science is a complex undertaking, but the committee believes that this responsibility belongs squarely with the senior leadership of NSF working with the NSB. At the same time, NSF and the NSB should make every effort to solicit input from the scientific community in producing the roadmap, for example, by placing a draft of the roadmap on NSF’s web site for public comment. All comments on the roadmap and NSF’s responses to the comments could also be placed on the web site. The resulting NSF roadmap should be widely distributed to the research community.
The final approval of the roadmap must lie with the NSB. It should work closely with the NSF staff and director to ensure the highest quality
of roadmap. The inclusion of a project on the 20-year roadmap would not necessarily guarantee funding in the president’s annual budget requests, in that priorities can change. However, the near-term projects should be closely reflected in the budget; if not, there should be an explanation. Also, depending on the state of affairs when NSF carries out the roadmapping process, one possible outcome is that the projects currently back-logged may not be considered to be the top priority at the time the roadmapping is conducted.
In the course of its work, the committee received a number of recommendations that it consider as models for the roadmapping processes. One example is that used by the high-energy physics community under the direction of the Department of Energy. Another is the roadmap developed by the astronomy community under the aegis of the National Aeronautics and Space Administration. We have adopted that idea, but we also hasten to point out that the process will have to be modified to mesh with NSF’s distinctive culture, budgets, and missions and, particularly, to encourage the involvement of scientists in fields that have traditionally not organized themselves in this fashion. Success will depend not only on NSF’s efforts but also on the good will of the members of the research community asked to think across disciplinary lines in the course of advising NSF on the creation of the roadmap.
2. The National Science Foundation, with the approval of the National Science Board, should base its annual MREFC budget submission to Congress on the roadmap. The annual budget submission should include the proposed yearly expenditures over the next 5 years for committed projects and for projects that will start in that period. It should supply a rank ordering of the proposed new starts and should include the rationale behind the proposed budget, the project ranking, and any differences between the budget submission and the roadmap.
The committee emphasizes that the final determination and approval of rankings across disciplines must be the responsibility of the NSF senior leadership subject to final approval by the NSB.
The observations and rationale used to rank one large facility project idea over another for inclusion in NSF’s annual budget requests should be clearly and publicly described so that policy makers and researchers understand the motivation for the decisions. NSF’s FY 2004 budget request and its followup letter to Congress are initial steps in that direction, but NSF should expand its discussion to respond to the need. The rationale should be prepared every year and should accompany, rather than follow, NSF’s annual budget requests.
Inevitably, the budgetary framework assumed in developing the
roadmap will not reflect actual annual budgets, so deviations from the roadmap necessary to prepare annual budget requests should be based on the above criteria, and the rationale for the decisions (or for deviation from the criteria) should be provided in the budget. Figure 3 summarizes this process by depicting the sequence of selection and ranking steps through which a large facility project must pass before being included in the president’s budget request as a candidate for MREFC funding.
3. To ensure that a large research facility project selected for funding is executed properly, on schedule, and within its budget, the National Science Foundation should enhance project preapproval planning and budgeting to develop a clear understanding of the project’s “technical definition” (also called “scope of work”) and the “implementation plan” needed to carry out the work.
Once a project is funded, there should be provision for a disciplined periodic independent review of the project’s progress relative to the original plan by a committee that includes internal and external engineering and construction experts and scientific experts and that will monitor the project’s status and provide its evaluation to the NSB and NSF.
After the construction phase, a committee with a different external and internal membership that includes scientists and people with expertise in managing large facilities should monitor facility operations annually (or as needed).
Finally, NSF has created a new position—Deputy Director, Large Facility Projects in NSF’s Office of Budget, Finance, and Award Management—to oversee the construction of these projects. Given the new nature and importance of this position, it should be reviewed by a committee of internal and external experts to evaluate its operation and effectiveness within a 2-year period.
If a project is highly ranked at the directorate level and is a candidate for a proposed start over the following 5 years, conceptual or preliminary engineering and plans for carrying out the project, in addition to the information described by the criteria, should be prepared before the NSB approves it for inclusion in the roadmap. The conceptual plans developed as a result of this process should describe how the project will be constructed and managed and should describe any constraints on its scope and implementation, such as funding, technology, development, or siting.
The conceptual plans should include the “technical definition” to provide sufficient information on the scope of the project. In complex projects, minor changes can evolve as more detail is developed, but there should be no major changes from this point out, without a thorough
analysis that weighs the potential advantage of such changes against the impact on the project’s cost and schedule and a review of that analysis by the National Science Board before initiation.
The “implementation plan” describes how the construction project will be accomplished. It includes such items as organization for implementation, management, contracting, scheduling, and budgeting. In addition, there should be plans and provisions for effective cost and schedule control, approval for changes, monitoring of progress, periodic review, and so on, all the way to commissioning and turning over the facility to the ultimate “owner and operator.”
As mentioned above, the roadmap should be the guide to funding project planning, engineering, and design activities to prepare a project proposal for MREFC funding. NSF has frequently been criticized for not supplying funding for this phase of a project.
During the construction of a large research facility, review committees should consist primarily of experts in constructing new and unique facilities and experts in the scientific and technical subjects that the project entails.
While the facility is in operation, the review committees should include people who have experience in managing large research facilities and, again, experts in the scientific and technical fields that use the facility. The value of informed, independent, and objective review in managing project construction and operations cannot be overemphasized.
The role of the deputy for large facility projects is to manage this process and bring all the various constituencies together so that the project happens on time, within budget, and with satisfactory performance. As a result, the person needs adequate and experienced project construction and management staff, access to qualified consultants and contractors, and the institutional authority to oversee the design engineering, construction, and operation phases adequately. Each project or program will have dedicated leadership, but it is this deputy who has principal responsibility to support the undertakings and for oversight and management. Because this deputy plays such a critical role, the office should be reviewed within 2 years to ensure that it is adequately staffed and providing the appropriate level of project oversight and leadership.
4. To ensure that potential international and interagency collaborations and ideas are discussed at the earliest possible stages, the Office of Science and Technology Policy in the Executive Office of the President should have a substantial early role in coordinating roadmaps across agencies and with other countries.
As noted in the remarks about the first recommendation, early discussions regarding potential interagency and international coordination
and collaboration can reduce the financial costs of a project to participating agencies and countries and can enhance its prospects of success.
5. Given the congressional emphasis in the most recent National Science Foundation reauthorization bill and the importance of the MREFC account to the research community and to the health of the US research enterprise, the NSF leadership and the NSB must give careful attention to the implementation of reforms in the MREFC account.
The NSF Authorization Act (HR 4664), which authorizes appropriations for NSF for fiscal years 2003-2007, states the following in Section 14(e):
(1) STUDY—Not later than 3 months after the date of the enactment of this Act, the Director shall enter into an arrangement with the National Academy of Sciences to perform a study on setting priorities for a diverse array of disciplinary and interdisciplinary Foundation-sponsored large research facility projects.
(2) TRANSMITTAL TO CONGRESS—Not later than 15 months after the date of enactment of this Act, the Director shall transmit to the Committee on Science and the Committee on Appropriations of the House of Representatives and to the Committee on Commerce, Science, and Transportation, the Committee on Health, Education, Labor, and Pensions, and the Committee on Appropriations of the Senate, the study conducted by the National Academy of Sciences together with the Foundation’s reaction to the study authorized under paragraph (1).
The committee suggests a six-step process (described in the next section) to implement the recommendations that we hope NSF and NSB will adopt. The committee believes that NSF and NSB reactions to the present report would best be illustrated by a memorandum that would describe the degree to which the report’s recommendations and proposed implementation plan are adopted and discuss the extent to which there is agreement or disagreement with the report’s recommendations. In addition, NSF and the NSB should describe what actions they plan to undertake to set priorities among NSF’s large research facility projects. This report should be provided not only to Congress but to the entire research community for public comment.