Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation (2004)

Wide assortments of materials were made available to the committee in the course of its work. The committee includes here excerpts from several of them that highlight the current set of principles and practices at NSF.

Statement of Dr. Anita Jones, vice chair, National Science Board Before the House Committee on Science Subcommittee on Research, September 6, 2001

Chairman Smith, Ranking Minority Member Johnson, members of the Committee, I appreciate the opportunity to testify today. My name is Anita Jones. I am Vice Chair of the National Science Board and Chair of the Board’s Committee on Strategy and Budget. I am also the Quarles Professor of Engineering and Applied Science at the University of Virginia. From 1993 to 1997 I served as Director of Defense Research and Engineering at the U.S. Department of Defense. In that position I was responsible for the science and technology program of the Department of

Defense, including the Defense Advanced Research Projects Agency and oversight of the Department’s laboratories.

On behalf of the National Science Board, I thank the Committee for its long-term support for science and engineering research and education activities, which have contributed so substantially to our Nation’s well being.

The National Science Board has two statutory roles: to serve as the governing board of the National Science Foundation, and to advise the Congress and the President on national policy issues for science and engineering research and education.

Today, my comments will focus on the Board’s role as governing board of the Foundation, specifically on our oversight and approval of the Foundation’s support for large-scale research facilities.

First, I would like to emphasize that the Foundation has an excellent record—spanning 50 years—of supporting such facilities, in terms of both the quality of their research and their management. Today, NSF invests over $1 billion annually in facilities and other infrastructure projects. With the exception of U.S. research facilities in the Antarctic, which are directly operated by the National Science Foundation, NSF typically makes awards to other organizations for the construction and operation of facilities.

The following are examples of major facilities:

• The Large Hadron Collider is a superconducting particle accelerator. Its purpose is to help scientists advance the fundamental understanding of matter. The Collider’s construction and operations are funded through an international collaboration.

• The Laser Interferometer Gravitational-Wave Observatory, or LIGO, will allow physicists and engineers to collaborate to test the dynamic features of Einstein’s theory of gravity and to study the properties of intense gravitational fields.

• The National Astronomy Center in Arecibo, Puerto Rico, supports observations in radio and radar astronomy and atmospheric sciences.

• Research facilities at the South Pole Station, currently under renovation, support a variety of diverse but important research activities that can only be conducted in the unique cold and pristine environment at the South Pole.

• The Ocean Drilling Program, involving 20 countries, supports research in areas including deep ocean structures, hydrology and geo-chemical cycles.

These five examples are all major research facilities. For the most part, they are the research instruments that make possible research advances that can be accomplished in no other way. They are all large; each one

opens new research frontiers that could not be entered without these tools. They are complex; each one involves challenging engineering tasks in its design, construction and operation. Hence, each is very costly. As I have illustrated, the U.S. frequently teams with international collaborators, not just to assure that the best research is pursued, but also to help make the facilities more affordable. And, very importantly, each facility has a very broad base of researchers who are the users; they frequently come from multiple disciplines.

Many scientific fields are on the edge of exciting discoveries that require such facilities. I anticipate that in the 21st century, the need for such large, complex research facilities will grow. In recent years, the Foundation’s portfolio of facilities has grown and diversified to include distributed projects and complex multidisciplinary projects like terascale computing systems and ocean observatories that challenge traditional management and oversight approaches.

The National Science Board plays a critical role in the oversight and approval of large NSF-supported facilities. The NSB is well constituted to exercise its oversight and approval responsibilities. Members of the Board include executives from industry and presidents of universities, individuals who have extensive experience in managing large, cutting-edge research facilities and instrumentation. Of course, the Board includes members who have used such facilities.

The Board conducts two activities that focus on the approval and oversight of facilities. They are the approval of large awards and the approval of candidates for the Major Research Equipment account. Typically, the Board hears briefings from NSF management at almost every NSB meeting on the subject of large facilities—existing and candidate.

The Board approves all major projects, including facilities, whose costs exceed one percent of the budget of the sponsoring directorate or office. The Board also approves new major programs whose budget exceeds three percent of the budget of the sponsoring directorate or office.

The Board’s Committee on Programs and Plans (CPP) reviews large projects at various stages of their development and makes recommendations to the full Board for the initiation of new awards and programs. In addition, the CPP reviews projects for adherence to the NSB approved criteria for merit review and the Board’s policy regarding the competition, recompetition and renewal of NSF awards. Throughout the imple-

mentation phase of a project, the CPP reviews its progress and informs the Board of its status and any issues that arise.

The Board’s Committee on Audit and Oversight (A&O) reviews specific financial and business management issues raised by the Inspector General and by Foundation management. Like the CPP, A&O informs the Board of any issues that arise.

The Major Research Equipment (MRE) account is an agency-wide capital asset account used to fund major science and engineering infrastructure projects that cost far more than one program’s budget could support. The costs of MRE facilities range from several tens to hundreds of millions of dollars. The Board sees these projects multiple times over their lifetime. The Board takes two kinds of actions. First, it authorizes a candidate project for possible inclusion by the Foundation in a future budget. Later, the Board approves specific funding for an organization or consortium to design and construct the facility.

Let me briefly outline how the Board oversees MRE projects. The Director selects candidates for the NSB to review during one of its five meetings throughout the year. The Board receives, for approval, candidate projects that may be included in a future budget request, subject to availability of funding. Board authorization signifies that the projects are meritorious and that planning is sufficiently advanced to justify funding. In giving its approval, the Board considers the intellectual merit, societal impacts of the projects, their importance to science and engineering, balance across disciplines, readiness to be implemented, and cost-benefit and risk analyses.

The Board authorizes the MRE projects for possible inclusion in future budgets, but does not rank-order them to preserve the Foundation’s flexibility in a given budget year. We believe that all projects authorized by the Board are of unquestioned excellence and worthy of Foundation support. When the Board approves the Foundation’s budget submission to the Office of Management and Budget, it reaffirms its support for any MRE projects included for funding.

After NSF has run a competition and detailed plans are in place for design, construction and operations, the project comes back to the Board for the award of funding at a specific level. Board oversight of MRE projects continues after an award is made. The Board’s CPP reviews a project’s progress at the midpoint of construction and whenever significant issues arise. If it appears that a project will exceed the Board’s approved dollar amount by over 20 percent or $10 million, whichever is less, the Director must return to the Board to request approval for a higher

level of funding. CPP also receives periodic reports on the status of all major projects.

As part of its oversight responsibilities, the Board began a dialogue with Foundation management about large facility issues more than a year and a half ago. Together, we have been discussing the improvement of the process for identifying candidate projects and for Foundation oversight of the management of construction and operation of such facilities.

The Foundation has created a new Large Facility Projects Management and Oversight Plan. That Plan, which was requested by the Administration, does two things. It incorporates and builds upon an existing facility management process. In addition, it strengthens financial oversight.

I would like to comment briefly on the Board’s participation in the development of the Plan. The full Board received a draft Plan for comment this summer. At the August 8-9 Board meeting, our Committee on Programs and Plans, which has the responsibility to review major projects and facilities, received a briefing on the Plan from the Deputy Director. Members of our Audit and Oversight Committee participated in those discussions. Board members were pleased with the direction, framework, and elements set forth in the briefing and encouraged Foundation management to proceed with the Plan’s development. The Board will continue to assess the Foundation’s progress in refining and implementing the Plan.

The Plan institutionalizes and builds on long-standing management practices.

• It codifies sound practices already in use, augments the existing MRE process, and documents principles of management.

• It ensures that project management will stay with the scientists and engineers, from planning through operation. The overall NSF Program Manager for a particular facility project is an individual in one of the research directorates of the Foundation.

• It strengthens Foundation oversight of financial and business functions. This requires organizational and managerial changes within the Foundation. In particular, it calls for the creation of a Deputy for Large Facility Projects who reports directly to the Chief Financial Officer and is responsible for “developing, implementing and managing, with NSF-wide input and concurrence, management oversight policies, guidelines and procedures.”

In summary, Mr. Chairman and members of the Committee:

• The National Science Board supports the general direction laid out in NSF’s Large Facility Projects Management and Oversight Plan. The Board will assess the Foundation’s progress in refining and implementing the elements of the Plan, particularly to ensure the integrity of the evaluation and oversight of the financial and business aspects of the facility project throughout its life.

• The implementation of the Plan will ensure that the Foundation, with Board oversight, has the policies and organization required for sound management of unique, complex, world-class research facilities.

Thank you for the opportunity to present these remarks. I would be glad to answer any questions you may have.

Question 1: The President’s fiscal year 2004 budget request represents a significant improvement over prior year requests with regard to directorate breakdown, full life-cycle costs, and prioritization of the “new start” projects in the Major Research and Facilities Construction (MREFC) account However, the budget does not include the criteria used to rank these projects and the relative value these criteria were given in establishing the prioritized list. This information is required to be annually submitted to Congress before any funds can be obligated from the MREFC account. Please provide us with this information. Also, please clarify how new projects will be reviewed and ranked, how they will be incorporated into the existing prioritized list, and when and how the revised list will be transmitted to Congress.

Answer: In FY 2004 budget request, $202.33 million in funding is requested for seven ongoing MREFC projects. No funds were requested for new start projects. This is consistent with current NSB policy, which requires that NSF give first priority to projects that have been started but not completed. The FY 2004 budget request identified three new starts for initiation in FY 2005 and FY 2006. In priority order, these are:

• Scientific Ocean Drilling in FY 2005;

• Rare Symmetry Violating Processes in FY 2006; and

• Ocean Observatories in FY 2006.

This specific case (i.e., the process that NSF used to prioritize these three projects) must first be viewed within the broader context of how NSF identifies, reviews, selects and prioritizes large facility projects.

In identifying new facility construction projects, the S&E community, in consultation with NSF, develops ideas, considers alternatives, explores partnerships, and develops cost and timeline estimates. By the time a proposal is submitted to NSF, these issues have been thoroughly examined.

Upon receipt by NSF, large facility proposals are first subjected to rigorous external peer review, focusing on the criteria of intellectual merit and the broad (probable) impacts of the project. Only the highest rated proposals—i.e., those that are rated outstanding on both criteria—survive this process and are recommended to an MREFC Panel comprised of the Assistant Directors and Office heads, serving as stewards for their fields and chosen for their breadth of understanding, and chaired by the Deputy Director.

The MREFC Panel uses a two-stage process. First, it selects the new start projects it will recommend to the Director for future NSF support, based on a discussion of the merits of the science within the context of all sciences that NS supports. Second, it places these recommended new start projects in priority order.

In selecting projects for future support, the Panel considers the following criteria:

• Significance of the opportunity to enable frontier research and education.

• Degree of support within the relevant S&E communities.

• Readiness of project, in terms of feasibility, engineering cost-effectiveness, interagency and international partnerships, and management.

Using these criteria, projects that are not highly rated are returned to the initiating directorates, and may be reconsidered at a future time. The Panel then places highly rated projects in priority order. This process is conducted in consultation with the NSF Director. The MREFC Panel and the Director use the following criteria to determine the priority order of the projects:

• How “transformative” is the project? Will it change the way research is conducted or change fundamental S&E concepts/research frontiers?

• How great are the benefits of the project? How: many researchers, educators and students will it enable? Does it broadly serve many disciplines?

• How pressing is the need? Is there a window of opportunity? Are there interagency and international commitments that must be met?

These criteria are not assigned relative weights, because each project has its own unique attributes and circumstances. For example, timeliness may be crucial for one project and relatively unimportant for another. Additionally, the Director must weigh the impact of a proposed facility on the balance between scientific fields, the importance of the project with respect to national priorities, and possible societal benefits.

In August the Director presents the MREFC priorities, including a discussion of the rationale for the priority order to the NSB, as part of the budget process. The NSB reviews the list and either approves or argues the order of priority. As part of its budget submission, NSF presents this rank-ordered list of projects to OMB. Finally, NSF submits a prioritized list of projects to Congress as part of its budget submission.

The three new start projects cited in the FY 2004 budget request are considered highly meritorious by the S&E community, the NSF and the NSB.

The Scientific Ocean Drilling (SOD) Project was ranked as the highest priority because delaying initiation of the project until FY 2006 would greatly impact this existing community of researchers, and because of the significant level of complementary international effort and planning that is already underway. This project will charter and modify a drill ship which will work in a new scientific program (Integrated Ocean Drilling Program [IODP], in concert, and complementary to, a deep drilling vessel to be constructed and operated by Japan. Some of the drilling to be done from the SOD vessel will be used to guide and plan drilling from the Japanese vessel, which is scheduled to begin operations in 2007. Additional international members who help finance our existing ocean drilling program are prepared to join the new program, but will have trouble maintaining and committing their financial contribution if drilling from the SOD vessel is delayed until 2007. At present, the Japanese vessel has been constructed and is undergoing outfitting. If the U.S. does not meet its commitment, there will be no conventional drill ship capabilities for use in the IODP, and critical studies of climate change and the ocean biosphere will be jeopardized.

The two remaining new start projects, RSVP and Ocean Observatories, were felt to be of equal value, but for different reasons. RSVP ranked second, primarily for reasons of balance across scientific fields. RSVP is very well designed, well reviewed, and addresses important scientific questions that have the potential to transform our basic understanding of the universe. There are, as with SOD, performance and cost risks associated with delays. The host laboratory, Brookhaven National Laboratory, would be forced to lay off key staff and then rehire and/or replace them following an extensive shut-down of beams planned for use by RSVP; and, the international collaborators may have difficulty maintaining (as SOD will) the large financial contributions currently committed to RSVP, on order of $10 million (US). Nevertheless, these considerations do not outweigh the funding and stewardship issues represented in SOD. If ini-

tiated in FY 2006, RSVP can still be implemented successfully and make major contributions to science.

The Ocean Observatories project will enable a large group of researchers to perform ocean science in new ways. It was ranked third among the new start projects because it is not as urgent as SOD or RSVP, and again, for reasons of balance across scientific fields.