Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation (2004)

This appendix begins with a set of tables that provide a comparative summary of the process used to set priorities for large research facilities for the most relevant unit of comparison at NSF, DOE, and NASA: NSF’s MREFC Account, DOE’s Office of Science, and NASA’s Office of Space Science. These tables summarize the command and advisory structure, the process used to identify projects, the project evaluation criteria, the process used to involve the scientific community, the prioritization process, and the use of strategic planning in each process.

The tables are followed by several in-depth descriptions of features of the planning and approval process in use at different institutions. Included are a detailed presentation of the process at NSF, the strategic planning process at NASA’s Office of Space Science, the approval and funding process at DOE’s Office of Science, the 20-year facilities outlook activity at DOE’s Office of Science, a discussion of the selection process for NSF Science and Technology Centers and NSF Engineering Research Centers, a description of DOD’s Office of the Director for Defense Research and Engineering, and strategic planning and prioritization processes in use by the United Kingdom and Germany.

About one-fifth of the NSF $5.3 billion budget in FY 2003 supports the development and provision of “tools,” which are intended to provide what NSF calls “a widely accessible, state-of-the-art science and engineering infrastructure.” Large facility projects are funded through the Major Research Equipment and Facilities Construction (MREFC) account and through other accounts encompassed in the tools budget category. The MREFC account represents about one-eighth of the Foundation’s proposed investment in tools in FY 2003, rising to about one-sixth in the budget estimates for FY 2004. Despite representing a relatively small portion of the total NSF budget, the large facility projects supported through the MREFC account are highly visible because of their size and geographic concentration, and many of the issues raised by these projects must also be considered in other NSF projects and programs.

The large facility projects supported by NSF are nearly as varied as the scientific research that the Foundation supports. Some facilities represent new and increasingly powerful versions of instruments that have been used for decades to study the natural world, such as telescopes or particle accelerators. Other large facilities use new ways of gathering information; examples are a new facility designed to measure gravity waves generated by such cosmic events as star collisions and supernovae

and a proposed facility that would detect high-energy neutrinos in a large volume of Antarctic ice to provide information about the astrophysical sources of extremely high energy cosmic rays. Some large facilities primarily serve specific scientific disciplines, such as optical telescopes and radio-telescopes for astronomy and observatory networks for oceanography. Other facilities enable research in a wide array of disciplines. For example, the ground facilities, ships, and aircraft stationed in Antarctica allow scientists to study the atmosphere, ice, oceans, and geology of the region.

Regardless of their detailed characteristics, all large facility projects are being affected by the accelerating development of information technologies. Increasing quantities and varieties of information are being gathered, rapidly analyzed, and interpreted. Information technologies are also changing the fundamental nature of many large facility projects. New information technologies are making it possible, for example, for many large facilities to consist of smaller instruments and research projects in widely distributed geographic locations. The George E. Brown, Jr. Network for Earthquake Engineering Simulation, which is intended to improve the seismic design and performance of the U.S. civil and mechanical infrastructure, will consist of 15 experimental equipment sites linked by a high-performance Internet system. Elements of EarthScope, a distributed project to study the structure and dynamics of North America, will operate in nearly every county in the United States during the project’s eight- to ten-year lifetime. The proposed National Ecological Observatory Network will consist of geographically distributed observatories linked to laboratories, data archives, and computer modeling facilities.

The origins of large facility projects are as varied as the projects themselves. Some arise as logical outgrowths of previous research or facilities. Others originate as a consequence of new scientific development where the need for a new facility becomes apparent where no such need existed before. In some cases, such as the provision of high-speed networks and computers, a large facility is required to enable other kinds of research. Other large facilities are focused on the acquisition of data that cannot be obtained in any other way.

The impetus for all new large facility projects originates within the scientific community, but ideas take various routes to fruition. The community processes vary greatly from field to field. Often, self-organizing groups within a field of science or engineering develop the initial ideas

for a new facility and set scientific objectives for the facility by prioritizing competing needs. At other times, facilities have been proposed at the initiative of an individual scientist or a small group of researchers with a bold vision. NSF Program Officers and staff foster these initiatives by providing funds for meetings and workshops that facilitate the scientific community’s internal evaluation and maturation of these concepts. In every case, the mission of the NSF is to seek out the best ideas and the best scientists and to empower their investigations.

This process of nurturing and maturation of a concept for a facility can take many years to fully develop, or it can come together as a funded proposal quite quickly, depending on the nature of the proposal, the immediacy of scientific need, and the potential payoffs scientifically and for society in general. The NSF’s role in this process is reactive and responsive to the scientific community, rather than prescriptive, insuring that the highest quality proposals, as determined by peer review within the scientific community, are brought forward for implementation. NSF Program Officers are the key people who make the requirements for approval of such projects clear to the community.

In identifying new facility construction projects, the science and engineering community, in consultation with NSF, develops ideas, considers alternatives, explores partnerships, and develops cost and timeline estimates. By the time a proposal is submitted to NSF, these issues have been thoroughly examined.

Upon receipt by NSF, large facility proposals are first subjected to rigorous external peer review, focusing on the criteria of intellectual merit and the broad (probable) impacts of the project. Only the highest rated proposals, i.e., those that are rated outstanding on both criteria, survive this process. These are recommended for further review

• by an MREFC Panel that comprises the assistant directors and office heads, serving as stewards for their fields and chosen for their breadth of understanding, and chaired by the deputy director acting in consultation with the director; and subsequently

• by the National Science Board.

Both the MREFC Panel and the National Science Board look for a consistent set of attributes in projects they recommend:

• The project represents an exceptional opportunity to enable frontier research and education.

• The impact on a particular field of research is expected to be transformational.

• The relevant research community places a high priority on the project.

• The resulting facility will be accessible to an appropriately broad user community.

• Partnership possibilities for development and operation are fully exploited.

• The project is technically feasible and potential risks are thoroughly addressed.

• There is a high state of readiness to proceed with development, in terms of engineering cost-effectiveness, interagency and international partnerships, and management.

The MREFC review panel evaluates the merit of a proposed project and then prioritizes it relative to other projects under consideration. It first selects the new projects it will recommend to the director for future NSF support, based on a discussion of the merits of the science within the context of all sciences that NSF supports. Using these criteria, projects that are not highly rated are returned to the initiating directorates and may be reconsidered at a future time. Then, highly rated projects are placed in priority order by the panel in consultation with the NSF director. The review panel and the director place particular emphases on the following criteria to determine the priority order of the projects:

• How “transformative” is the project? Will it change the way research is conducted or alter fundamental science and engineering concepts or research frontiers?

• How great are the benefits of the project? How many researchers, educators and students will it enable? Does it broadly serve many disciplines?

• How pressing is the need? Is there a window of opportunity? Are there interagency and international commitments that must be met?

These criteria are not assigned relative weights because each project has its own unique attributes and circumstances. For example, timeliness may be crucial for one project and relatively unimportant for another. Additionally, the director must weigh the impact of a proposed facility on the balance between scientific fields, the importance of the project with respect to national priorities, and possible societal benefits.

After considering the strength and substance of the MREFC Panel’s recommendations, the balance among various fields and disciplines, and

other factors, the director selects the candidate projects to bring before the NSB for consideration. The NSB reviews individual projects on their merits and authorizes the Foundation to pursue the inclusion of selected projects in future budget requests. In August of each year, the director presents the priorities, including a discussion of the rationale for the priority order, to the NSB, as part of the budget process. The NSB reviews the list and either approves or argues the order of priority. As part of its budget submission, NSF presents this rank-ordered list of projects to OMB. Finally, NSF submits a prioritized list of projects to Congress as part of its budget submission.

Except for its facilities in the Antarctic, NSF does not directly operate research facilities. Rather, it makes awards to other organizations, such as universities, consortia of universities, or nonprofit organizations, to construct, operate, and manage the facilities. NSF enters into partnerships with those organizations, the details of which are most often defined through cooperative agreements, to accomplish this. The cooperative agreement defines the scope of work to be undertaken by the awardee and establishes the project-specific terms and conditions by which the NSF will maintain oversight of the Project. NSF has the final responsibility for oversight of the development, management, and performance of the facilities.

Each large facility project supported by NSF has a program manager in NSF who is the primary person responsible for all aspects of project oversight and management of the project within the foundation. The program manager carries out these responsibilities in accordance with an internal management plan (IMP) that has been crafted specifically for this project. The IMP defines a project advisory team (PAT) that consists of NSF personnel with expertise in the scientific, technical, management, and administrative issues associated with the project. The team works with the program manager to ensure the establishment of realistic cost, schedule, and performance goals for the project. The team also helps to develop terms and conditions of awards for constructing, acquiring, and operating a large facility. The NSF’s director for large facility projects works closely with the program manager, providing expert assistance on non-scientific and non-technical aspects of project planning, budgeting, implementation, and management to further strengthen the oversight capabilities of the foundation. The deputy also facilitates the use of best

management practices by fostering coordination and collaboration throughout NSF to share application of lessons learned from prior projects.

The awardee designates one person to be the project director. This person has the overall control and responsibility for the project within the awardee organization. Throughout the Implementation stage, the awardee executes and manages the project—either construction or acquisition—in accordance with the cooperative agreement between the awardee institution and the NSF. This phase of the project includes all installation, testing, commissioning, and acceptance. Oversight by the NSF during this phase is accomplished through periodic reviews, written reports by the awardee to the foundation that include documentation of technical and financial status using “Earned Value” reporting methods, annual work plans, periodic external reviews, and site visits.

By the end of the implementation stage, a proposal is submitted for operations and maintenance to the program manager. The program manager reviews proposals in accordance with the merit review procedures contained in Chapter V of the NSF’s Proposal and Award Manual and presents a recommendation for funding to his or her division director and assistant director—office head. The Director’s Review Board (or DRB) reviews proposals for awards exceeding the Director’s Review Board threshold (see Chapter VI of the NSF’s Proposal and Award Manual). Following DRB, the NSF director recommends awards above the National Science Board (NSB) threshold for approval to the NSB. The NSB reviews and approves awards recommended by the director. Following this, the assistant director—office head, through the division director, authorizes the program manager to recommend the making of an award in accordance with the proposal processing procedures contained in Chapter VI of the Proposal and Award Manual.

The program manager, with the Division of Grants and Agreements, drafts the cooperative agreement that will govern the project in accordance with the procedures contained in Chapter VIII of the Proposal and Award Manual. The Division of Grants and Agreements makes the award once the cooperative agreement is executed by it and the awardee.

Using the recommendations received from the MREFC Panel, the NSF director selects candidate projects to be considered by the NSB during one of its meetings in the year. According to the Guidelines, the director uses the following criteria in making this selection:

• Strength and substance of the information provided to the MREFC Panel.

• The relationship to NSF goals and priorities, including NSF’s educational mission.

• Appropriate balance among various fields, disciplines, and directorates, based upon a consideration of needs and opportunities.

• Guidance from the NSB on overall decision boundaries for the MREFC account, provided at the annual MREFC planning discussion (May).

• Opportunities to leverage NSF funds.

The NSB’s Committee on Program and Plans (CPP) takes the lead in reviewing the proposed project; a member of the committee leads the discussion. The criteria considered by CPP are these:

• Need for such a facility.

• Research that will be enabled.

• Readiness of plans for construction and operation.

• Construction budget estimates.

• Operations budget estimates.

After the CPP reviews the project, it makes recommendations to NSB for approving its inclusion in future budget requests and for approving actual project implementation.

Once the NSB has approved a project for funding, the director may recommend the project for inclusion in a future budget request to OMB. In August of each year, the NSB reviews the NSF budget, which includes the list of projects being submitted to OMB for funding. For projects included in the budget request, a capital asset plan and justification must be prepared, following a format developed by OMB. The capital asset plan and justification provides a summary of how much the project will cost to build and operate, information on its management and cost, schedule, and performance goals and milestones.

The list of major projects in the budget may be modified during negotiations between OMB and NSF. During that process, other parts of the executive branch, such as the White House Office of Science and Technology Policy, may provide input on the projects included in the budget.

After submission of the president’s budget to Congress in February of each year, congressional subcommittees and committees examine the proposed expenditures and begin the appropriations process. Congressional appropriators make decisions about whether to fund each of the large facility projects proposed for NSF in the President’s budget. In addition, because of budgetary constraints, the NSF director and OMB may decide not to request funds for large facility projects that the NSB has approved for inclusion in the budget.

By 2001, the NSB had approved six large facility projects that had not yet been funded. Concerns were expressed in Congress and elsewhere that political pressures rather than scientific merit would increasingly determine which projects received appropriations. In 2001, Congress asked NSF to rank the six projects in order of priority. NSF responded by dividing the projects into two categories of three projects each, with no ordering within a category. In its appropriations for FY 2003, Congress provided funds for two of the three projects in the high-priority category. In the 2004 budget request, NSF further ranked the projects, requesting funding for the remaining high-priority project in that fiscal year and proposing to start funding for the other three in FY 2005 and FY 2006.

The recent focus on NSF’s setting of priorities among large facility projects continues a long-running discussion of the best way for NSF to support such undertakings. In the June 12, 2002, letter to NAS President Bruce Alberts that led to the present study, six senators stated that “funding requests by the Foundation for large facility projects appear to be ad hoc and subjective.” The letter directed the National Academies to “review the current prioritization process and report to us on how it can be improved.”

In the FY 2002 House conference report, Congress provided guidance as to the use of MREFC and R&RA expenditures. It stated that the purpose of the MREFC account is to provide resources for the acquisition, construction, and commissioning of large-scale research equipment and facilities, whereas the R&RA account is to fund planning, design, operations, and maintenance costs. Unless an exemption is granted, MREFC funding can no longer be used to fund planning and design costs, as has occurred in the past.

The NSF funds two programs for creating university-based research centers: The Science and Technology Centers (STC) and the Engineering Research Centers (ERC) programs. While the specific program goals of

these two programs differ substantively, the overall review, renewal, and oversight schemes are similar. Both programs provide initial funding for five years with the possibility of extension to ten based on reviews performed during the initial funding period. Center selection follows an extensive review process wherein a program solicitation is sent to request preliminary proposals for consideration. After initial review, invitations are sent requesting full proposals from those groups whose pre-proposals meet review criteria. Additional review criteria are then imposed to decide the award recipients. The following describes specific details of the individual programs.

The NSF created the STC program in 1987 “to fund important basic research and education activities and to encourage technology transfer and innovative approaches to interdisciplinary programs” [1]. Since its inception, the program has funded four classes of centers in 1989, 1991, 2000, 2002, and the NSF released a program solicitation for preliminary proposals to the Class of 2005 in June 2003 [2, 3]. Twenty-three centers have completed the full 10 years of funding, and 11 centers currently operate on NSF funds. Two centers were closed prematurely due to management issues [2]. For consideration in the STC program, proposals must demonstrate cross-disciplinary research goals, an extensive education program, and a means of enabling knowledge transfer to industry or other interested parties such as government [2]. An example of an STC is the Science and Technology Center for Adaptive Optics at the University of California, Santa Cruz, which began funding in FY2000. The center brings together astronomers and vision scientists “to develop new instruments optimized for adaptive optics” with applications as diverse as imaging planets around nearby stars and 3-D construction of optic nerve fibers [4].

STC awards are made following a multistage review process. A program solicitation announces the request for preliminary proposals. These proposals are reviewed by “panels of individuals intellectually distinguished in their fields and experienced in integrative science, mathematics, engineering and technology research” [3]. The panels examine the preliminary proposals based on the merit review criteria common to all NSF programs:

• Intellectual merit.

• Broader impacts.

• Integration of research and education.

• Integration of diversity.

The preliminary proposals must also address three STC-specific review criteria:

• The value-added from funding the activity as a center.

• The efficacy of the proposed leadership and management plans.

• The integrative nature of the proposed center.

Groups whose preproposals demonstrate the most promise are then invited to submit full proposals. In the award selection process for the Class of 2005, 159 preproposals resulted in 37 invitations for full proposals [2]. The full proposals are again judged by mail and panel review according to the same criteria but with a special emphasis on the integrative nature of the center [3]. A subselection deemed most “worthy” then undergo site visit review where added emphasis is placed on the proposed management and leadership plan. An external ad hoc STC Advisory Committee makes a priority list of recommended centers based on the above criteria, “the potential national impact and legacy of the proposed activity, the balance of awards among scientific fields, geographical distribution, and the combined ability of the proposed Centers to meet the objectives of the STC Program.” NSF management uses the list to make funding recommendations to the NSF director and the Director’s Review Board. According to Bruce Umminger, senior scientist in the NSF Office of Integrative Activities (OIA), the stringent review process reflects the award size, $1.5 million–4 million/year, and the high visibility of the centers [2].

Following the award, centers continue to receive NSF oversight through their respective NSF directorates in coordination with the OIA. During its first 5 years of funding, each center undergoes annual reviews on which support for the following year is contingent. An in-depth review in the fourth year determines whether the center can renew for an additional 5 years of funding. The renewal request is evaluated through an ad hoc mail review and a formal on-site visit. In the event that a center is not renewed, additional funding at a decreased level is provided for 1 year. After a successful fourth-year review and renewal, centers continue to undergo NSF review at least every 18 months through the reduced, phase-out period of funding in years 9 and 10.

NSF created the ERC program in 1985 “to develop a government-industry-university partnership to strengthen the competitive position of U.S. firms in world trade and change the culture of engineering research and education in the U.S.” [5]. The most recent crop of centers were

announced October 3, 2003, bringing the number of engineering research centers currently under NSF support to 24 [6]. Before the announcement of the four new centers in 2003, the program had supported the creation of a total of 37 new centers over its lifetime [7]. Three elements structure the backbone of the ERC program: cross-disciplinary and systems-oriented research, education and outreach, and industrial collaboration and technology transfer [8]. The technological emphases of the current centers include bioengineering; design, manufacturing, and product development systems; earthquake engineering; and microelectronic systems and information technology. Among the newly named ERCs is the Engineering Research Center for Extreme Ultraviolet (EUV) Science and Technology (EUV ERC), headquartered at Colorado State University. EUV ERC will “explore the interface of physics, electrical engineering, chemistry, and biology using [EUV] light.” The center already has begun partnerships with members of the semiconductor, laser, and advanced optics industries [6].

Similar to the STC program, ERC awards result from an extensive review process. Preliminary proposals are solicited through a program announcement inviting prospective teams to “develop a ten-year vision for advances in an emerging, potentially revolutionary or transforming engineered system” [7]. These preproposals are then reviewed by a panel of outside experts. At the time of submission, proposers are invited to suggest names of appropriate or inappropriate reviewers. When selecting reviewers, NSF places extra emphasis on finding persons from outside academe, from minority-serving institutions, and from related disciplines. The preproposals are evaluated according to the standard NSF criteria (listed above in the STC description) and a set of ERC-specific criteria [7]:

• Potential of the proposed engineered system to spawn new industries; transform the industrial base, service delivery system, or infrastructure; and have societal impact.

• Research plan that targets critical systems goals and challenging scientific and technical barriers and proposes projects to address them.

• Indication of an extensive understanding of the state of knowledge and the state of the art.

• An education plan that includes curriculum development at all levels.

• An outreach program that reaches a broad spectrum of faculty, teachers, and students.

• Convincing rationale for industrial partners and plan for including them in all aspects of the project.

• Appropriate institutional configuration that is well integrated across institutions in the case of multiuniversity centers.

• Available expertise to address all aspects of center research and

capable leadership, faculty, and students representing a diverse mix of sex, race, and ethnicity.

• An effective organizational structure and management plan.

• Necessary equipment, facilities, and laboratory space.

• A commitment to the interdisciplinary, educational, and diversity-building goals of the ERC program.

After initial review, full proposals are invited from a smaller selection of proposers. The full proposals are then reviewed according to additional criteria that include [7]:

• Proposed center space that can encourage interdisciplinary collaboration and house center management.

• Commitments from industry to become fee-paying members of the center.

• Industrial agreements that indicate a centerwide collaboration rather than a collection of individual projects and that facilitate technology transfer between the center and industry [7].

Reviewers are asked to submit a summary rating and recommendation on whether to fund each proposal. An NSF program officer assigned to each proposal uses the reviewers’ advice to formulate a recommendation. These recommendations are sent to the Division of Engineering Education and Centers (EEC), where the director decides whether to accept the recommendation. The recommended proposals are forwarded to the Division of Grants and Agreements, where they are reviewed for business, financial, and policy implications, and the final funding decision is made [7].

After the initial award, all ERCs continue to receive NSF oversight and review. Each ERC is required to submit an annual report of progress and plans, and members of each ERC’s leadership attend an annual meeting in Washington, D.C., to discuss progress, receive updates, and provide advice on the program. ERCs must also collect and submit progress indicator data to NSF. Like the STC program, the first ERC award provides funding for 5 years, with awards of up to $2.5 million/year. The annual reports submitted by the centers undergo outside merit review that forms the basis for determining funding levels for the following year. During either the third or sixth years, a center may submit a proposal for renewal to extend the award to 10 years. In the event that such a proposal does not receive approval, funding is phased out over a 2-year period “to protect the graduate students” [7]. A center that successfully applies for renewal will received decreased funding in years 9 and 10 to facilitate the transition to self-sufficiency [7].

Center for Advanced Materials for Water Purification, Urbana, IL

University of Illinois at Urbana Champaign (sponsor), Stanford University, and Clark Atlanta University

Center for Biophotonics Science and Technology, Sacramento, CA

University of California, Davis (sponsor), University of California, San Francisco, University of California, Berkeley, Stanford University, and the Lawrence Livermore National Laboratory

Center for Embedded Network Sensing, Los Angeles, CA

University of California, Los Angeles (sponsor), University of Southern California, University of California, Riverside, the NASA Jet Propulsion Laboratory, California State University, Los Angeles, and the California Institute of Technology

Center for Integrated Space Weather Modeling, Boston, MA

Boston University (sponsor), Alabama A&M University, Dartmouth College, Rice University, Stanford University, University of California, Berkeley, University of Colorado, Boulder, University of Texas, El Paso, National Center for Atmospheric Research, NOAA Space Environment Center, and Science Applications International Corporation

Center on Material and Devices for Information Technology Research, Seattle, WA

University of Washington (sponsor), University of Arizona, California Institute of Technology, University of Southern California, University of California, Berkeley, and University of California, Santa Barbara

National Center for Earth-surface Dynamics, Minneapolis, MN

University of Minnesota, Twin Cities (sponsor), Fond Du Lac Tribal and Community College, Massachusetts Institute of Technology, University of California, Berkeley, Princeton University, and Science Museum of Minnesota

Engineering Research Center for Extreme Ultraviolet Science and Technology (EUV ERC), Fort Collins, CO

Colorado State University (headquarters), University of Colorado at Boulder, University of California, Berkeley, and Lawrence Berkeley National Laboratory

Engineering Research Center for Environmentally Beneficial Catalysis (CEBC), Lawrence, KS

University of Kansas in Lawrence (headquarters), University of Iowa in Iowa City, and Washington University at St. Louis

Engineering Research Center for Collaborative Adaptive Sensing of the Atmosphere (CASA), Amherst, MA

University of Massachusetts at Amherst (headquarters), Colorado State University, University of Oklahoma, University of Puerto Rico at Mayaguez, National Severe Storms Laboratory, Oak Ridge National Laboratory, and Massachusetts Department of Education

Engineering Research Center for Biomimetic Microelectronic Systems (BMES), Los Angeles, CA

University of Southern California (headquarters), California Institute of Technology, and University of California, Santa Cruz

[1] NSF Fact Sheet on Science and Technology Centers: Integrative Partnerships.

[2] National Science Board meeting discussion by Bruce Umminger, October 16, 2003.

[3] NSF Program Solicitation for STC (NSF 03-550), March 3, 2003.

[4] NSF Press Release (NSF PR99-45), July 29, 1999.

[5] Report from the Engineering Education and Centers Division of the NSF Directorate for Engineering, The Engineering Research Centers (ERC) Program: An Assessment of Benefits and Outcomes, December 1997.

[6] NSF Press Release (NSF PR03-115), October 3, 2003.

[7] NSF Program Solicitation for ERC (NSF 02-24), November 29, 2001.

[8] NSF website on ERC, www.eng.nsf.gov/eec/erc/directory/erc_a.htm.

The Department of Energy’s Office of Science (SC) has a long history of initiating and supporting large-facility projects. Although the success of individual project management has been varied, the current DOE guidelines provide a robust framework for the formal development of project ideas.

On the basis of input from the SC advisory committees, staff perform an initial evaluation of all proposal applications to ensure that required information is provided, that the proposed effort is technically sound and feasible, and that the effort is consistent with program funding priorities. For applications that pass the initial evaluation, the office reviews and evaluates each application received on the basis of criteria set forth below and in accordance with the merit-review system. Evaluators are selected on the basis of their professional qualifications and expertise. They evalu-

ate new and renewal applications according to the following criteria, listed in descending order of importance⁶.

1. Scientific or technical merit or educational benefits of the project.

2. Appropriateness of the proposed method or approach.

3. Competence of applicant’s personnel and adequacy of proposed resources.

4. Reasonableness and appropriateness of the proposed budget.

5. Other appropriate factors, established and set forth by the office in a notice of availability or in a specific solicitation.

DOE considers, as part of the evaluation, other available advice or information and such program-policy factors as ensuring an appropriate balance among program subjects. In addition to the evaluation criteria, the recipient’s performance under an existing award during the evaluation of a renewal application are considered. Applications are chosen for award on the basis of the findings of the technical evaluations, the importance and relevance of the proposed application to the office’s mission, and funds availability. Cost reasonableness and realism are also considered to the appropriate extent.

For projects over $5 million, the procedure is more formal. Special guidelines from DOE’s Office of Engineering and Construction Management apply, and the Office of Science’s Construction Management Support Division is more directly involved. Projects are tracked by their progress along so-called critical decisions (CDs). A CD is a formal determination or decision at a specific point in a project phase that allows the project to proceed to the next phase and commit resources. CDs are required during the planning and execution of a project, for example, before commencement of conceptual design, commencement of construction, or start of operations. CDs for traditional construction projects include the following⁷:

• CD-0, Approve Mission Need

  • Authorizes use of program funds for conceptual design studies.

  • Requires preconceptual planning document.

  • Requires mission-need justification document and external independent review.

• CD-1, Approve Preliminary Baseline Range

  • Allows expenditure of project engineering and design funds for design work.

• CD-2, Approve Performance Baseline

  • Establishes baseline budget for construction.

  • Continues design process.

  • Authorizes development of request for construction funding.

  • Requires review of contractor management system.

  • Requires external independent review of performance baseline.

• CD-3, Approve Start of Construction

  • Approves expenditure of funds for construction.

  • Requires readiness of final design and procurement packages.

  • Requires external independent review of execution readiness.

• CD-4, Approve Start of Operations or Project Closeout

  • Allows start of operations or closeout of project.

  • Includes readiness review and acceptance report.

To ensure that resources are allocated to the most scientifically promising experiments, DOE and its national laboratories seek external input by using a variety of advisory bodies. The FACA-chartered advisory committees provide advice to DOE on a continuing basis regarding the direction and management of the national energy research program. SC comprises six science offices and the associate director of each office is advised by its own corresponding advisory committee. The advisory committees meet regularly to advise the sponsoring agencies (for instance, the High Energy Physics Advisory Panel is jointly sponsored by DOE and NSF) on their research programs, assess their scientific productivity, and evaluate the scientific case for new facilities. Each advisory committee solicits input from the community during its regular long-range planning exercises. The call for project solicitations is made at meetings of professional societies and usually at community workshops that are considering related issues.

The Office of Science has six program advisory committees, each FACA-chartered. Each advisory committee provides valuable, independent advice to DOE on the complex scientific and technical issues that arise in the planning, management, and implementation of its program. Recommendations include advice on establishing research and facilities priorities, determining proper program balance among disciplines, and identifying opportunities for interlaboratory collaboration, program integration, and industrial participation. The committee includes mainly representatives of universities, national laboratories, and industries involved in energy-related scientific research. Particular attention is paid to obtaining a diverse membership with a balance of disciplines, interests,

experience, points of view, and geography. Members are appointed by the secretary of energy; the committees are staffed by DOE personnel.

The director and the associate directors periodically charge each advisory panel to perform assessments or to address particular questions. The committees typically convene ad hoc subpanels to respond to the charge; an ad hoc report is filed with the parent advisory committee on completion. The committee can adopt or modify the subpanel report, with an accompanying written justification, before forwarding its final recommendation to the commissioning office.

One of the most important functions of the advisory committees is the development of long-range plans that express communitywide priorities for research. The most recent such plan was submitted in January 2003 at the request of the director as part of the 20-year facilities roadmap initiative and presented a roadmap for each field, laying out the science opportunities that each planning subpanel could envision as possibilities for the next 20 years. Large facility projects first appear in the community, work their way into the frequent but irregular advisory committee long-range plans, and eventually undergo development, typically at one of the national laboratories to leverage existing resources and expertise.

The DOE’s Office of Science began to prioritize future major facilities in the fall of 2002. The Associate Directors of the Office of Science⁹ were asked to list major facilities required for world scientific leadership in their respective programs out to 2023. Each Associate Director was given a funding “envelope” under which they were to include their estimated research budgets as well as the major facility planning, construction, and operating costs.¹⁰

Forty-six facilities were identified and phased to conform to perceived scientific opportunities over this 20 year period. Internal hearings were held, with each Associate Director describing the nature of the recommended facilities, together with the scientific rationale behind their choices. This process was completed in December 2002.

The program prioritizations by the Associate Directors were then submitted in mid-January 2003 to the respective program’s Advisory Committee, with a request for an analysis of the relative scientific opportunities associated with each of the facilities proposed by their respective Associate Director, and with any additions they felt important that may have been omitted. In some cases, the Committees were requested specifically to work together to capture the interdisciplinary needs that might be missed if a Committee focused too narrowly on its own traditional discipline. The Office of Science Advisory Committees are chartered to bring to each program the full breadth of perspectives of the U.S. scientific community. Of the 118 people that sit on the Office of Science Advisory Committees, 64 percent are from universities, 15 percent from DOE laboratories, 10 percent from industry, 3 percent from other government agencies, and 8 percent from other types of institutions.

The Advisory Committees recommended 53 major facilities for construction, and assessed each according to two criteria: scientific importance and readiness for construction. Against the first criteria, the committees divided their facilities into three categories: highest scientific importance, secondary scientific importance, and hard-to-assess scientific importance. The Committees also categorized the facilities into “near-term,” “mid-term,” and “far-term” according to their readiness for construction.

The results were plotted in a matrix illustrated in Figure D-1. “Highest scientific importance” was divided into categories A, B, and C, depending upon readiness for construction. “Secondary scientific importance” was labeled as category D, and “hard-to assess scientific importance” as category E.

With this input from the Advisory Committees, the challenge remained to prioritize the facilities across scientific disciplines.¹¹ The

FIGURE D-1. DOE’s Office of Science facilities matrix.

Director of the Office of Science addressed this challenge by prioritizing the 53 facilities according to his assessment of their scientific promise and their fit with the Department’s missions. The costs associated with the Office of Science’s base research programs and the other responsibilities were added, and the entirety was made to fit under an aggressive funding envelope (see footnote 2 in this appendix) extended through 2023. Twenty-eight projects survived, along with a contingency in the “out-years,” recognizing the need for flexibility over a 20-year period.

The Twenty Year Outlook represents a snapshot—the DOE Office of Science’s best guess today at how the future of science and the need for scientific facilities will unfold over the next two decades. We know, however, that science changes. Discoveries, as yet unimagined, will alter the course of research and so the facilities needed in the future.

For this reason, the Outlook should be assessed periodically in light of the evolving state of science and technology. The Outlook will also serve

as a benchmark, enabling an evaluation of facilities proposed in the future against those on this list. Future revisions should maintain the funding envelope used to guide this list, enforcing fiscal discipline upon discussions and requiring the elimination of facilities in order to accommodate more important or exciting prospects.

The DOE’s Office of Science recognizes that the breadth and scope of the vision encompassed by these 28 facilities reflects a most aggressive and optimistic view of the future of the Office. Nevertheless, we believe that it is necessary to have and discuss such a vision. See Figure D-2. Despite the uncertainties, it is important for organizations to have a clear understanding of their goals and a path toward reaching those goals. The Outlook offers just such a vision.

The 28 facilities are listed by priority [above]. Some are noted individually; however others, for which the advice of the Advisory Committees was insufficient to discriminate among relative priority, are presented in “bands.” In addition, the facilities are roughly grouped into near-term priorities, mid-term priorities, and far-term priorities (and color-coded red, blue, and green, respectively) according to the anticipated R&D timeframe of the scientific opportunities they would address.

Each facility listing is accompanied by a “peak of cost profile,” which indicates the onset, years of peak construction expenditure, and completion of the facility. Because many of the facilities are still in early stages of conceptualization, the timing of their construction and completion is subject to the myriad considerations that come into play when moving forward with a new facility.

NASA’s strength is in its strategic planning, which uses community input to formulate priorities in a public and independent process. As a mission agency, the Office of Space Science (OSS) sets goals and objectives that are developed through a careful process. Project proposals are measured against them and organized for development. On the front end, announcements of opportunity for particular types of projects are used to solicit proposals. These can be as general as a “dark energy probe” or as specific as a specific instrument set for a planned mission. On the implementation and operation end, NASA is relatively successful through the use of its subordinate but independent centers (typically

Goddard Space Flight Center for OSS). Finally, NASA projects are spearheaded by, essentially, principal investigators, who include a management plan in the proposal and work closely with the host center.

NASA engages in an agencywide strategic planning process every 3 years, producing an updated 5-year plan that includes a detailed roadmap accommodating the projected budget envelope. That process is designed to satisfy NASA’s compliance with the Government Performance and Results Act of 1993 (GPRA), but it has become a powerful tool for identifying and shaping the mission of the agency, tying together budget, performance, and vision. To be synchronized with the triennial revision of the agency strategic plan mandated by GPRA, OSS revises its own strategic plan at the same interval. Each unit measures the consistency of its projects against the overall NASA objectives, the OSS objectives, and its own goals to produce a roadmap. The roadmaps are considered jointly by the enterprise at a closed but broadly representative retreat where determinations about relative importance and priority are made. OSS then drafts its strategic plan; this involves another series of open planning meetings that synthesize the different roadmaps, the broader goals of the enterprise, and additional information about budget expectations. NASA headquarters uses each enterprises’s strategic plan to create the final and authoritative NASA strategic plan.¹³ This final document becomes the basis for the agency’s annual budget request for the next few years.

The NASA vision communicates the agency’s mandate in the 21st century. The NASA mission lays out a clear path to the future. The mission provides a framework for developing goals that each unit of NASA must achieve. NASA has seven strategic goals (expanding on the mission) that enable each enterprise to focus planning, manage programs, and measure results. Each of the agency’s six enterprises uses the strategic goals to define its programs. The agency goals are further broken down into enterprise objectives. (These are, essentially, projections of the agency’s mission onto the subspace spanned by OSS.)

The strategic-plan development process depends on active involvement of outside parties, especially the space-science research community. The National Research Council (through its Board on Physics and Astronomy, Space Studies Board, and their discipline subcommittees) develops long-range strategic-program assessments and recommendations. The Space Science Advisory Committee (SScAC), on the basis of inputs

from its own subcommittees, provides the enterprise with roadmaps that integrate National Research Council and additional community inputs with technical, budget, and programmatic factors.

A narrative description of the process used to develop the FY 2000 Space Science Strategic Plan is an instructive example. This description is from the The Space Science Enterprise Strategic Plan published in November 2000; note that the Space Science Enterprise refers to the Office of Space Science.

The Space Science Enterprise strategic plan serves several purposes. It facilitates a consensus process in the science community that focuses on goals and priorities for the future. It serves OSS by providing a reference for programmatic decision making, and it provides input to the overall agency strategic plan and material to meet GPRA requirements. The strategic plan becomes a tool for use by OMB and Congress in program and budget advocacy. Finally, it provides a handbook for the public on what space science is going to do and why.

The Department of Defense (DOD) Science and Technology (S&T) Program supports the fundamental research, development, and demonstrations in science and technologies identified as important to military capabilities and operations. The extent of the S&T Program includes the development of the nation’s high technology weapons systems, the technology base upon which those system rely, and the equipment to both support and prepare military personnel. The program also plays an important role in developing certain key technologies that transfer to commercial applications and help to grow the overall economy. Basic research (6.1), applied research (6.2), and advanced technology development (6.3) together comprise DOD’s Science and Technology (S&T) program. S&T projects seek new ways of accomplishing tasks of military value and the underlying scientific and engineering principles involved.

Within DOD, the Office of the Director of Defense Research and Engineering is primarily responsible for the basic research plan of the department, “to ensure that the warfighters today and tomorrow have superior and affordable technology to support their missions, and to give them revolutionary war-winning capabilities.”

Project proposals solicited by the Directorate of Defense Research and Engineering are typically judged on the following criteria:

• DOD mission priority.

• Military advantage gained by exploiting the requested S&T.

• Merit of scientific study.

• Potential for significant progress.

• Appropriateness of solution for meeting requirements.

DOD uses defense technology objectives (DTO) to provide focus for the development of technologies that address identified military needs across the department. Each DTO identifies a specific technology advancement that will be developed or demonstrated, with expected date of availability, specific benefits resulting from it, and the amount of funding needed. The DTO process is used to comply with GPRA. The output of this process includes budget and management decisions.

The methodology used for evaluating the S&T program is known as technology area reviews and assessments (TARA). TARA is the department’s official response to GPRA, and it is a mechanism to evaluate science and technology programs through expert peer reviews. The evaluation of basic and applied research is carried out by internal agency panels of experts and by TARA review panels. Each panel consists of 10-12 technical experts from academe, industry, and nonprofit research organizations. Most TARA team members are recognized experts from the National Academies, the Defense Science Board, the scientific advisory boards of the military departments, industry, and academe. Each is chaired by a senior executive appointed by the deputy under secretary for S&T.

These teams are asked to evaluate the programs for quality, advances in leading the state of the art in research areas, and for their scientific vision. The department requires that two-thirds of each panel be experts from outside DOD. One-third of each panel’s members are refreshed at the time of each reviewing cycle.

The Defense Science Board operates by forming task forces consisting

of Board members and other consultants and experts to address those tasks referred to it by formal direction. The products of each task force typically consist of a set of formal briefings to the board and appropriate DOD officials and a written report containing findings, recommendations, and a suggested implementation plan. The board reports directly to the secretary of defense through the under secretary of defense for acquisition, technology and logistics (USD [AT&L]) while working in close coordination with the DDR&E to develop and strengthen the department’s research and development strategies for the 21st century. The board met for the first time on September 20, 1956. Its initial assignment examined basic research, component research, and technology advancement programs and their administration in S&T areas of interest to the DOD. On December 31, 1956, a charter was issued specifying the Board as advisory to the assistant secretary of defense (research and development). The subsequent consolidation of the offices of the assistant secretaries of defense for R&D and applications engineering in 1957 resulted in the board’s reconstitution as adviser to the secretary of defense through the assistant secretary of defense (research and engineering).

The mission of the board is to advise, in response to formal requests, the secretary of defense, the deputy secretary of defense, the under secretary of defense for acquisition, technology and logistics, and the chairman of the Joint Chiefs of Staff on matters relating to science, technology, research, engineering, manufacturing, and acquisition process and other matters of special interest to the DOD. The board specifically concerns itself with pressing and complex technology problems facing DOD in such areas as research, engineering, and manufacturing. It seeks to ensure the identification of new technologies and new applications of technology in these areas for the strengthening of national security. The board does not advise on individual procurements.

DARPA fulfills a unique role within the Department of Defense. As a Defense Agency, DARPA reports to the Secretary of Defense. The Director, Defense Research and Engineering has been assigned to be DARPA’s Principal Staff Assistant (PSA). DARPA is the Secretary of Defense’s only research agency not tied to a specific operational mission. DARPA sup plies technological options for the entire Department. DARPA is designed

to be the “technological engine” for transforming the Department of Defense.

DARPA’s decision-making process is somewhat unusual for a government agency. It is informal, flexible, and yet highly effective because it focuses on making decisions on specific technical proposals based on the factors discussed above. There are two reasons for this. DARPA is a small, flat organization rich in military technological expertise. There is just one porous management layer (the Office Directors) between the program managers and the Director. With less than 20 senior technical managers, it is easy to make decisions. This management style is essential to keeping DARPA entrepreneurial, flexible, and bold. DARPA’s management philosophy is to pursue fast, flexible, and informal cycles of “think, propose, discuss, decide, and revise.” This approach may not be possible for most government agencies, but it has worked well for DARPA.

The Basic Process: DARPA uses a top-down process to define problems and a bottoms-up process to find ideas, involving the staff at all levels. DARPA’s upper management and program managers identify “DARPAhard” problems by talking to many different people and groups. This process includes:

• Specific assignments

• Requests for help

• Discussions with senior military leaders

• Research into recent military operations

• Discussions with defense agencies

• Discussions with the intelligence community

• Discussions with other government agencies or outside organizations

• Visits to Service exercises or experiments.

During DARPA’s program reviews, which occur throughout the year, DARPA’s upper management looks for new ideas from program managers (or new program managers with ideas) for solving these problems. At the same time, management budgets for exploring highly speculative technology that have far-reaching military consequences.

Program managers get ideas from many different sources, such as:

• Their own technical communities;

• Suggestions from DOD-wide advisory groups, including the Defense Science Board and Service science boards;

• Suggestions from DARPA-sponsored technical groups

• Suggestions from industry or academia, often in response to published Broad Area Announcements or open industry meetings

• Breakthroughs in DARPA programs and/or U.S. or international

research

During reviews of both proposed and ongoing programs, DARPA’s assessment is often guided by a series of questions. These seemingly simple queries help reveal if a program is right for DARPA.

• What is the project trying to do?

• How is it done now and what are the limitations?

• What is truly novel in the approach that will remove those limitations and improve performance? By how much?

• If successful, what difference will it make??

• What are the midterm exams required to prove the hypothesis?

• What is the transition strategy?

• How much will it cost?

• Are the programmatic details clear?

On behalf of the research councils of the United Kingdom (RCUK), the UK Office of Science and Technology (OST) released the second edition of its Large Facilities Strategic Roadmap in June 2003.¹⁷ Several excerpts from that document are provided here. In 2001, the UK initiated a new process for treating proposals for large facilities. It begins with the production of a roadmap that contains projects currently identified as being of the highest strategic importance. There is no commitment to funding and no guarantee that the UK will participate in all the projects in the roadmap.

From Section 1, “Executive Summary”:

• Where there could be an international dimension to the proposed

facility and therefore opportunity to share costs and develop relationships to benefit the UK science programme;

• Where the facility supports the requirements of research communities of more than one Research Council;

• Where the capital investment is greater than the sum of £25 million, when it represents a significant element of an individual Research Council’s budget line.

The roadmap is scheduled to be updated every 2 years; the June 2003 version represents the second iteration; it contains projects in 10 strategic fields. The roadmap is the first stage of a gateway process that requires all large capital investments to be managed as discrete projects subject to review and independent scrutiny at all stages in their life cycle.

From Section 5, “OGC Gateway Process and How it is Used in Large Science Facilities”:

In the case of scientific projects, the first stage must involve an independent assessment of scientific value, including some form of peer review that addresses the following criteria:

• Importance (depth) of science knowledge to be delivered by project.

• Breadth of science knowledge that will benefit from investment.

• Match with international positioning of UK science.

• Strength of opportunity for training (links to numbers of users).

• Contribution to or from UK technology-industry base.

• Opportunity for spin-off and exploitation.

See additional material in Appendix E concerning the criteria used in the strategic roadmapping process.

Germany has a distinctive science system, with responsibilities shared between federal and state governments. At the federal government level, the Ministry of Education and Research (BMBF) is the key player. Various nonprofit bodies play important roles in distributing funding; for example the Deutsche Forschungsgemeinschaft is a major source of project funding for universities, and the Max Planck Gesellschaft operates 81 research institutes.

National policy coordination and policy advice are provided by two joint federal-state bodies:

• The Bund-Länder-Kommission für Bildungsplanung und Forschungs-förderung, which is a forum for discussions between federal and state ministries.

• The Wissenschaftsrat, or Science Council, an independent body whose members are appointed by the federal president and that advises federal and state governments on all matters of higher education and research policy. Its members include representatives of the science community and nominated representatives of state and federal governments.

Over the last 20 years, the BMBF has set up ad hoc groups to recommend priorities for funding large facilities that have been proposed by the scientific community whenever the need was felt (their reports have become known by the names of the chairs: Mayer-Leibnitz, Pinkau, and Grossman). Such recommendations go to the minister, who makes final decisions (subject to discussion with the Finance Ministry and at cabinet level when appropriate).

In 2000, the BMBF asked the Wissenschaftsrat (henceforth referred to as the Science Council) to review nine proposals for large facilities for basic research (each priced at at least 15 million euros); and in January 2001, the Science Council established a working group for this purpose. The group included scientists at universities and research establishments in Germany, the United States, and Switzerland and “individuals involved in and representing national and international scientific administrations.” The group established six subpanels composed of 57 external experts, including 37 from abroad.

The Science Council’s working group not only evaluated specific projects but laid out a general framework for such work and proposed criteria. It seems likely that, as proposed in the report, the Science Council will be asked to carry out such reviews at regular intervals in the future, replacing the previous ad hoc procedure.

The council established a working group with six expert subpanels that assessed projects in particular fields according to the following criteria:

• The probability of fundamental new insights or the possibilities of decisive scientific advances that could be achieved only with the large-scale facility.

• The large-scale facility’s technical feasibility and degree of technical innovation.

• The scientific and technical competence of the institutions involved.

• The already existing or anticipated acceptance of the (potential) users with immediately relevant and related expertise.

• The fulfillment of various objectives of importance for research (transfer, international perspectives, and promoting young scientists).

The working group then considered all the projects, taking into account:

• Scientific potential of the research program.

• Fulfillment of science and technology policy goals as formulated in 10 general “theses on the significance of large-scale facilities for basic scientific research.”

• Degree of maturity of the technical concept and, connected to it, the possible timeframe for implementing the individual components.

• The context of further national and international scientific development of the research fields to which they belong and their interaction with other disciplines.

Finally, the working group divided the nine projects into three categories, the first meriting unconditional support, the second with specific points yet to be clarified, and the third requiring additional development.

See Appendix E for additional information on these criteria.

This page intentionally left blank.