Appendix D
Approval Processes in Other Agencies and Other Countries

This appendix begins with a set of tables that provide a comparative summary of the process used to set priorities for large research facilities for the most relevant unit of comparison at NSF, DOE, and NASA: NSF’s MREFC Account, DOE’s Office of Science, and NASA’s Office of Space Science. These tables summarize the command and advisory structure, the process used to identify projects, the project evaluation criteria, the process used to involve the scientific community, the prioritization process, and the use of strategic planning in each process.

The tables are followed by several in-depth descriptions of features of the planning and approval process in use at different institutions. Included are a detailed presentation of the process at NSF, the strategic planning process at NASA’s Office of Space Science, the approval and funding process at DOE’s Office of Science, the 20-year facilities outlook activity at DOE’s Office of Science, a discussion of the selection process for NSF Science and Technology Centers and NSF Engineering Research Centers, a description of DOD’s Office of the Director for Defense Research and Engineering, and strategic planning and prioritization processes in use by the United Kingdom and Germany.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Appendix D Approval Processes in Other Agencies and Other Countries This appendix begins with a set of tables that provide a comparative summary of the process used to set priorities for large research facilities for the most relevant unit of comparison at NSF, DOE, and NASA: NSF’s MREFC Account, DOE’s Office of Science, and NASA’s Office of Space Science. These tables summarize the command and advisory structure, the process used to identify projects, the project evaluation criteria, the process used to involve the scientific community, the prioritization process, and the use of strategic planning in each process. The tables are followed by several in-depth descriptions of features of the planning and approval process in use at different institutions. Included are a detailed presentation of the process at NSF, the strategic planning process at NASA’s Office of Space Science, the approval and funding process at DOE’s Office of Science, the 20-year facilities outlook activity at DOE’s Office of Science, a discussion of the selection process for NSF Science and Technology Centers and NSF Engineering Research Centers, a description of DOD’s Office of the Director for Defense Research and Engineering, and strategic planning and prioritization processes in use by the United Kingdom and Germany.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation NSF   Relevant unit for agency comparison MREFC account; annual budget about $150 million Command structure The director presides over seven program directorates spanning different fields of science and engineering. Additional offices provide administrative, financial, and management support. The director is advised by several NSF internal advisory councils of staff. As chartered, the National Science Board is the agency’s governing board that establishes policies, oversees strategic planning, approves new programs and major awards, and oversees the general operations of NSF. Advisory structure Each directorate is advised by an advisory committee of external experts. Committees of visitors are used on a periodic basis to review and improve program operations within each directorate. Origin of projects Nominally, all projects come from the community. Large facility project ideas can be identified at community meetings, NSF-sponsored workshops, or by NSF program managers. Strategic planning NSF does not operate as a mission agency; rather, it strives to act as a facilitator for innovation and creativity. As required by GPRA the agency produces a 5-year strategic plan, last written in 2003. NSF does not regularly engaged in roadmapping activities across directorates. New standards are now requiring MREFC projects to provide life-cycle cost and management schedules. However, NSF’s strategic plan does identify cross-cutting themes to attract attention and encourage fields of research. For instance, the “People, Ideas, Tools” theme of NSF plays a large role in directing its programs. Also, NSF identifies “cross-cutting investment areas,” which are selected for substantial investment over the next few years. Project evaluation criteria Need for such a facility. Research that will be enabled. Readiness of plans for construction and operation. Construction budget estimates. Operations budget estimates. Community involvement The NSF internal champion for a project is typically program staff for some sector of the scientific community. At the simplest level, MREFC projects arise from solicited proposals. Once a project has started to gain momentum, the channels for community input in project development are not standard, but typically involve workshops. Prioritization process Projects are recommended to the director by staff for consideration by the NSF internal MREFC review panel. The MREFC review panel evaluates the merit of a proposed project and then prioritizes it relative to other projects under consideration. The review panel and the director place particular emphases on the following criteria to determine the priority order of the projects:

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation How “transformative” is the project? Will it change the way research is conducted or alter fundamental science and engineering concepts or research frontiers? How great are the benefits of the project? How many researchers, educators and students will it enable? Does it broadly serve many disciplines? How pressing is the need? Is there a window of opportunity? Are there interagency and international commitments that must be met?   The director then selects projects for review by the NSB Committee on Programs and Plans. The board authorizes the director to seek funding for a project in a future funding request. At the discretion of the director, projects are included in the agency’s budget request. DOE   Relevant unit for agency comparison Office of Science (SC); annual budget about $3.3 billion, of which about 25 percent ($825 million) goes to major user facilities Command structure The SC director presides over six offices separated by scope of scientific research and one workforce development program. Additional offices provide management and administrative support. The director makes the final decisions regarding projects and planning on the basis of detailed documentation provided by each of the six program offices and their advisory committees. Because SC operates many large facilities, a recent reorganization has provided a specific Office of Engineering and Construction Management that centralizes project planning and oversight. In contrast with NSF, the SC director does not have the final word; SC’s activities take place in the larger context of the entire DOE. Advisory structure There are six program advisory committees, each FACA-chartered. Members are scientific experts in the fields appointed by the secretary of energy; the committees are staffed by SC personnel. The SC director periodically charges each advisory panel to perform an assessment or to generate recommendations. The FACA committees typically spin off subpanels to address specific tasks assigned by the director. Origin of projects Large facility projects are usually based at national laboratories and therefore span many different fields of science research. Members of the scientific community develop project plans, usually in conjunction with one of the national laboratories. The SC advisory committees often solicit community input and convene planning workshops to identify community needs. Strategic planning A strategic plan was produced according to GPRA for 2001. SC is developing an updated mission statement and strategic plan that promise to improve on the spirit of GPRA. The SC director regularly charges each advisory committee to generate a long-range plan containing priorities and recommendations. These then become templates for the direction of each program office. SC does not traditionally generate a roadmap, although the current director has recently completed an effort to produce a 20-year facilities outlook.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Project evaluation criteria Scientific or technical merit of the educational benefits of the project. Appropriateness of the proposed method or approach. Competence of personnel and adequacy of proposed resources. Reasonableness and appropriateness of the proposed budget. Other appropriate factors, established and set forth by SC in a notice of availability or in the specific solicitation. Community involvement Community involvement is substantial and focused at the advisory committee level. They fall under FACA, so all full advisory-panel meetings are open to the public, encouraging an open process and public comment. Commissioned long-range plans involve substantial community input, often with town meetings, series of workshops, and calls for project ideas. Prioritization process SC relies on the expert opinions of the community to set priorities, particularly through the use of the independent advisory committees and extensive peer-reviewed competition for grants and contacts. Inputs to priority-setting process by the advisory panels include scientific opportunity, projected investment opportunity, DOE mission needs, and administration and departmental priorities. NASA   Relevant unit for agency comparison Office of Space Science (OSS); annual budget about $4 billion Command structure The OSS Associate Administrator presides over four, mostly exclusive divisions focusing on different types of space science. The Associate Administrator makes final planning decisions on the basis of community and advisory committee inputs. Advisory structure Associate administrator is advised by the Space Science Advisory Committee (SScAC), a FACA-chartered committee. Smaller non-FACA committees representing each division (theme) report to SScAC. These committees are composed of science and engineering experts in the field and generally provide tactical advice. Strategic advice can be provided by independent bodies such as the NRC’s Committee on Astronomy and Astrophysics or specific blue ribbon panels. Origin of projects Large missions are ranked by external working groups (often the National Research Council) over long timeframes; OSS issues research solicitations for specific missions and instrumentation to further develop plans and opportunities. Strategic planning A 5-year strategic plan for the entire agency is updated every 3 years (GPRA) with input and synthesis at each level. A combination of top-down vision and bottom-up mission proposals produces an agencywide plan that is coherent and responsive. OSS develops a formal Strategic Plan that includes a roadmap and a detailed rationale for the set of projects recommended over the next 5 years and beyond.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Project evaluation criteria Scientific or technical merit, including competence. Relevance to NASA’s objectives (defined in planning process). Realism and reasonableness of cost and management plans. Community involvement Origin of proposals, priority-setting procedures at division level, and advisory committees at all levels. The research community is an integral component of the strategic planning process that produces a set of science priorities that are then incorporated into the eventual roadmap. Prioritization process Division level: often outsourced to the community. Office level: community participation in “shootout” among division priorities under auspices of SScAC with roadmapping and budget input from NASA personnel. NATIONAL SCIENCE FOUNDATION PRIORITY-SETTING PROCESS FOR MREFC ACCOUNT PROJECTS Introduction1 About one-fifth of the NSF $5.3 billion budget in FY 2003 supports the development and provision of “tools,” which are intended to provide what NSF calls “a widely accessible, state-of-the-art science and engineering infrastructure.” Large facility projects are funded through the Major Research Equipment and Facilities Construction (MREFC) account and through other accounts encompassed in the tools budget category. The MREFC account represents about one-eighth of the Foundation’s proposed investment in tools in FY 2003, rising to about one-sixth in the budget estimates for FY 2004. Despite representing a relatively small portion of the total NSF budget, the large facility projects supported through the MREFC account are highly visible because of their size and geographic concentration, and many of the issues raised by these projects must also be considered in other NSF projects and programs. The large facility projects supported by NSF are nearly as varied as the scientific research that the Foundation supports. Some facilities represent new and increasingly powerful versions of instruments that have been used for decades to study the natural world, such as telescopes or particle accelerators. Other large facilities use new ways of gathering information; examples are a new facility designed to measure gravity waves generated by such cosmic events as star collisions and supernovae 1   The text for this section has been reviewed and modified by NSF to reflect its practices as accurately as possible.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation and a proposed facility that would detect high-energy neutrinos in a large volume of Antarctic ice to provide information about the astrophysical sources of extremely high energy cosmic rays. Some large facilities primarily serve specific scientific disciplines, such as optical telescopes and radio-telescopes for astronomy and observatory networks for oceanography. Other facilities enable research in a wide array of disciplines. For example, the ground facilities, ships, and aircraft stationed in Antarctica allow scientists to study the atmosphere, ice, oceans, and geology of the region. Regardless of their detailed characteristics, all large facility projects are being affected by the accelerating development of information technologies. Increasing quantities and varieties of information are being gathered, rapidly analyzed, and interpreted. Information technologies are also changing the fundamental nature of many large facility projects. New information technologies are making it possible, for example, for many large facilities to consist of smaller instruments and research projects in widely distributed geographic locations. The George E. Brown, Jr. Network for Earthquake Engineering Simulation, which is intended to improve the seismic design and performance of the U.S. civil and mechanical infrastructure, will consist of 15 experimental equipment sites linked by a high-performance Internet system. Elements of EarthScope, a distributed project to study the structure and dynamics of North America, will operate in nearly every county in the United States during the project’s eight- to ten-year lifetime. The proposed National Ecological Observatory Network will consist of geographically distributed observatories linked to laboratories, data archives, and computer modeling facilities. Origins of Concept and Development of Proposals2 The origins of large facility projects are as varied as the projects themselves. Some arise as logical outgrowths of previous research or facilities. Others originate as a consequence of new scientific development where the need for a new facility becomes apparent where no such need existed before. In some cases, such as the provision of high-speed networks and computers, a large facility is required to enable other kinds of research. Other large facilities are focused on the acquisition of data that cannot be obtained in any other way. The impetus for all new large facility projects originates within the scientific community, but ideas take various routes to fruition. The community processes vary greatly from field to field. Often, self-organizing groups within a field of science or engineering develop the initial ideas 2   The text for this section has been reviewed and modified by NSF to reflect its practices as accurately as possible.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation for a new facility and set scientific objectives for the facility by prioritizing competing needs. At other times, facilities have been proposed at the initiative of an individual scientist or a small group of researchers with a bold vision. NSF Program Officers and staff foster these initiatives by providing funds for meetings and workshops that facilitate the scientific community’s internal evaluation and maturation of these concepts. In every case, the mission of the NSF is to seek out the best ideas and the best scientists and to empower their investigations. This process of nurturing and maturation of a concept for a facility can take many years to fully develop, or it can come together as a funded proposal quite quickly, depending on the nature of the proposal, the immediacy of scientific need, and the potential payoffs scientifically and for society in general. The NSF’s role in this process is reactive and responsive to the scientific community, rather than prescriptive, insuring that the highest quality proposals, as determined by peer review within the scientific community, are brought forward for implementation. NSF Program Officers are the key people who make the requirements for approval of such projects clear to the community. In identifying new facility construction projects, the science and engineering community, in consultation with NSF, develops ideas, considers alternatives, explores partnerships, and develops cost and timeline estimates. By the time a proposal is submitted to NSF, these issues have been thoroughly examined. Establishing Priorities for Large Projects3 Upon receipt by NSF, large facility proposals are first subjected to rigorous external peer review, focusing on the criteria of intellectual merit and the broad (probable) impacts of the project. Only the highest rated proposals, i.e., those that are rated outstanding on both criteria, survive this process. These are recommended for further review by an MREFC Panel that comprises the assistant directors and office heads, serving as stewards for their fields and chosen for their breadth of understanding, and chaired by the deputy director acting in consultation with the director; and subsequently by the National Science Board. Both the MREFC Panel and the National Science Board look for a consistent set of attributes in projects they recommend: 3   The text for this section has been reviewed and modified by NSF to reflect its practices as accurately as possible.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation The project represents an exceptional opportunity to enable frontier research and education. The impact on a particular field of research is expected to be transformational. The relevant research community places a high priority on the project. The resulting facility will be accessible to an appropriately broad user community. Partnership possibilities for development and operation are fully exploited. The project is technically feasible and potential risks are thoroughly addressed. There is a high state of readiness to proceed with development, in terms of engineering cost-effectiveness, interagency and international partnerships, and management. The MREFC review panel evaluates the merit of a proposed project and then prioritizes it relative to other projects under consideration. It first selects the new projects it will recommend to the director for future NSF support, based on a discussion of the merits of the science within the context of all sciences that NSF supports. Using these criteria, projects that are not highly rated are returned to the initiating directorates and may be reconsidered at a future time. Then, highly rated projects are placed in priority order by the panel in consultation with the NSF director. The review panel and the director place particular emphases on the following criteria to determine the priority order of the projects: How “transformative” is the project? Will it change the way research is conducted or alter fundamental science and engineering concepts or research frontiers? How great are the benefits of the project? How many researchers, educators and students will it enable? Does it broadly serve many disciplines? How pressing is the need? Is there a window of opportunity? Are there interagency and international commitments that must be met? These criteria are not assigned relative weights because each project has its own unique attributes and circumstances. For example, timeliness may be crucial for one project and relatively unimportant for another. Additionally, the director must weigh the impact of a proposed facility on the balance between scientific fields, the importance of the project with respect to national priorities, and possible societal benefits. After considering the strength and substance of the MREFC Panel’s recommendations, the balance among various fields and disciplines, and

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation other factors, the director selects the candidate projects to bring before the NSB for consideration. The NSB reviews individual projects on their merits and authorizes the Foundation to pursue the inclusion of selected projects in future budget requests. In August of each year, the director presents the priorities, including a discussion of the rationale for the priority order, to the NSB, as part of the budget process. The NSB reviews the list and either approves or argues the order of priority. As part of its budget submission, NSF presents this rank-ordered list of projects to OMB. Finally, NSF submits a prioritized list of projects to Congress as part of its budget submission. Project Implementation and Oversight4 Except for its facilities in the Antarctic, NSF does not directly operate research facilities. Rather, it makes awards to other organizations, such as universities, consortia of universities, or nonprofit organizations, to construct, operate, and manage the facilities. NSF enters into partnerships with those organizations, the details of which are most often defined through cooperative agreements, to accomplish this. The cooperative agreement defines the scope of work to be undertaken by the awardee and establishes the project-specific terms and conditions by which the NSF will maintain oversight of the Project. NSF has the final responsibility for oversight of the development, management, and performance of the facilities. Each large facility project supported by NSF has a program manager in NSF who is the primary person responsible for all aspects of project oversight and management of the project within the foundation. The program manager carries out these responsibilities in accordance with an internal management plan (IMP) that has been crafted specifically for this project. The IMP defines a project advisory team (PAT) that consists of NSF personnel with expertise in the scientific, technical, management, and administrative issues associated with the project. The team works with the program manager to ensure the establishment of realistic cost, schedule, and performance goals for the project. The team also helps to develop terms and conditions of awards for constructing, acquiring, and operating a large facility. The NSF’s director for large facility projects works closely with the program manager, providing expert assistance on non-scientific and non-technical aspects of project planning, budgeting, implementation, and management to further strengthen the oversight capabilities of the foundation. The deputy also facilitates the use of best 4   The text for this section has been reviewed and modified by NSF to reflect its practices as accurately as possible.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation management practices by fostering coordination and collaboration throughout NSF to share application of lessons learned from prior projects. The awardee designates one person to be the project director. This person has the overall control and responsibility for the project within the awardee organization. Throughout the Implementation stage, the awardee executes and manages the project—either construction or acquisition—in accordance with the cooperative agreement between the awardee institution and the NSF. This phase of the project includes all installation, testing, commissioning, and acceptance. Oversight by the NSF during this phase is accomplished through periodic reviews, written reports by the awardee to the foundation that include documentation of technical and financial status using “Earned Value” reporting methods, annual work plans, periodic external reviews, and site visits. By the end of the implementation stage, a proposal is submitted for operations and maintenance to the program manager. The program manager reviews proposals in accordance with the merit review procedures contained in Chapter V of the NSF’s Proposal and Award Manual and presents a recommendation for funding to his or her division director and assistant director—office head. The Director’s Review Board (or DRB) reviews proposals for awards exceeding the Director’s Review Board threshold (see Chapter VI of the NSF’s Proposal and Award Manual). Following DRB, the NSF director recommends awards above the National Science Board (NSB) threshold for approval to the NSB. The NSB reviews and approves awards recommended by the director. Following this, the assistant director—office head, through the division director, authorizes the program manager to recommend the making of an award in accordance with the proposal processing procedures contained in Chapter VI of the Proposal and Award Manual. The program manager, with the Division of Grants and Agreements, drafts the cooperative agreement that will govern the project in accordance with the procedures contained in Chapter VIII of the Proposal and Award Manual. The Division of Grants and Agreements makes the award once the cooperative agreement is executed by it and the awardee. NSF Director and National Science Board5 Using the recommendations received from the MREFC Panel, the NSF director selects candidate projects to be considered by the NSB during one of its meetings in the year. According to the Guidelines, the director uses the following criteria in making this selection: 5   The text for this section has been reviewed and approved by the National Science Board.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Strength and substance of the information provided to the MREFC Panel. The relationship to NSF goals and priorities, including NSF’s educational mission. Appropriate balance among various fields, disciplines, and directorates, based upon a consideration of needs and opportunities. Guidance from the NSB on overall decision boundaries for the MREFC account, provided at the annual MREFC planning discussion (May). Opportunities to leverage NSF funds. The NSB’s Committee on Program and Plans (CPP) takes the lead in reviewing the proposed project; a member of the committee leads the discussion. The criteria considered by CPP are these: Need for such a facility. Research that will be enabled. Readiness of plans for construction and operation. Construction budget estimates. Operations budget estimates. After the CPP reviews the project, it makes recommendations to NSB for approving its inclusion in future budget requests and for approving actual project implementation. NSF Director and Office of Management and Budget Once the NSB has approved a project for funding, the director may recommend the project for inclusion in a future budget request to OMB. In August of each year, the NSB reviews the NSF budget, which includes the list of projects being submitted to OMB for funding. For projects included in the budget request, a capital asset plan and justification must be prepared, following a format developed by OMB. The capital asset plan and justification provides a summary of how much the project will cost to build and operate, information on its management and cost, schedule, and performance goals and milestones. The list of major projects in the budget may be modified during negotiations between OMB and NSF. During that process, other parts of the executive branch, such as the White House Office of Science and Technology Policy, may provide input on the projects included in the budget.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Goddard Space Flight Center for OSS). Finally, NASA projects are spearheaded by, essentially, principal investigators, who include a management plan in the proposal and work closely with the host center. NASA engages in an agencywide strategic planning process every 3 years, producing an updated 5-year plan that includes a detailed roadmap accommodating the projected budget envelope. That process is designed to satisfy NASA’s compliance with the Government Performance and Results Act of 1993 (GPRA), but it has become a powerful tool for identifying and shaping the mission of the agency, tying together budget, performance, and vision. To be synchronized with the triennial revision of the agency strategic plan mandated by GPRA, OSS revises its own strategic plan at the same interval. Each unit measures the consistency of its projects against the overall NASA objectives, the OSS objectives, and its own goals to produce a roadmap. The roadmaps are considered jointly by the enterprise at a closed but broadly representative retreat where determinations about relative importance and priority are made. OSS then drafts its strategic plan; this involves another series of open planning meetings that synthesize the different roadmaps, the broader goals of the enterprise, and additional information about budget expectations. NASA headquarters uses each enterprises’s strategic plan to create the final and authoritative NASA strategic plan.13 This final document becomes the basis for the agency’s annual budget request for the next few years. The NASA vision communicates the agency’s mandate in the 21st century. The NASA mission lays out a clear path to the future. The mission provides a framework for developing goals that each unit of NASA must achieve. NASA has seven strategic goals (expanding on the mission) that enable each enterprise to focus planning, manage programs, and measure results. Each of the agency’s six enterprises uses the strategic goals to define its programs. The agency goals are further broken down into enterprise objectives. (These are, essentially, projections of the agency’s mission onto the subspace spanned by OSS.) The strategic-plan development process depends on active involvement of outside parties, especially the space-science research community. The National Research Council (through its Board on Physics and Astronomy, Space Studies Board, and their discipline subcommittees) develops long-range strategic-program assessments and recommendations. The Space Science Advisory Committee (SScAC), on the basis of inputs 13   The degree of top-down vs. bottom up planning at this stage varies, depending in part on NASA’s leadership. For instance, the 2003 NASA strategic plan was released far in advance of the OSS strategic plan, forcing OSS to play the lead role in synthesizing the theme roadmaps with the overall NASA vision. However, the 5-year plan is revised every 3 years, so the overarching strategic plan is still based heavily on grass-roots priorities.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation from its own subcommittees, provides the enterprise with roadmaps that integrate National Research Council and additional community inputs with technical, budget, and programmatic factors. 2000 Strategic Planning Process14 A narrative description of the process used to develop the FY 2000 Space Science Strategic Plan is an instructive example. This description is from the The Space Science Enterprise Strategic Plan published in November 2000; note that the Space Science Enterprise refers to the Office of Space Science. Work on the 2000 plan began in late 1998, when the Enterprise’s Science Board of Directors initiated the development of science and technology roadmaps for each Enterprise science theme (Astronomical Search for Origins, Structure and Evolution of the Universe, Solar System Exploration, and Sun-Earth Connection). Those roadmaps—which were developed by roadmapping teams that included scientists, engineers, technologists, educators, and communicators of science—address science goals, strategies for achieving the goals, missions to implement the strategies, technologies to enable the missions, and opportunities for communicating with the public. Each roadmapping team was built from or overseen by its theme subcommittee of the Space Science Advisory Committee. The teams each held a series of meetings to obtain science priority views from community scientists, hear advocacy presentations for specific missions, examine technology readiness for alternative mission options, and discuss relative science priorities, balance, and optimal activity sequencing in light of this community input. One technique used to foster convergence was to take straw polls among team members during successive meetings. At the end of the roadmapping period, each of the four theme roadmapping activities submitted a summary document outlining science and mission recommendations to the Space Science Advisory Committee and to Enterprise Headquarters management. Enterprise management then combined the mission recommendations of the roadmapping teams into an integrated mission plan, guided by the current OMB 5-year budget profile, realistic estimates of most likely future resource availability beyond that, and additional agency-level and administration guidance. Likewise, science goals in the roadmaps were used to examine and restate those presented in the 1997 Enterprise Plan. An integrated roadmap was presented and discussed at a planning workshop that expanded the membership of the Space Science Advisory Committee with other community members and representatives of the 14   The Space Science Enterprise Strategic Plan, NASA, November 2000, p. 48.  

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation technology, education, and public-outreach communities. Attendees at the workshop also analyzed and revised the proposed updated science objectives and derived a new set of shorter-term research activities. The resulting consensus mission plan and goals, objectives, and research activities serve as the nucleus for the current Strategic Plan. A draft of this Plan was provided to the Space Studies Board and its committees for review and feedback, and guidance received was used in finalizing the Plan. The findings and recommendations of the Academy’s recently completed ten-year astronomy and astrophysics survey were consulted to assure consistency with the draft Plan. Finally, the Space Science Advisory Council had an opportunity to review the revised Plan and suggest changes before the Plan went to press. The Space Science Enterprise strategic plan serves several purposes. It facilitates a consensus process in the science community that focuses on goals and priorities for the future. It serves OSS by providing a reference for programmatic decision making, and it provides input to the overall agency strategic plan and material to meet GPRA requirements. The strategic plan becomes a tool for use by OMB and Congress in program and budget advocacy. Finally, it provides a handbook for the public on what space science is going to do and why. DEPARTMENT OF DEFENSE OFFICE OF THE DIRECTOR OF DEFENSE RESEARCH AND ENGINEERING The Department of Defense (DOD) Science and Technology (S&T) Program supports the fundamental research, development, and demonstrations in science and technologies identified as important to military capabilities and operations. The extent of the S&T Program includes the development of the nation’s high technology weapons systems, the technology base upon which those system rely, and the equipment to both support and prepare military personnel. The program also plays an important role in developing certain key technologies that transfer to commercial applications and help to grow the overall economy. Basic research (6.1), applied research (6.2), and advanced technology development (6.3) together comprise DOD’s Science and Technology (S&T) program. S&T projects seek new ways of accomplishing tasks of military value and the underlying scientific and engineering principles involved. Within DOD, the Office of the Director of Defense Research and Engineering is primarily responsible for the basic research plan of the department, “to ensure that the warfighters today and tomorrow have superior and affordable technology to support their missions, and to give them revolutionary war-winning capabilities.”

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation Project Selection Project proposals solicited by the Directorate of Defense Research and Engineering are typically judged on the following criteria: DOD mission priority. Military advantage gained by exploiting the requested S&T. Merit of scientific study. Potential for significant progress. Appropriateness of solution for meeting requirements. Program Evaluation15 DOD uses defense technology objectives (DTO) to provide focus for the development of technologies that address identified military needs across the department. Each DTO identifies a specific technology advancement that will be developed or demonstrated, with expected date of availability, specific benefits resulting from it, and the amount of funding needed. The DTO process is used to comply with GPRA. The output of this process includes budget and management decisions. The methodology used for evaluating the S&T program is known as technology area reviews and assessments (TARA). TARA is the department’s official response to GPRA, and it is a mechanism to evaluate science and technology programs through expert peer reviews. The evaluation of basic and applied research is carried out by internal agency panels of experts and by TARA review panels. Each panel consists of 10-12 technical experts from academe, industry, and nonprofit research organizations. Most TARA team members are recognized experts from the National Academies, the Defense Science Board, the scientific advisory boards of the military departments, industry, and academe. Each is chaired by a senior executive appointed by the deputy under secretary for S&T. These teams are asked to evaluate the programs for quality, advances in leading the state of the art in research areas, and for their scientific vision. The department requires that two-thirds of each panel be experts from outside DOD. One-third of each panel’s members are refreshed at the time of each reviewing cycle. Defense Science Board The Defense Science Board operates by forming task forces consisting 15   Portions of this section have been excerpted from Implementing the Government Performance and Results Act for Research: A Status Report, National Academy Press, 2000.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation of Board members and other consultants and experts to address those tasks referred to it by formal direction. The products of each task force typically consist of a set of formal briefings to the board and appropriate DOD officials and a written report containing findings, recommendations, and a suggested implementation plan. The board reports directly to the secretary of defense through the under secretary of defense for acquisition, technology and logistics (USD [AT&L]) while working in close coordination with the DDR&E to develop and strengthen the department’s research and development strategies for the 21st century. The board met for the first time on September 20, 1956. Its initial assignment examined basic research, component research, and technology advancement programs and their administration in S&T areas of interest to the DOD. On December 31, 1956, a charter was issued specifying the Board as advisory to the assistant secretary of defense (research and development). The subsequent consolidation of the offices of the assistant secretaries of defense for R&D and applications engineering in 1957 resulted in the board’s reconstitution as adviser to the secretary of defense through the assistant secretary of defense (research and engineering). The mission of the board is to advise, in response to formal requests, the secretary of defense, the deputy secretary of defense, the under secretary of defense for acquisition, technology and logistics, and the chairman of the Joint Chiefs of Staff on matters relating to science, technology, research, engineering, manufacturing, and acquisition process and other matters of special interest to the DOD. The board specifically concerns itself with pressing and complex technology problems facing DOD in such areas as research, engineering, and manufacturing. It seeks to ensure the identification of new technologies and new applications of technology in these areas for the strengthening of national security. The board does not advise on individual procurements. Defense Advanced Research Projects Agency16 DARPA fulfills a unique role within the Department of Defense. As a Defense Agency, DARPA reports to the Secretary of Defense. The Director, Defense Research and Engineering has been assigned to be DARPA’s Principal Staff Assistant (PSA). DARPA is the Secretary of Defense’s only research agency not tied to a specific operational mission. DARPA sup plies technological options for the entire Department. DARPA is designed 16   Excerpted from Strategic Plan Defense Advanced Research Projects Agency, February 2003, p. 5. Available on the DARPA web site: (http://www.darpa.mil/body/pdf/DARPAStrategicPlan2003.pdf).

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation to be the “technological engine” for transforming the Department of Defense. DARPA’s decision-making process is somewhat unusual for a government agency. It is informal, flexible, and yet highly effective because it focuses on making decisions on specific technical proposals based on the factors discussed above. There are two reasons for this. DARPA is a small, flat organization rich in military technological expertise. There is just one porous management layer (the Office Directors) between the program managers and the Director. With less than 20 senior technical managers, it is easy to make decisions. This management style is essential to keeping DARPA entrepreneurial, flexible, and bold. DARPA’s management philosophy is to pursue fast, flexible, and informal cycles of “think, propose, discuss, decide, and revise.” This approach may not be possible for most government agencies, but it has worked well for DARPA. The Basic Process: DARPA uses a top-down process to define problems and a bottoms-up process to find ideas, involving the staff at all levels. DARPA’s upper management and program managers identify “DARPAhard” problems by talking to many different people and groups. This process includes: Specific assignments Requests for help Discussions with senior military leaders Research into recent military operations Discussions with defense agencies Discussions with the intelligence community Discussions with other government agencies or outside organizations Visits to Service exercises or experiments. During DARPA’s program reviews, which occur throughout the year, DARPA’s upper management looks for new ideas from program managers (or new program managers with ideas) for solving these problems. At the same time, management budgets for exploring highly speculative technology that have far-reaching military consequences. Program managers get ideas from many different sources, such as: Their own technical communities; Suggestions from DOD-wide advisory groups, including the Defense Science Board and Service science boards; Suggestions from DARPA-sponsored technical groups Suggestions from industry or academia, often in response to published Broad Area Announcements or open industry meetings Breakthroughs in DARPA programs and/or U.S. or international

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation research During reviews of both proposed and ongoing programs, DARPA’s assessment is often guided by a series of questions. These seemingly simple queries help reveal if a program is right for DARPA. What is the project trying to do? How is it done now and what are the limitations? What is truly novel in the approach that will remove those limitations and improve performance? By how much? If successful, what difference will it make?? What are the midterm exams required to prove the hypothesis? What is the transition strategy? How much will it cost? Are the programmatic details clear? SUMMARY OF UK RESEARCH COUNCILS’ PRIORITY-SETTING PROCESS On behalf of the research councils of the United Kingdom (RCUK), the UK Office of Science and Technology (OST) released the second edition of its Large Facilities Strategic Roadmap in June 2003.17 Several excerpts from that document are provided here. In 2001, the UK initiated a new process for treating proposals for large facilities. It begins with the production of a roadmap that contains projects currently identified as being of the highest strategic importance. There is no commitment to funding and no guarantee that the UK will participate in all the projects in the roadmap. From Section 1, “Executive Summary”: It is a tool by which Research Councils UK (RCUK) and its members can assess strategically the most expensive and complex scientific facilities with which UK researchers are or may wish to be involved. The road-map includes facility “projects” identified by members of RCUK as a priority for consideration which meet one or more of the following criteria: Where there could be an international dimension to the proposed 17   The excerpts provided here are taken from the UK Office of Science and Technology Web site, where the Large Facilities Strategic Roadmap is posted at http://www.ost.gov.uk/research/funding/lfroadmap/index.htm.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation facility and therefore opportunity to share costs and develop relationships to benefit the UK science programme; Where the facility supports the requirements of research communities of more than one Research Council; Where the capital investment is greater than the sum of £25 million, when it represents a significant element of an individual Research Council’s budget line. The roadmap is scheduled to be updated every 2 years; the June 2003 version represents the second iteration; it contains projects in 10 strategic fields. The roadmap is the first stage of a gateway process that requires all large capital investments to be managed as discrete projects subject to review and independent scrutiny at all stages in their life cycle. From Section 5, “OGC Gateway Process and How it is Used in Large Science Facilities”: The OST and members of RCUK use the Office of Government Commerce’s Gateway process to help procure large scale scientific facilities. All new procurement projects in civil Central Government—including NDPBs—are subject to the Gateway process, which examines a project at critical stages in its lifecycle to provide assurance [that] it can progress successfully to the next stage. The process has a series of Gateway Reviews, as follows: 0. To confirm the overall strategic assessment 1. To confirm the business justification 2. To confirm the procurement method and sources of supply 3. To confirm the investment decision - before letting any contracts 4. To confirm “readiness for service” 5. To confirm “in service benefits” In the case of scientific projects, the first stage must involve an independent assessment of scientific value, including some form of peer review that addresses the following criteria: Importance (depth) of science knowledge to be delivered by project. Breadth of science knowledge that will benefit from investment. Match with international positioning of UK science. Strength of opportunity for training (links to numbers of users). Contribution to or from UK technology-industry base. Opportunity for spin-off and exploitation. See additional material in Appendix E concerning the criteria used in the strategic roadmapping process.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation SUMMARY OF GERMAN MINISTRY FOR EDUCATION AND RESEARCH’S PRIORITY-SETTING PROCESS18 Germany has a distinctive science system, with responsibilities shared between federal and state governments. At the federal government level, the Ministry of Education and Research (BMBF) is the key player. Various nonprofit bodies play important roles in distributing funding; for example the Deutsche Forschungsgemeinschaft is a major source of project funding for universities, and the Max Planck Gesellschaft operates 81 research institutes. National policy coordination and policy advice are provided by two joint federal-state bodies: The Bund-Länder-Kommission für Bildungsplanung und Forschungs-förderung, which is a forum for discussions between federal and state ministries. The Wissenschaftsrat, or Science Council, an independent body whose members are appointed by the federal president and that advises federal and state governments on all matters of higher education and research policy. Its members include representatives of the science community and nominated representatives of state and federal governments. Over the last 20 years, the BMBF has set up ad hoc groups to recommend priorities for funding large facilities that have been proposed by the scientific community whenever the need was felt (their reports have become known by the names of the chairs: Mayer-Leibnitz, Pinkau, and Grossman). Such recommendations go to the minister, who makes final decisions (subject to discussion with the Finance Ministry and at cabinet level when appropriate). In 2000, the BMBF asked the Wissenschaftsrat (henceforth referred to as the Science Council) to review nine proposals for large facilities for basic research (each priced at at least 15 million euros); and in January 2001, the Science Council established a working group for this purpose. The group included scientists at universities and research establishments in Germany, the United States, and Switzerland and “individuals involved in and representing national and international scientific administrations.” The group established six subpanels composed of 57 external experts, including 37 from abroad. 18   This text has been drawn from the final report of the Science Council, Statement on Nine Large-scale Facilities for Basic Scientific Research and on the Development of Investment Planning for Large-scale Facilities, 2003. The report is also available on the Science Council’s Web site at http://www.wissenschaftsrat.de/texte/5385-02.pdf.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation The Science Council’s working group not only evaluated specific projects but laid out a general framework for such work and proposed criteria. It seems likely that, as proposed in the report, the Science Council will be asked to carry out such reviews at regular intervals in the future, replacing the previous ad hoc procedure. The council established a working group with six expert subpanels that assessed projects in particular fields according to the following criteria: The probability of fundamental new insights or the possibilities of decisive scientific advances that could be achieved only with the large-scale facility. The large-scale facility’s technical feasibility and degree of technical innovation. The scientific and technical competence of the institutions involved. The already existing or anticipated acceptance of the (potential) users with immediately relevant and related expertise. The fulfillment of various objectives of importance for research (transfer, international perspectives, and promoting young scientists). The working group then considered all the projects, taking into account: Scientific potential of the research program. Fulfillment of science and technology policy goals as formulated in 10 general “theses on the significance of large-scale facilities for basic scientific research.” Degree of maturity of the technical concept and, connected to it, the possible timeframe for implementing the individual components. The context of further national and international scientific development of the research fields to which they belong and their interaction with other disciplines. Finally, the working group divided the nine projects into three categories, the first meriting unconditional support, the second with specific points yet to be clarified, and the third requiring additional development. See Appendix E for additional information on these criteria.

OCR for page 147
Setting Priorities for Large Research Facility Projects Supported by the National Science Foundation This page intentionally left blank.