5

Research Prioritization: Analysis and Findings

As a mission agency with numerous objectives and a broad range of programs, NASA presents a complex multidimensional prioritization problem with its research programs. Not only are diverse scientific disciplines represented within the agency's research portfolio, but planning and execution are strongly affected by the challenging and costly space environment in which the research is done, institutional imperatives for technology innovation, and the distributed field center network.

Several studies on the space research priority-setting problem have been carried out with varying degrees of success. The Space and Earth Sciences Advisory Committee of the NASA Advisory Council released an influential report1 in 1986 that outlined a framework of factors to be considered in prioritizing missions. In 1992 the NRC Space Studies Board Task Group on Priorities in Space Research laid out the rationale2 for a more general cross-disciplinary prioritization of scientific objectives and broad initiatives. The arguments presented therein for an active role by scientists in the priority-setting process were generally well received. A specific implementing methodology proposed in a second report3 for effecting this broadened prioritization, on the other hand, was not as well accepted as the underlying principles laid out in the predecessor report.

The present committee's task group on research prioritization therefore set out to reexamine the existing process of priority setting for science programs within NASA, with special attention to the roles of advisory groups and the agency's management organization. The central thrust was to reexamine these roles and to clarify the necessary functions on the basis of the following fundamental premises and overall goals.

PRINCIPLES OF RESEARCH PRIORITIZATION

Axioms

The committee made the following assumptions in analyzing the priority-setting problem:

1  

Space and Earth Sciences Advisory Committee of the NASA Advisory Council, The Crisis in Space and Earth Sciences, 1986.

2  

NRC, Space Studies Board, Setting Priorities for Space Research: Opportunities and Imperatives, National Academy Press, 1992.

3  

NRC, Space Studies Board, Setting Priorities for Space Research: An Experiment in Methodology, National Academy Press, 1995.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 48
MANAGING THE SPACE SCIENCES 5 Research Prioritization: Analysis and Findings As a mission agency with numerous objectives and a broad range of programs, NASA presents a complex multidimensional prioritization problem with its research programs. Not only are diverse scientific disciplines represented within the agency's research portfolio, but planning and execution are strongly affected by the challenging and costly space environment in which the research is done, institutional imperatives for technology innovation, and the distributed field center network. Several studies on the space research priority-setting problem have been carried out with varying degrees of success. The Space and Earth Sciences Advisory Committee of the NASA Advisory Council released an influential report1 in 1986 that outlined a framework of factors to be considered in prioritizing missions. In 1992 the NRC Space Studies Board Task Group on Priorities in Space Research laid out the rationale2 for a more general cross-disciplinary prioritization of scientific objectives and broad initiatives. The arguments presented therein for an active role by scientists in the priority-setting process were generally well received. A specific implementing methodology proposed in a second report3 for effecting this broadened prioritization, on the other hand, was not as well accepted as the underlying principles laid out in the predecessor report. The present committee's task group on research prioritization therefore set out to reexamine the existing process of priority setting for science programs within NASA, with special attention to the roles of advisory groups and the agency's management organization. The central thrust was to reexamine these roles and to clarify the necessary functions on the basis of the following fundamental premises and overall goals. PRINCIPLES OF RESEARCH PRIORITIZATION Axioms The committee made the following assumptions in analyzing the priority-setting problem: 1   Space and Earth Sciences Advisory Committee of the NASA Advisory Council, The Crisis in Space and Earth Sciences, 1986. 2   NRC, Space Studies Board, Setting Priorities for Space Research: Opportunities and Imperatives, National Academy Press, 1992. 3   NRC, Space Studies Board, Setting Priorities for Space Research: An Experiment in Methodology, National Academy Press, 1995.

OCR for page 48
MANAGING THE SPACE SCIENCES In conformance to the Space Act of 1958, science will continue to have a central role among NASA activities. Because of the rich return from earlier missions, there are more opportunities for new discovery than there are resources available to exploit them. To fulfill its scientific mandate, NASA will continue to receive appropriated funds for scientific research, and, as an agency, will have the responsibility of distributing and administering these science funds. Because opportunities will outstrip these resources, this responsibility will entail priority setting. Judgments about scientific merit will have a major role in the agency 's administration of resources allocated to it for scientific research. Practicing scientists are the most qualified arbiters of scientific merit. Goals The following goals have been identified for priority setting in NASA's space research programs: Conforming to its charter in the Space Act, the scientific research sponsored and managed by NASA should vigorously advance knowledge in the areas in which it is undertaken. Scientific efforts sponsored by NASA should be proposed and evaluated in the context of research sponsored and conducted elsewhere and should be competitive with this other research in interest, quality, and importance. In order to optimize the return on expended resources for science while addressing other technological objectives for the agency laid out in the Space Act, NASA's scientific research program should appropriately balance innovation with risk of failure. This will help set the directions for technological development and ensure that benefits of new capabilities are realized for the science programs. NASA's research portfolio should be opportunistic, not only in the technological sense (goal 3 above), but also in terms of focusing on areas most ripe for significant advances with the resources and capabilities available. Prioritization should consider all relevant factors, including opportunities to contribute to policymaking and other public needs, in optimizing scientific return for the resources invested. PRIORITIZATION OF SCIENCE AT NASA Defining the Problem The space sciences, like any natural science, begin with scientific ideas and goals that are transformed into research programs4—the means of carrying out the necessary experimental or observational research. In space science, though, the cost and complexity of many missions are much greater than in most other areas of science. Therefore, the relative practical priority of the scientific goals, that is, the strategy and timing of attacking them, might be more strongly determined by mission considerations than is the case in other areas of science. This is especially true in light of the extraordinarily long lead times that are associated with some of the large space science missions. Figure 5.1 presents a schematic life cycle for the priority-setting process. Figure 5.2 shows relationships between various key players in this process. As is seen below, the level of scientific and technical detail involved in the inner feedback loop in Figure 5.1 will vary depending on where it is being done in the flow diagram of Figure 5.2. In some instances, scientists on disciplinary committees can 4   By “program” is meant the collective activities in a discipline or subdiscipline (or even across disciplines) that include both missions and ground-based research—all the activities directed toward a goal or set of goals.

OCR for page 48
MANAGING THE SPACE SCIENCES FIGURE 5.1 Prioritization life cycle. FIGURE 5.2 Key priority-setting participants. complete the loop with a general knowledge of technology options and “ballpark” estimates of cost. But before final decisions are made within the NASA program offices, this loop has to be closed in great detail. The outer feedback loop recognizes the fact that the Administration or Congress sometimes reduces outlays to the point where missions have to be rescoped or rethought entirely. There are many participants, tasks, criteria, and processes involved in the life cycle in these diagrams. The details vary from discipline to discipline. In microgravity, for example, the “space platforms, ” principally the Shuttle and Space Station, correspond to “missions” (Figure 5.1) and are responsive to

OCR for page 48
MANAGING THE SPACE SCIENCES FIGURE 5.3 Coupling between science and mission platform. national priorities outside of science; here, scientists focus almost entirely on prioritizing scientific goals within these constraints. In the classical space sciences (astronomy, planetary science, space physics), on the other hand, the scientific goals and the designs of missions are tightly bound, and the scientific community must conduct at least the initial trade-offs all the way down to “Prioritize Missions.” In the case of the “Great Observatories,” prioritization of some scientific goals takes place even after launch when user time on the instruments is allocated. In all disciplines, after in-depth studies of design, cost, and timing, the initial recommendations from the scientific community may repeat the cycle. Thus, the disciplines present varying circumstances: a space platform built for other purposes that can accommodate scientific experiments (e.g., Shuttle, Space Station); new launch vehicles and space platforms being designed to accommodate a variety of science observations and experiments (e.g., the New Millennium spacecraft); and space platforms that must be designed explicitly for the particular scientific goal being pursued (e.g., larger observatories, planetary surface exploration systems). The strength of the coupling between scientific goal, program, and mission (illustrated schematically in Figure 5.3) must be recognized by the science prioritization process. For areas in which coupling is strong, the process loop of Figure 5.1 must be traversed early in the prioritization process at least at the level of conceptual design and ballpark cost estimates. For weak coupling, the loop is not traversed until late in the process. The entire process of establishing space science priorities is dependent on a number of parameters: disciplinary level at which priorities are established, organizational level at which priorities are set, participants in the process, criteria to be applied, approach or process to be used, and tasks to be performed. Disciplines The committee used the word “discipline” to signify the six areas so designated in the current NASA program structure. These areas and the current NASA major offices having responsibility for each are as follows: Office of Space Science (OSS) Astrophysics Space physics Planetary science

OCR for page 48
MANAGING THE SPACE SCIENCES Office of Life and Microgravity Sciences and Applications (OLMSA) Life science Microgravity science Office of Mission to Planet Earth (OMTPE) Earth science A discipline is an area, such as those represented in university hierarchies by academic departments, in which there is a coherent, communicating body of scientists who understand the scientific and technical aspects of one another's fields of research well enough to judge overall quality but who cannot easily trade places in performing research. The disciplines have different characteristics that affect the manner in which priorities are eventually established. Four (astrophysics, space physics, planetary science, and Earth science) are observational disciplines,5 while two (life science and microgravity science) are laboratory disciplines. Further, “microgravity” is, strictly speaking, a variable, not a discipline; it addresses the question of what can be learned in a variety of disciplines by experimentation in a low-gravity environment. A few characteristics of NASA's six space science disciplines are as follows: Astrophysics—Depends on remote-sensing techniques; NASA's program is a substantial part of the whole field of astronomy (certainly in dollar terms), which is therefore heavily dependent on NASA; may require large, expensive instruments. Space Physics—Uses both remote-sensing and in situ techniques; field is strongly dependent on NASA, although it varies somewhat by subdiscipline (e.g., small for atmospheric science, moderate for solar physics, large for magnetospheric physics); can probably manage well with “smaller, faster, cheaper” missions. Planetary Science—Uses both remote-sensing and in situ techniques; almost entirely dependent on NASA; requirements for interplanetary flight and planetary capture or landing lead to mission complexity. Life Science—Laboratory science; NASA supports only a very small part of the total research field; this small part includes operational support (e.g., long-term effects of spaceflight on human biology) and operational (or strategic) research, as well as fundamental research. Operational (strategic) research in the life sciences is research that addresses problems directly related to the presence of humans in space and their short- and long-term ability to survive and function in that environment.6 Microgravity Science—Laboratory science; a loose collection of several disciplines (fluid mechanics and transport, biology and biotechnology, materials, combustion, and physics), some of which are phenomenological. The microgravity science portion of these disciplines is always a very small fraction of the total reach of the discipline. Individual microgravity studies are relatively small in scope and (relatively) small in incremental costs because the platform (Shuttle, Space Station) is presumed available. Earth Science—Mainly remote sensing of Earth, ocean, and atmosphere parameters best examined from space; small part of the total field. One of the most important effects of these differences is the degree to which the design of missions influences a determination of priorities of scientific goals. For example, scientists concerned with 5   The committee recognizes the laboratory and theory components of these four sciences and is here referring to those areas of these sciences that are conducted in or from space by NASA. NASA will need to ensure that the ground-based research essential for understanding space observations is carried out. 6   NRC, Space Studies Board, Committee on Space Biology and Medicine, letter to NASA's Life and Biomedical Sciences and Applications Division Director Joan Vernikos, July 27, 1995.

OCR for page 48
MANAGING THE SPACE SCIENCES microgravity studies come from a variety of different traditional disciplines. Therefore, at the discipline advisory committee stage in the setting of priorities, scientists deliberating on scientific priorities within the program will be judging the relative scientific merits of projects outside the traditional boundaries of their personal disciplines. Although they cannot judge with the same precision as a corresponding panel in other space disciplines, participants are expected to obtain necessary information from one another and be able to make judgments regarding the relative importance of microgravity studies outside their own discipline. They should be able to explain this judgment to one another and through such discussions arrive at relative science priority rankings. A microgravity experiment of great significance to protein crystal growth, for example, ought to have higher priority than an experiment in fluid flow of only modest significance to that field. Competent scientists with adequate breadth of knowledge can have constructive discussions, even arguments, about such issues and arrive at a conclusion if asked to do so. In astronomy, on the other hand, the commonly agreed-on science frontiers or goals are often expressed as new observational capabilities, that is, as a mission. This is because there is usually an array of frontier science questions that share a common need for a new observational capability. As the process of prioritization engages the organizational entities shown in Figure 5.2, it passes through various levels of “disciplinarity”: Subdisciplinary level—A field in which there is a communicating body of scientists who regard each other as active, working experts in the field, who believe that their own work can be adequately understood and judged by one another, and who could do one another's research. Disciplinary level—A field in which there is a communicating body of scientists who understand the scientific and technical aspects of one another's fields of research, who could not necessarily easily trade places in the research they perform, but who can readily understand one another's work and can easily cooperate in joint research. (Note: In the NASA lexicon, the “discipline” of microgravity is, strictly speaking, inconsistent with this definition, but it can be accommodated.) Cross-disciplinary level—Fields in which there are communicating bodies of scientists with general knowledge of one another's fields of expertise but (usually) having insufficient initial understanding to collaborate closely without reciprocal education and tutelage. At this cross-disciplinary level, two new elements arise in addressing priorities: (1) The need to “interleave” the priorities put forth by the disciplines. An important new element enters at this level: the conventional notion of “peer review ” cannot be relied on exclusively, because areas with different sets of peers or expert constituencies are being intercompared. (2) Factors outside of pure scientific merit become more important. In fact, some of these, entering for the first time, may alter the order of priority established at the disciplinary level. Agency level—Total scope of all research performed by NASA. Structure and Participants Figure 5.2 depicts the general context for setting science priorities for NASA. In the past, disciplinary committees of the NRC's Space Studies Board (SSB), tapping into a broad constituency of scientists, would issue reports describing the important scientific goals to be pursued in their fields. These reports would be taken up by internal NASA groups, often working at the subdisciplinary level, and incorporated into plans for various space missions. A number of these missions were very large—at the billion dollar level or above—and required many years to be brought to fruition. The expenditures on such a mission would include space vehicles and launch costs. Further, the projects involved considerable expenditure on continuing data collection, research, and analysis. Not too many of these large projects could be pursued concurrently, and so they would be ordered into a “mission queue” designed to meet the needs of the various scientific disciplines sequentially. Within NASA, the Associate Administrator for Space Science and Applications orchestrated the process of transforming

OCR for page 48
MANAGING THE SPACE SCIENCES disciplinary scientific goals into this mission queue. The scientists in any discipline could be reasonably certain that, periodically, “their time would come.” In establishing the priority and sequence of these projects, the OSSA associate administrator not only used internal NASA resources, but also continued to tap into the broad community of scientists for advice and counsel. Occasionally, as convened by NASA in 1991 at Woods Hole, Massachusetts, scientists from internal and external committees in various fields assembled to discuss and argue about the entire set of plans and priorities. In general, this system was perceived to work well. The various disciplines felt that their arguments and suggestions had been heard even when their own programs did not head the resulting queue. However, the environment was one of budgets more or less adequate to accommodate much of what the scientific community thought important to do. While there were delays and occasional mission failures, few projects were canceled in midstream for budgetary reasons. The future will be different. It will likely be a future of decreasing budgets, smaller missions, stronger emphasis on controlling costs, and greater competition between disciplines. Thus, the task of establishing priorities and making choices will be more difficult, and possibly more contentious, than in the past. In addition, NASA's science programs are now distributed among several organizations. Subdisciplinary and disciplinary trade-offs are made in separate units before choices are presented to each associate administrator for disciplinary and cross-disciplinary selection. Final decisions for major initiatives are then made by the Administrator with advice from the Chief Scientist. The process in the various disciplines is not managed uniformly. While this variation may be logical from the point of view of the different national goals and purposes served by the major science offices, it is not clear that an appropriate balance of science and other objectives results. Moreover, the process has become less transparent to the scientific community at a time when its input may be more rather than less useful. The scientific community has not had to choose the highest-priority scientific goals from among many desirable goals, to participate actively in cross-disciplinary choices, or to help identify projects that should be stopped. Yet all of these things are going to be part of the future scene. In addition, innovative ways of conducting space science missions must be sought to enhance the ability to advance the space sciences. These novel methods of doing space science will be ever more important as budgets become tighter. The scientific community must find ways to help optimize these choices. Otherwise, decisions will be made primarily on administrative or political grounds. Thus, the scientific community and NASA each face a challenge: the scientific community must address issues not addressed in the past, and NASA must establish and maintain a system of prioritization that is transparent and open in the new and more challenging climate of today. Meeting these challenges will be necessary if NASA's space science programs are to be strengthened and energized. As NASA faces ever-increasing budgetary pressure, it is becoming increasingly important not only to prioritize among science initiatives and missions, but also to address issues of efficiency and productivity. The present NASA culture for accomplishing missions is sometimes inefficient, and many aspects of mission design, oversight, quality assurance, and management need to be remodeled and reduced. The top priority is to execute the highest-quality set of NASA science missions and initiatives possible within whatever budget is available. As a consequence, the real work of prioritizing among missions, both ongoing and new, must be done in an environment where an ever-increasing fraction of the science budget is devoted to science rather than to supporting functions. More must be done for less, rather than permitting an inefficient culture to squeeze out existing and future science initiatives. The participants in this priority-setting process fall into three categories, as described below. NRC Participants Space Studies Board and its discipline committees—Mirroring NASA's historical administrative structure, the Space Studies Board (SSB) has divided space science into six disciplines: astronomy and

OCR for page 48
MANAGING THE SPACE SCIENCES astrophysics, solar and space physics, planetary science, Earth science, microgravity, and life science. Each of these disciplines is overseen by a committee reporting to the Board; in two cases (astronomy and astrophysics, and solar and space physics) committee oversight is shared with another NRC board. The committees are charged with identifying research opportunities, setting and prioritizing the overall scientific goals within their discipline, and evaluating NASA's success in implementing these strategies. NRC ad hoc committees—In the case of astronomy and astrophysics, the overall prioritization has been done once per decade by an ad hoc NRC committee. Unlike the committees that oversee the other five disciplines, the NRC astronomy and astrophysics survey committees prioritize missions and projects from major to relatively modest sizes. NASA Advisory Participants NASA advisory committees—NASA has established an extensive advisory committee system that, in the Office of Space Science at least, operates at both the associate administrator (cross-disciplinary) level and the division (disciplinary) level. That is, at the major office level, each associate administrator has an advisory committee (cross-disciplinary) that spans the fields in his office and reports directly to the NASA Advisory Council. Likewise, each division director within the major office has a disciplinary subcommittee that reports upward to the full committee as well as to the disciplinary division. Typically, the chair of each panel sits on the next highest level body. The primary task of these internal committees is advising NASA on science programs in their areas of competence. These programs are developed on the basis of scientific merit, technical readiness, and programmatic factors operative at their levels. Developing these programs entails prioritizing and integrating proposals submitted by their subordinate panels. The panels also serve some of the review functions of a university visiting committee, continually assessing the progress and status of their sponsoring NASA organizations. The role of the NASA committees is primarily tactical and program planning, in contrast to the strategic focus of NRC committees. The NASA committees are also subject to the Federal Advisory Committee Act. As a result of these distinctions, meetings of the two groups of committees are quite different. The NASA committee meetings and deliberations have a much stronger programmatic flavor and typically address nearer-term and more specific planning issues. They are also always attended by a senior NASA manager and have more direct access to agency scientists, engineers, and budget information. The NASA advisory committees consist primarily of scientists drawn from the external community, but may include NASA scientists with particularly relevant expertise. While individual external community scientists may serve on the SSB (or its committees) and NASA committees (or their subcommittees) serially, simultaneous participation in both SSB and NASA committee structures is precluded by custom. This is to avoid the perception that any individual has excess influence or is serving as an advisor to both strategic planning and program execution at the same time. NASA ad hoc committees—In order to provide regular access to space, NASA has established the Explorer program in astrophysics and space physics and the Discovery program in planetary science. In these programs, missions are prioritized by ad hoc program peer review panels during investigator and team selection. The degree to which SSB strategies are used in setting the priorities depends on details of the proposals under review and the extent to which SSB science priorities are accepted by the members of the ad hoc panel. NASA Management Participants Associate administrators—The associate administrators have the responsibility for managing the entire scientific program within their purview. In the Office of Space Science and the Office of Life and Microgravity Sciences and Applications, this involves setting priorities across disciplines. The associate

OCR for page 48
MANAGING THE SPACE SCIENCES administrators are also responsible for coordinating with other federal agencies working in related areas. Finally, they serve as advocates for their programs to the Administrator in structuring funding priorities for the agency. Science Council—This group comprises the science associate administrators and the Chief Scientist, is chaired by the latter, and acts as an advisory body to the Administrator. Chief Scientist—The Chief Scientist provides advice to the Administrator on all issues relating to science within the agency. This official chairs the Science Council. The Chief Scientist tracks the choices made within the science programs of NASA and, with the Science Council, provides a forum for their coordination and balance. Administrator—The Administrator of NASA makes final decisions that not only incorporate the best scientific and technical advice that has come up through the previous channels, but also reflect NASA strategy, national goals, and policy direction provided by the Administration and Congress. THE PRIORITIZATION PROCESS Key Elements The scientific community strongly supports the practice of peer review for selection of research support and refereeing of results for publication. The scientific community regularly uses peer review to render judgment about the relative scientific value of research projects. Peer review implies judgment by others who are as expert in the field of inquiry as the one proposing work or reporting results. Scientists are comfortable with judgments rendered by peers, and were this process fully adequate for establishing priorities, the issue would be simple. But it is not. The scientific community is uncomfortable with priority setting in which factors beyond pure scientific merit come into play, partly because it implies judgment by people who are not experts. Moreover, the degree of discomfort grows as the degree of scientific expertise of the decision makers goes down. Nevertheless, in the real world, priorities are virtually synonymous with budgets, so they cannot be escaped. Thus, if scientists are to play an expanded role in establishing priorities, they must engage in deliberations outside their areas of primary scientific expertise. They must also become accustomed to the intrusion of nonscientific factors. Beginning with decision making within the narrowest boundaries, priority-setting mechanisms in use include the following: Peer review—Scientific peers (i.e., experts in a common subdiscipline) determine a project's scientific merits, novelty, originality, importance, timeliness, methodologies, uncertainties, and conceptual reservations and reach agreement about a project's importance in relation to other projects. Peer judgment—After peer review, projects must often be evaluated by a broader-based group, still in the same general discipline, who understand one another 's work. This evaluation involves interleaving the recommendations of subdiscipline groups. The group's decisions are based primarily on scientific merit, but technical feasibility and cost, if known in some degree, can enter this review. Science advocacy and challenge—Any proposed scientific effort has advocates who can argue its merits to the groups or authorities making priority decisions. Such advocacy provides informed support in the face of critical evaluation so that the scientific community will consider the proposal to have had fair representation. Technical evaluation—At some level a decision has to be made not only on scientific merit, but also on technical feasibility. This decision will involve cost, whether needed technology exists or can be reasonably developed, and its effect on other projects. This decision process is not totally separable from scientific merit, and there will be elements of technical evaluation at all decision levels.

OCR for page 48
MANAGING THE SPACE SCIENCES Management review—At the program and agency levels, if not at lower levels, there should be a comprehensive and vigorous confrontation of competing proposals. It is here that all of the factors going into priority setting must come together and the factors outside of science, technology, and cost (e.g., foreign policy, national needs, NASA strategies) must be considered by line management. Final review must ensure that all prior evaluations were made competently and in accordance with the agency's policies and procedures and that all significant criteria have been taken into account. Analytic/algorithmic methods—The Space Studies Board has explored this type of process in the past.7 The process involves specifying a set of criteria and a set of entities to be evaluated. Each entity is evaluated or ranked by each criterion and the results combined by an agreed-upon method to obtain an overall rank or priority. Obviously, there can be many alternative ways of doing this, ranging from hard numbers assigned at every step to much “softer” rankings and combinations. The main virtue of an algorithmic approach is that it forces an analysis of the substance and importance of the criteria to be used in prioritization. The danger is that the process can degenerate into an exercise in numerology. Anyone who wishes to use such a process should be careful to verify that the final outcome is consistent with his or her overall judgment. The committee developed the following specific recommendations for the prioritization process: Recommendation 5-1: A clear set of criteria for prioritizing scientific goals and missions should be established by NASA and adopted by all participants. Even though different subsets of the criteria may be applied by the various parties in the system, all of these participants should know what the complete set is, and, ideally, agree with it. Recommendation 5-2: Goals and priorities within subdisciplines, and to some extent within disciplines, have been set in the past by panels of scientists; feasibility is demonstrated by a number of NRC committee reports. It is highly important that scientific criteria be applied by scientists. Recommendation 5-3: Scientific goals need to be kept foremost in view at all times, even as NASA establishes priorities across the disciplines. As the scientific goals and priorities get modified by considerations of missions and/or broader national interests, including the priorities of other federal agencies, the scientific community needs to be continually involved in formulating these compromises and modifications. Recommendation 5-4: Processes should be fair and sufficiently open and transparent to ensure credibility among nonparticipants. Broad participation is desirable—a significant and respected part of the scientific community should be involved in major priority-setting exercises. These characteristics will also ensure credibility among the participants themselves. Some of the participants at lower levels of priority setting should also participate at higher levels. Recommendation 5-5: Processes and criteria at each priority-setting step must be able to identify new initiatives and rank ongoing efforts. The task of identifying new initiatives will occur automatically provided the formal exercise of priority setting is done periodically. Ongoing efforts must be rigorously included in the processes to ensure their continued priority. Management must face the task of canceling programs or projects that are failing or whose priority has dropped. 7   NRC, Space Studies Board, Setting Priorities in Space Research: An Experiment in Methodology, National Academy Press, 1995.

OCR for page 48
MANAGING THE SPACE SCIENCES Recommendation 5-6: Participation by scientists should continue up the hierarchy as far as scientific goals and criteria continue to play a role in priority setting. At each level, parties must have confidence that they are dealing only with propositions that have “passed scientific muster ” below. There are separate and complementary roles for in-house NASA committees and for external advisors. Recommendation 5-7: At each level in the hierarchy, priority-setting processes should incorporate (1) appropriate criteria, (2) competent and forceful advocacy, (3) strong challenge by dispassionate adversaries, and (4) involved discussion, all leading to consensus and/or concurrence. The processes should be public and widely understood and should include knowledgeable but disinterested individuals. NASA must buy in to the process, be a significant part of it, and accept the results. Interaction between NASA and the groups engaged in the process is necessary. A key element of the process is review of proposals for scientific investigation by peers of the proposers. There are, however, two issues that must be dealt with in the proper management of peer review. The first is that the process must be free of any taint of conflict of interest. The use of experts in the same or related fields carries with it the risk that the reviewers may influence the outcome in ways that are favorable either to their own programs or to their close collaborators. The second issue relates specifically to NASA programs where NASA center scientists may compete for support with outside scientists. It is essential that the review process, whether for flight missions or laboratory research, not create real or perceived advantages for center scientists. If program management functions are moved from Headquarters to field centers, it is particularly important that Headquarters retain responsibility for the selection process in order to ensure that no bias is introduced into the evaluation. Recommendation 5-8: Peer review for scientific investigations should be continued. Control of the peer review process should remain with Headquarters, and adequate Headquarters staff should be retained for this purpose. The process should be structured such that proposals from either outside or inside the agency are not placed at a real or apparent disadvantage. One of the charges to this committee is to ensure the preservation of funding opportunity for highly innovative or high-risk research in an environment of budget constraints and well-entrenched existing constituencies. An innovative proposal typically suggests a pilot project designed to explore a scientific or technological idea that falls outside contemporary frameworks of inquiry or design. In principle, applying a criterion for scientific merit such as “Potential for New Discoveries and Understanding, and Uniqueness” in a rigorous, systematic way would ensure high rankings for highly innovative research. However, innovative and high-risk research does not always enter the system by conventional means, and even when it does, it may not be easy to rank against more conventional proposals during the peer review process. The most effective way to ensure that innovative, high-risk proposals get due attention is to build some flexibility and discretion into the system. Such a proposal, which might well be considered “high risk” by peer-review panels, would be considered for funding if it meets two criteria: the possible payoff of a successful outcome would more than compensate for the perceived level of risk; and the proposed experiments are well-designed and in the hands of competent investigators. Program managers would announce the opportunity to submit such proposals to the scientific and technological communities. Review panels would also be encouraged to forward to the relevant program manager proposals that they perceive to fall into this category. The funding of innovative research should be adequate to test and explore an idea but should not

OCR for page 48
MANAGING THE SPACE SCIENCES become a permanent source of funding for it. At some point, after appropriate seeding, the research either should be terminated or should enter the normal channels of program definition and prioritization. Because proposals for innovative research may arise from many different sources and because, by definition, their precise nature is not easily anticipated, they need to be supported by an explicit budgetary item. Thus NASA budgets should include a line devoted to small grants for innovative ideas in space science, and the funds should not be used for any other purpose (e.g., for funding or “topping off” marginal projects in established budgets). The difficulty of maintaining room for innovative research will be much greater in the future than in the past because of intensified budgetary pressures. Recommendation 5-9: Innovative research that may lead to new and important scientific findings should be fostered through allocation of limited discretionary funding for innovative, high-risk, high-return ideas falling outside current frameworks of inquiry or design. This research is highly important and deserves special management attention, including that of the Chief Scientist. This recommendation is not intended to allow circumventing of peer review for the major parts of any science program. As an aside, small initiatives play an extremely important role in providing the opportunity for innovative ideas or unproved concepts to be included in the science enterprise. They should be judged principally on scientific merit and potential payoff. Tasks The tasks that must be performed to arrive at a final set of prioritized scientific goals and missions are suggested in Figure 5.1 and elaborated on below: Identify scientific goals—In any advancing field of science, research participants are knowledgeable about the most compelling unanswered questions. These questions lead to the definition of common scientific goals that would be most likely to advance scientific understanding. Peer review is the proper means of screening scientific goals to identify those items that are legitimate and important scientific goals. Prioritize scientific goals—Prioritization can take many forms. It may be a rigorous numerical ranking of items under consideration, or it may be no more than the identification, without ranking, of those items that exceed a defined threshold of importance. It is, basically, devising and using an explainable method—of whatever nature—to discriminate between things that are more important and those that are less important, and doing this to the extent necessary to make the decisions that must be made. Part of the entire process is deciding what issues (criteria) need to be addressed in determining importance. Define programs—Given scientific goals and priorities, it is possible to structure the content of a program, including both flight missions and ground-based research, appropriate to the prioritized scientific goals. It is important that the science program not be warped excessively to conform to the current high-priority flight missions, which, after all, address only certain of the priorities. Identify technology options—In order to define the missions necessary to pursue scientific goals, alternative technical approaches must be identified and judged. Space science projects must always be alert to new technology that might make missions more effective scientifically and less expensive. Define missions—With an understanding of science priorities, overall program structure, and technology options, it should be possible to devise an optimal set of missions to carry them out. In astronomy the missions are essentially defined by the scientific goals as noted. In the microgravity sciences the mission is set by nonscientific national priorities, and the science rides “piggyback” on them. In other disciplines the coupling varies in depth and strength.

OCR for page 48
MANAGING THE SPACE SCIENCES Prioritize missions—The primary consideration in prioritizing missions should be the quantity of high-quality science that can be achieved per dollar of expenditure. Evaluate technology, feasibility, and cost—This task is represented by the inner feedback loop in Figure 5.1. It is the means by which scientific goals and missions become integrated and, as noted earlier, varies in intensity and depth from discipline to discipline. Design payloads, platforms, vehicles—Conceptual versions of this task have to be performed as missions are defined to deduce feasibility and cost; these are iterated with prioritization of scientific goals. After the feedback loop in Figure 5.1 has been traversed enough to converge on science and mission priorities, the final, detailed design of payloads, platforms, and vehicles is the last step in planning the observational or experimental science. After collecting the data and observations, the investigators must be able to carry out their analysis and evaluation. SUMMARY OF PRIORITIZATION TASKS, TOOLS, AND PARTICIPANTS Summary Matrix Table 5.1 collects and summarizes the elements of prioritization. This matrix illustrates the approximate distribution and relationship of the factors involved in priority setting and is not meant as a hard prescription. The degree of scientific involvement from the subdiscipline to the agency level will vary depending on discipline, and tasks duplicated from one level to the next will vary in style and depth. During its analysis, the committee also considered the advisory functions traditionally carried out by the boards and committees of the National Research Council (see box). TABLE 5.1 Matrix of Factors Involved in Prioritization and Science Program Level Level Tasks Participants Criteria Approach Subdiscipline and discipline Identify and prioritize scientific goals Identify technology options SSB disciplinary committees NRC ad hoc committees NAC committees NASA program associate administrators Scientific merit Programmatic merit (initial) Peer review Peer judgment Science advocacy Initial technical evaluation Cross-discipline Prioritize scientific goals (interleave disciplines) Define programs and prioritize missions Evaluate technology feasibility, costs NAC advisory committees NASA program associate administrators NASA Chief Scientist NASA Science Council Scientific merit Programmatic merit Science advocacy and challenge Management review Agency Prioritize and select missions NASA Administrator NASA Chief Scientist NASA Science Council Scientific merit Programmatic merit Political and social merit Science advocacy and challenge Program advocacy and challenge Management review NOTE: At each level the process deals only with propositions that have the heritage of scientific merit from earlier reviews.

OCR for page 48
MANAGING THE SPACE SCIENCES Role of the National Research Council National Research Council committees, including the Space Studies Board, should continue to be the channel and surrogate for participation by the broader scientific community and should be used by NASA to solicit and draw on the best ideas and judgment of this community. These committees should be made up of scientists not supported by NASA, as well as those who are. The establishment of space science priorities should begin, as it has in the past, with the efforts of the scientific community through the NRC and the Space Studies Board. This prioritization will address broad research thrusts, not individual proposals or technical approaches. The disciplinary committees of the Board have been effective in the past in suggesting important scientific goals for NASA to pursue. This process should be continued. Clear statements of priorities among suggested scientific goals should be made by all NRC committees that address priorities. The disciplinary committees should give some weight, in the beginning, to costs and timing. Scientists can often help in finding low- or lower-cost ways of addressing scientific goals. There is no reason to leave this factor totally out of consideration, even at the early stages of formulating scientific goals. Information about mission constraints, to the extent known, should be obtained by NRC disciplinary committees. An important output of the disciplinary committees should be a relatively small number of important scientific goals, major or moderate-sized programs, and the scientific justification for them. Arguments in support of their importance should be founded on defined criteria. Recommendations for NASA Management The committee offers the following additional prioritization-related management recommendations for NASA. Recommendation 5-10: The NASA Advisory Council and its committees should continue to play a major advisory role in determining program priorities. The Council and committees should be composed primarily of external members, with internal NASA scientists included as appropriate. Recommendation 5-11: The Chief Scientist should attend all key internal NASA meetings concerned with priorities and ensure that adequate scientific representation is maintained throughout the prioritization process and that a properly balanced set of recommendations reaches the Administrator. The Chief Scientist needs to be able to argue issues directly to the Administrator of NASA, independent of decisions by committees or managers at lower levels. Recommendation 5-12: Within NASA Headquarters, there must be a capable scientific staff to support management priority setting in order to help ensure compatibility of program content and science priorities. These scientists must also interface with field center managers and external investigators to ensure science program integrity.