Executive Summary

Beginning in the early to mid-1990s, NASA moved toward mission lines that offer scientists the opportunity to lead their own space science missions. Before that, scientists had taken responsibility for science instruments and data analysis on a mission but NASA had managed the projects and developed the spacecraft.1 As a first step, NASA introduced the Discovery Program and developed it into a competitive, peer-reviewed mission line moving toward planetary science exploration under the principal investigator (PI) mode.2 Then it transitioned the Explorer Program, the oldest of its competitive mission lines, to the PI-led mode as well. Explorer missions are focused on goals in solar and space physics and in astrophysics; Discovery missions address solar system exploration and the goals of NASA’s Origins and Astrobiology programs. The PI-led approach gives scientists more autonomy and freedom in the decision making and management of a developing space mission but at the same time enforces a strict cost cap that constrains competition for the selection and subsequent development of the PI-led mission. In the last 5 years, NASA has introduced two additional PI-led mission lines: Mars Scout provides mission opportunities for the Mars Exploration Program, and New Frontiers invites proposals for targeted solar system exploration.

Thirteen PI-led projects have successfully achieved—or are about to achieve—their mission, and eight others are currently in various stages of development. Two suffered technical failures and one was canceled. In addition, the PI-led mission lines have had to adjust to the changing environment at NASA and in society as a whole. Recently, PI-led mission costs and schedules have increased so much that NASA is considering what lessons might be learned from the different PI-led programs and whether the programs can be improved. To that end, NASA asked the Space Studies Board of the National Research Council to explore the factors contributing to the successes and challenges of PI-led missions. The Committee on Principal-Investigator-Led Missions in the Space Sciences undertook this task with the understanding that such missions are an essential, scientifically productive component within NASA’s suite of missions that complements the strategic missions emerging from the decadal survey and roadmap processes. The importance of

1  

Spacecraft projects that are managed and developed by NASA are referred to as core missions throughout this report.

2  

In this report, program refers to a PI-led mission line and project refers to an individual PI-led mission.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences Executive Summary Beginning in the early to mid-1990s, NASA moved toward mission lines that offer scientists the opportunity to lead their own space science missions. Before that, scientists had taken responsibility for science instruments and data analysis on a mission but NASA had managed the projects and developed the spacecraft.1 As a first step, NASA introduced the Discovery Program and developed it into a competitive, peer-reviewed mission line moving toward planetary science exploration under the principal investigator (PI) mode.2 Then it transitioned the Explorer Program, the oldest of its competitive mission lines, to the PI-led mode as well. Explorer missions are focused on goals in solar and space physics and in astrophysics; Discovery missions address solar system exploration and the goals of NASA’s Origins and Astrobiology programs. The PI-led approach gives scientists more autonomy and freedom in the decision making and management of a developing space mission but at the same time enforces a strict cost cap that constrains competition for the selection and subsequent development of the PI-led mission. In the last 5 years, NASA has introduced two additional PI-led mission lines: Mars Scout provides mission opportunities for the Mars Exploration Program, and New Frontiers invites proposals for targeted solar system exploration. Thirteen PI-led projects have successfully achieved—or are about to achieve—their mission, and eight others are currently in various stages of development. Two suffered technical failures and one was canceled. In addition, the PI-led mission lines have had to adjust to the changing environment at NASA and in society as a whole. Recently, PI-led mission costs and schedules have increased so much that NASA is considering what lessons might be learned from the different PI-led programs and whether the programs can be improved. To that end, NASA asked the Space Studies Board of the National Research Council to explore the factors contributing to the successes and challenges of PI-led missions. The Committee on Principal-Investigator-Led Missions in the Space Sciences undertook this task with the understanding that such missions are an essential, scientifically productive component within NASA’s suite of missions that complements the strategic missions emerging from the decadal survey and roadmap processes. The importance of 1   Spacecraft projects that are managed and developed by NASA are referred to as core missions throughout this report. 2   In this report, program refers to a PI-led mission line and project refers to an individual PI-led mission.

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences these small and medium Discovery- and Explorer-class missions was noted in several previous NRC reports;3,4,5 one of them, a 2004 report,6 stated: The Explorer program contributes vital elements that are not covered by the mainline … missions. Explorers fill critical science gaps in areas that are not addressed by strategic missions, they support the rapid implementation of attacks on very focused topics, and they provide for innovation and the use of new approaches that are difficult to incorporate into the long planning cycles needed to get a mission into the strategic mission queues…. The Explorers also provide a particularly substantial means to engage and train science and engineering students in the full life cycle of space research projects. Consequently, a robust … science program requires a robust Explorer program. Input from PIs, project managers (PMs), and others led the committee to the following overall finding: Finding. The space science community believes that the scientific effectiveness of PI-led missions is largely due to the direct involvement of PIs in shaping the decisions and the mission approach to realizing the proposed science concepts. In this report the committee recommends practices and incentives for improving the overall conduct of PI-led missions. In particular, it recommends adjustments to the selection and implementation processes that aim to strengthen the mission-line programs so that they can continue to provide one of the best science returns per taxpayer dollar for NASA, the scientific community, and the public. The committee’s findings and recommendations are presented below and organized into five themes: the selection process, funding profiles, international contributions, program management, and project management. SELECTION PROCESS Information gathered by the committee indicates that the scientific and technical communities invest excessive effort in preparing proposals for PI-led mission programs and that few institutions can or should maintain the infrastructure support (administrative, management, cost estimation) that is required for responding to announcements of opportunity (AOs) for PI-led missions. The review panels involved in evaluating and selecting PI-led mission proposals need to be able to make their decisions based on a more concise set of essential information and in the end to select from proposals that have made a short list and that have been better developed because proposers received funding to prepare mission concept studies. As a result of the large number of detailed proposals submitted in response to AOs, for which NASA conducts separate science and technical merit reviews, the selection process can be inefficient and ineffective. The administrative, management, and cost analysis efforts and the time involved in preparing proposals (for which the chance of success is only 10 percent or less) are unnecessarily exhausting proposers and reviewers, depleting their resources, and resulting in selections that in some cases are destined for cost and schedule problems from the start. NASA may wish to reconsider the basic ideas behind the technical, management, cost (TMC) experiment of 1999, TMC-lite, which tried out a selection 3   National Research Council (NRC), 2004, Review of Progress in Astronomy and Astrophysics Toward the Decadal Vision, Letter Report, Washington, D.C.: The National Academies Press, p. 10. 4   NRC, 2003, New Frontiers in the Solar System: An Integrated Exploration Strategy, Washington, D.C.: The National Academies Press, pp. 191-192. 5   NRC, 2001, Astronomy and Astrophysics in the New Millennium, Washington, D.C.: National Academy Press, pp. 194-195. 6   NRC, 2004, Solar and Space Physics and Its Role in Space Exploration, Washington, D.C.: The National Academies Press, p. 20.

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences process aimed at reducing the information required in a proposal and, thus, the burden on the proposer. NASA may also wish to consider emphasizing certain scientific targets or concept areas in the AOs as a means of reducing the number of proposals submitted—and another means of reducing the burden on proposers. On the other hand, the concept studies that will be required after the provisional selection round of competition need to be more mature in project design definition and TMC planning in order to provide a sound basis for final evaluation and selection. Proposals and Reviews Finding. The PI-led mission selection process could be made more efficient and effective, minimizing the burden on the proposer and the reviewer and facilitating the selection of concepts that become more uniformly successful projects. Recommendation 1. NASA should consider modifying the PI-led mission selection process in the following ways: Revise the required content of the mission proposals to allow informed selection while minimizing the burden on the proposing and reviewing communities by, for example, reconsidering the TMC-lite approach and eliminating the need for content that restates program requirements or provides detailed descriptions such as schedules that would be better left for postselection concept studies, Alter the order of the review process by removing low- to medium-ranking science proposals from the competition before the TMC review, and Allow review panels to further query proposers of the most promising subset of concepts for clarification, as necessary. Finding. The still-competitive but already funded concept study stage (Phase A) of selected, short-listed PI-led missions is the best stage for the accurate definition of the concept details and cost estimates needed to assist in final selection. Recommendation 2. NASA should increase the funding for and duration of concept studies (Phase A) to ensure that more accurate information on cost, schedule, and technical readiness is available for final selection of PI-led missions. Finding. Community-based studies of science opportunities and priorities can be used to focus AO proposals on specific topics of great interest and to guide the choices of selection officials. Recommendation 3. NASA should make explicit all factors to be considered in the selection of PI-led missions—for example, targets and/or technologies that are especially timely and any factors related to allocating work among institutions and NASA centers. Proposing Team Experience and Leadership The committee finds that the importance of team experience and interpersonal and institutional interactions cannot be overstated. The officials who select PI-led missions need to be able to evaluate and duly weigh the teaming aspect of a proposed mission. Along the same lines, the members of a chosen

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences mission, especially the PI and the technical PMs, need to seek out experienced teammates, especially individuals who have worked on other PI-led missions, suborbital projects, core missions, and/or technology development projects such as complex technical systems or instruments. NASA can help to make these experiences available to younger scientists and engineers and also to foster the transfer of information from active mission teams to potential proposers. Finding. The combined relevant experience of the PI and the PMs in PI-led missions is critical to mission success. Programs can emphasize the importance of experience in their selections and create opportunities for prospective PIs and PMs to gain such experience. Recommendation 4. NASA should develop PI/PM teams whose combined experience and personal commitment to the proposed implementation plan can be evaluated. NASA should also provide opportunities for scientists and engineers to gain practical spaceflight experience before they become involved in PI-led or core NASA missions. These opportunities could become available as a result of revitalizing some smaller flight programs, such as the sounding rocket and University-class Explorer programs. Technology Readiness Based on its interviews and data-gathering efforts, the committee identified underdeveloped technologies as a major source of cost and schedule problems for PI-led missions. At the same time, the committee found that opportunities—for example, availability of competed funds—for developing technologies for PI-led missions outside the actual mission were limited. Explicit, competed technology development components for each PI-led program (Discovery, New Frontiers, Explorer, Mars Scout) could help ensure that a pipeline of technology developments, from the breadboard to the brassboard levels (closer to flight-ready design), will be available for use on future PI-led missions. Such competed technology development efforts would diminish the likelihood that untested technologies will be used in a PI-led mission. Finding. As a rule, PI-led missions are too constrained by cost and schedule to comfortably support significant technology development. Those missions that include technology development inevitably have cost and schedule problems. Regular technology development opportunities managed by PI-led programs could lead to a technology pipeline that would help to enable successful mission selection and implementation. Recommendation 5. NASA should set aside meaningful levels of regular funding in PI-led programs to sponsor relevant, competed technology development efforts. The results from these program-oriented activities should be made openly available on the program library Web site and in articles published in journals or on the World Wide Web. FUNDING PROFILES Project funding profiles—schedules for spending a project’s funds for development, implementation, and operations—have been mandated in some AOs, resulting in funding increments that force the PI to follow a development schedule that may be inefficient or even risky. For instance, funds spent early on instrumentation or systems technologies can ensure that the instruments have been tested sufficiently before being integrated onto the spacecraft. The selection process could include an evaluation of the funding profile established by the proposer(s). NASA could consider adjusting a project’s cost caps if it cannot secure funds on the schedule proposed for the selected mission.

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences Finding. Funding profiles represent a special challenge for PI-led missions because they are planned at the mission concept stage with the goals of minimizing costs and achieving schedules. However, like all NASA missions, PI-led missions are subject to the availability of NASA funding, annual NASA budgetary cycles, and agency decisions on funding priorities, all of which can disrupt the planned funding profiles for PI-led missions. Recommendation 6. NASA and individual mission PIs should mutually agree on a funding profile that will support mission development and execution as efficiently as possible. If NASA must later deviate from that profile, the mission cost cap should be adjusted upward to cover the cost of the inefficiency that results from the change in funding profile (see Recommendation 10). INTERNATIONAL CONTRIBUTIONS International contributions have an important impact on the science capabilities of PI-led missions, often providing major pieces of the science instrument payload. Yet these collaborations are viewed as risky because it is difficult to get foreign entities to commit funds before a proposal has been selected and to conduct technical exchanges in the face of International Traffic in Arms Regulations (ITAR) requirements. While the increased national emphasis on ITAR, with its sometimes poorly defined restrictions on technology and technical information exchange, has hurt many NASA mission programs, its impact on the highly cost-constrained PI-led missions can be even more damaging, especially as it discourages the involvement of international team members. University and student participation in PI-led missions, ostensibly an advantage of the PI-led approach, can also be compromised because, based on ITAR concerns, NASA in its contracts with universities, private industry, or other entities restricts the access of some individuals to certain technical information. Finding. International contributions have an important positive impact on the science capabilities of PI-led missions but are faced with an increasingly discouraging environment, in part due to ITAR. In addition, logistical difficulties associated with foreign government budgetary commitments and the timing of proposals and selections persist. The result is both real and perceived barriers to teaming and higher perceived risk for missions including international partners. Recommendation 7. NASA PI-led-mission program officials should use recent experiences with ITAR to clarify for proposers (in the AO) and for selected projects (e.g., in guidance on writing technical assistance agreements and transferal letters7) the appropriate application of ITAR rules and regulations. PROGRAM MANAGEMENT Role of the Program Office The PI-led mission program offices provide support and oversight functions for PI-led projects. Each of the offices, which are staffed by NASA personnel, has a different location, style of operating, and approach to assisting the PI-led projects in its program. The Explorer Program Office at Goddard Space Flight Center (GSFC) has, through its long history and NASA center infrastructure, provided substantial project assis- 7   Transferal letters are documents that describe relationships between NASA and non-U.S. institutions or funding agencies, for example.

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences tance. The Discovery Program Office has been relocated on more than one occasion and is in a state of flux, which has led to difficulties for some Discovery missions. A recently merged Discovery and New Frontiers Program Office is in the process of being reestablished at Marshall Space Flight Center (MSFC), and the relatively new Mars Scout Program Office at the Jet Propulsion Laboratory is managing its first mission. Finding. The PI-led program offices can play a critical positive role in the success of PI-led missions if they are appropriately located and staffed, and are able to offer enabling infrastructure for projects and NASA Headquarters from the proposal through the implementation stages. Recommendation 8. NASA should ensure stability at its program offices, while providing sufficient personnel and authority to enable their effectiveness, both in supporting their missions and in reporting to and planning with NASA Headquarters. Program Oversight Practices NASA oversight of all missions, including PI-led missions, has been increasing over the past decade, largely in response to the failures of non-PI-led Mars missions in the 1990s, followed by the Columbia shuttle disaster. This increase in oversight has meant cost and schedule difficulties for PI-led missions and has challenged their style of management by adding formal technical and management reviews by NASA-appointed review teams. PI-led mission PMs argue that such reviews can introduce risk into PI-led projects, because they repeatedly distract the team from the planned implementation tasks. Even when compensated for the costs of these reviews, PI-led project managers view NASA-mandated reviews as less useful to their projects than informal peer reviews of subsystems in which small numbers of experts external to the project provide technical assessments and advice. NASA needs to consider both the appropriate level of oversight for PI-led missions and adjustments to the cost caps to cover the cost of additional reviews. Such actions would be especially timely in view of the recent establishment of the Independent Technical Authority (ITA), a new technical oversight organization whose impact on NASA science missions is still undetermined, according to NASA interviewees. Finding. NASA oversight of PI-led missions, as well as of all missions, increased following a string of mission failures in the late 1990s and is again increasing following the Columbia shuttle disaster. Some of the added oversight, and especially the style of that oversight, appears excessive for robotic missions as small as the PI-led missions. Increases in oversight also strain project resources and personnel to the point of adding risk rather than reducing it. Recommendation 9. NASA should resist increasing PI-led mission technical and oversight requirements—as, for example, on quality assurance, documentation, ITA-imposed requirements, or the use of independent reviews—to the level of requirements for larger core missions and should select missions whose risks are well understood and that have plans for adequate and effective testing. Finding. There is confusion about the processes in place for adjusting PI-led mission cost caps and schedules to accommodate oversight requirements introduced after selection. Recommendation 10. NASA should clarify the change-of-scope procedures available for projects to negotiate the cost and schedule impacts of any changes in requirements initiated by NASA Headquarters

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences or a PI-led program office, including the addition of reviews, documentation, reporting, and/or increased standards. The schedule impact of negotiating changes of scope should also be evaluated. Threat of Cancellation If a PI-led mission is projected to exceed its cost cap for reasons that NASA Headquarters judges to be within the project, the PI-led program office (Explorer, Discovery/New Frontiers, Mars Scout) and NASA Headquarters may call a termination review. The Program Management Council of NASA Headquarters’ Science Mission Directorate conducts these reviews to determine the cause of the cost overrun and the appropriate response. Possible responses to overruns include allowing the mission to proceed, often at additional cost; changing project management and/or contractors in consultation with the PI; descoping the mission (removing systems or instruments); or terminating it. The committee learned that termination reviews are no longer regarded as mission-threatening, because very few missions have been canceled even though some PI-led (and most core) missions do grow beyond their initial cost cap. Moreover, canceling a mission after substantial investment has been made is not reasonable if the mission has no fatal technical issues or additional cost or schedule requirements. However, a PI-led mission is more vulnerable than a core mission to cancellation or descopes because its cost cap was a key factor in its winning the competition. The committee considers termination reviews as an effective management tool for missions that overrun their cost caps, provided that both NASA and the project teams recognize that such reviews raise the prospect of Headquarters-mandated changes to the mission capability. Lessons learned from these reviews should be used to inform other active PI-led program and project leaders. A related concern is science instrument descopes that have been decided without the PI’s agreement and outside the termination review process. Finding. The threat of cancellation in a termination review is no longer an effective way of keeping PI-led missions within their cost caps, because few missions have been canceled as a result of exceeding their cost caps. Nevertheless, a termination review is taken seriously because it reflects negatively on project management performance and raises the possibility of science descopes. Project leaders need to be made aware of problems that lead to termination reviews so that they can avoid them. Recommendation 11. NASA should continue to use the existing termination review process to decide the fate of PI-led missions that exceed their cost cap. It should develop lessons learned from termination reviews and make them available to other PI-led projects. Finding. High-impact decisions such as descopes made by NASA outside the termination review process undermine a PI’s authority and can cause a mission to lose science capability. Recommendation 12. NASA should not descope mission capabilities (including science instruments) without the PI’s agreement or outside the termination review process. PROJECT MANAGEMENT Technical and Programmatic Failures The committee found that potentially valuable lessons learned in both the technical and management areas of PI-led missions are neither easily located nor widely discussed despite being resources of which

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences every PI-led mission leader should be aware. The PI-led program offices can help disseminate lessons-learned information. For example, the well-regarded engineering practice “test as you fly,” which replicates in-flight conditions as closely as possible in ground subsystems tests, can be reinforced, useful peer reviewer names shared, and design and parts information quickly aired. Such practices could allow a return to fewer technical requirements, such as prevailed in the early days of PI-led missions. Finding. Lessons learned from experience in both PI-led and other missions can be extremely valuable for reducing risk and inspiring ideas about how to do things better. Much useful lessons-learned documentation is available on the Web but is not collected in a coherent library or directory. A modest effort by the program offices to locate these distributed documents, provide a centralized Web site containing links, and advertise its existence would allow these lessons to be more widely used. Recommendation 13. NASA PI-led program officials and PI-led mission teams should study lessons-learned documentation to benefit from the experiences of previous PI-led missions. NASA should make such lessons learned easily and widely available and update them continuously, as is done on the Discovery Program Web site posted by the Langley Research Center. Team Interactions The ability of PI-led mission team members, especially the PI and the PM, to work together has a critical impact on the progress of these projects. PIs need to choose a PM they can acknowledge as a technical lead and on whom they can rely. If unresolvable differences arise and appropriate efforts at resolution fail, the PI should have the authority to replace the PM with the concurrence of the relevant program office. Similarly, project leadership and the relevant NASA center and/or industrial teammates should communicate openly and be able to ensure that all team members function in their designated roles. NASA should enable and support PIs in adjusting the composition of project leadership and teams if that becomes necessary. Similarly, the supporting institution supplying the PM, including a NASA center, should not have the authority to replace the PM without the PI’s agreement. Finding. The leaders of PI-led missions occasionally find they must replace a manager or a key team member to reach their goals. While the cost and schedule impacts of such a major change must be considered, a change in project management needs to be allowed if it is for the good of the mission. The PI should make all final decisions on project management personnel. Recommendation 14. NASA and the PIs should include language in their contracts that acknowledges the PI’s authority to make the final decisions on key project personnel. Cost, Schedule, and Science Performance The committee encountered difficulties in obtaining a consistent set of mission cost performance summaries, a situation that apparently stems from differences in the ways in which the different PI programs and projects keep cost and budget records. While many records contain useful mission budgetary and schedule information, the committee was unable to obtain the kind of moderately detailed data that would normally be expected to be readily available for NASA’s own internal use or for an analysis of historical trends. Consolidating records into a few standard templates for mission programs, including PI-led missions, would facilitate analyzing the cost and schedule performance of those missions.

OCR for page 1
Principal-Investigator-Led Missions in the Space Sciences The information that could be obtained on cost and schedule performance in PI-led missions indicated that they face the same cost growth drivers as core or strategic missions but that any such growth in PI-led missions is more visible within NASA because the cost caps are enforced so much more strictly. The cost growth, in percent, of PI-led missions is in any case documented as being, on average, less than that for core missions (see Chapter 5). The perception of a cost growth problem specific to PI-led mission lines is thus not supported by the records. On the other hand, their science performance appears to be competitive with that of core missions, although more highly focused, with science analysis phase (Phase E) investments in PI-led projects averaging around 10 percent of the mission cost. The guest investigator opportunities funded by some PI projects, as well as supplemental resources from NASA supporting research and technology and data analysis programs, benefit science outcomes. Finding. The summary cost and schedule performance records for PI-led and other missions are not kept in a consistent way, making external comparative analyses difficult. Science activities on PI-led missions seem to be competitive with those on core missions to the extent that the data sets are made available and science analysis is supported. Recommendation 15. NASA should maintain and make available for assessment consistent and official documentation of project costs and reasons for cost growth on all PI-led (and other) missions. PI-LED MISSIONS AND THE VISION FOR SPACE EXPLORATION In considering the recommendations provided here in their entirety, the committee recognizes that NASA is already at least partially implementing (or attempting to implement) some of the items, such as moderating ITAR impacts on space science missions and considering enhanced concept study phases (Phase A’s). The Discovery/New Frontiers Program Office is currently undergoing changes, and the technology development issues for space science missions are under scrutiny. Nevertheless, the committee believes these issues should be emphasized here. As the committee completes this report, NASA Headquarters and its programs are undergoing significant change in response to the President’s Vision for Space Exploration. The Science Mission Directorate now consists of four subdivisions: Heliophysics, Planetary Science, Earth Science, and Astronomy and Physics. Earth Science has its own line of PI-led missions.8 The space science PI-led mission lines described in this report have the potential to address some of the high-priority science recommended in NRC decadal surveys. They also have the potential for application to the Vision for Space Exploration, particularly for missions related to the exploration of the Moon and Mars and for characterizing the solar-activity-related radiation environment. Subjects relevant to the Vision that match or complement the objectives and/or instrument capabilities of desirable missions in the decadal surveys may be especially strategic targets for PI-led missions at this time in NASA’s history. The committee believes that its report provides some useful suggestions and recommendations that would help NASA administrators, agency program managers, centers, and the science community as they continue to exploit this most grass-roots of NASA mission lines. 8   NRC, 2004, Steps to Facilitate Principal-Investigator-Led Earth Science Missions, Washington, D.C.: The National Academies Press.