Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 57
Principal-Investigator-Led Missions in the Space Sciences 7 Conclusions and Recommendations Overall, the committee was impressed by the strong support offered by everyone it interviewed for PI-led missions, regardless of any problems or difficulties they encountered. Many experts remarked that the PI-led projects put science first in making choices and setting priorities. The personal involvement inspired by these mission lines brings to mind the pioneers of the space age. However, in spite of the great enthusiasm, the committee identified several sources of stress for the projects that emerged as key problems in cost and schedule management or as personnel and/or institutional conflicts. Described below are the committee’s findings and recommendations on the most pervasive issues, which generally fall into one of three categories: the selection process, program management, and project management. SELECTION PROCESS Proposals and Reviews The task of creating a comprehensive Step 1 mission concept proposal is tremendously demanding and expensive and is not funded by NASA. While participating NASA centers generally have access to other sources of support for PI-led proposal activities,1 the burden of an effort with, typically, less than a 10 percent chance of selection must be borne by the proposers—namely, industry and the science community. The committee confirmed that only a small minority of organizations have institutional funds available to support proposal efforts for PI-led missions. The low probability of generating a successful proposal seriously hinders the participation of the smaller aerospace and technology companies that NASA would like to attract.2 At least the Step 1 selectees obtain NASA funding to refine their concepts in a later 1 NASA centers negotiate with NASA Headquarters on bid and proposal funds, which come from center overhead funds. 2 NASA AOs for PI-led missions explicitly encourage that proposers include small aerospace companies. For example, Discovery AO NNH04ZSS0020, April 16, 2004, states as follows: Additionally, the Discovery Program requires proposers to set goals for the participation of Small Disadvantaged Businesses (SDBs), Women-Owned Small Businesses (WOSBs), Veteran-Owned Small Businesses (VOSBs), Historically Black Colleges and Universities
OCR for page 58
Principal-Investigator-Led Missions in the Space Sciences TABLE 7.1 Reproposals Related to Discovery Mission Announcements of Opportunity 1996 1998 2000 2004 Overall reproposals 56% (9 of 16) 43% (10 of 23) 50% (13 of 26) 60% (9 of 15) Phase A reproposals 80% (4 of 5) Phase A studies were reproposals; 3 were selected 100% (3 of 3) Phase A studies were reproposals; 2 were selected Not applicable Reproposals selected for Phase B 60% (3 of 5) 66% (2 of 3) Not applicable SOURCE: Susan Niebur, NASA Headquarters, Science Mission Directorate. definition phase (Phase A) and have incentives to promote their ideas and report their progress in open and accessible reports and presentations. Although the experts interviewed by the committee deemed the level of support and the duration of Phase A inadequate, these initial awards allow substantial refinement and further detail in key areas of the mission concept.3 It is notable that more than half of the selected concepts at both stages of selection were new versions of concepts that had previously been proposed but were not selected. According to NASA Headquarters, about 60 percent of Explorer proposals (including the 1998 MIDEX, the 1999 SMEX, the 2001 MIDEX, and the 2003 SMEX) were reproposals; 60 percent of the Explorer proposals selected for Phase A studies were reproposals, and 75 percent of Explorer proposals that resulted in missions downselected to Phase B or extended Phase A were reproposals.4 Table 7.1 provides similar statistics for the Discovery Program. A Step 1 selection process based on a succinct scientific and technical implementation description should be possible. For high-ranked proposals, this approach could be supplemented with follow-up questions from the selection committee. In 1999, NASA tried a TMC-lite review and selection process for the 2003 SMEX opportunity: Step 1 proposals were not required to provide so many details of cost and schedule, nor were they required to include formal approvals of their needed commitments and budgets. Thus, rigorous TMCO reviews were not done at Step 1, giving science the most weight in the initial selection. A larger number of concepts were provided Step 2 or Phase A funding. At the end of Phase A, the concepts were all deemed more risky than concepts that had been subjected to the original two-step full TMCO evaluation process. Proposers later suggested that TMC-lite required almost the same level of investment and team effort except for the formal endorsements. NASA Headquarters perceived that the missions selected from the 2003 SMEX round did not fare well, and one of the two selected missions was canceled at Phase B. NASA program officials concluded that the original selection process was superior and abandoned the TMC-lite review. At least one interviewed official indicated that NASA funded Phase A studies for missions that were viewed by (HBCUs), and other Minority Educational Institutions (MEIs) in proposed procurements (see Section 5.8). Participating Scientist Programs (PSPs), Data Analysis Programs (DAPs), and/or Guest Observing programs (GOs) that involve more members of the community in the data analysis and/or mission operation are encouraged, as described in Section 5.2.5. NASA 2003 AO-03-OSS-02 for SMEX and MoOs states, similarly: The PI and team members shall agree to use their best efforts to assist NASA in achieving its goal for the participation of small disadvantaged businesses (SDB’s), women-owned small businesses (WOSB’s), historically black colleges and universities (HBCU’s), and other minority institutions (OMI’s) in NASA procurements. Investment in these organizations reflects NASA’s commitment to increase the participation of minority concerns in the aerospace community and is to be viewed as an investment in our future. 3 For the core mission Solar Dynamic Observatory, nearly 10 percent of the total mission cost was spent to establish the cost cap (technical baselines, cost, and schedule), which is significantly more than is spent for PI missions. 4 Paul Hertz, assistant associate administrator for science, Science Mission Directorate, NASA Headquarters.
OCR for page 59
Principal-Investigator-Led Missions in the Space Sciences selection panels as unlikely to advance. This information suggests the TMC-lite concept may not have been optimally applied.5 Based on the information gathered, the committee believes that revisiting the TMC-lite process and exploring other approaches might be one way to reduce the burden on proposers and might offer NASA an efficient and less burdensome proposal process. (In fact, despite the full TMCO process—parallel science and TMCO reviews—that is currently applied to PI-led missions, some of these missions still face cost overruns and problems with technical readiness (see Chapter 5).) TMC-lite also could reduce the barriers to entry and improve the competitive field for PI-led concepts, because PIs from smaller or less-well-endowed institutions may be better able to participate in a TMC-lite process.6 As discussed above, a TMC-lite Step 1 selection process that includes an option for selection committees to follow up with questions for highly ranked proposals is one possibility. The composition of the review panel may also contribute to the effectiveness of the selection process. NASA needs to rely on expert advice at every stage of the selection process to make the best possible decisions. A panel that does not have enough information from a proposal to make a confident decision may benefit from being able to question the proposers for clarification.7 Similarly, a review panel that does not believe it has the internal expertise to properly evaluate a concept should be able to take on additional advisors. Better-informed expert opinion and fewer Phase A selectees, who would be funded more generously and given more time in Phase A, could help to stop recent cost and schedule growth in PI-led missions. The committee considered whether the unrestricted nature of the solicitations for concept proposals is an aspect of PI-led mission programs that could be modified. The very large number of possible scientific concepts increases the difficulty of proposing and of choosing from among competing missions. There are many more scientifically worthy mission goals than can be accomplished by any program with limited resources and budgetary constraints. In the current evaluation process, technical, management, cost, and other (TMCO) selection panels spend time on detailed assessments no matter what the ranking or timeliness of the science, while a science panel works in parallel to prioritize without knowing the concept’s TMCO feasibility. This approach greatly taxes review panels and detracts from the depth of review for those concepts that are finally selected. In addition, the proposers never benefit from the full TMCO review, because the panel shares only a limited amount of information with the proposers in verbal debriefings. Moreover, even if the science panel uses community-based studies and NASA roadmaps as one measure of the desirability of a particular concept, they are often so broad in scope that they do not help to narrow the field.8 For New Frontiers missions, a few key programmatic/strategic mission targets are specified, although details of the science investigations and mission implementation are left open. The relative success of this practice in achieving the desired goals of the PI-led mission programs remains to be demonstrated. Nevertheless, the committee learned that in the case of New Frontiers, the ability of the proposing and selecting groups to focus on fewer targets, and thus fewer proposals, has simplified the selection process, with only a small subgroup of the scientific and technical communities engaged.9 The practice of focusing on fewer targets need not be viewed as restricting the number of targets but rather as highlighting desirable 5 The committee’s information on the PI-led selection process was acquired, in large part, from R. Wayne Richie, who at the time was Mars acquisition manager at NASA Langley. 6 Those PIs from smaller institutions should be able to attract the necessary infrastructure support once they have successfully passed the first selection stage. 7 For example, a panel may wish to contact proposers in cases where a limitation on the number of pages or a misunderstanding of the AO requirements is considered to be the main problem rather than the quality of the proposals. 8 See SRM3 Solar System Exploration Strategic Roadmap, available at <www.lpi.usra.edu/opag/announcements.html>, accessed August 8, 2004. 9 In the case of New Frontiers, selection review panels were larger because of the higher dollar value of the mission.
OCR for page 60
Principal-Investigator-Led Missions in the Space Sciences targets. A proposal that successfully sells itself to the selectors but is not addressing a highlighted target need not be any less competitive. The selection process is a critical part of PI-led mission programs and can sometimes guarantee the success or failure of missions up front. Based on data gathered during the study, the committee concluded that the selecting panels, both scientific and technical, are spread thin and that there are too many mission concepts to sort through and too many viable options for PI-led missions. Moreover, in spite of all of the effort expended on the Step 1 proposals (concept proposals) and selection (see Chapter 3), PIs, PMs, and NASA officials considered that Phase A study results were often too superficial to ensure successful performance in subsequent phases. The combination of a menu containing fewer but higher-priority science topics or targets and longer, better-funded Phase A studies could set selected PI-led missions on a more solid path to keeping their cost, schedule, and science goals.10 The narrower range of science topics or targets is in subtle respects already evident in programs other than New Frontiers, first in the choices that NASA centers and the aerospace industry make in teaming on PI-led proposals and second in the final decisions by NASA Headquarters selecting official(s). Proposed missions are judged not in isolation but in the context of the core missions and NASA roadmaps as well as the distribution of missions among the various competing disciplines, centers, and institutions.11 It is important that the PI-led missions continue to complement the strategic missions identified in the decadal surveys and roadmaps and provide opportunities for pursuing scientific objectives that have not been decades in the planning. Some interviewees suggested that other considerations—including existing workload, recent performance at an implementing organization, and program office mission preferences—may also influence the outcome. The committee believes that proposers would benefit from more specific guidance, when possible, on NASA priorities and program directions at the time of the AO releases. Recommendation 1. NASA should consider modifying the PI-led mission selection process in the following ways: Revise the required content of the mission proposals to allow informed selection while minimizing the burden on the proposing and reviewing communities by, for example, reconsidering the TMC-lite approach and eliminating the need for content that restates program requirements or provides detailed descriptions such as schedules that would be better left for postselection concept studies, Alter the order of the review process by removing low- to medium-ranking science proposals from the competition before the TMC review, and Allow review panels to further query proposers of the most promising subset of concepts for clarification, as necessary. Recommendation 2. NASA should increase the funding for and the duration of concept studies (Phase A) to ensure that more accurate information on cost, schedule, and technical readiness is available for final selection of PI-led missions. Recommendation 3. NASA should make explicit all factors to be considered in the selection of PI-led missions—for example, targets and/or technologies that are especially timely and any factors related to allocating work among institutions and NASA centers. 10 Paul Graf, Aerospace Solutions LLC, comments to the committee on November 18, 2004. Dr. Graf indicated that missions that had spent 10-15 percent of total mission costs on Phase A/B experienced lower project overruns. 11 See SRM3 Solar System Exploration Strategic Roadmap, available at <www.lpi.usra.edu/opag/announcements.html>, accessed August 8, 2004.
OCR for page 61
Principal-Investigator-Led Missions in the Space Sciences Proposing Team Experience and Leadership Sufficient numbers of PI-led missions have now been through the implementation phase to have produced a pool of scientists, engineers, and managers with the unique perspective of experience. Prospective reviewers who are no longer actively competing for PI missions would be especially capable of recognizing potential difficulties in a proposing team’s approach. Based on the committee’s interviews, the importance of the team leader’s experience and team “chemistry” to the success of PI-led missions cannot be overstated. For example, inexperienced PIs and/or PMs from different institutions who have not worked together on a technical flight project are more likely to experience institutional and personnel issues12 that interfere with project implementation than those who have previous experience with missions or technical projects. Teams that include instrument providers from different institutions with various administrative and management styles could face communication problems or contractual difficulties unless they have experience working with a variety of external team members. PMs need to work with their PIs to propose the implementation plan for their own missions. The committee believes that the selection process does not sufficiently weigh the teaming aspect in proposal evaluations. As it stands, most reviewers generally do not have firsthand knowledge of the proposers and have not themselves participated in a PI-led project. Use of noncompeting members of successful PI-led mission teams as reviewers could provide insight on this sometimes subtle matter of project internal organization during the selection process. While the basics of mission implementation success are common to both core and PI-led missions, the cost cap issue must figure prominently into every decision or change made in a PI-led project. PIs and PMs reported to the committee that the effectiveness of PIs and their teams is dependent on personality, background, and institutional culture. In particular, prior experience in space hardware development provides the opportunity for mutual understanding to grow between scientists, engineers, and managers as well as an appreciation of the realities of flight projects, including the persistence and time commitment needed on the part of the PI and PM. Several previous PI-led space science missions, especially in the Explorer line, had the advantage of PIs and PMs with significant experience in either orbital or suborbital hardware development, usually for scientific instrumentation. Successful PIs who lacked space hardware construction experience had often acquired management experience from both space missions and ground-based projects and realized the necessity of identifying a capable PM who could manage the technical implementation. However, there is concern that opportunities to gain experience, both for PIs and their engineering and management teams, are dwindling. For example, the sounding rocket program has dramatically diminished since many of the current PIs gained their experience (see Figure 7.1), and small satellite programs such as the University Explorer (UNEX) program have not been carried forward, primarily because of the limited availability of small launchers in the United States. The balloon program serves only a limited set of subdisciplines, primarily Earth-Sun System and astrophysics. The planetary science discipline has traditionally focused on instrumentation designed for ground-based, rather than space-based, observatories. Thus, opportunities for training future PIs and PMs need to be acknowledged as an essential part of the PI-led mission programs themselves. Deputyships and apprenticeships have been mentioned in some but not all AOs of PI-led programs. PI-led missions provide a unique opportunity to gain experience on the entire mission development process, from the science concept to instrument development, to spacecraft design, to mission operations, and—finally—to data analysis. Involvement in this end-to-end development process bolsters the cadre of experienced people who will become future PIs and PMs. Thus, these missions play a key role in training leaders for future spaceflight programs. However, prospective PIs and PMs also need to aggressively seek experi- 12 Institutional and personnel issues include, for example, contractual language, documentation practices, style of management, and treatment of personnel.
OCR for page 62
Principal-Investigator-Led Missions in the Space Sciences FIGURE 7.1 NASA sounding rocket launches, 1999-2003. SOURCES: Available at <www.univ.perp.fr/fuseurop/a/nasa.htm> and <www.wff.nasa.gov/news/index_news.php>. ence relevant to PI-led missions in suborbital, core mission, and/or technology development programs to make themselves competitive. Recommendation 4. NASA should develop PI/PM teams whose combined experience and personal commitment to the proposed implementation plan can be evaluated. NASA should also provide opportunities for scientists and engineers to gain practical spaceflight experience before they become involved in PI-led or core NASA missions. These opportunities could become available as a result of revitalizing some smaller flight programs, such as the sounding rocket and University-class Explorer programs. Technology Readiness PI-led mission AOs encourage the infusion of advanced technologies but not necessarily their development.13 Several experts reported that project technology development efforts often lag planned progress 13 NASA AO AO03-OSS-02 for Explorer Program SMEX and MoOs states as follows: “Instructions for the advanced technology component of the proposal are contained in Appendix B. A detailed advanced technology infusion and transfer implementation plan will be developed by each selected investigation as part of its Phase A concept study.”
OCR for page 63
Principal-Investigator-Led Missions in the Space Sciences owing to unexpected design failures, fabrication or testing issues, or other glitches. One approach to mitigating this risk is to follow several parallel technology development paths. In two cases (RHESSI and THEMIS), the project teams funded alternative or backup subsystem developments for their projects and ended up using them.14 The committee found that attempts by mission projects to use promising but immature technology is a frequent cause of PI-led missions (and others) exceeding the cost cap.15 For example, GALEX team members reported that several technological challenges had affected the mission schedule.16 Schedule erosion is either rationalized or unrecognized by the project management until it is too late to recover without a significant cost impact. In 2004, the crosscutting technology program at NASA was moved to the Exploration Systems Mission Directorate and its focus was changed to exploration systems technology. This program had previously resided in the Aerospace Technology Directorate and, before that, in the Science Mission Directorate. The individual missions are currently responsible for funding any focused or directed technology that they require. The existence of Category 3 selections (projects that exhibit compelling science but are not technologically mature enough for selection), combined with evidence of cost overruns due to technology development factors (see Chapter 5), indicates a need for program-specific technology development opportunities. The committee viewed Category 3 selections as awkward: They are not mission selections, nor are they selections for an explicitly competed technology development program. The committee believes there should be a cleaner division between mission competitions and selections and technology development competitions and selections. It would be highly desirable, in the committee’s view, for all major new technology development efforts for PI-led mission programs to be undertaken outside of project implementation, because such development could have potentially significant adverse impacts on mission costs and schedules. In the recent past, both the Explorer and Discovery programs provided the opportunity for selection as Category 3 projects, which received funding to advance their necessary new technology to a level adequate for future PI-led mission selection. However, because they were selected as a discretionary option within the mission competition, they were not openly competed as technology development efforts. The programs could include, or be tightly coupled to, substantial advertised technology development opportunities that fund promising new technologies not yet incorporated into a particular mission. A proposer should not need to write an entire mission proposal to advance the technological readiness level of a mission concept. One effective way to assure that major new technology developments apply to PI-led mission program lines is to have the PI program sponsor the technology development. For example, the unique new technologies required by the PI-led planetary science missions might be best sponsored by the Discovery/New Frontiers Program Office. As a case in point, Kepler required development of a photometer, which was possible only because the PI was a civil servant, had time to work on the project, and was able to obtain funding from a NASA center director’s discretionary fund. Of course there are also efforts external to the PI-led program that can be taken advantage of as well—for example, the Mars Scout Program benefits from having access to technologies developed for the Mars Exploration Program. If each PI program regularly sponsored new technology opportunities (for example, as program elements in the AOs), the technology pipeline for PI-led mission programs would be much better assured. 14 For example, RHESSI developed other versions of its roll angle sensor and 20-micron grids and needed to use them. THEMIS used its backup detectors and is developing other backup systems. 15 NRC, 2000, Assessment of Mission Size Trade-offs for NASA’s Earth and Space Science Missions, Washington, D.C.: National Academy Press, pp. 27-28. 16 On GALEX, for example, the aspheric beamsplitter and its unique multilayer coating involved unanticipated technical hurdles, and the detectors involved novel technology that presented special difficulties.
OCR for page 64
Principal-Investigator-Led Missions in the Space Sciences Proposals for multimission advanced technology developments could be solicited through a more general NASA research announcement (NRA), but with PI-led and other current mission PIs and PMs included in the selection process. Proposals for both the breadboard and brassboard (closer to flight-ready design) levels of development could be solicited. (Interviewees identified brassboard-level technologies as a needed but missing component of the current AO opportunities.) The areas of technology focus for a particular program or NRA solicitation could be identified by a panel of scientists and engineers, including those involved in PI-led missions and science mission prioritization studies. The results of the competed technology developments could be made available to the proposing community via the PI-led program library, which is accessible on the Web.17 A technology program such as the one described above was successful within the Explorer Program in the late 1990s but was discontinued when NASA Headquarters cut technology funding (~$5 million per year) from the Explorer budget. Recommendation 5. NASA should set aside meaningful levels of regular funding in PI-led programs to sponsor relevant, competed technology development efforts. The results from these program-oriented activities should be made openly available on the program library Web site and in articles published in journals or on the World Wide Web. Funding Profiles Funding profiles are the estimated schedules for spending funds over the course of a project’s design, development, and operations. In the past, the Discovery and New Frontiers programs predefined their funding profiles. In so doing, the program dictates the progression of mission development and implementation based on the funds that it is expected to have rather than the best schedule for the mission. The committee agreed with the concerns of some interviewees about the imposition of funding profiles in the Discovery and New Frontiers AOs. PI-led missions, particularly planetary missions, are already constrained in many dimensions, including launch timing, management practices, and total costs. Several of those interviewed mentioned that predefined funding profiles could contribute to cost and schedule overruns and increased risks when funding that should be spent early in the project is deferred to fit the required profile. The Discovery and New Frontiers programs could consider the strategy used in the Explorer Program, which does not impose funding profiles in its AOs.18 That is, proposers are free to let funding profiles be dictated by the nature of the mission, and the profiles are considered by program officials as part of the selection. Recommendation 6. NASA and individual mission PIs should mutually agree on a funding profile that will support mission development and execution as efficiently as possible. If NASA must later deviate from that profile, the mission cost cap should be adjusted upward to cover the cost of the inefficiency that results from the change in funding profile (see Recommendation 10). International Contributions During the development of this report the committee often heard from interviewees that international collaborations are viewed by NASA as introducing significant difficulties and risks into mission projects. 17 The Web site of the Discovery Program library can be accessed at <centauri.larc.nasa.gov/discovery/DPL>. 18 The Explorer Program Office has established nominal funding profiles based on the actual spending profiles of prior missions, but these are used only to define a realistic AO release schedule that does not artificially constrain the proposers.
OCR for page 65
Principal-Investigator-Led Missions in the Space Sciences NASA now requires a firm commitment of funding by an international team member at the time of proposal submission, a requirement that does not usually suit the funding allocation schedules of the would-be team member. Further, because NASA AOs and times of selection are not known well in advance, this requirement means the foreign team member’s government must commit funds for an indeterminate length of time for an opportunity that has only a small probability of being selected (about 5 to 10 percent) (see Chapter 3). The loss of international contributions to PI-led missions would have major science impact—for example, Dawn’s camera to image Ceres and Vesta is a German contribution and RHESSI’s hard x-ray telescope is provided by Switzerland. In addition to the funding commitment schedule, there are three other serious obstacles to international collaboration: (1) the constrained access of foreign-born students and faculty in the United States to technical information about the missions; (2) restrictions limiting the participation of foreign-born students, engineers, and scientists; and (3) the difficulty of procuring hardware from non-U.S. vendors. The first problem is by far the most serious. In a university environment where equal access to information by all members of the academic community is the rule, it is difficult to develop, test, and calibrate an instrument that is considered to be on the munitions list and thus ITAR controlled.19 The ITAR rules involved in implementing a mission are confusing and poorly defined. Moreover, because ITAR does not provide clearly defined procedures for the space research community, it is easy to either ignore the rules or overapply them. Because science instruments are valuable only to the extent that their calibration is well understood, appropriate calibration facilities and personnel to operate them must be available. Such university test facilities often involve students, engineers, and technicians who are foreign born, and ITAR is unclear on how the rules apply. Even in cases where ITAR licenses can be obtained, the time required to acquire the licenses and document all transactions, and the difficulty and delicacy of communications dealing with problems in technical development, can seriously increase cost and schedule delays for PI-led mission projects. The committee was not tasked with providing a critical analysis of issues arising from ITAR that are imposed by agencies and policy makers outside NASA. As noted in the Space Studies Board report The Sun to the Earth—and Beyond: A Decadal Research Strategy in Solar and Space Physics,20 NASA should consider how ITAR-related mandates impact all missions with an international component. In the highly constrained PI-led programs, the additional burdens due to ITAR can be magnified. Moreover, the committee believes that the value of respecting and maintaining U.S. international teaming relationships in space science missions, including PI-led missions, and the value of including universities and their students in these missions should not be underestimated. The PI-led programs themselves can help in this regard by communicating and documenting the best approaches and practices gleaned from their international teaming experience as part of their lessons learned. Unfortunately, in the current environment, international collaborations, in spite of their demonstrated benefits, will likely remain a perceived risk for PI-led missions. Recommendation 7. NASA PI-led-mission program officials should use recent experiences with ITAR to clarify for proposers (in the AO) and for selected projects (e.g., in guidance on writing technical assistance agreements and transferal letters21) the appropriate application of ITAR rules and regulations. 19 In 1999, space science satellites became part of the U.S. munitions control list (USML) and therefore subject to ITAR, which is administered by the U.S. Department of State. ITAR includes regulations that control the export of USML items, and this jurisdiction “authorizes the President to control the export and import of defense articles and defense services” that are designated on the USML (ITAR, Sections 120.9 and 120.1). As part of its data-gathering effort, the committee solicited input from PI and PM interviewees on the impact of ITAR on PI-led missions. 20 NRC, 2003, The Sun to the Earth—and Beyond: A Decadal Research Strategy in Solar and Space Physics, Washington, D.C.: The National Academies Press, pp. 159-161. 21 Transferal letters are documents that describe relationships between NASA and foreign institutions or funding agencies, for example.
OCR for page 66
Principal-Investigator-Led Missions in the Space Sciences PROGRAM MANAGEMENT Program and project management are at the heart of the many concerns about PI-led missions. A mission failure, whether technical or programmatic, can often be traced to a management failure. Likewise, successes often stem from good management. Most of the comments the committee received from the PI-led program community revolved around management issues. Even though the AOs solicit innovative management plans, NASA’s program management often imposes itself on PI-led projects. Most PI-led programs share a common overall structure: a headquarters arm that is responsible for policy and the selection of missions and a program office that is responsible for the final formulation and implementation of missions. The details of program office structure and location are unique to the particular PI-led program (see Chapter 2). The following section presents the committee’s findings on organizations and practices within the space science PI-led mission programs. Role of the Program Office The inputs received by the committee from PIs and the space science community22 emphasized the need for a well-run and stable program office that enables project implementation and the benefits of such an office. In particular, several PIs, PMs, and other interviewees said the Explorer Program Office was a significant factor in the success of PI-led missions under its aegis. The committee examined the program management approach for the various PI-led mission lines to learn which factors affected their ability to provide the needed support for their projects and for NASA Headquarters. Based on the information provided to it and as discussed in Chapter 4, the committee finds that in the past, instability in the Discovery Program Office (DPO) led to confusion in the projects about the lines of authority. Multiple communication paths between Headquarters and the DPO sometimes resulted in conflicting direction to PI-led projects. Policy changes were not communicated until problems became apparent during status reviews. Discovery projects still succeeded, although DPO instability made it more difficult to manage Discovery projects efficiently and effectively. It remains to be seen if the new Discovery/New Frontiers Program Office at MSFC will have the authority, adequate staffing, MSFC support, and unambiguous lines of communication between Headquarters and the projects needed to manage the extensive list of PI-led missions for which it is responsible. The Discovery/New Frontiers Program Office appointees confirmed that they have been studying the Explorer Program Office as they work to set up their operation. The committee heard no specific comments on the operation of the Mars Scout Program Office at JPL, but that office is still managing its first mission. Recommendation 8. NASA should ensure stability at its program offices, while providing sufficient personnel and authority to enable their effectiveness, both in supporting their missions and in reporting to and planning with Headquarters. Program Oversight Practices NASA acknowledges that AOs evolve and adapt to prevailing budgetary, programmatic, and agency climates. NASA Headquarters continuously updates and clarifies requirements in each successive AO release based on agency goals and on issues arising in the past and in ongoing projects. PI-led missions 22 Inputs for this study from the space science community were elicited by notices published in newsletters and by questions sent directly to PIs.
OCR for page 67
Principal-Investigator-Led Missions in the Space Sciences typically take 5 or more years from the AO release date to complete their formulation and implementation phases. Many PIs and PMs cited problems with the increased oversight that was imposed on missions under development during the late 1990s, when NASA activities as a whole were being reconsidered and their style revised to comply with faster-better-cheaper directives.23 PIs and PMs reported to the committee that the primary manifestation of this new NASA oversight approach was the additional layer of reviews (on top of those that were originally planned) by the program offices or NASA Headquarters.24 For example, the RHESSI and GALEX Explorer projects initially were allowed to establish their own system review teams. During implementation, however, these review teams were replaced by NASA-led, independent review teams. The usefulness of a given review depends on many factors, including, for example, the reviewers, timing, level of detail, and level of formality. A majority of the project PIs and PMs interviewed by the committee said that many of the additional reviews directed by the program offices, especially the formal reviews, were more disruptive than beneficial. The staff time needed to prepare, present, and close a formal review is considerable and takes away from the primary bench-level engineering and management tasks. Given the tight budgets and schedules involved in PI-led missions, nonproductive and/or excessive reviews can hinder mission success more than they help NASA oversight. In at least one case, more engineers had to be hired because the senior engineers were too busy supporting reviews. PIs and PMs who spoke to the committee suggested that despite being impractical, inappropriate, or unaffordable, a number of the requests for action (RFAs) submitted by the formal review panels still demand a careful, serious, and time-consuming response. Review panels also do not always seem to be aware of the style of mission they are reviewing, especially its constraints. At present, AOs do not address those Headquarters or program requirements that may be introduced after selection. Changes of scope are not uncommon, and there are standard procedures for addressing such changes. The PI-led mission PI and PM should not be held accountable when their authority is superseded and new oversight is imposed. In center-led core missions, changes of scope are typically assessed for their impact, and appropriate actions in response to the change are negotiated between the various team members in the project, the center(s), and NASA Headquarters. NASA routinely adjusts the budget of rescoped missions, usually on an annual cycle. PI-led missions are programmatically limited in their ability to address changes in scope because there is no such yearly adjustment to the cost cap. Inexperienced PIs may try to absorb the budgetary impacts of such reviews within their project budgets, sometimes unsuccessfully, to preserve their good performance record. The above oversight changes have collectively eroded PI authority. PIs who led missions within the last 3 years say that the authority of PIs to make decisions about their missions has been seriously challenged by the new NASA oversight requirements and constraints. This shift has occurred because NASA has become more risk averse, even for PI-led missions, whose AOs solicit management innovation and initiative. For example, PIs need to be able to manage their own cost and schedule margins and reserves and to determine any modifications, such as descopes,25 to science instruments or spacecraft capabilities as long 23 See NASA, Mars Program Independent Assessment Team, Summary Report, March 14, 2000; Mars Climate Orbiter Mishap Investigation Board Phase I Report, November 10, 1999, available at <ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf>. Also, on faster-better-cheaper, see “A view, a vision, an imperative,” speech by Daniel S. Goldin, NASA administrator, American Geophysical Union (AGU) meeting, NASA Goddard Space Flight Center, December 16, 1998; “A challenge to change: NASA’s nonlinear path to the future,” speech by Daniel S. Goldin, NASA administrator, American Astronautical Society, Goddard Memorial Symposium, Arlington, Va., March 10, 1993. 24 Reviews can be useful tools for evaluating project performance and are described in NPR 7120.5B, now updated as NPR 7120.5C, a NASA document that provides a guide for PI-led mission managers. 25 NASA requires that PIs include a descope plan when proposing for PI-led missions. This plan outlines the systems, instruments, or spacecraft components, mission operations, and schedule elements that will be eliminated from the mission should the mission face cost or schedule problems. Any eliminations are planned with the goal of preserving the minimum acceptable science for the mission.
OCR for page 68
Principal-Investigator-Led Missions in the Space Sciences as minimum science goals and cost and schedule constraints are met. NASA oversight has also increased as the cost caps and complexities of the PI-led missions grow (as, for example, in the New Frontiers line). NASA needs to consider where the practical limits of the PI-led approach lie in mission implementation so that missions do not become PI-led in name only. It may be appropriate to reconsider guidelines for the PI-led mission lines in terms of levels of PI authority and control, which might depend, for example, on the cost and/or complexity of the mission. For example, a mission with many instruments, a complicated spacecraft design, unusual environmental constraints, or a long cruise will likely benefit more from NASA oversight than a simple Earth orbiter with one or two sensors. A one-size-fits-all oversight philosophy is wasteful of time and effort. Other recent important changes are the program requirements for given levels of unencumbered budgetary reserves in the cost estimates for PI-led projects. Costs are estimated at a very early development stage (pre-Phase A in most cases) using the level of reserves required for the project specified in the AOs. These estimates are then used as the baseline for the project’s cost cap. The TMCO reviewers of the Phase A study or the concept study may recommend to the program what they consider is an adequate reserve for the mission, but the selection cost cap usually stands. Given the number of changes that can occur during the formulation and implementation of a space mission, not only within the project itself but also in the external environment, requiring a fairly high cost reserve level at the start of Phase B should help reduce the chance that a given project cannot meet its cost cap. However, the cost reserve is not a change that can be easily arranged within an ongoing project without major impacts. The requirement to increase reserves during implementation has occurred in at least one Discovery project, Dawn. A change in reserves constitutes a major change of scope for which all original constraints on the project (cost, schedule, technical content) need to be adjusted so that the change does not impair a mission’s performance. In addition, the pressures to increase reserves under a given cost cap naturally put pressure on the science content of the missions and may exacerbate the tendency to include overly optimistic cost estimates in mission proposals. The majority of past PI-led projects used NASA-provided facilities such as launch vehicle services, navigation, and tracking and data distribution services. These services need to be budgeted based on the information available at the time of the AO release. The PI of a selected mission has no control over or knowledge of how the cost of these services might increase over time (assuming the same level of service originally specified in the proposal). In the mid- to late 1990s, in particular as NASA began full-cost accounting, the cost of these services (e.g., for NASA personnel, for facilities use, and for launch vehicles) changed significantly. (See Figures 5.1 and 5.2 for examples of increases in launch vehicles costs and inflation.) In some early cases, cost increases were expected to be absorbed by the projects. More recently, NASA-provided services such as launch vehicles and DSN use have been removed from project cost caps. This practice needs to be standardized and explicitly described in future AO releases for potential users of these services. Moreover, the full-cost accounting of center personnel and other center contributions continues to frustrate PI teams’ ability to predict, track, and report costs. PI-led mission programs need to assist their projects in the cost management task by ensuring that PIs and PMs have needed insight and cost data from their NASA center team members. Currently, several practices in place partly address the need for adjusting the cost cap when doing so can be justified. These include the configuration change request (CCR) and the Program Operating Plan (POP) processes. A CCR can be requested by the project and may be forwarded by the program to Headquarters. However, when it receives a CCR for a funding increase beyond the cost cap, Headquarters may decide to call a termination review. It is not apparent that the CCR process can be used for changes imposed by NASA for added oversight or other requirements. POPs constitute NASA Headquarters’ formal acknowledgment and approval of changes that have been requested. PIs and PMs, and the programs
OCR for page 69
Principal-Investigator-Led Missions in the Space Sciences themselves, are not generally clear on when and if the CCR and POP processes can be used to address externally driven changes in scope and, hence, costs. It is important to consider the constraints on PI-led missions when instituting new or additional mission requirements, such as those that may be imposed by the newly established Independent Technical Authority (ITA). Appropriate actions on the part of the PI-led projects need to be discussed within the programs and projects as soon as possible to limit schedule disruption and risks. Recommendation 9. NASA should resist increasing PI-led mission technical and oversight requirements—as, for example, on quality assurance, documentation, ITA-imposed requirements, or the use of independent reviews—to the level of requirements for larger core missions and should select missions whose risks are well understood and that have plans for adequate and effective testing. Recommendation 10. NASA should clarify the change-of-scope procedures available for projects to negotiate the cost and schedule impacts of any changes in requirements initiated by NASA Headquarters or a PI-led program office, including the addition of reviews, documentation, reporting, and/or increased standards. The schedule impact of negotiating changes of scope should also be evaluated. Threat of Cancellation NASA guidelines regarding PI-led missions dictate that a termination review may be convened whenever a mission is projected to exceed its agreed-to cost cap or is found to be performing below the agreed-to science performance floor.26 According to NASA officials who were interviewed, before calling a termination review, consideration is given to the causes of the cost growth to rule out external factors. A termination review is generally regarded as a serious matter, not only because the fate of a mission hangs in the balance but also because the need to reallocate resources to save the mission under review—if this is the decision—will likely impact the program as a whole. In particular, cost and schedule overruns in one mission can have negative cost and schedule consequences for all others in the queue. Ideally, the primary goal of a termination review is to save the mission from cancellation if it can be saved by the application of reasonable and available resources. To achieve this goal, the NASA Headquarters’ Science Mission Directorate PMC conducts a careful assessment of the problem(s) that triggered the termination review. If required for an accurate assessment of the situation, this review should include informed parties (engineers, scientists, etc.) who are not affiliated with the PI team or the program office. The objective of the review is twofold: first, to determine the cause(s) of the problem(s); second, to ascertain what steps and associated costs are needed to correct the problem(s). If corrective action is deemed appropriate, the action is tailored to the objective of getting the mission to launch at minimum additional cost or schedule slip. Actions currently range from providing more funds to the mission, to the application of NASA center expertise or facilities, to the removal of the PM if it is determined that costly errors are traceable to mission-threatening technical management decisions. The committee’s interviews revealed that because a project does not always view a termination review as mission-threatening, such reviews can lose their impact. There is a tendency to assume that sufficient prior investment will save a mission regardless of its problems. However, PI-led missions have suffered forced descopes or rescopes in termination reviews. PI-led missions are especially vulnerable: Because they are competed, the cost cap is regarded as a harder ceiling than it is for core missions, which can grow 15 percent. For these reasons, termination reviews are avoided whenever possible. Toward this end, NASA 26 Paul Hertz, Science Mission Directorate, NASA Headquarters.
OCR for page 70
Principal-Investigator-Led Missions in the Space Sciences could improve the collection and dissemination of lessons learned from previous termination reviews. PIs and PMs who are educated about the problems that led to termination reviews for previous missions will, presumably, be more attentive to the early onset of similar problems on their own mission and thus be more likely to address the problems before they grow. A related concern of the committee is that decisions such as science instrument descopes have been made outside the termination review process without the PI’s agreement. It is the committee’s opinion that due process needs to be followed before such a high-impact decision is made. Recommendation 11. NASA should continue to use the existing termination review process to decide the fate of PI-led missions that exceed their cost cap. It should develop lessons learned from termination reviews and make them available to other PI-led projects. Recommendation 12. NASA should not descope mission capabilities (including science instruments) without the PI’s agreement or outside the termination review process. ROLE OF PROJECT MANAGEMENT Technical and Management Failures The technical failures described in Chapter 6 can largely be attributed to inadequate or incomplete design and testing. Apart from CONTOUR, where the fault most likely resulted from inadequate analysis of the physics of radiative and conductive coupling of the rocket plume to the body of the spacecraft, the problems entailed errors at such a deep level of engineering detail that the relevant PI and PM experience and engagement concerned mainly decisions about the adequacy of peer review and testing. Since similar problems have caused the loss of core missions, it is not clear that the fact that they were PI-led—with the potential for not enough funds and/or time for carrying out additional reviews, analyses, and tests—contributed to these failures. That earlier lessons may not have been learned by PI-led projects is a particular concern: It is the only thing suggesting that lead personnel in center-led missions, who are selected based on their experience and stage of professional advancement, might be more aware of NASA failure histories. A few PIs reported that they would have benefited from a lessons-learned database but did not have one or were not aware of one. However, the committee found many lessons-learned documents on the Internet (see Appendix G). Also, many PI-led mission PMs are located at centers where they benefit from the same institutional culture and training opportunities. While lessons learned cannot possibly cover all topics or needs, or substitute for experience, modest efforts to document key points of potentially useful information is a service that each generation of PIs and PMs from all institutions can perform for the next. It would be prudent of every PI, PM, and lead NASA center to review lessons learned at all phases of their mission, including the proposal phase and especially Phase A, or to at least be aware of their contents. Program offices should see that links to such materials are provided in their online program libraries27 and AOs and that selected lessons learned are discussed at preproposal conferences and program annual retreats involving the PIs, PMs, and prospective proposers. 27 The Discovery, Explorer, Mars Scout, and New Frontiers programs’ Web sites include program libraries that offer resources for PIs, including guidelines and requirements documents and NASA mission strategies and policies. See <explorer.larc.nasa.gov/explorer/mel.html>, <discovery.larc.nasa.gov/discovery/dpl.html>, <explorer.larc.nasa.gov/explorer/sel.html>, <centauri.larc.nasa.gov/ma
OCR for page 71
Principal-Investigator-Led Missions in the Space Sciences The PI-led mission series has had an implicit philosophy that it is acceptable to incur a somewhat higher level of risk than can be tolerated by the large center-led programs.28 However, because NASA had to deal with a variety of highly visible failures since the mid-1990s and with the advent of the larger New Frontiers program, the extent of tolerable risk remains poorly defined and is diminishing, if not vanishing. The balancing of risk versus cost remains a delicate art; it is not clear that the risk management tools available would reduce mission risk or help to quantify it. The committee believes that the thorough testing of what is to be flown on the spacecraft and flying only those systems that have been tested, a practice known as “test as you fly,” is the best answer. Alternatively, robust simulation tools are needed for systems and functions that are impractical to test, such as propulsive maneuvers and entry-descent-landing sequences. Informed risk-taking may be an appropriate compromise, in which case judicious use of focused peer reviews and testing are given the highest priority by both the program and the project. The selection process outlined in Chapter 2 suggests that programmatic failures are often guaranteed at selection by the choice of an overly ambitious and/or technologically underdeveloped concept. Proposers and/or programs may view the PI-led mission line as the only way to get an important mission flown, and NASA Headquarters may therefore accept greater risk than they otherwise might in selecting a mission. In the competition, such missions can appear the most exciting from a science, technology, or public interest perspective. If the selection panels identify underdeveloped critical technologies and/or instruments and in spite of this information NASA selects the mission, the project’s potential for failure in the form of cost and schedule overruns should not be surprising. On the other hand, overly conservative selections would negate the purpose of the PI-led program. Thus there is some intermediate ground in which common sense and science goals together dictate selection. Selection panels and selecting officials need to strive to identify that territory. Recommendation 13. NASA PI-led program officials and PI-led mission teams should study lessons-learned documentation to benefit from the experiences of previous PI-led missions. NASA should make such lessons learned easily and widely available and update them continuously, as is done on the Discovery Program Web site posted by Langley Research Center. Team Interactions Nearly every interview or written contribution of PIs and PMs to the committee emphasized that the ability of the team members, especially the PI and the PM, to work together is critical to success. Thus, the practice of forming a team, especially team leadership, out of individuals who have never worked together can pose significant risks. Trust, as well as the ability to communicate openly and effectively, is essential. While maturity and experience are important advantages in team relationships, differences in priorities, styles, and institutional and professional backgrounds can interfere with the smooth functioning of a team that has no time for interpersonal conflict. The projects need to recognize when their personnel, even (or especially) at the top levels, do not work well together and move to resolve passing disagreements and to change team members or their responsibilities when necessary. The program offices need to be supportive when a change at the PM, major contractor, or contributor level is involved. Being supportive may include participating in negotiations and enabling personnel replacements. 28 Daniel S. Goldin, NASA administrator, “A view, a vision, an imperative,” speech to the American Geophysical Union meeting on December 16, 1998; Daniel S. Goldin, NASA administrator, “A challenge to change: NASA’s nonlinear path to the future,” speech to the American Astronautical Society on March 10, 1993.
OCR for page 72
Principal-Investigator-Led Missions in the Space Sciences PIs who have limited experience with flight project management could benefit from aggressively educating themselves and surrounding themselves with supporting team members of the highest caliber. Choosing a high-caliber, experienced PM who is cognizant of technical personnel and providers and who can ensure the provision of systems engineering experience on the project often ensures a strong and balanced technical team. The PI must have confidence in the ability of his/her PM. If the PI is him-/herself technically expert, the PI should consider choosing a PM with whom he/she can work well and who accepts the PI’s participation in the technical leadership. The committee learned through its interviews with PIs and PMs that in many cases, PMs have not worked with their PIs before, and even in situations where the pairing is not a first-time one, they have generally not worked together on a project as large as an entire mission. The committee found that PMs who are used to working with a variety of other people, including scientists, are most likely to work well with team members in the course of managing the mission. At the same time, PMs should be prepared to accept the PI as the ultimate decision maker for the mission. The undermining of a PI’s leadership is especially hard to avoid when interactions with contractors occur without the PI present, a circumstance that may be more likely if the PI and PM are not collocated. The same issues hold for lead centers, which must make day-to-day project decisions in order to move forward, even when the PI is not present or available for consultation. Sensitivity to this potential source of problems needs to be encouraged at all levels, from PIs to PMs to participating center directors. Much of the solution lies in good communication, but a clear, contractual statement of PI authority would avoid ambiguity in the assignment of specific authority. Recommendation 14. NASA and the PIs should include language in their contracts that acknowledges the PI’s authority to make the final decisions on key project personnel. Cost, Schedule, and Science Performance To understand and analyze the reasons behind cost and schedule changes for PI-led missions, it is essential that the details of cost increases or decreases be documented in a consistent manner. The committee was unable to analyze these details either because there were no documents at NASA or because of difficulty in obtaining financial records that clearly explain any changes. Financial records on PI-led projects are not consistent or easily obtained. The cost details contained in this report were gathered from various documents, including PDRs, CDRs, and POPs. Detailed information at the spacecraft sub-system level was not available. According to the committee’s investigations, the science return from PI-led missions, though difficult to quantify, appears to be at least comparable in influence to the return from core missions. PI-led projects invest, on average, roughly 10 percent of their cost caps in Phase E (see Table 5.4), which includes MO&DA. The committee was not able to obtain information on the distribution of these Phase E funds between MO (nonscience) and DA (science) commitments. However, recent PI-led missions all seem to support either internally funded science programs (funded within cost caps) or other externally funded science opportunities (such as NASA supporting research and technology, guest investigator, or data analysis programs)—a practice that usually has a significant positive impact on the overall level of science output, given the small size of PI-led mission science teams. The main role of the PI in preserving the proposed mission science seems to occur in the mission implementation phases, when PIs have sometimes successfully fought to avoid undesirable descopes. On the other hand, this often involves transferring project funds from Phase E to Phase C-D, which has an undesirable impact on realizing the mission goals within the cost cap. There is a danger that Phase E funds too often become the spare margin for technical
OCR for page 73
Principal-Investigator-Led Missions in the Space Sciences implementation cost overruns. The result is that other programs or PI-led program elements, such as DAPs, must sometimes pick up Phase E data analysis costs for the highly constrained PI-led missions. Recommendation 15. NASA should maintain and have available for assessment consistent and official documentation of project costs and reasons for cost growth on all PI-led (and other) missions. In considering all the recommendations presented, the committee recognizes that NASA is already at least partially implementing (or attempting to implement) some of the recommendations, such as moderating ITAR impacts on space science missions and considering enhanced Phase A’s (Recommendation 7). The Discovery/New Frontiers Program Office is currently undergoing changes, and the technology development issues for space science missions are under scrutiny. Nevertheless, the committee deems these issues sufficiently important to be emphasized here. As the committee completes this report, NASA Headquarters and its programs are undergoing significant changes in response to the Vision for Space Exploration.29,30 The Science Mission Directorate now consists of four subdivisions: Heliophysics, Planetary Science, Earth Science, and Astronomy and Physics. Earth Science has its own line of PI-led Explorers.31 The space science PI-led mission lines described in this report have the potential to address some of the high-priority science recommended in NRC decadal surveys. They also have the potential for application to the Vision for Space Exploration—particularly for missions related to the exploration of the Moon and Mars and for characterizing the solar-activity-related radiation environment. Subjects relevant to the Vision for Space Exploration that match or complement the objectives and/or instrument capabilities of desirable missions in the decadal surveys may be especially strategic targets for PI-led missions at this time in NASA’s history. The committee believes that its report provides some useful suggestions and recommendations that would help NASA administrators, agency program managers, centers, and the science community in the continuation and exploitation of this most grass-roots of NASA mission lines. 29 President’s Commission on Implementation of United States Space Exploration Policy, 2004, A Journey to Inspire, Innovate, and Discover, Washington, D.C. 30 Doug Cooke, deputy associate administrator, Exploration Systems Mission Directorate, NASA, “NASA’s exploration architecture,” presentation to the Space Studies Board on November 9, 2005. 31 NRC, 2004, Steps to Facilitate Principal-Investigator-Led Earth Science Missions, Washington, D.C.: The National Academies Press.
OCR for page 74
Principal-Investigator-Led Missions in the Space Sciences This page intentionally left blank.
Representative terms from entire chapter: