6
Lessons Learned from PI-Led Mission Experiences

Spacecraft missions are intrinsically complex: Success requires literally thousands or millions of things to go right. This chapter describes the types of events that can go wrong and the potential for either the cancellation or the loss of a spacecraft and/or its science goals and the potential for development to far exceed the proposed and accepted cost and schedule envelope set for the mission. In most failures the root cause can be traced to decisions made and/or events that took place, or should have taken place, before launch or even at selection. In most cases of technical failure, the failure was traced to a single item that was not noticed because the testing program was faulty or inadequate. In all cases, management decisions played a role, but the question remains whether those decisions were affected by the fact that the mission was led by a PI—for example, cost and/or schedule constraints associated with PI-led missions, divisions of authority and responsibility unique to such missions, and oversight practices for them.

TECHNICAL FAILURES

Although summary information provides a broad picture of the PI-led mission experience and outcome, specific examples help to put the issues into a real perspective. The examples described here are not the only cases of technical problems in the PI-led missions but were perhaps the most visible and documented.

Comet Nucleus Tour

The Comet Nucleus Tour (CONTOUR) was a Discovery mission with Joseph Veverka of Cornell University as PI.1 CONTOUR’s goals depended on its traveling to several comets to obtain comparative sets of data on their composition and appearance. The spacecraft was built by APL. CONTOUR was launched July 3, 2002, into an eccentric Earth orbit. On August 15, 2002, the integral solid rocket motor (SRM) was fired to leave Earth orbit. The anomaly causing the loss of the mission occurred in association with that event. The mission design did not provide for telemetry coverage of the SRM burn, nor was there optical

1  

Information in this section is based on the Mishap Investigation Board Report, May 31, 2003.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 52
Principal-Investigator-Led Missions in the Space Sciences 6 Lessons Learned from PI-Led Mission Experiences Spacecraft missions are intrinsically complex: Success requires literally thousands or millions of things to go right. This chapter describes the types of events that can go wrong and the potential for either the cancellation or the loss of a spacecraft and/or its science goals and the potential for development to far exceed the proposed and accepted cost and schedule envelope set for the mission. In most failures the root cause can be traced to decisions made and/or events that took place, or should have taken place, before launch or even at selection. In most cases of technical failure, the failure was traced to a single item that was not noticed because the testing program was faulty or inadequate. In all cases, management decisions played a role, but the question remains whether those decisions were affected by the fact that the mission was led by a PI—for example, cost and/or schedule constraints associated with PI-led missions, divisions of authority and responsibility unique to such missions, and oversight practices for them. TECHNICAL FAILURES Although summary information provides a broad picture of the PI-led mission experience and outcome, specific examples help to put the issues into a real perspective. The examples described here are not the only cases of technical problems in the PI-led missions but were perhaps the most visible and documented. Comet Nucleus Tour The Comet Nucleus Tour (CONTOUR) was a Discovery mission with Joseph Veverka of Cornell University as PI.1 CONTOUR’s goals depended on its traveling to several comets to obtain comparative sets of data on their composition and appearance. The spacecraft was built by APL. CONTOUR was launched July 3, 2002, into an eccentric Earth orbit. On August 15, 2002, the integral solid rocket motor (SRM) was fired to leave Earth orbit. The anomaly causing the loss of the mission occurred in association with that event. The mission design did not provide for telemetry coverage of the SRM burn, nor was there optical 1   Information in this section is based on the Mishap Investigation Board Report, May 31, 2003.

OCR for page 52
Principal-Investigator-Led Missions in the Space Sciences coverage of it. No telemetry was received from CONTOUR following the SRM burn. From August 16 to 21, large ground-based telescope images revealed three objects near the expected CONTOUR trajectory. A Department of Defense analysis supported the conclusion that the spacecraft had broken apart. The Mishap Investigation Board was unable to determine with certainty the cause of the failure because no data were available from the SRM burn but concluded that the probable cause was overheating of the spacecraft by the SRM exhaust plume. CONTOUR’s spacecraft design had the SRM embedded deeply in the structure. The root causes were identified as project reliance on analysis by similarity, an inadequate system engineering process, and an inadequate review process. The board listed a number of significant factors, including the lack of telemetry during a critical event (the board was unanimous that this was unacceptable unless absolutely unavoidable); significant reliance on subcontractors without adequate oversight, insight, and review; inadequate communications between the prime contractor (APL) and the SRM vendor; and limited understanding of SRM plume heating and operating conditions for CONTOUR. The board also noted a lack of rigor in the engineering process and documentation and an inadequate level of detail in the technical reviews. In addition, the board made recommendations on the standards for reviews, including their depth and timing. It also recommended against relying heavily on previous analysis unless it has been shown to be applicable and appropriate to the current use and recommended recognizing and addressing the limits of critical expertise and experience within a project team. Genesis Genesis, a solar wind sample return mission, launched August 8, 2001. The PI is Donald Burnett of Caltech. The mission’s main problem occurred during the scheduled aircraft “snatch” of the sample return capsule with its parachute on September 8, 2004. The parachute failed to deploy on reentry, and the capsule crashed in the Utah desert. In a preliminary report,2 the Mishap Investigation Board identified a probable cause of the mishap as incorrect orientation—it had been installed upside down—of the “gravity switch,” which should have triggered parachute deployment during deceleration in Earth’s atmosphere. “This single cause has not yet been fully confirmed, nor has it been determined whether it is the only problem within the Genesis system,” said the board chair.3 The board is working to confirm this proximate cause and to determine why it happened and why it was not caught by the test program. TERRIERS The Tomographic Experiment using Radiative Recombinative Ionospheric EUV and Radio Sources (TERRIERS) mission was led by Daniel Cotton of Boston University as part of the Student Explorer Demonstration Initiative (STEDI) administered by the University Space Research Association (USRA) for NASA. (STEDI missions were the predecessors of the UNEX line of Explorers.) TERRIERS launched successfully from Vandenberg Air Force Base on May 18, 1999. It achieved a nearly perfect orbit (550 × 530 km and an inclination of about 98 degrees). TERRIERS operations began with the first contact of the spacecraft around 9 hours after launch. Then, after the first three contacts it became clear that the spacecraft’s attitude control system was not functioning properly, since the solar array was pointing away from the Sun and the system was nutating. Sometime shortly thereafter the spacecraft ran out of battery power. Within a week or so the list of possible causes was 2   Genesis Mishap Investigation Board, 2004, Interim Report, October. 3   Amir Alexander, 2005, “Investigation uncovers likely cause of Genesis mishap: Stardust team confident of safe return.” Available at <www.planetary.org/news/2004/genesis_stardust_1015.October15,2005.html>.

OCR for page 52
Principal-Investigator-Led Missions in the Space Sciences pared down to one lone cause of failure: a sign flip in one of the three torque coil actuators. The fix for this problem was fairly simple given the flexibility of the flight software. Numerous attempts to contact the spacecraft continued through December 1999, with no positive results. Failures of Core Missions The technical failures of PI-led missions are similar to the failures of core missions. For example, there have been three failures in the Mars program during the era of PI-led missions. Two faster-better-cheaper (FBC) Mars missions failed, resulting in the end of that concept. For the Mars Polar Lander, hindsight revealed several potentially fatal problems; the most likely root cause was software responding to an erroneous early touchdown signal from the legs, causing engine shutoff. For the Mars Climate Orbiter, the root cause was the use of English rather than metric units in a navigation routine. Only Mars Observer, a pre-FBC core mission, appears to have had a specific hardware cause: probable catastrophic failure of the propulsion system during Mars orbit insertion. Mishap investigation boards for these core missions reported conclusions in line with those for the PI-led mission technical failures. A reader unaware of the difference in management approaches would, in the committee’s view, not be able to distinguish between the PI and core mission mishap reports.4 PROGRAMMATIC AND PROJECT MANAGEMENT FAILURES PI-led missions are selected for final formulation, development, and flight at the end of their concept (Phase A) study, which specifies a proposed total mission cost (below or equal to the program cost cap) that the PI agrees not to exceed, and that becomes the cost cap for the mission. The PI also commits to a schedule associated with the cost in the Phase A report. The key date in the schedule is the mission launch date. The launch date is allowed to shift during development as long as the cost cap is not exceeded. Of course, the planetary launch windows must be achieved. Normally, any significant slip in the launch date will also increase cost. Thus, adherence to the cost cap is the figure of merit for program management. When project and/or program managers project that a confirmed mission will exceed the cost cap, NASA considers the reasons for the overrun, any extenuating circumstances, and whether to hold a termination review. (A mission can be canceled before the confirmation review.) This criterion is the same as that for core missions. NASA recognizes that certain cost increases are beyond any reasonable control of the project—for example, launch vehicle cost increases and launch slips caused by conflicts with other missions. In such cases, a termination review may not be called and the project cost cap may be adjusted to cover the costs of the delay. If a termination review is called, it is conducted by the NASA Headquarters Program Management Council (PMC), using information provided by the project and the program. The PMC may solicit advice from NASA’s scientific advisory committee prior to its review. Information provided to the committee from its interviews with NASA personnel indicated that in termination reviews cost caps are taken more seriously for PI-led missions than for core missions owing to the competitive nature of the PI-led mission program. The decision to terminate rests with the associate administrator of the Science Mission Directorate. To date, only two PI-led missions, the Full Sky Astronometric Mapping Explorer (FAME) and the Cooperative Astrophysics and Technology Satellite (CATSAT) mission, were terminated because their projected costs unacceptably exceeded the cap. Similarly, the STEDI mission SPIDR was canceled for technical reasons and never confirmed. Inner Magnetosphere Explorer (IMEX) was canceled 4   NRC, 2000, Assessment of Mission Size Trade-offs for NASA’s Earth and Space Science Missions, Washington, D.C.: National Academy Press, pp. 27–30.

OCR for page 52
Principal-Investigator-Led Missions in the Space Sciences when it began to experience unsupportable cost growth. That growth stemmed, in part, from problems in securing a piggyback position on a U.S. Air Force launch and from increasing NASA requirements during Phase A development. IMEX was canceled after the Phase A (concept study) review and did not undergo formal confirmation review.5 (See Figure 3.1, which outlines the key milestones for PI-led missions.) Thus, although termination reviews typically apply to confirmed missions, a mission can be canceled prior to confirmation. Several other missions have been through one or more termination reviews, resulting in adjustments to the mission or cost cap rather than termination. However, one mission (Dawn) lost a key science instrument by an executive decision made outside the termination review process. Experience with spaceflight projects has shown that it is extremely rare that a mission, either PI-led or core, is completed within its estimated cost. According to data published by the Congressional Budget Office (CBO),6 overruns of several tens of percent are commonplace (Tables 5.1 and 5.2 show overruns for a subset of PI-led missions and core missions). The CBO data indicate that core missions have a higher frequency of cost overruns and that the overruns are greater, percentage-wise, than those of PI-led missions. The vagaries of space systems development—including parts failures, unexpected problems with new technology (usually instrumentation), component or system failures during environmental testing, incomplete or inaccurate cost estimations, and institutional overhead increases—mean significant cost risks for any spaceflight project. Even if prudent budget reserves are included to mitigate the cost risks, a 10 percent overrun is not unusual (see Chapter 5). To date, 13 PI-led missions succeeded in achieving their missions, 8 in the Explorer Program and 5 in the Discovery Program. Five of them exceeded 10-15 percent growth in projected cost at confirmation. Table 6.1 lists these missions and the associated costs. The reasons provided for the cost growth in each mission (see Tables 5.1 and 5.2) do not allow us to separate increases due to external factors from increases due to project overruns. Thus the committee could not identify the source of possible program or project management failures in any of the missions in Table 6.1. It also could not obtain detailed information on the use of termination reviews in the PI-led mission programs or the number of times they were called. However, it was clear that NASA uses them as a management tool for controlling PI-led mission cost growth. TABLE 6.1 Cost Growth of Recent PI-Led Missions Mission Estimated Cost (million $) Percent Increase (%) At Confirmation At Launch GALEX 77.6 108.2 39.4 Swift 166.7 241.3 44.5 Genesis 216.6 272.3 25.7 MESSENGER 314.0 422.5 34.6 Deep Impact 279.2 332.5 19.1   SOURCE: NASA Headquarters. 5   Personal telephone communication between the committee chair and Keith Goetz, IMEX project manager, University of Minnesota, on September 24, 2005. John Wygant of the University of Minnesota was the PI for IMEX. 6   Congressional Budget Office, 2004, A Budgetary Analysis of NASA’s New Vision for Space Exploration, Washington, D.C., September.

OCR for page 52
Principal-Investigator-Led Missions in the Space Sciences Of the missions in Table 6.1, GALEX, a mission to map galaxies and other objects in the ultraviolet wavelength, is considered a stunning scientific success and is achieving its data-gathering goals, but it must nevertheless be viewed as having had an unfavorable development history (see Table 5.2). Likewise, Swift is in orbit and obtaining its intended data on gamma ray bursts in spite of the technology challenges7 and cost overruns. Genesis samples are being recovered and are expected to achieve a significant portion of the anticipated science return,8 and MESSENGER, in spite of cost growth and major delays, will undertake the first comprehensive survey of Mercury since Mariner 10 in the early to mid-1970s and, as such, it set the scientific context for subsequent investigations by the European and Japanese Bepi Colombo mission. Deep Impact was launched and achieved its targeted cometary impact goal. Clearly, it is necessary to ask at what point problems that might cause the cost cap to be exceeded should justify mission cancellation. This question is discussed in Chapter 7, which presents the committee’s conclusions. 7   Swift had problems with the Burst Alert Telescope, which ultimately cost twice the estimated cost and resulted in a 16-18 month delay. 8   For more on Genesis and science activities, see <genesismission.jpl.nasa.gov/mission/release_nasajsc.html> and <genesismission.jpl.nasa.gov>.