1
Size and Historic Causes of Cost Growth
STUDY BACKGROUND
The Science Mission Directorate (SMD) of the National Aeronautics and Space Administration (NASA) conducts both Earth and space science missions. The latter type encompasses missions in planetary sciences, heliophysics, and astrophysics.
Cost growth in NASA Earth and space science missions is a longstanding problem with a wide variety of interrelated internal and external causes, both technical and programmatic. Many different organizations, both public and private, have examined the cost growth of NASA Earth and space science missions,1 but the results of these studies have not been assessed as a whole. In response to this situation, the NASA Authorization Act of 2008 (P.L. 110-422) directed the NASA administrator to sponsor an “independent external assessment to identify the primary causes of cost growth in the large-, medium-, and small-sized Earth and space science spacecraft mission classes, and make recommendations as to what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s science spacecraft mission programs.”
NASA subsequently requested that the National Research Council (NRC) conduct a study to:
-
Review the body of existing studies related to NASA space and Earth science missions and identify their key causes of cost growth and strategies for mitigating cost growth;
-
Assess whether those key causes remain applicable in the current environment and identify any new major causes; and
-
Evaluate effectiveness of current and planned NASA cost growth mitigation strategies and, as appropriate, recommend new strategies to ensure frequent mission opportunities.
The Committee on Cost Growth in NASA Earth and Space Science Missions was established to conduct this study. As part of this effort, NASA also asked the NRC to “note what differences, if any, exist with regard to Earth science compared with space science missions.” NASA identified a list of relevant cost studies and related analyses to use as primary references for the study (see Table 1.1).
TABLE 1.1 Primary References Provided to the Study Committee
NASA Cost Studies |
|
1 |
R.E. Bitten, D.L. Emmons, and C.W. Freaner. 2006. Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines. IEEE Paper #1545. December. |
2 |
NASA Langley Research Center (LaRC). 2007. Cost/Schedule Performance Study for the Science Mission Directorate. Final Report. Prepared by the NASA LaRC Science Support Office and Science Applications International Corporation. NASA LaRC, Hampton, Va. October. |
3 |
NASA. 2007. “Cost and Schedule Growth at NASA.” Presentation provided to the committee by Director of the Cost Analysis Division Tom Coonce, Office of Program Analysis and Evaluation, NASA, Washington, D.C. November. |
4 |
NASA. 2008. “SMD Cost/Schedule Performance Study Summary Overview.” Presentation by B. Perry and C. Bruno, NASA Science Support Office; M. Jacobs, M. Doyle, S. Hayes, M. Stancati, W. Richie, and J. Rogers, Science Applications International Corporation. January. |
5 |
C.W. Freaner, R.E. Bitten, D.A. Bearden, and D.L. Emmons. 2008. “An Assessment of the Inherent Optimism in Early Conceptual Designs and Its Effect on Cost and Schedule Growth.” Paper presented at the Space Systems Cost Analysis Group/Cost Analysis and Forecasting/European Aerospace Cost Engineering Working Group 2008 Joint International Conference, European Space Research and Technology Centre, Noordwijk, The Netherlands, May 15-16. European Space Agency, Paris, France. |
6 |
B. Mlynczak and B. Perry, Science Support Office, NASA. 2009. “SMD Earth and Space Mission Cost Driver Comparison Study. Final Report and Presentation.” March. |
Related Analyses |
|
7 |
General Accounting Office. 1992. Space Missions Require Substantially More Funding Than Initially Estimated. GAO/NSIAD-93-97. Washington, D.C. December. |
8 |
National Research Council. 1997. Reducing the Costs of Space Science Research Missions: Proceedings of a Workshop. National Academy Press, Washington, D.C. |
9 |
J. McCrillis, Office of the Secretary of Defense, Cost Analysis Improvement Group. 2003. “Cost Growth of Major Defense Programs.” Presentation to the Annual Department of Defense Cost Analysis Symposium, January 30, Williamsburg, Va. |
10 |
Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. 2003. Report of the Defense Science Board/Air Force Scientific Advisory Board Joint Task Force on Acquisition of National Security Space Programs. Department of Defense, Washington, D.C. May. |
The committee generally concurs with the consensus viewpoints expressed in the primary references as a whole. However, as detailed below, in some areas the prior studies reached different conclusions.
This chapter describes the size and historic causes of cost growth in NASA Earth and space science missions, based primarily on the above references. As an illustrative example, this chapter also describes the results of a detailed assessment of cost and schedule data for the 40 missions included in Primary Reference 1.2 Chapter 2 describes the committee’s own assessment of key problems, as they currently exist, and recommended solutions. A full copy of the study statement of task, the tasking letter from NASA, and the legislation that prompted this study appear in Appendix A. The expertise of the study committee is summarized in Appendix B. The findings and recommendations contained in the primary references are summarized in Appendix C.
ANNOUNCEMENT OF OPPORTUNITY AND DIRECTED MISSIONS
NASA implements two separate and distinct classes of Earth and space science missions: announcement of opportunity (AO) missions, such as the Mars Exploration Rover mission, and directed missions, such as the Hubble Space Telescope mission. NASA headquarters competitively selects AO missions from proposals submitted in response to periodic AOs by teams led by a principal investigator, who is commonly affiliated with a university
2 |
Primary Reference 1 was selected because it assesses more NASA missions than any of the other primary references. (See Table 1.2.) |
but may work in industry or for NASA. These teams seek to address one or more scientific questions within the scope of a particular AO. Teams commonly include a NASA center, such as the Goddard Space Flight Center (GSFC) or the Jet Propulsion Laboratory (JPL), as well as an industrial partner. NASA then uses a moderately complex peer review process to evaluate these “Step 1” proposals. This typically leads to the selection of two or three proposals for further development in what is referred to as Phase A. Phase A concludes with the submission of “Step 2” proposals, from which NASA selects a single mission for implementation. AO missions may carry one or several instruments, and they typically cost from $100 million to several hundred million dollars. Only rarely does the cost exceed $1 billion.
NASA headquarters determines the scientific goals and requirements for directed missions, which are sometimes referred to as facility class missions or flagship missions. Headquarters then directs a particular NASA center, usually GSFC or JPL, to implement the mission. The spacecraft for directed missions usually constitute large scientific facilities that produce a variety of data using multiple instruments, which may be quite large. Some of the instruments may be selected using an AO process. NASA headquarters establishes a science working group for directed missions to guide the science aspects of the missions as they move through each phase of development and into operations. Directed missions may cost one to several billion dollars.
SIZE OF COST GROWTH3
Finding. Size of Cost Growth. Historic studies of cost growth indicate the following:
-
Past studies of cost growth in NASA Earth and space science missions calculated values for average cost growth ranging from 23 percent to 77 percent.
-
Different studies reach different conclusions because they examine different sets of missions and because of differences in how cost growth is calculated.
-
Relatively little cost growth occurs between preliminary design review (PDR) and critical design review (CDR). A majority of cost growth occurs after CDR, with the rest occurring prior to PDR.
-
For one large set of 40 missions, 80 percent of the total cost growth (in absolute dollar terms) was caused by only 11 missions.
-
The size of the cost growth of Earth and space science missions has been comparable.
Range of Cost Growth
The 10 primary references examined by this study (listed in Table 1.1) address cost growth in both NASA science missions (Primary References 1 to 8) and Department of Defense (DOD) programs (Primary References 9 and 10). Table 1.2 compares the average cost and schedule growth reported by some of these studies.
While the lowest reported average cost growth in the 10 primary references was 23 percent, the maximum was 77 percent. The average across all 10 was 51 percent. The primary references confirmed that extensive cost growth exists in many—but not all—NASA Earth and space science missions, but the extent of the cost growth differed significantly from one mission to the next, and from one study of cost growth to the next. However, regardless of the wide range in cost growth reported by the various studies, unwanted cost and schedule growth have certainly made it more difficult to accomplish NASA’s Earth and space science missions than was originally anticipated when these missions were authorized.4
TABLE 1.2 Average Reported Cost and Schedule Growth of Past NASA and Department of Defense Missions
Primary Reference |
Missions |
Number of Missions or Programs |
Average Cost Growth (%) |
Average Schedule Growth |
1 |
NASA |
40 |
27 |
22 % |
2, 4 |
NASA |
15 |
23 |
13 months |
3 |
NASA |
25 |
68 |
56 % |
5 |
NASA |
10 |
76 |
36 % |
7 |
NASA |
29 |
77a |
|
9 |
DOD |
142 |
32 |
|
NOTE: Primary References 2 and 4 examined the same data. Primary References 6, 8, and 10 did not calculate average cost or schedule growth. Primary Reference 7 concluded that (1) cost increased by more than 50 percent for three-fourths of the missions examined and (2) cost increased by more than 100 percent for one-third of the missions examined. a The 77 percent cost growth in Primary Reference 7 represents the median cost growth observed. |
Different Studies—Different Results
It is difficult to reconcile fully the different values of cost growth identified in the primary references. The committee identified two primary reasons for these differences:
-
Differences in how cost growth is calculated, and
-
Differences in the sets of missions and programs examined.
By its very definition, cost growth is a relative measure comparing an initial estimate of mission costs against actually incurred costs at a later time. The times at which the initial and final cost estimates are made differ across studies. Figure 1.1 shows the sequence of program phases, key decision points (KDPs), and key mission reviews for a typical NASA mission. Figure 1.1 also shows how three different references calculate cost growth. Primary Reference 1 only considers growth in phases A through D, except for launch. Primary Reference 2 assesses cost growth from the beginning of Phase B (KDP B) through end of mission. Primary Reference 7, on the other hand, measures cost growth from the first time a mission appears as a line item in the NASA budget submitted to Congress until the completion of the mission (end of Phase F).
In general, the earlier the initial estimate, the more cost growth will occur. In addition, including more of the later phases (such as launch, operations, and data analysis) in the cost growth assessment increases the total cost assigned to each mission and the absolute value of the cost growth (in dollars). In particular, a key reason that Primary References 1, 2, and 4 show an average cost growth of 23 to 27 percent, whereas Primary Reference 7 shows a 77 percent average cost growth, is that the start and end points of the cost growth measurements made by Primary Reference 7 are very different from those for Primary References 1, 2, and 4. These differences make it very difficult to derive a single, reliable value for the average cost growth of NASA Earth and space science missions based on previous studies.
Cost growth estimates can also be distorted by missions with exceptionally long development schedules, by changes in mission scope, and by the cost of extended mission operations. For example, Gravity Probe B was under continuous development for 40 years before it was launched in 2004 (NRC, 1995). The Gravity Probe B cost and schedule growth data used by Primary Reference 1 assume that development began in fiscal year 1994, although NASA had previously expended $128 million developing mission concepts and technology.5
Costs are reduced when a mission is descoped, but the value of the mission is also reduced. Conversely, the extra value obtained from an increase in mission scope or an extension of mission operations is presumably worth the cost, or else the broader scope or extended operations would not be approved. In both cases, programmatic changes obscure the extent of cost growth that the missions would have experienced if they had been implemented as originally planned. For example, Aqua, an Earth Observing System (EOS) mission formerly known as EOS-PM,
was significantly descoped in 1992 when the EOS budget was reduced by NASA. Primary Reference 2 reports that Aqua cost growth is 7 percent (based on changes from CDR through launch). Primary Reference 6, however, reports that Aqua cost growth is 23 percent (based on changes during phases B, C, and D). Similar discrepancies are reported for the Spitzer Space Telescope mission (formerly known as the Space Infrared Telescope Facility, SIRTF), which also was significantly descoped. SIRTF cost growth during phases B though D is reported as 68 percent in Primary Reference 2 and 52 percent in Primary Reference 6. Cost growth for Chandra (formerly known as the Advanced X-ray Astrophysics Facility) is reported as 5 percent in Primary Reference 2 (for phases B through D), as negative 2 percent in Primary Reference 6 (for phases B through D), and as 194 percent in Primary Reference 7 (from project initiation through the current state of development when that report was completed in 1992; Chandra was subsequently descoped). These examples suggest that extreme care must be taken when comparing cost growth as measured by different studies, and they highlight the need for common metrics.
Another key complicating factor is that different studies assess different sets of missions and programs. As seen in Table 1.2, the primary references that assessed NASA’s performance examined between 10 and 40 missions. Primary Reference 9 examined 142 DOD projects covering a wide range of applications; the vast majority were not space launch missions.
Table 1.3 lists the missions assessed by each of the primary references that examined NASA missions.6 Note
that Primary Reference 7 examined some missions, such as the Advanced Communications Technology Satellite, that are not Earth or space science missions.
As shown in Table 1.3, some missions are only discussed in one study while others are included in multiple studies. For example, there is a core group of 12 missions that are included in Primary References 1, 2, 4, and 6. These missions are listed below and offer the best chance for a more in-depth comparison of cost growth calculations across different studies:
-
Messenger (Mercury Orbiter),
-
Cloud Satellite (CLOUDSAT),
-
Galaxy Evolution Explorer (GALEX),
-
Swift Gamma-Ray Burst Mission,
-
Spitzer Space Telescope (formerly SIRTF),
-
Mars Reconnaissance Orbiter (MRO),
-
Solar Terrestrial Relations Observer (STEREO),
-
Comet Nucleus Tour (CONTOUR),
-
Deep Impact,
-
EOS-Aqua,
-
Mars Exploration Rover (MER), and
-
Reuven Ramaty High Energy Solar Spectral Imager (RHESSI).
Seven of these 12 are also included in Primary Reference 5. Accordingly, the cost growth experience of these 12 missions is overemphasized in any simple averaging of the results of the primary references. On the other hand, Primary References 1, 3, and 7 assess a large number of missions that are not included in any of the other primary references, and this undoubtedly contributes to the large variance in average cost growth determined by the primary references.
Cost Growth Versus Schedule Growth
To assess the overall cost growth of past missions, it is helpful to consider how missions cluster in terms of their relative cost and schedule growth. Figure 1.2 shows the distribution of the 40 missions examined by Primary Reference 1 in terms of development cost and schedule growth. Missions are grouped as Type I, II, and III based on the cost growth thresholds defined in the NASA Authorization Act of 2005:7
-
Type I, minor or no cost growth (less than 15 percent),
-
Type II, significant cost growth (15 to 30 percent), and
-
Type III, excessive cost growth (more than 30 percent).
Of the 40 missions in Primary Reference 1, Figure 1.2 shows that 19 missions (roughly half) are Type I, 8 missions (one-fifth) are Type 2, and 13 missions (one-third) are Type III.8 In other words, half of the Earth and space science missions in this large sample were completed within the 15 percent cost growth target needed to avoid congressional notification.
7 |
The NASA Authorization Act of 2005 requires congressional notifications for cost growth of 15 percent or more during phases B and C or schedule delays of 6 months or more. Reauthorization is required for cost growth of 30 percent or more or $1 billion or more during phases B and C. These reporting requirements do not apply to all of the missions plotted in Figure 1.2; per statute, they only apply to programs with an estimated life-cycle cost greater than $100 million, and some missions in Figure 1.2 occurred prior to fiscal year 2005. |
8 |
Many of the figures and tables in this report depict data from Primary Reference 1, which were used for these illustrative examples because those data assessed more missions than any of the other studies that examined NASA missions, and because the committee was able to obtain the detailed data used by the authors of Primary Reference 1 in order to conduct the committee’s own supplemental analyses. |
TABLE 1.3 NASA Missions Examined by the Primary References
|
|
|
Primary Reference |
||||||
Mission Name |
Program/Class |
Launch Year |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
Messenger |
Discovery |
2004 |
X |
X |
|
X |
X |
X |
|
CLOUDSAT |
Earth System Science Pathfinder |
2006 |
X |
X |
|
X |
X |
X |
|
GALEX |
Explorer |
2003 |
X |
X |
|
X |
X |
X |
|
Swift Gamma-Ray Burst Mission |
Explorer/Medium-class Explorer |
2004 |
X |
X |
|
X |
X |
X |
|
Spitzer Space Telescope (formerly SIRTF) |
Great Observatory |
2003 |
X |
X |
|
X |
X |
X |
|
MRO |
Mars |
2005 |
X |
X |
|
X |
X |
X |
|
STEREO |
Solar Terrestrial Probe |
2006 |
X |
X |
|
X |
X |
X |
|
CONTOUR |
Discovery |
2002a |
X |
X |
|
X |
|
X |
|
Deep Impact |
Discovery |
2005 |
X |
X |
|
X |
|
X |
|
EOS-Aqua |
Earth Observing System |
2002 |
X |
X |
|
X |
|
X |
|
MER |
Mars |
2003 |
X |
X |
|
X |
|
X |
|
RHESSI |
Small Explorer |
2002 |
X |
X |
|
X |
|
X |
|
EO-1 |
New Millennium |
2000 |
X |
|
|
|
X |
X |
|
EOS-Aura |
Earth Observing System |
2004 |
X |
|
|
|
|
X |
|
CALIPSO |
Earth System Science Pathfinder |
2006 |
X |
|
|
|
|
X |
|
GRACE |
Earth System Science Pathfinder |
2002 |
X |
|
|
|
|
X |
|
THEMIS |
Explorer/Medium-class Explorer |
2007 |
X |
|
|
|
|
X |
|
Landsat-7 |
Landsat Program |
1999 |
X |
|
|
|
|
X |
|
FAST |
Small Explorer |
1996 |
X |
|
|
|
|
X |
|
TRACE |
Small Explorer |
1998 |
X |
|
|
|
|
X |
|
TRMM |
Mission to Planet Earth |
1997 |
X |
|
|
|
|
|
X |
Gravity Probe B |
Astrophysics |
2004 |
X |
|
|
|
|
|
|
Genesis |
Discovery |
2001 |
X |
|
|
|
|
|
|
Lunar Prospector |
Discovery |
1998 |
X |
|
|
|
|
|
|
NEAR |
Discovery |
1996 |
X |
|
|
|
|
|
|
Stardust |
Discovery |
1999 |
X |
|
|
|
|
|
|
Mars Pathfinder |
Discovery - Mars |
1996 |
X |
|
|
|
|
|
|
ICESAT |
Earth Observing System |
2003 |
X |
|
|
|
|
|
|
SORCE |
Earth Observing System |
2003 |
X |
|
|
|
|
|
|
ACE |
Explorer |
1997 |
X |
|
|
|
|
|
|
HETE-II |
Explorer |
2000 |
X |
|
|
|
|
|
|
IMAGE |
Explorer |
2000 |
X |
|
|
|
|
|
|
MAP |
Explorer |
2001 |
X |
|
|
|
|
|
|
FUSE |
Explorer (Origins) |
1999 |
X |
|
|
|
|
|
|
MCO |
Mars |
1998a |
X |
|
|
|
|
|
|
MGS |
Mars |
1998 |
X |
|
|
|
|
|
|
MPL |
Mars |
1999a |
X |
|
|
|
|
|
|
DS-1 |
New Millennium |
1998 |
X |
|
|
|
|
|
|
SWAS |
Small Explorer |
1998 |
X |
|
|
|
|
|
|
WIRE |
Small Explorer |
1999a |
X |
|
|
|
|
|
|
|
|
|
Primary Reference |
||||||
Mission Name |
Program/Class |
Launch Year |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
TIMED |
Solar Terrestrial Probe |
2001 |
X |
|
|
|
|
|
|
EOS-Terra |
Earth Observing System |
1999 |
|
|
|
X |
|
||
New Horizons (Pluto) |
New Frontiers |
2006 |
|
X |
|
X |
X |
X |
|
Chandra X-ray Observatory (formerly AXAF) |
Great Observatory |
1999 |
|
X |
|
X |
|
X |
X |
ACRIMSAT |
Solar Science |
1999 |
|
X |
|
X |
|
|
|
Dawn |
Discovery |
2007 |
|
|
X |
|
X |
X |
|
Fermi Gamma-ray Space Telescope (formerly GLAST) |
NASA/DOE/Japan |
2008 |
|
|
X |
|
|
X |
|
IBEX |
Small Explorer |
2008 |
|
|
X |
|
|
X |
|
SOFIA |
Astrophysics (airborne) |
2007 |
|
|
X |
|
|
|
|
TWINS A/B |
Heliophysics |
2008 |
|
|
X |
|
|
|
|
LCROSS |
ESMD |
2009 |
|
|
X |
|
|
|
|
LRO |
ESMD/SMD |
2009 |
|
|
X |
|
|
|
|
Kepler |
Discovery |
2009 |
|
|
X |
|
|
|
|
Glory |
Earth Observing - Directed |
2010c |
|
|
X |
|
|
|
|
OCO |
Earth System Science Pathfinder |
2009a |
|
|
X |
|
|
|
|
Planck |
ESA |
2009 |
|
|
X |
|
|
|
|
Herschel |
ESA Horizon 2000 Program |
2009 |
|
|
X |
|
|
|
|
WISE |
Explorer/Medium-class Explorer |
2009 |
|
|
X |
|
|
|
|
JWST |
Great Observatory |
2014c |
|
|
X |
|
|
|
|
Aquarius |
Instrument |
2010c |
|
|
X |
|
|
|
|
OSTM |
Joint NASA/CNES/NOAA |
2008 |
|
|
X |
|
|
|
|
MSL |
Mars |
2011c |
|
|
X |
|
|
|
|
CINDI |
Mission of Opportunity |
2008 |
|
|
X |
|
|
|
|
Space Technology 7 |
New Millennium |
cancelled |
|
|
X |
|
|
|
|
Space Technology 8 |
New Millennium |
cancelled |
|
|
X |
|
|
|
|
M3-Foton |
Russian-ESA |
2007 |
|
|
X |
|
|
|
|
MMS |
Solar Terrestrial Probe |
2014c |
|
|
X |
|
|
|
|
NPOESS Preparatory Project |
NASA/NOAA/USAF |
2011c |
|
|
X |
|
|
|
|
Cassini-Huygens |
NASA-ESA |
1997 |
|
|
|
|
|
X |
|
Phoenix |
Mars Scout |
2007 |
|
|
|
|
|
X |
|
AIM |
Small Explorer |
2007 |
|
|
|
|
|
X |
|
Magellan |
Venus |
1989 |
|
|
|
|
|
|
X |
NSCAT |
NASA/Japan |
1996 |
|
|
|
|
|
|
X |
ACTS |
Communications |
1993 |
|
|
|
|
|
|
X |
GOES I-M |
Operational weather satellites |
1994-2001 |
|
|
|
|
|
|
X |
EUVE |
Explorer |
1992 |
|
|
|
|
|
|
X |
XTE |
Explorer/Medium-class Explorer |
1995 |
|
|
|
|
|
|
X |
Galileo |
Planetary |
1989 |
|
|
|
|
|
|
X |
|
|
|
Primary Reference |
||||||
Mission Name |
Program/Class |
Launch Year |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
CGRO |
Great Observatory |
1991 |
|
|
|
|
|
|
X |
HST |
Great Observatory |
1990 |
|
|
|
|
|
|
X |
Mars Observer |
Mars |
1992a |
|
|
|
|
|
|
X |
TDRS-7 |
NASA Communications |
1995 |
|
|
|
|
|
|
X |
TOPEX |
NASA/CNES |
1992 |
|
|
|
|
|
|
X |
Ulysses |
NASA/ESA |
1990 |
|
|
|
|
|
|
X |
ASRM |
Space Shuttle |
cancelled |
|
|
|
|
|
|
X |
ATP |
Space Shuttle |
2002 |
|
|
|
|
|
|
X |
TSS-1 |
Space Shuttle |
1992 |
|
|
|
|
|
|
X |
FTS |
Space Station |
cancelled |
|
|
|
|
|
|
X |
Freedom (Space Station), now ISS |
Space Station |
1998-2011 |
|
|
|
|
|
|
X |
AFE |
Technology development |
cancelled |
|
|
|
|
|
|
X |
Collaborative Solar Terrestrial Research |
International Solar-Terrestrial Physics |
1992 |
|
|
|
|
|
|
X |
Global Geospace Science Program (Polar and Wind missions) |
International Solar-Terrestrial Physics |
1994/1996 |
|
|
|
|
|
|
X |
Landsat-D |
Earth Observing |
1982 |
|
|
|
|
|
|
X |
OMV |
Technology development |
cancelled |
|
|
|
|
|
|
X |
NOTE: Acronyms are defined in Appendix D. a Mission failed. b Initially part of the study, but then dropped from the analysis. c Planned launch date (has not yet occurred). |
Figure 1.2 also shows a substantial linear correlation (R2 = 0.64) between cost growth and schedule growth. This correlation would be even stronger if not for the effect of missions with fixed launch windows. For example, launch windows for missions to Mars open every 780 days, and mission staff are highly motivated to be ready to go during the assigned window. If that window is missed, the only alternatives are to cancel the mission or sit on the ground for 2 years waiting for the next window (resulting in a large increase in both cost and schedule). Hence, in Figure 1.2, it is not surprising to see that the Mars Reconnaissance Orbiter (MRO) mission, the Mars Exploration Rover (MER) mission, and the Comet Nucleus Tour (CONTOUR) mission (which had a single, 25-day launch window) had significant cost growth, even though schedule growth was zero or nearly so. In fact, the pressure that such missions experience to meet the launch window may drive cost growth higher than it would otherwise be if the mission had a more flexible launch window.
For many missions, there is substantial correlation between cost and schedule growth. However, even when that is the case, schedule delays are not necessarily the primary cause of cost growth. In some cases, schedule delays are a secondary effect of other factors, such as a failure to adequately fund the program or problems with launch vehicle or launch pad availability. In these cases, additional effort is required to replan and manage the mission, and additional costs are incurred due to extension of supporting areas such as program review and cost accounting. In other cases, it may even be necessary to store the spacecraft and supporting systems and keep the launch and support crew trained and capable of supporting a launch. The costs associated with these efforts are sometimes referred to as “sustaining cost” or “capability retention cost,” and they represent real effort that is required for a successful program.
In most cases, however, schedule growth and cost growth arise from development problems that require both time and money to solve. Earth and space science programs are very complex, with extensive interactions among
tasks, so that schedule delays in one development area impact the schedule and scope of other tasks. Furthermore, delays in one task cause additional effort to replan and manage all of the interdependent project tasks. Thus, cost growth may occur in many areas of a project because of a schedule delay in one area or on one task. When that happens, the total cost growth might be attributed to schedule growth, but the root cause is a development problem in one or more areas. In any case, as discussed below, there are many causes of cost growth in addition to schedule growth.
Absolute Cost Growth
Additional insight can be gained by considering absolute cost growth (in dollars) in addition to relative cost growth (as a percentage of the initial estimate). Figures 1.3a and 1.3b show in decreasing order the absolute cost growth in excess of reserves for the 40 missions from Primary Reference 1. The two figures are identical, except that the first shows the distribution of Earth science and space science missions, and the second shows the distribution of AO and directed missions. Figure 1.3c also shows the initial cost estimate and launch date for each of these missions.9 The average development cost for each of these 40 missions is $215 million. This does not include the cost of launch, mission operations, data analysis, or inflation. If savings from missions that under run their budgets are allowed to partially offset the cost growth of other missions, the total net absolute cost growth for these 40 missions is $1.3 billion. This corresponds to the total development cost of about 6 missions of average size.
As is often the case, about 20 percent of the population causes 80 percent of the problem. In particular, with regard to the 40 missions examined by Primary Reference 1, 11 of them (28 percent of the total) account for about 80 percent of the total cost growth (in absolute terms), and 14 missions (35 percent of the total number of missions) account for about 92 percent of the total cost growth; the other 26 missions (two-thirds of the total number) account for only 8 percent of the total cost growth (see Table 1.4).
Based on data from 1970 through 1999, there is not a consistent trend for cost growth of NASA missions over time. As shown in Table 1.5, cost growth for NASA missions increased from the 1970s to the 1980s, but then decreased in the 1990s. More recently, Figure 1.3c and Table 1.4 indicate that, for this set of 40 missions, cost and cost growth for missions launched in 2003 and later are generally higher than for missions launched in 2002 and earlier. The 15 missions launched in 2003 and later had an average initial cost of $244 million, whereas the 25 missions launched in 2002 and earlier had an average initial cost of $146 million, and the 15 missions launched in 2003 and later, as a whole, have a much higher cost growth (29 percent/$72 million) than the 25 missions launched in 2002 and earlier (6 percent/$8 million). In addition, all but 2 of the 14 missions with the most cost growth (in dollars) were launched in 2003 or later, and all but 3 of the 26 missions with the least cost growth were launched in 2002 or earlier.
A comparison of Figures 1.2 and 1.4 shows that the slopes of the linear trend lines are nearly identical (y/x = 1.22 in Figure 1.2 and 1.23 in Figure 1.4), indicating that the relationship between schedule growth and cost growth is virtually the same for both small and large missions. However, the impact of even modest cost growth in a large mission can still be substantial.
Figure 1.3b shows that both AO and directed missions experience significant cost growth (in absolute terms). Similarly, Figures 1.3a and 1.4 show that both Earth and space science missions experience a similar mix of cost growth in both absolute terms and as a percentage of initial cost. The 10 Earth science missions in this set of 40 projects exhibited a total cost growth of 13 percent (an average of $37 million per mission), while the 30 space science missions showed a total cost growth of 20 percent (an average of $31 million per mission). Primary Reference 6 concurs that Earth science missions are not significantly different from other Science Mission Directorate (SMD) missions in terms of cost or cost growth (and this is supported by Figure 1.4). Primary Reference 6 concludes that cost growth is more closely associated with increases in spacecraft mass and higher levels of mission complexity rather than with mission type. In any case, as difficult as it is to develop an accurate and consistent view of mission costs and cost growth based on historic studies, it is even more difficult to determine with certainty the cause and effect relationship among the many factors involved.
9 |
For missions with multiple launches, the date of the first launch is listed in Figure 1.3c. |
Cost Growth by Phase
One of the clear and consistent findings in the primary references is that cost growth does not accumulate uniformly across mission phases; rather, the bulk of cost growth occurs post-CDR (see Figure 1.5). Some post-CDR cost growth is driven by external factors, such as delays in the availability of launch vehicles. However, most post-CDR cost growth is due to internal project development issues, even though (1) CDR is intended to be the final milestone of the design phase and (2) spacecraft configuration should be frozen at CDR. CDR for many missions may be held prematurely—driven by schedule rather than driven by design maturity. CDR approval of an immature design can cause downstream problems during Phase D such as integration difficulties and late changes.
Primary Reference 2 found that, excluding external impacts, cumulative average cost growth to CDR is only 7 percent, but this grows to 28 percent by launch. So 75 percent of cost growth occurs after CDR. Primary Reference 5 also analyzed the evolution of cost growth over time and concluded that “it is important to notice that, unlike the mass and power growth time trends, cost growth is typically not recognized until after CDR. This is counter to standard industry guidelines that recommend a decreasing percentage reserve on a reduced cost-to-go basis. The
substantial cost growth after CDR implies that a greater percentage reserve on cost to go should be held. Alternately, it may mean that cost growth is occurring earlier in the project lifecycle, but isn’t recognized until later” (p. 9).10 Similarly, Primary Reference 6 reported that the highest percentage of schedule growth occurs after the start of integration and testing, i.e., during Phase D, and that this phenomenon is consistent across Earth science, heliophysics, astrophysics, and planetary missions. If risk has not been sufficiently reduced by CDR, then cost and schedule uncertainty remains high, especially because the underlying causes of some post-CDR cost growth may have originated prior to CDR without being recognized.
Figure 1.5 suggests that, for one set of 20 missions examined by the committee, there is a slight decrease in cost growth as we go from initial cost estimates at the start of the program to PDR from about 48 percent to about 35 percent cost growth. Thus about 10-15 percent cost growth can be attributed to a lack of design maturity pre-PDR.
TABLE 1.4 Breakdown of Cost and Cost Growth for the 40 missions from Primary Reference 1
|
Total Initial Cost |
Total Cost Growth |
||
|
(billion $) |
(%) |
(billion $) |
(%) |
14 missions with the most cost growth |
3.9 |
53 |
1.2 |
92 |
26 missions with the least cost growth |
3.4 |
47 |
0.1 |
8 |
Total for all 40 missions |
7.3 |
100 |
1.3 |
100 |
|
Average Initial Cost |
|
Average Cost Growth |
|
|
(million $) |
|
(million $) |
(%) |
15 missions launched in 2003 and later |
244 |
72 |
29 |
|
25 missions launched in 2002 and earlier |
146 |
8 |
6 |
|
SOURCE: Based on data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009, providing source data for Primary Reference 1. |
TABLE 1.5 Decadal Trends in Cost Growth for NASA Missions
Surprisingly, the cost growth between PDR and CDR is not very large: only about 5 percent for AO missions and about 12 percent for directed missions.
Therefore the majority of cost growth, typically 20 to 30 percent of the initial estimate, occurs post-CDR. This is an important finding, which is supported by many of the primary references. This finding is symptomatic that a number of projects experience “surprises” post-CDR that significantly increase mission cost, even though the design of the instruments and spacecraft is supposed to be “frozen” at CDR. The reasons for the post-CDR cost growth will be discussed in the following sections.
CAUSES OF COST GROWTH
Finding. Causes of Cost Growth. Past studies identify a wide range of factors that contribute to cost and schedule growth of NASA Earth and space science missions. The most commonly identified factors are as follows:
-
Overly optimistic and unrealistic initial cost estimates,
-
Project instability and funding issues,
-
Problems with development of instruments and other spacecraft technology, and
-
Launch service issues.
In addition, any problem that causes schedule growth contributes to and magnifies total mission cost growth, and cost growth in one mission may induce organizational replanning that delays other missions in earlier stages of implementation, further amplifying overall cost growth. Effective implementation of a comprehensive, integrated cost containment strategy, as recommended herein, is the best way to address this problem.
The historic causes of cost growth in the primary references are discussed below and summarized in Table 1.6.11 In addition to the four most common causes listed in the above finding, the primary references have identified a wide range of additional factors that contribute to cost growth, such as poor contractor performance and a tendency to over-engineer. Primary Reference 10, which examined DOD projects, also noted the erosion of capabilities to lead and manage the space acquisition process.
The committee generally relied on the assessments made in the primary references, rather than begin its assessment using raw data collected by the authors of the earlier studies, in part because the raw data were not available to the committee. However, spot checks of data for key missions supported the conclusions reached by the primary references and this study. For example, the Spitzer Space Telescope mission (formerly known as the Space Infrared Telescope Facility, SIRTF) had the largest absolute cost growth of the 40 missions assessed by Primary Reference 1. Cost growth problems encountered by Spitzer included many of the factors cited in the above finding and in Table 1.6: early planning deficiencies; problems with development, integration, and/or testing of the spacecraft as well as all three major instruments; launch vehicle problems; schedule delays associated with all of the above; and cost growth of project-level management functions (Mlynczak and Perry, 2009).
The primary references are generally consistent in their conclusions, although, in some cases, different studies disagree about the significance of the impact of some factors on cost growth. For example, Primary Reference 6 uses data-driven correlation analyses to assert that there is little or no correlation between mission cost growth and planned cost reserves or the percent of funds spent in Phase B. Primary References 2 and 4 (which, for the most part, examined the same missions as Primary Reference 6) concur that there is no correlation between cost growth and planned cost reserves. However, several committee members have observed that cost reserves are often “manufactured” during the early stages of a program by unrealistically reducing baseline budgets (without reserves), so that the total budget can include a cost reserve of the expected magnitude without descoping the mission. This manufactured reserve does not represent a true contingency fund in excess of likely project costs. The committee concludes that the lack of correlation in some studies between cost reserve and cost growth (in excess of the cost reserve) indicates that more care is needed in establishing cost reserves in excess of realistic cost estimates.
Primary References 2, 4, and 6 also point out a lack of correlation between funds spent in Phase B and cost growth. Primary Reference 2, however, recommends including a level of cost reserves that is commensurate with mission implementation risk. In addition, during Phase B, many projects are still in a competitive mode, and competitive pressures encourage (overly) optimistic assessments of the cost and schedule impacts of addressing uncertainties and overcoming potential problems.
One reason to designate a larger percentage of mission budgets for Phase B technology development is to better
TABLE 1.6 Causes of Cost Growth Based on the Primary References
Causes of Cost Growth |
Page Number in Primary Reference |
||||||||
Initial Cost Estimates |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
9 |
10 |
Inherent optimism in initial design and estimates (in part because of the emphasis that is placed on science, not cost; in part because mission complexity is underestimated), which leads to unrealistic proposals and estimates |
9 |
38 |
4 6 |
15 |
1 5 |
|
12 |
58 |
2 |
Incomplete initial budget estimates (e.g., cost of launch services, mission operations, and/or data analysis not included) |
|
|
|
|
|
|
17 |
58 |
|
Cost has replaced mission success as the primary driver in managing acquisition processes, resulting in excessive technical and schedule risk. |
|
|
|
|
|
|
|
iii |
|
Assumptions about heritage hardware, software, and commercial-off-the-shelf equipment did not materialize. |
|
|
6 |
|
|
|
|
|
|
The space acquisition system is strongly biased to produce unrealistically low cost estimates throughout the acquisition process. These estimates lead to unrealistic budgets and unexecutable programs. Government budget space acquisition programs should use a most probable (80/20) cost, with a 20-25 percent management reserve for development programs included within this cost. |
|
|
|
|
|
|
|
|
iii |
Inflation |
|
|
|
|
|
|
17 |
|
|
Project Instability and Funding Issues |
|
|
|
|
|
|
|
|
|
Program instability/changes (to mission requirements, spacecraft, instruments, launch vehicles, upper stage propulsion systems, trajectories, and/or operations) |
|
25 |
6 12 |
|
|
|
11 15 |
58 |
2 |
Technical and programmatic uncertainty at the beginning of a project (not enough time or resources available in phases A and B; TRL lower than claimed; risk identification and mitigation prior to CDR needs attention; uncertainty in spacecraft mass, power, pointing accuracy, data transmission rates, and so on) |
15 |
36 |
6 12 19 |
|
12 13 |
|
|
|
|
Budget constraints, lack of stable funding, and/or inadequate initial funding profile |
|
25 |
6 |
15 |
|
|
11 15 |
|
|
Weak independent validation of cost and schedule |
|
|
6 |
|
|
|
|
|
|
Inadequate definition of technical and management aspects of projects prior to NASA and OMB approval |
|
|
|
|
|
|
11 |
|
|
Inadequate cost and schedule reserves |
12 |
|
6 |
|
|
|
|
|
|
Technical complexity |
|
|
|
|
|
|
15 |
|
|
Adverse impacts of financial problems experienced by other NASA missions |
|
|
6 |
|
|
|
|
|
|
Development of Instruments and Other Spacecraft Technology |
|
|
|
|
|
|
|
|
|
Instrument development problems caused, for example, by instrument designs that lack technical details and/or fail to identify technical challenges |
11 |
29 |
6 15 |
8 |
12 13 |
117 |
|
|
|
The primary contributor to internal cost growth (i.e., growth caused by factors within the control of a given project) is instrument development problems. |
4 11 |
|
|
|
|
|
|
|
|
Increase in spacecraft development costs, especially integration and test costs |
|
31 |
|
|
|
114 |
|
|
|
Increases in spacecraft mass, power requirements, and complexity |
|
|
|
|
4 |
|
|
|
|
Causes of Cost Growth |
Page Number in Primary Reference |
||||||||
Increases in payload cost (except for planetary missions) |
|
|
|
|
|
114 117 |
|
|
|
Launch Service Issues |
|
|
|
|
|
|
|
|
|
Launch service issues (primarily higher launch vehicle prices or selection issues) |
|
25 |
6 |
15 |
|
|
|
|
|
Launch vehicle delays resulting in overall schedule delays |
|
25 |
6 |
|
|
|
17 |
|
|
The primary contributor to external cost growth (i.e., growth caused by factors beyond the control of a given project) is problems with the readiness of the launch vehicle. |
4 |
|
|
|
|
|
|
|
|
Other Factors |
|
|
|
|
|
|
|
|
|
Schedule growth that leads to cost growth |
11 |
38 |
12 |
|
|
|
19 |
59 |
|
Poor contractor performance |
|
|
|
|
|
|
15 |
|
|
Industry has failed to implement proven practices on some programs. |
|
|
|
|
|
|
|
|
4 |
Industry guidelines do not in general adequately predict the uncertainty in the initial physical and programmatic parameters claimed in proposals. |
|
|
|
|
|
13 |
|
|
|
Tendency to over-engineer |
|
|
6 |
|
|
|
|
|
|
Increase in project-level management costs |
|
27 |
|
8 |
|
|
|
|
|
Failure to achieve anticipated cost and/or schedule savings from mission descopes |
|
30 |
6 |
|
|
|
|
|
|
Increases in mission operations and data analysis costs for science enhancements and/or extended missions |
|
44 |
|
|
|
|
|
|
|
Hardware from foreign partners (late delivery or increased costs) |
|
36 |
6 17 |
|
|
|
|
|
|
ITAR requirements |
|
44 |
|
|
|
|
|
|
|
Department of Defense capabilities to lead and manage the space acquisition process have seriously eroded over time. |
|
|
|
|
|
|
|
|
3 |
NOTE: Primary Reference 8 is focused on reducing the absolute costs of NASA space science missions. It does not directly address cost growth, and so its results are not included in this table. Primary References 9 and 10 focus on Department of Defense systems. |
define system designs, reconcile mission goals and budgets, retire risk prior to PDR, and reduce uncertainties in the project plan. If the baseline cost is established after PDR (that is, after the end of Phase B), increased costs identified in Phase B do not show up as cost growth. Primary References 2, 4, and 6, however, define baseline mission costs at the beginning of Phase B. As a result, technology investments made during Phase B cannot improve the understanding of the technical baseline, which is necessary to improve the accuracy of the initial cost estimate.
Initial Cost Estimates
The primary references have concluded that mission requirements are typically formulated on the basis of science return on the particular mission much more than cost. In addition, it is easy to underestimate the complexity of mission hardware and software and/or make unrealistically optimistic assumptions about how easy it will be to develop and manufacture systems for a new mission based on similarities to systems used in past missions. In a competitive environment, bidders are motivated to argue that science goals can be achieved as inexpensively as possible in order to maximize the science return for a given assumed budget. In addition, initial cost estimates may not include all costs, such as the cost of launch services, mission operations, and/or data analysis. As a result, initial cost estimates and mission proposals tend to be overly optimistic and unrealistic to the point that it becomes very difficult to implement them within the initial budgets. Yet, once projects get under way, cost constraints
may become so important that efforts to minimize cost growth may result in excessive technical and schedule risk. Inflation also becomes a significant factor for missions with a long life cycle, either by design or because of schedule growth.
Project Instability and Funding Issues
The primary references have concluded that project instability (associated with changes to mission requirements, spacecraft, instruments, launch vehicles, upper stage propulsion systems, trajectories, and/or operations) are a major cost growth factor. Budget constraints, lack of stable funding, inadequate initial funding profiles, insufficient cost and schedule reserves, and weak independent validation of cost and schedule likewise contribute to cost growth. In many cases, project instability and funding issues are intertwined. At the beginning of a project, actual technology readiness levels may be lower than anticipated, leading to uncertainty in mission requirements in terms of spacecraft mass, power, pointing accuracy, data transmission rates, and overall mission complexity. Technical and programmatic uncertainty tends to persist if the funding in phases A and B is insufficient to resolve risks adequately. As a result, missions are sometimes approved by NASA and OMB before cost and schedule requirements are well understood. Even when a particular mission is well understood and well planned, changes and delays are sometimes imposed when, in a constrained cost environment, cost growth experienced by one NASA mission may require NASA to delay or curtail other missions. Other sources of external changes, which are beyond the control of a given project manager, are changes in overall requirements or processes that may be imposed, for example, by an evolving view of what constitutes acceptable risk. As unanticipated problems and changes arise, either internally or externally, the resulting instability tends to redirect the attention of the project management team from the technical challenges of implementing the mission to work on replanning efforts. These replanning activities themselves generally result in increased cost and schedule. Project instability and funding issues can be especially difficult to avoid for large missions that rely on annual funding appropriations for a mission life cycle that may span a decade or more.
Development of Instruments and Other Spacecraft Technology
Earth and space science satellites are highly customized, and their design is often driven by the nature of their instruments and their requirements in terms of power, mass, pointing accuracy, thermal control, and so on. The primary references conclude that instrument development problems may be the largest element of mission cost growth that can be attributed to factors within the control of a given project. Common problems are instrument designs that lack technical details and/or fail to identify technical challenges. Cost growth can also arise from problems in the development of other spacecraft technologies and systems, especially in the area of integration and testing and as a result of increases in spacecraft mass, power requirements, and/or complexity.
Launch Service Issues
Launch service issues include higher launch vehicle prices and/or launch vehicle selection issues. In addition, launch vehicle delays are a common cause of delays in the overall schedule. In fact, Primary Reference 1 concluded that problems with the readiness of the launch vehicle are the primary contributor to external cost growth (i.e., growth caused by factors beyond the control of a given project).
DIFFERENCES BETWEEN EARTH AND SPACE SCIENCE MISSIONS
A casual look at partial cost data or personal experience may indicate different cost growth potential for missions from different disciplines, because different classes of missions face different challenges. For example, Earth science missions typically have more complex, more costly, and more massive instruments than space science missions do. Earth science missions also have more stringent requirements in terms of pointing accuracy, resolution, stability, and so on, although astrophysics missions also have stringent pointing requirements, and planetary
spacecraft and instrument technology must be able to survive long cruise phases and radiation environments that are sometimes quite extreme. Earth science missions also dedicate a higher fraction of mission costs to instruments than do other missions, their instruments cost the most, and their spacecraft are typically more complex than most space science spacecraft are. Space science missions that leave Earth orbit have greater incentives to minimize spacecraft mass and power, and the average cost and the average spacecraft mass for space science missions are less than those for Earth science missions. However, as specifically addressed by Primary Reference 6, Earth science missions have not shown a systematic difference in cost or cost growth compared to other SMD missions.12 Both Earth and space science missions have shown good correlation between (1) instrument schedule growth and instrument cost growth, (2) instrument cost/schedule growth and mission cost/schedule growth, and (3) the absolute costs of instruments and instrument complexity.
Primary Reference 1 and other studies (Bitten, 2008; Bearden, 2008) confirm that the complexity of Earth science missions results in long development cycles (and associated large development costs as a fraction of total mission costs). Because Earth science missions rarely have highly restricted launch windows, it is easier to postpone launch (and slip the mission schedule) as necessary to solve technical problems. Restricted launch windows motivate mission teams to avoid schedule delays if at all possible. For some missions, this has involved descoping. Schedule-constrained missions (which are mostly planetary missions) typically respond to late-breaking technical problems not by increasing schedule but by increasing the workforce (and costs). In some cases, despite the best efforts of the mission staff, risk created by late-breaking problems is not fully resolved prior to launch. As a result, schedule-constrained missions tend to have less cost growth—and higher failure rates—than do other missions.