2
Key Problems and Solutions

In its detailed examination of the primary references, other related studies, and presentations by NASA and industry personnel, the committee did not identify any major new individual causes of cost growth. However, the committee has identified three key problem areas and suggests action to address them. First, there is the issue of cost realism, including the tendency to underestimate program difficulty; uncertainty in the definition/establishment of cost and schedule baselines; and the use and capability of NASA cost-estimating methodologies. Second, there are a series of process-based issues, including project selection and formulation, risk identification and mitigation, and the review process. The committee also believes that NASA needs to invigorate the technology base, particularly technology supporting instrument development. Third, a principal factor external to programs is launch vehicle selection and cost.

In addition to the many specific causes of cost growth identified in the primary references, the integrated effects of these causes also leads to cost and schedule growth. Therefore, the committee recommends that NASA develop an overall strategy for dealing with the issues that cause cost and schedule growth.

COST REALISM

Cost Estimates

Finding. Unrealistic Initial Cost and Schedule Estimates. People and organizations tend to optimize their behavior based on the environment in which they operate. The current system incentivizes overly optimistic expectations regarding cost and schedule. As a result, initial cost estimates generally underestimate final costs by a sizable amount.


Recommendation. Independent Cost Estimates. NASA should strengthen the role of its independent cost-estimating function as follows:

  • Expanding and improving NASA’s ability to conduct parametric cost estimates, and

  • Obtaining independent parametric cost estimates at critical design review (in addition to system requirements review and preliminary design review), comparing them to other estimates available from the project and reconciling significant differences.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 30
2 Key Problems and Solutions In its detailed examination of the primary references, other related studies, and presentations by NASA and industry personnel, the committee did not identify any major new individual causes of cost growth. However, the committee has identified three key problem areas and suggests action to address them. First, there is the issue of cost realism, including the tendency to underestimate program difficulty; uncertainty in the definition/establish - ment of cost and schedule baselines; and the use and capability of NASA cost-estimating methodologies. Second, there are a series of process-based issues, including project selection and formulation, risk identification and mitigation, and the review process. The committee also believes that NASA needs to invigorate the technology base, particularly technology supporting instrument development. Third, a principal factor external to programs is launch vehicle selection and cost. In addition to the many specific causes of cost growth identified in the primary references, the integrated effects of these causes also leads to cost and schedule growth. Therefore, the committee recommends that NASA develop an overall strategy for dealing with the issues that cause cost and schedule growth. COST REALISM Cost Estimates Finding. Unrealistic Initial Cost and Schedule Estimates. People and organizations tend to optimize their behavior based on the environment in which they operate. The current system incentivizes overly optimistic expectations regarding cost and schedule. As a result, initial cost estimates generally underestimate final costs by a sizable amount. Recommendation. Independent Cost Estimates. NASA should strengthen the role of its independent cost- estimating function as follows: • Expanding and improving NASA’s ability to conduct parametric cost estimates, and • Obtaining independent parametric cost estimates at critical design review (in addition to system require - ments review and preliminary design review), comparing them to other estimates available from the project and reconciling significant differences. 0

OCR for page 30
 KEY PROBLEMS AND SOLUTIONS NASA project staff generally estimate mission costs using detailed engineering analyses of labor and material requirements, vendor quotes, subcontractor bids, and the like. Non-advocate independent cost estimates in NASA are generally parametric cost estimates using statistical cost-estimating relationships based on historical relation - ships among cost and technical and programmatic variables (mass, power, complexity, and so on). In both cases, mission cost estimates are created by summing costs at lower levels of a project’s work breakdown structure (WBS) to obtain total project costs. Parametric cost models rely on observations rather than opinion, are an excellent tool for answering what-if questions quickly, and provide statistically sound information about the confidence level of cost estimates. In contrast, the process used within NASA to generate cost estimates based on detailed engineer- ing assessments does not provide a statistical confidence level and, in retrospect, has generally been less accurate than parametric cost models have been.1 Parametric cost models implicitly assume that the cost of current projects will be influenced by the same factors that influenced the cost of past missions, which generally seems to be the case with NASA Earth and space science missions. Parametric cost models automatically include allowances for many of the technical and programmatic problems that NASA development projects encounter (e.g., schedule delays, requirements changes, unforeseen technology challenges, and so on) because the historic database used by the models is comprised of past projects that had these types of problems. Parametric cost models should be used as the primary source of cost estimates in Phase A, and they remain applicable in phases B, C, and D (NASA, 2008). Parametric cost models are particularly useful when the parameters used to describe the missions are relatively accurate representations of the as-flown mission. The accuracy of parametric cost models produced by NASA for Earth and space science missions could be improved, for example, by better processes for (1) generating realistic system data early in the process and (2) validating models to improve their accuracy. Uncertainty in the lower-level WBS cost elements is inevitable. The uncertainties can be thought of as ranges or probability distributions. Most WBS element cost probability distributions are skewed to the high side—in cost, there is more room for cost growth than cost reduction. In such positively skewed distributions, the cost associated with the peak of the cost probability distribution (the “most likely value” or mode) is less than the median value (where there is a 50 percent expectation that actual costs will be less than the estimate). Arithmetically summing the most likely values—which has been a common practice throughout NASA history—leads to a total mission cost estimate that is less than the median. In practice, summing the most likely values generally leads to a total cost estimate that has only a 20 to 30 percent chance of being sufficient; meaning that there is a 70 to 80 percent probability that actual costs will exceed the estimate. This is a situation that pretty well describes the history of NASA cost outcomes. For example, the final cost of 83 percent of the 40 missions assessed in Primary Reference 1 exceeded their initial costs (including reserves). Beginning with the implementation of NASA Procedural Requirements (NPR) 7120.5C in 2005 (NASA, 2005), NASA required projects to statistically sum WBS costs using Monte Carlo or other mathematical techniques in order to obtain total project costs with higher confidence levels. This policy was further amplified with NPR 7120.5D (NASA, 2007) and NASA Policy Directive (NPD) 1000.5 (NASA, 2009). For NASA science missions, NPD 1000.5 requires cost estimates for individual projects to be calculated at a confidence level of 50 percent or higher, as necessary to ensure that programs are budgeted so that there is at least a 70 percent confidence level that the program budget will be sufficient for all of the missions included therein. 2 NPD 1000.5 also stipulates that programs and projects should perform statistical analyses necessary to ensure that the joint cost and schedule confidence levels achieve or exceed these thresholds. The committee endorses the current practice of funding programs at the 70 percent confidence level (thereby allowing sufficient reserves to be held at both the project and program levels). However, only projects that have 1 A detailed list of the strengths and weaknesses of various cost-estimating methods appears in NASA, 008 NASA Cost Estimating Hand- book, Washington, D.C. 2 For programs with a single large mission, this means that the mission must be budgeted to the 70 percent confidence level. However, for programs with multiple missions, mathematical analysis of current cost-estimating methods demonstrates that a program can achieve a 70 confidence level even though budgets for the missions included in that program have a (slightly) lower confidence level. The more missions in the program, the larger the difference in confidence levels can be.

OCR for page 30
 CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS recently entered implementation comply with all of these requirements. If the policy works well, projects complet - ing development in the coming years will have much improved cost outcomes. The primary references have shown that missions often expand beyond their initial scope (as defined at SRR and PDR). For example, the 10 missions assessed by Primary Reference 5 demonstrated average increases in spacecraft mass and power requirements of 43 percent and 42 percent, respectively. NASA’s cost models are kept up to date with cost and technical data from the latest projects that have flown. However, there may still be substantial uncertainty in the design of a new project in terms of the readiness level of proposed technologies, realistic mass and power requirements, design heritage, and so on. Early in a mis - sion, it is common to make optimistic assumptions, which means that missions are often proposed and accepted with initial design margins that are significantly smaller than historical experience would dictate. For example, Primary Reference 8 notes that most small spacecraft have very small design margins to save cost. NASA could reduce cost and schedule growth by establishing design rules and other guidelines with minimum required levels of reserves in mass, power, and other key design parameters. Reserves should reflect a more realistic assessment of software and hardware technical heritage (or lack thereof) and other historical experience. Cost and schedule estimates should be based on these risk adjusted designs. Inputs to cost models could be improved by using the NASA Cost Analysis Data Requirement (CADRe) database to help define model parameters at various project milestones (e.g., how much growth should be expected after PDR, in terms of mass, power, complexity, and so on). In addition, parametric cost estimators should have access to independent and experienced systems engineers to obtain more realistic inputs to the models. Cost Growth Methodology Finding. Measurement of Cost Growth. The measurement of cost growth has been inconsistent across pro- grams, NASA centers, and congressional appropriation and oversight bodies. Recommendation. Measurement of Cost Growth. NASA, Congress, and the Office of Management and Budget should consistently use the same method to quantify and report cost. In particular, they should use as the baseline a life-cycle cost estimate (that goes through the completion of prime mission operations) produced at preliminary design review. Although NASA Procedural Requirements (NPR) 7120.5D (NASA, 2007) delineates KDP C (PDR) as the first official cost commitment to stakeholders, there continues to be considerable confusion regarding what cost and schedule estimate should be considered the baseline from which future cost and schedule growth should be mea - sured. As detailed in Chapter 1, different reports and studies reach different conclusions regarding the magnitude of cost growth experienced by NASA Earth and space science missions, in part because they define baseline costs differently. The Government Accountability Office (GAO) and Congress generally considers the baseline to be the first time a mission appears as a budget line item in an appropriations bill. This is often before PDR. In addition to the project advocacy cost estimate presented at PDR, cost estimates are presented to various Program Manage - ment Councils and independent estimates are generated by subject matter experts and/or the Program Analysis and Evaluation Office. The content of these various estimates also differs—some include Phase A and B, some start with Phase C, some (but not all) include launch costs and/or mission operations, and some include NASA oversight and internal project management costs. These differences make it difficult to develop a clear understanding of trends in cost and schedule growth. However, as noted in Chapter 1, despite the inconsistencies noted in historical cost and schedule growth data, cost and schedule growth are clearly a problem, and urgent action should be initiated to reduce cost and schedule growth in parallel with efforts to avoid similar inconsistencies in the future.

OCR for page 30
 KEY PROBLEMS AND SOLUTIONS DEvELOPMENT PROCESS Management of Announcement of Opportunity Missions and Directed Missions Finding. Differences between announcement of opportunity (AO) missions and directed missions are as follows: • The impact of cost growth in AO missions, which are managed within a mission budget line (e.g., Dis - covery), is limited to other missions within the line. To a much larger degree, cost growth in larger, “flagship” missions has the potential to diminish NASA’s Earth and space science enterprise as a whole. • AO missions generally are selected at a very low level of system maturity and are very vulnerable to technology and instrument development problems. Recommendation. Management of Large, Directed Missions. NASA headquarters’ project oversight func- tion should pay particular attention to the cost and schedule of its larger missions (total cost on the order of $500 million or more), especially directed missions (which form a single line item). Recommendation. Management of Announcement of Opportunity (AO) Missions. NASA should continue to emphasize science in the AO mission selection process, while revising the AO mission selection process to allocate a larger percentage of project funds for risk reduction and improved cost estimation prior to final selection. Recommendation. Incentives. NASA should ensure that proposal selection and project management processes include incentives for program managers, project managers, and principal investigators to establish realistic cost estimates and minimize or avoid cost growth at every phase of the mission life cycle, for both directed missions and announcement of opportunity missions. As noted in Chapter 1, the primary references indicate that extensive cost growth exists in many—but not all—NASA Earth and space science missions. In addition, cost and schedule growth have made it more difficult to accomplish those missions. For a variety of reasons (e.g., inherent optimism, excessive technical and schedule risk, and faulty assumptions about the ease of adapting heritage technologies and systems), initial estimates are typically too low and highly uncertain. Neither the primary references nor this committee conclude that deficiencies in the qualification or expertise of NASA project managers or PIs have been a significant cause of cost growth of NASA Earth and space science missions as a whole. Even so, a project manager or PI who is personally determined to control costs can be of great assistance in avoiding cost growth. People and organizations tend to optimize their behavior based on the environment in which they operate, and so it is important to motivate and reward vigilance in accurately predict - ing and controlling costs. However, the differing nature and goals of directed and AO missions calls for different management approaches. In particular, the impact of cost growth in AO missions that are managed within a mission budget line (e.g., Discovery) is limited to other missions within the line. To a much larger degree, cost growth in larger, “flagship” missions has the potential to diminish NASA’s Earth and space science enterprise as a whole. Prospective PIs for AO missions compete with other PIs proposing different missions within selected science areas with cost caps and schedule constraints defined by the AO. These constraints are intended to limit the scope and complexity of the proposed missions. However, experience to date has shown that the potential science return is the predominant factor in selection of AO missions; cost realism is given relatively little weight as long as the proposed cost is within the cost cap. AO missions tend to have a smaller scope, less complexity, and less spacecraft mass than directed missions do. As a result, the average cost and schedule for AO missions is less than for directed missions. However, the selection process for AO missions is designed to solicit mission concepts with a much lower level of technological and system maturity for mission instruments and spacecraft than is the case for directed missions. This increases

OCR for page 30
 CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS TABLE 2.1 Average Cost and Schedule Growth (in Excess of Reserves) for Missions from Primary Reference 1 Average Average Cost Initial Schedule Number of Initial Cost Growth Cost Schedule Growth Schedule Missions (million $) (million $) Growth (%) (months) (months) Growth (%) All 0 missions Directed 18 296 44 15 60 13 21 AO 22 90 23 25 44 10 23  missions with largest cost growth Directed 9 362 89 25 64 14 22 AO 5 127 77 60 49 22 45 technological risks that must be overcome during the developmental process. As a result, AO missions tend to have higher percentage cost and schedule growth than do directed missions, but this is offset by the lower cost and shorter schedules they begin with. For example, Primary Reference 1 analyzed the cost growth (in excess of reserves) for 40 missions, and the results are charted in Figures 1.3a, b, and c and summarized in Table 2.1. As shown, on average, cost growth for AO missions was higher than for directed missions (AO, 25 percent; directed, 15 percent). However, this is more than offset by the higher average cost of directed missions, so that directed missions, on average, had a higher absolute cost growth (AO, $23 million; directed, $44 million). Among the 14 missions with the most cost growth (which accounted for 92 percent of the total cost growth, in absolute terms, for these 40 missions), the 5 AO mis - sions had an average cost growth that was much higher than for 9 directed missions (AO, 60 percent; directed, 25 percent). Even so, because the average cost for the high-growth directed missions was almost three times the average cost for the high-growth AO missions, the absolute value of the cost growth of the high-growth directed missions remained higher than for the high-growth AO missions (AO, $77 million; directed, $89 million). Simi - larly, directed missions, on average, had longer schedules than AO missions had. For the full set of 40 missions, however, AO and directed missions had approximately the same schedule growth, both in terms of percentage and in absolute terms. For the missions with high cost growth, on the other hand, AO missions had more schedule growth than directed missions had in terms of percentage (AO, 45 percent; directed, 22 percent) and in absolute terms (AO, 22 months; directed, 14 months). The primary references have identified key reasons for cost and schedule growth that go beyond initial under- estimates. Prior to CDR, important factors are unstable budgets, design evolution leading to increased complexity, excessive reviews, and difficulty in maturing new technologies. Following CDR, where most cost and schedule growth occurs, the most troublesome factors are associated with late delivery of instruments, hardware and soft - ware integration and testing, and launch vehicle availability and cost. These issues are more readily addressed with directed missions where the strategic and project planning processes are more likely to assure that risks associated with technology development, instrument development, and design maturity are sufficiently retired prior to PDR. AO missions are selected at a very low level of system maturity (typically after expending only 1 to 2 percent of total mission costs), and they are more susceptible to technology and instrument development problems. NASA manages the Discovery, Explorer, Mars Scout, New Frontiers, and other similar science programs as separate line items at NASA headquarters and at the various centers. As a result, the impact of cost growth in individual projects is accommodated within the line item, typically by delaying less mature projects. This is a reasonable approach as long as the cost growth of each individual AO mission can be accommodated within the overall budget for its respective science program; the alternative of attempting to force each AO mission to stay within its original, often highly optimistic budget estimate is more problematic. Managing AO missions primarily by program line item rather than by mission can thereby accomplish the overall goal of limiting cost growth in the NASA science budget while still achieving valuable science missions. A similar argument does not hold for the directed missions, which are much more expensive and each of which forms a single line item. In particular, the impact of a large cost overrun in these missions tends to be much more severe, with the potential to impact many other missions within the SMD, and other directorates as well.

OCR for page 30
 KEY PROBLEMS AND SOLUTIONS Cost growth of large (expensive) missions clearly creates a much larger budgetary problem and has a much larger impact on other missions. Of the 40 missions in Primary Reference 1, 92 percent of the total cost growth was accounted for by just 14 missions. In addition, 8 of the 10 missions with the highest initial cost estimates were among the set of 14 with the most cost growth (in absolute terms). Directed missions come in all sizes. In Primary Reference 1, 4 of the 10 smallest missions were directed. However, most large missions are directed missions. In Primary Reference 1, the 9 largest missions (in terms of initial cost estimates) were all directed; the most expensive AO mission comes in at number 10. Therefore, one way to help reduce overall cost growth would be for NASA to monitor the most expensive directed missions more closely than other missions. When assessing trends for large missions compared to small, the results are similar to the above comparison of directed and AO missions, perhaps because almost all of the most expensive missions are directed. Of the 40 missions in Primary Reference 1, the 10 with the highest initial cost had a cost growth of 15 percent (in excess of reserves), compared to 48 percent for the 10 smallest missions. Nonetheless, the 10 missions with the largest initial costs had total cost growth of $649 million, dwarfing the total cost growth of the 10 smallest missions ($210 million). In fact, the 10 largest missions accounted for more than half of the total cost growth. The extent to which a relatively small number of missions run up the total cost growth of NASA Earth and space science missions would be even worse if some missions currently in development, such as the Mars Science Laboratory (cost growth of approximately $660 million) and the James Webb Space Telescope (cost growth of approximately $1.5 billion), were included (GAO, 2010.)3 Given the dominant role that large missions play in determining total cost growth of NASA’s Earth and space science missions, it would be prudent to pay special attention to missions whose planned cost exceeds a specified threshold. The average initial cost of the 9 most expensive directed missions in Primary Reference 1 was $474 mil - lion and the initial cost for number 9 (the TRMM mission) was $253 million. However, these costs do not include the cost of launch, mission operations, data analysis, or inflation (these missions were developed and launched mostly in the 1990s). For directed missions, cost realism should be emphasized from the start, with special empha - sis given to missions with costs on the order of $500 million or more. This should begin with the decadal survey process and continue with disciplined program and project management: technology development and validation should precede the commitment to a given subsystem; instruments should be thoroughly tested before defining the spacecraft interfaces; PDR and CDR should be rigorous; and viable descope options should be identified in all project phases (and implemented as necessary to hold the line on cost and schedule growth). Thus, for NASA to be an effective, cost-conscious agency, every effort should be made to complete directed missions within cost, on schedule, and with the expected performance. For AO missions, cost growth could be addressed by changes to the AO selection process. Currently, the NASA AO selection process has two steps. During Step 1, an AO is released to solicit proposals for new mission concepts addressing questions of fundamental importance to the science community. At this point, proposing institutions (with no funding from NASA) will spend less than 1 percent of the stated cost cap on defining requirements, developing the concept, and maturing the required technologies to prepare a proposal. During Step 1, the emphasis is on science, because the initial selection of two or three proposals for Step 2 is primarily on the basis of value 3The GAO recently released a report assessing the cost and schedule growth of selected NASA Earth and Space Science projects currently in implementation (GAO, 2010). The average development cost growth of 10 projects that had been in the implementation phase for sev - eral years was $121 million, corresponding to 18.7 percent of their total initial budgets. However, one of these projects, a directed mission known as the Mars Science Laboratory (MSL), has had a cost growth of $662.4 million, corresponding to 68.4 percent of the initial cost. Interestingly, the summary table in the GAO study credits the James Webb Space Telescope (JWST) with zero cost growth, because in 2009 the mission was rebaselined, and the mission remains within the new baseline cost that was established at that time. However, the GAO study also reports that the JWST was replanned in 2006 after a cost growth of $1 billion, and costs increased another $500 million when the program was rebaselined in 2009. This reaffirms the potential for directed missions to have absolute cost growth (in dollars) that far exceeds the cost growth of AO missions even in situations where the average cost growth of directed and AO missions is similar. The cost growth of the MSL and JWST missions (totaling more than $2 billion) could have entirely funded a great many AO missions, and the accompanying slip in MSL’s launch date has severely impacted other Mars missions already in development as well as planned missions in the next decade. Thus, while it is certainly important to try to control cost growth on all missions, it is absolutely crucial to control cost growth on large directed missions, because when their costs get out of control, the potential impact is enormous.

OCR for page 30
 CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS and quality of the science being proposed. During Step 2, NASA provides supplemental funding to the initial winners to further define the mission concept, technology, and so on. Still, even by the point of final downselect to a single mission concept, less than 2 percent of the stated cost cap will have been spent. To be selected for an AO mission, a PI must propose a mission that will generate as much new scientific information as possible, and there is little or no penalty (in terms of the proposal evaluation process) for unrealistic cost estimates as long as the estimate falls within the cost “cap.” In addition, NASA makes only a small financial investment in helping PIs who reach Step 2 to refine their requirements, costs, and schedule as well as to reduce risk by maturing key technologies. This lack of funds early on in the process leads to overly optimistic cost esti - mates because little effort can be expended to identify and retire risks. Furthermore, the cost-capped mission model encourages PIs and their contractors to provide unrealistic, over-optimistic cost estimates and incentives to control costs after selection are ineffective. The result is poor cost performance. In addition, ongoing and planned missions suffer as they are slowed down to make funds available to cover overruns of earlier projects. These problems could be alleviated by a revised AO selection process that postpones final commitment to execute a selected mission until a reliable cost estimate can be prepared as follows: 1. Continue to select missions based primarily on science return but recognize that the “cost cap” of the mis - sion is better described as a cost target. This change in philosophical approach recognizes that AO missions seldom come in at or below their cost “cap” because of the competitive nature of the process, inadequate funds early in the process for technology maturation and risk reduction, and the difficulty of estimating the cost of immature system concepts for new one-of-a-kind missions. Why call it a cost “cap” when initial cost estimates are so uncertain and the cost growth of AO missions is generally accommodated, one way or the other? The Vegetation Canopy Lidar mission and the Full-sky Astrometric Mapping Explorer mission were cancelled because of cost increases, but this rarely occurs. 2. After downselecting a single mission, instead of proceeding with Phase B, as is the current practice, con - tinue with an extended Phase A. Phase A is the best opportunity to improve the accuracy of the mission concept and cost estimates (NRC, 2006). During this extended Phase A, NASA should provide sufficient funding (up to 5 percent of target cost) to reduce risk, improve cost estimates and associated cost risks, and identify potential descopes. Risk reduction should focus on technology development of the highest risk elements of the proposed approach, including development of test hardware as appropriate. Descopes would be exercised at the end of the risk reduction phase to lower the proposed cost closer to the cost target, if necessary. The amount allocated to this task should be enough to reduce risk by a significant amount, but it should be small enough that the effort could be terminated without undue concern about wasting funds. 3. At the end of the extended Phase A, develop an independent cost estimate and assess the technological maturity of the high-risk elements of the proposed approach. This would ensure that there is a good understanding of the residual risk and realism of the cost estimate prior to the final confirmation of the selected mission. Before confirming the proposed mission for development, the PI should demonstrate—and NASA should concur—that (1) the mission is affordable (relative to the cost target); (2) it has a realistic cost estimate; (3) the technology needed by the high-risk elements is adequately mature; and (4) the budget includes sufficient cost reserves. If a mission fails to meet these four criteria, then necessary corrective action should be taken (e.g., by continuing the extended Phase A to address residual questions and concerns) or the proposed mission should be terminated; in no case should NASA allow AO missions to proceed into Phase B without meeting these criteria. Concurrent with efforts to improve management processes for AO and directed missions, NASA should establish incentives for PIs, project managers, and program managers to minimize costs and avoid cost growth throughout mission life, while still meeting or exceeding minimum mission requirements, including science return and mission duration. For example, NASA might encourage mission managers to minimize cost growth by estab - lishing a policy that some percentage of unexpended reserves would routinely be made available for extended mission operations.

OCR for page 30
 KEY PROBLEMS AND SOLUTIONS Technology and Instrument Development Technology Development Finding. Technology Development. The limited time and resources available in phases A and B to mature new technology and solidify system design parameters contribute to cost growth through higher risk and unrealistic cost estimates. Recommendation. Technology Development. NASA should increase the emphasis in phases A and B on technology development, risk reduction, and realism of cost estimates. A recurring theme in the primary references is technical and programmatic uncertainty at the beginning of a project. Not enough time or resources are available in phases A and B to mature new technologies or to solidify principal system design parameters.4 NASA NPR 7120.5D requires that “during formulation, the project establishes performance metrics, explores the full range of implementation options, defines an affordable project concept to meet requirements specified in the Program Plan, develops needed technologies, and develops and documents the project plan” (NASA 2007, Section 2.3.4). Despite these procedural requirements, the primary references iden - tify an ongoing need to improve technical and programmatic definition at the beginning of a project. This would require more time and funding for Phase A and Phase B, especially for complex projects, to allow more time for development of technology, baseline costs, funding profiles, and the overall implementation plan. Instrument Development Finding. Instrument Development. Delays and cost increases for instrument development are pervasive and impact a large number of missions. Recommendation. Instrument Development. NASA should initiate instrument development well in advance of starting other project elements and establish a robust instrument technology development effort relevant to all classes of Earth and space science missions to strengthen and sustain the nation’s instrument development capability. Recommendation. Decadal Surveys. NASA should ensure that guidance regarding the development of instru- ments and other technologies is included in decadal surveys and other strategic planning efforts. In particular, future decadal surveys should prioritize science mission areas that could be addressed by future announcements of opportunity and the instruments needed to carry out those missions. Under NASA sponsorship, the National Research Council now performs decadal surveys for all SMD scientific disciplines, providing retrospective and forward-looking assessments of status and opportunities as well as recom - mendations for scientific and programmatic priorities for future investments. These surveys are broadly based and widely respected. To further improve the quality and utility of the surveys, a workshop on decadal science strategy surveys was held in November 2006 “… to review lessons learned from the most recent surveys, and to identify potential approaches for future surveys that can enhance their realism, utility, and endurance” (NRC, 2007). The workshop addressed several issues that are germane to the current discussion: • When a large, flagship mission encounters significant cost growth, it has dramatic impact on SMD’s bal - anced portfolio of large, medium, and small projects. • Estimates of program cost and technology maturity have often proved to be overly optimistic. 4 Program phases are described in Chapter 1. See Figure 1.1.

OCR for page 30
8 CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS • Instrument development is a leading contributor to cost risk, so that programs for reducing risk related to instruments are especially important. Instrument development problems were also identified as a major cause of cost growth in many of the primary references. Of particular importance are instrument development problems caused, for example, by instrument designs that lack technical details and/or fail to identify technical challenges. Shrinkage of the U.S. industrial base supporting space system development is exacerbating problems with the development of new instruments, particularly because Earth and space science missions generally require special purpose, one-of-a-kind compo - nents. It is not necessary to complete the development of new instruments before the missions that would use such instruments are initiated. However, the committee concurs with Primary References 1 and 3, which recommend increasing support for early instrument development to increase technical maturity and reduce risk. The commit - tee concurs that NASA should (1) establish a robust instrument technology development program and (2) initiate instrument development well in advance of starting other program elements, especially in cases where there is a distinct separation between the payload and spacecraft bus with well understood interfaces. Currently, NASA supports several programs for early instrument development in selected program areas: the Mars Instrument Development Program, the Planetary Instrument Development Program, and the Instrument Incubator Program for Earth science. These programs seem to be quite successful and could serve as models for additional and more robust efforts to meet the needs of all Earth and space science mission areas for new instru - ments and instrument technologies. Cost growth caused by problems with the development of instruments and other advanced spacecraft tech - nologies could also be addressed by expanding the scope of future decadal surveys to include instrument technol - ogy development. The statements of task for future surveys (and other strategic planning efforts) should ask for recommendations concerning the development of future instrument concepts and other technologies of particular importance to the next generation of science missions. The scope of these efforts should not be limited to directed missions. In addition, decadal surveys should also prioritize science mission areas that could be addressed by future AOs and the instruments needed to carry out these missions. As noted above, AO mission concepts are generally selected with a much lower level of instrument technological maturity than for directed missions, and develop - ment of instrument technology has been a particular problem for some AO missions. Identifying key priorities and investing in the technological needs of both directed and AO missions would make the best use of limited instrument development funds. Major Reviews Finding. External Project Reviews. NASA has increased the size and number of external project reviews to the point that some reviews are counterproductive. The large number of non-consensus reviews exacerbates this problem. Recommendation. External Project Reviews. NASA should reassess its approach to external project reviews to ensure that (1) the value added by each review outweighs the cost (in time and resources) that it places on projects; (2) the number and the size of reviews are appropriate given the size of the project; and (3) major reviews, such as preliminary design review and critical design review, occur only when specified success criteria are likely to be met. The primary references, other reports (see Box 2.1), as well as presentations to the committee by industrial repre- sentatives and NASA program personnel, alluded to the disruptive effect of an excessive number of external reviews. Primary Reference 2 noted that “large review teams and frequent reviews are too much—particularly for small missions” (p. 70). For example, the current (external) Standing Review Board (SRB) approach has been extended downward to all missions having a baseline cost of $250 million or more; this encompasses most missions. In addition, apparently because of conflict of interest issues that arose in NASA’s Constellation program, many boards now conduct non-consensus reviews. Simply stated, if a board includes any members who are not govern -

OCR for page 30
 KEY PROBLEMS AND SOLUTIONS BOX 2.1 Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs The National Research Council’s 2009 report Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs1 addressed many problems and issues similar to those facing NASA project managers. This study focused on concerns about the growing number of program and techni- cal reviews of Air Force programs. The benefit of these reviews was sometimes outweighed by the time that program staff spent supporting them instead of tending to program execution. The resulting report did not single out specific reviews for modification or elimination. Rather, it focused on ways in which to improve the overall management of the review process as follows: • Improving front-end planning. The Air Force Senior Acquisition Executive should establish a process to plan, coordinate, and execute reviews at all levels of the organization. Lack of effective coordination and a multiplicity of informal pre-reviews resulted in significant costs in both time and money for program managers. In addition, to ensure that the review remains well focused, the report recommended that objec- tives, metrics, and success criteria be provided to the program manager well in advance of the review, with a follow-up report addressing each of these areas. • Better synchronizing of the reviews. By focusing more on milestones and decision points and seeing that reviews are properly timed based on the readiness of the program to move forward, there should be a reduction in pre-reviews, cutbacks in costs, and fewer schedule delays. • Ensuring a clear basis of need. Before any new review is put in place, the senior acquisition ex- ecutive should compare its objectives with existing reviews to see if those might be modified, perhaps by broadening the review group. • Seeing that appropriate subject matter experts participate. The absence of reviewers with critical skill sets significantly reduces their effectiveness. • Clearly documenting and disseminating the output of the review. Documenting the outcomes of reviews and providing lessons learned should support a focus on results and foster an environment of continuous improvement. Implementing the above recommendations was expected to afford the Air Force Senior Acquisition Executive greater control over the review process and reduce the costs and time associated with oversight. Inasmuch as NASA project and management officials encounter similar issues, NASA should consider taking similar action, as appropriate and relevant to NASA’s operations. 1 NRC, Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs, The National Academies Press, Washington, D.C., 2009. ment employees, the chair is not allowed to drive the board to a consensus position. Rather, the chair is expected to assimilate the various points of view of the board members in a single document that represents his/her own opinion, augmented by highlighting each member’s individual views. This can be a difficult task, because delibera - tions intended to achieve a consensus viewpoint are not allowed. Additionally, each member of a non-consensus board writes an individual report that is appended to the chair’s report. Project staff are then expected to respond to the comments of all board members, even though the comments may be inconsistent or even contradictory. It may be difficult to meet the requirements associated with holding consensus reviews, but holding a small number of consensus reviews is likely to be more valuable—and more cost-effective—than is the current practice of holding a large number of non-consensus reviews. Another option for reducing the number of reviews would be to make greater use of subject matter experts as in-plant monitors in lieu of some reviews.

OCR for page 30
0 CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS Other troubling aspects of NASA’s current approach to conducting reviews include the following: • Review boards have grown in size, sometimes to the point of outgrowing existing conference facilities, and program offices have difficulty finding new facilities large enough to accommodate the large number of participants. With some relatively small projects, the number of reviewers sometimes exceeds the size of the project staff. • The number of reviews has increased also. For example, SRBs are conducted at the mission level. In preparation for each SRB, a series of Integrated Independent Review Team Reviews is conducted for each major element of the mission (e.g., spacecraft, ground system, and each instrument). Currently, a relatively simple mission with a spacecraft, a ground system, and three instruments can expect to have more than 30 external independent reviews during its life cycle, in addition to the internal reviews conducted by NASA staff. This large number of reviews constitutes a major distraction that adds significant cost. • Given the importance of cost and schedule growth, SRBs are now charged with assessing programmatic as well as technical issues. However, it is difficult for one body to accomplish both of these tasks efficiently and effectively. • Large numbers of reviews can diffuse responsibility and accountability, creating an environment where NASA senior managers can become dependent on review teams with many outside members who sometimes do not understand NASA, the field center in question, and/or the mission being reviewed. NPR 1720.5 is quite specific about the composition, scheduling, and intent of project reviews, and it calls for cost and schedule to be reviewed as part of PDR and CDR. Well-conducted reviews allow program personnel and the sponsoring organization to identify problem areas and required mitigation steps. However, the involvement of large numbers of reviewers who are unfamiliar with the mission is not conducive to project progress. Also, if a mission has not progressed far enough for a review to accomplish its intended purpose, the review is a wasted effort. Major reviews would be more effective in reducing cost and schedule growth if NASA consistently exercised greater care in the conduct of major reviews by assuring that missions are ready for review, programmatically and technologically, and that the review board is appropriately selected and well prepared to conduct a given review. Proceeding with a review when a project is not ready or the review board is poorly constituted is likely to increase cost and schedule growth in subsequent project phases. Launch vehicles Finding. Launch Vehicles. Problems with the procurement of launch vehicles and launch services are a sig - nificant source of cost growth. Recommendation. Launch Vehicles. Prior to preliminary design review, NASA should minimize mission- unique launch site processing requirements. NASA should also select the launch vehicle with appropriate margins as early as possible and minimize changes in launch vehicles. One of the primary references identifies launch service issues and delays in launch readiness as the primary external factor influencing cost growth of NASA missions. Specific factors include increases in the cost of expend - able launch vehicles, vendor issues such as strikes, weather-related issues at the launch site, problems with launch site facility capabilities or range availability, and delays in the availability of a given launch vehicle. In addition, if a mission is required to change launch vehicles, the costs can be substantial. Differences in launch vehicle interface requirements can require substantial changes to spacecraft design and testing requirements. These problems could be minimized by taking a more disciplined approach to launch that fully recognizes the potential cost of setting unique requirements for launch site processing and/or changing launch vehicles. COMPREHENSIvE, INTEGRATED STRATEGy FOR COST AND SCHEDuLE CONTROL Finding. Comprehensive, Integrated Cost Containment Strategy. Recent changes by NASA in the develop- ment and management of Earth and space science missions are promising. These changes include budgeting

OCR for page 30
 KEY PROBLEMS AND SOLUTIONS programs to the 70 percent confidence level and specifying that decadal surveys include independent cost estimates. However, it is too early to assess the effectiveness of these actions, and NASA has not taken the important step of developing a comprehensive, integrated strategy. Recommendation. Comprehensive, Integrated Cost Containment Strategy. NASA should develop an inte- grated, comprehensive strategy to contain cost and schedule growth and enable more frequent science oppor- tunities than would be possible in an environment with large cost growth. This strategy should include recent changes that NASA has already implemented as well as other actions recommended in this report. Internal Factors A consistent approach for defining cost and schedule growth and applying rigorous independent parametric cost and schedule estimates is necessary for a clear and consistent understanding of how individual projects are performing. Cost estimates become more accurate as risks and uncertainties are reduced through the maturation of critical technologies for instruments and other key systems, subsystems, and components. An increased emphasis on technology development in phases A and B will help missions avoid the cost and schedule growth—and the false expectations—that arise from sometimes optimistic assumptions about how easy it will be to develop new technology for a particular mission. The selective application of reviews that are tailored to add value to each mission and rigorously comply with NASA Space Flight Program and Project Management Requirements (NPR 7120.5D; NASA, 2007) would provide the necessary insight into mission status and highlight issues that may require special attention. In addition, larger missions (with costs on the order of $500 million or more) deserve special attention because cost and schedule growth of these missions can have a major impact on the stability of NASA’s overall Earth and space science mission portfolio and schedule. Although cost growth in smaller mis - sions only rarely has the same kind of effect, a proactive effort to improve the cost and schedule performance of smaller missions is also encouraged to conserve resources and improve confidence in NASA’s plans and project management capabilities. As discussed in this study and summarized in the recommendations and supporting text above, the elements of the recommended cost and schedule strategy should include the following internal agency actions: • Standardize cost baseline definition. • Improve independent cost estimating capability. Adopt and report using parametrically developed and experience-based cost estimates. • Increase the emphasis on early development of technologies for instruments and other spacecraft systems, within existing missions and as independent research activities. • Tighten use of the provisions of NPR 7120.5 regarding reviews and planning documentation. • Provide concrete incentives for program managers, project managers, and PIs to establish realistic cost estimates and minimize or avoid cost growth at every phase of mission life for both directed and AO missions. • Focus NASA headquarters’ project oversight function on the cost and schedule of its larger missions (costs on the order $500 million or more), particularly those which form a single line item. In addition to the issues discussed previously in this study, the effectiveness of the recommended comprehen - sive cost containment strategy would be enhanced by addressing key workforce, infrastructural, and organizational issues. To be successful, NASA must have access to a pool of talented personnel and key infrastructure, internally and externally, in national laboratories, universities, and industry. With constrained financial resources and ambi - tious plans for the future, optimizing the use of these assets is increasingly important. NASA may wish to consider opportunities to improve the effectiveness of its center structure by enhancing centers of excellence and sustaining redundant activities and capabilities only when necessary, for example, to maintain a competition of ideas or an assured national capability in critical areas. The recommendations contained in this report, especially the call for a comprehensive, integrated strategy for controlling cost and schedule growth, would enable NASA to conduct more frequent Earth and space science mis -

OCR for page 30
 CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS sions, to enhance accountability, and to build confidence in NASA’s Earth and space science endeavors, internally and within its external constituencies. External Factors The primary references conclude that most cost and schedule growth is caused by internal factors that NASA has the ability to control. However, external factors also contribute to cost and schedule growth. The recom - mended comprehensive strategy to control cost and schedule growth should address key external factors, such as the following: • Industrial base. Consolidation within the shrinking U.S. space industrial base, which is largely dependent on the U.S. national security and NASA budgets, has reduced the number of competitors for NASA (and DOD) missions (CSIS, 2008). The limited, occasional demand for some technologies and systems that are critical to Earth and space science missions can result in a shortage of suppliers that makes it more difficult and costly for NASA to develop advanced instruments and subsystems for new missions. To improve the nation’s space technology capabil- ity, particularly in instrument development, NASA and other government technology organizations could leverage technology development across U.S. government space activities while also engaging in the national discussion on industrial base issues, especially with regard to export controls that are limiting the global competitiveness of the U.S. space industry and promoting the development of foreign competitors in areas traditionally dominated by U.S. industry (AFRL, 2007). • Workforce. As noted in the report Rising Above the Gathering Storm, “an educated, innovative, motivated workforce—human capital—is the most precious resource of any country” in the world (NAS-NAE-IOM, 2007, p. 30). That study recommended making the United States the most attractive setting in the world in which to study and perform research. Earth and space science missions clearly need many critical skills that can only be developed through a combination of advanced education and work experience. Critical shortages are already developing in some areas, such as program management, systems engineering, and software development capabilities (AFRL, 2007). Any reduction in key workforce skill areas would be a potentially serious problem. NASA should continue and, as appropriate, increase its support and development of the United States’ future science and engineering workforce through scholarships, fellowships, and internships. • Launch vehicles. Implementation of a standard set of launch-vehicle-to-payload interfaces for use by all U.S. government agencies, as well as more standard launch vehicle production lines, would likely reduce mission cost and schedule and increase the probability of mission success. Greater standardization would also help avoid the sometimes substantial cost of modifying systems when launch vehicles are changed during the development of a particular mission. Collaborative efforts by NASA and other agencies to develop a coordinated strategy for access to space, as well as related technologies and vehicles, would help assure reliable, timely, and affordable access to space. • Funding Stability. As program execution becomes more disciplined and cost and schedule growth is reduced, it will become increasingly important to maintain a funding profile that is consistent with the established, baselined program. A concerted effort by individual NASA missions, NASA’s Earth and space science programs, OMB, and Congress to improve budget stability will maximize the programmatic efficiency and scientific results produced by NASA. Strategic Benefits NASA sets the strategic direction of its Earth and space science programs using decadal surveys, the SMD science plan, and supporting road maps. A comprehensive, integrated approach to control cost and schedule growth is also essential. The primary references include dozens of specific causes, dozens of specific recommendations, and dozens of findings concerning this problem (see Table 1.6 and Appendix C). The primary references are generally con - sistent and comprehensive, and so the individual causes of cost growth and the necessary correction action are

OCR for page 30
 KEY PROBLEMS AND SOLUTIONS not a mystery. However, rather than simply picking and choosing from among these long lists of causes, findings, and recommendations, development of an integrated strategy offers the best chance that future actions will work in concert to minimize or eliminate cost and schedule growth. Internally, an integrated cost containment strategy would improve the definition of baseline costs and enhance the utility of NASA’s independent cost-estimating capabilities. Early development of technologies and more effec - tive program reviews would improve the ability to identify and effectively manage risks and uncertainties. Exter- nally, NASA has the opportunity to collaborate with other federal agencies, OMB, and Congress to sustain and improve critical capabilities and expertise in the industrial base and the nation’s science and engineering workforce, to address cost and schedule risk associated with launch vehicles, and to improve funding stability. Successful implementation of a comprehensive, integrated strategy for cost and schedule growth of NASA Earth and space science missions will benefit both NASA and the nation, while enabling NASA to more efficiently and effectively carry out these critical missions.