Summary
STUDY BACKGROUND
Cost growth in Earth and space science missions conducted by the Science Mission Directorate (SMD) of the National Aeronautics and Space Administration (NASA) is a longstanding problem with a wide variety of interrelated causes. To address this concern, the NASA Authorization Act of 2008 (P.L. 110-422) directed the NASA administrator to sponsor an “independent external assessment to identify the primary causes of cost growth in the large-, medium-, and small-sized Earth and space science spacecraft mission classes, and make recommendations as to what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s science spacecraft mission programs.” NASA subsequently requested that the National Research Council (NRC) conduct a study to:
-
Review the body of existing studies related to NASA space and Earth science missions and identify their key causes of cost growth and strategies for mitigating cost growth;
-
Assess whether those key causes remain applicable in the current environment and identify any new major causes; and
-
Evaluate effectiveness of current and planned NASA cost growth mitigation strategies and, as appropriate, recommend new strategies to ensure frequent mission opportunities.
As part of this effort, NASA also asked the NRC to “note what differences, if any, exist with regard to Earth science compared with space science missions.”
COST GROWTH—MAGNITUDE AND CAUSES
NASA identified 10 cost studies and related analyses that this study uses as its primary references (listed in the References chapter and in Table 1.1). The committee generally concurs with the consensus viewpoints expressed in these studies as a whole, but in some areas, the studies reached different conclusions. For example, the prior studies calculated values for average cost growth ranging from 23 percent to 77 percent. Different studies reach different conclusions because they examine different sets of missions and calculate cost growth based on different criteria. By definition, cost growth is a relative measure reflecting comparison of an initial estimate of mission costs against costs actually incurred at a later time. But studies use initial estimates made at different points in
mission life cycles (see Figure S.1), as well as cost estimates that cover different phases of mission life cycles. For example, some studies consider only development costs (up to but not including launch), but other studies consider all costs through the end of each mission.
In general, the earlier the initial estimate, the more the cost will grow. In addition, including a larger share of the later phases of a mission (such as launch, operations, and analysis of data collected by a mission) increases the total cost assigned to each mission and the absolute value of the cost growth (in dollars). These differences make it very difficult to derive a single, reliable value for the average cost growth of NASA Earth and space science missions on the basis of previous studies.
The primary references also indicate that most cost growth occurs after critical design review. This implies that the required level of cost reserves remains substantial, even late in the development process. In addition, a relatively small number of missions cause most of the total cost growth. For one large set of 40 missions, 92 percent of the total cost growth (in dollars) was caused by only 14 missions (one-third of the total number). Conversely, the 26 missions with the least cost growth (two-thirds of the total number) accounted for only 8 percent of the total cost growth (see Figure S.2).
The primary references identify a wide range of factors that contribute to cost and schedule growth of NASA Earth and space science missions. The most commonly identified factors are the following:
-
Overly optimistic and unrealistic initial cost estimates,
-
Project instability and funding issues,
-
Problems with development of instruments and other spacecraft technology, and
-
Launch service issues.
Additional factors identified in the primary references include schedule growth that leads to cost growth. Schedule growth and cost growth are well correlated because any problem that causes schedule growth contributes to and magnifies total mission cost growth. Furthermore, cost growth in one mission may induce organizational replanning that delays other missions in earlier stages of implementation, further amplifying overall cost growth. Effective implementation of a comprehensive, integrated cost containment strategy, as recommended herein, is the best way to address this problem.
COMPREHENSIVE, INTEGRATED STRATEGY FOR COST AND SCHEDULE CONTROL
NASA sets the strategic direction of its Earth and space science programs using decadal surveys, the SMD science plan, and supporting road maps. A comprehensive, integrated approach to cost and schedule growth is also essential.
The primary references identify dozens of specific causes, make dozens of specific recommendations, and include dozens of additional findings concerning cost growth. The primary references, as a whole, are generally consistent and comprehensive, and so the individual causes of cost growth and the necessary corrective actions are not a mystery. However, rather than simply picking and choosing from among the many suggested causes, findings, and recommendations, development of a comprehensive, integrated strategy offers the best chance that future actions will work in concert to minimize or eliminate cost and schedule growth. An effective strategy would substantially reduce cost growth (beyond reserves) on individual missions and programs so that whatever growth does occur is offset by other missions and programs completed for less than the budgeted amount. This approach would allow NASA to execute the Earth and space science mission portfolio for the appropriated budget. Achieving this goal will require NASA to address both internal and external factors.
Internally, a comprehensive, integrated cost containment strategy would improve the definition of baseline
costs and enhance the utility of NASA’s independent cost-estimating capabilities. Early development of technologies and more effective program reviews would improve the ability to identify and effectively manage risks and uncertainties. Externally, NASA has the opportunity to collaborate with other federal agencies, the Office of Management and Budget, and Congress to sustain and improve critical capabilities and expertise in the industrial base and the nation’s science and engineering workforce; to address cost and schedule risk associated with launch vehicles; and to improve funding stability.
Successful implementation of a comprehensive, integrated strategy to control cost and schedule growth of NASA Earth and space science missions would benefit both NASA and the nation, while enabling NASA to more efficiently and effectively carry out these critical missions.
Finding. Comprehensive, Integrated Cost Containment Strategy. Recent changes by NASA in the development and management of Earth and space science missions are promising. These changes include budgeting programs to the 70 percent confidence level1 and specifying that decadal surveys include independent cost estimates. However, it is too early to assess the effectiveness of these actions, and NASA has not taken the important step of developing a comprehensive, integrated strategy.
Recommendation. Comprehensive, Integrated Cost Containment Strategy. NASA should develop a comprehensive, integrated strategy to contain cost and schedule growth and enable more frequent science opportunities. This strategy should include recent changes that NASA has already implemented as well as other actions recommended in this report.
KEY PROBLEMS
In addition to developing a comprehensive, integrated cost containment strategy, and as detailed below, NASA should address specific issues related to cost realism and the development process for Earth and space science missions.
Cost Realism
Cost Estimates
NASA project staff generally estimate mission costs using detailed engineering analyses of labor and material requirements, vendor quotes, subcontractor bids, and the like. Non-advocate independent cost estimates in NASA are generally parametric cost estimates using statistical cost-estimating relationships based on historical relationships among cost and technical and programmatic variables (mass, power, complexity, and so on). In both cases, mission cost estimates are created by summing costs at lower levels of a project’s work breakdown structure to obtain total project costs. Parametric cost models rely on observations rather than opinion, are an excellent tool for answering “what-if” questions quickly, and provide statistically sound information about the confidence level of cost estimates. In contrast, the process used within NASA to generate cost estimates on the basis of detailed engineering assessments does not provide a statistical confidence level and, in retrospect, has generally been less accurate than parametric cost models in estimating the cost of NASA Earth and space science missions.2
A project manager or principal investigator who is personally determined to control costs can be of great assistance in avoiding cost growth. People and organizations tend to optimize their behavior based on the environment in which they operate. Unfortunately, instead of motivating and rewarding vigilance in accurately predicting and controlling costs, the current system incentivizes overly optimistic expectations regarding cost and schedule.
For example, competitive pressures encourage (overly) optimistic assessments of the cost and schedule impacts of addressing uncertainties and overcoming potential problems. As a result, initial cost estimates generally are quite optimistic, underestimating final costs by a sizable amount, and that optimism sometimes persists well into the development process.
Recommendation. Independent Cost Estimates. NASA should strengthen the role of its independent cost-estimating function by
-
Expanding and improving NASA’s ability to conduct parametric cost estimates, and
-
Obtaining independent parametric cost estimates at critical design review (in addition to system requirements review and preliminary design review), comparing them to other estimates available from the project and reconciling significant differences.
Cost Growth Methodology
The measurement of cost growth has been inconsistent across programs, NASA centers, and Congress. The Government Accountability Office and Congress generally consider the baseline to be the first time a mission appears as a budget line item in an appropriations bill, which is often before preliminary design review. The contents of NASA estimates also differ—some estimates include Phase A and B, some start with Phase C, some (but not all) include launch costs and/or mission operations, and some include NASA oversight and internal project management costs. These differences make it difficult to develop a clear understanding of trends in cost and schedule growth.
Recommendation. Measurement of Cost Growth. NASA, Congress, and the Office of Management and Budget should consistently use the same method to quantify and report cost. In particular, they should use as the baseline a life-cycle cost estimate (that goes through the completion of prime mission operations) produced at preliminary design review.
Development Process
Management of Announcement of Opportunity Missions and Directed Missions
NASA implements two separate and distinct classes of Earth and space science missions—announcement of opportunity (AO) missions and directed missions. NASA headquarters competitively selects AO missions from proposals submitted in response to periodic AOs by teams led by a principal investigator (PI), who is commonly affiliated with a university but may work in industry or for NASA. NASA headquarters determines the scientific goals and requirements for directed missions, which are sometimes referred to as facility class missions or flagship missions. Headquarters then directs a particular NASA center, usually Goddard Space Flight Center or the Jet Propulsion Laboratory, to implement the mission.
The differing nature and goals of directed and AO missions call for different management approaches. AO missions are on average much smaller than directed missions are, and the impact of cost growth in AO missions, which are managed within a mission budget line (e.g., Discovery), is limited to other missions within the line. Flagship missions, however, are typically much larger than AO missions are, and so cost growth in these missions has a much greater potential to diminish NASA’s Earth and space science enterprise as a whole.
Recommendation. Management of Large, Directed Missions. NASA headquarters’ project oversight function should pay particular attention to the cost and schedule of its larger missions (total cost on the order of $500 million or more), especially directed missions (which form a single line item).
Recommendation. Management of Announcement of Opportunity (AO) Missions. NASA should continue to emphasize science in the AO mission selection process, while revising the AO mission selection process to allocate a larger percentage of project funds for risk reduction and improved cost estimation prior to final selection.
Recommendation. Incentives. NASA should ensure that proposal selection and project management processes include incentives for program managers, project managers, and principal investigators to establish realistic cost estimates and minimize or avoid cost growth at every phase of the mission life cycle, for both directed missions and announcement of opportunity missions.
Technology and Instrument Development
NASA Procedural Requirements (NPR) 7120.5, NASA Space Flight Program and Project Management Requirements, requires that “during formulation, the project establishes performance metrics, explores the full range of implementation options, defines an affordable project concept to meet requirements specified in the Program Plan, develops needed technologies, and develops and documents the project plan” (NASA, 2007, Section 2.3.4). However, despite these requirements, the primary references identify an ongoing need to improve technical and programmatic definition at the beginning of a project. The limited time and resources typically available in phases A and B to mature new technology and solidify system design parameters contribute to cost growth through higher risk and unrealistic cost estimates.
Instrument technology is particularly important because Earth and space science missions generally require special-purpose, one-of-a-kind components. Delays and cost increases for instrument development are pervasive and impact a large number of missions. This problem is exacerbated by shrinkage of the U.S. industrial base that supports space system development.
Recommendation. Technology Development. NASA should increase the emphasis in phases A and B on technology development, risk reduction, and realism of cost estimates.
Recommendation. Instrument Development. NASA should initiate instrument development well in advance of starting other project elements and establish a robust instrument technology development effort relevant to all classes of Earth and space science missions to strengthen and sustain the nation’s instrument development capability.
Recommendation. Decadal Surveys. NASA should ensure that guidance regarding the development of instruments and other technologies is included in decadal surveys and other strategic planning efforts. In particular, future decadal surveys should prioritize science mission areas that could be addressed by future announcements of opportunity and the instruments needed to carry out those missions.
Major Reviews
NASA has increased the size and number of external project reviews to the point that some reviews are counterproductive and disruptive, especially for small missions. Large numbers of reviews diffuse responsibility and accountability, creating an environment where NASA senior managers can become dependent on review teams with many outside members who sometimes do not understand NASA, the field center in question, and/or the mission being reviewed. In addition, major reviews are sometimes conducted as scheduled even though a project may not have progressed as rapidly as expected and, as a result, cannot achieve the intended review criteria, programmatically and/or technologically.3
Recommendation. External Project Reviews. NASA should reassess its approach to external project reviews to ensure that (1) the value added by each review outweighs the cost (in time and resources) that it places on projects; (2) the number and the size of reviews are appropriate given the size of the project; and (3) major reviews, such as preliminary design review and critical design review, occur only when specified success criteria are likely to be met.
Launch Vehicles
Problems with the procurement of launch vehicles and launch services are a significant source of cost growth. Specific factors include increases in the cost of expendable launch vehicles, vendor issues such as strikes, weather-related issues at the launch site, problems with launch-site-facility capabilities, and delays in the availability of a given launch vehicle. In addition, if a mission is required to change launch vehicles, the costs can be substantial.
Recommendation. Launch Vehicles. Prior to preliminary design review, NASA should minimize mission-unique launch site processing requirements. NASA should also select the launch vehicle with appropriate margins as early as possible and minimize changes in launch vehicles.
DIFFERENCES BETWEEN EARTH AND SPACE SCIENCE MISSIONS
Different classes of missions face different challenges. Earth science missions typically have more complex, more costly, and more massive instruments than do space science missions, because Earth science missions also have more stringent requirements in terms of pointing accuracy, resolution, stability, and so on, although astrophysics missions also have stringent pointing requirements, and planetary spacecraft and instrument technology must be able to survive long cruise phases and radiation environments that are sometimes quite extreme. Space science missions that leave Earth orbit have greater incentives to minimize spacecraft mass and power, and the average cost and average spacecraft mass of these missions are lower than those for Earth science missions. However, the size of the cost growth of Earth and space science missions has been comparable. Both Earth and space science missions have shown good correlation between (1) instrument schedule growth and instrument cost growth, (2) instrument cost/schedule growth and mission cost/schedule growth, and (3) the absolute costs of instruments and instrument complexity.