B
Implementing the CATE Process
IMPLEMENTING THE CATE PROCESS
In the 2008 NASA Authorization Act, Congress mandated a “lifecycle cost and technical readiness” review of proposed NASA projects. Such reviews have become an important part of the decadal survey process. The cost and technical evaluation (CATE) process described in this appendix has been developed for the decadal surveys in order to provide an independent, standardized process to produce a figure-of-merit for technical and cost risk that aids in science prioritization. CATE, as implemented by the Aerospace Corporation in partnership with the decadal survey committee of the National Academies of Sciences, Engineering, and Medicine, is based on historical and continuously updated and validated databases and methods. It is designed to evaluate diverse mission concepts of varying design maturity.
The CATE methodology incorporates cost growth based on the historical record and the state of the design or development. CATE assesses the technical risk and readiness. It then monetizes the technical risks into potential design growth and associated cost and schedule threats. The objective of the CATE process is to perform a cost and technical risk analysis for a set of concepts that may have a broad range of maturity, and to assure that the analysis is consistent, fair, and informed by historical data.
Nature of System Development
The CATE is intended to forecast the potential cost of the final system as built, one that has typically undergone multiple iterations and that is often very different from the one initially conceived. A CATE estimate is therefore often more conservative than a conventional independent cost estimate because it accounts for design evolution and changes due to typical and unforeseen changes and challenges.
Missions evolve in cost and complexity, as notionally illustrated in Figure B.1. Experience shows that in most cases the cost of a space mission is not well understood until it has completed its preliminary design review (PDR). Even then, unexpected mass, cost, and schedule growth ordinarily occurs during the later phases of design and development. Historically, such changes have resulted in cost growth. For example, 20 space missions from 2008 to 2010 showed 37 percent and 41 percent mass and power growth, respectively, from Phase B to launch. The
FIGURE B.1 Cost and complexity evolution during the life cycle of a project.
associated cost and schedule growth was 56 percent and 38 percent respectively.1 Historical data like these were used to develop the CATE heuristics used to predict potential design evolution and, ultimately, potential cost growth.
Typically, concepts evaluated via the CATE process are early in their life cycle, and therefore liable to undergo significant design changes. The CATE process attempts to capture this potential growth by defining figures of merit based on cost and complexity. Fair evaluation requires that the relative maturity of concepts be taken into account. Some pre-Phase A concepts are more mature than others because greater resources have been devoted to their formulation. The CATE allows comparison of one concept of low maturity relative to another that has undergone several iterations and review. Also, recognizing that the organization that proposed or studied the mission may not be the one that ultimately implements it, the CATE process makes no assumptions about the eventual choice of an organization to implement the mission.
Application of CATE within the Decadal Survey Process
The CATE process operates within the committee and panel structure of a decadal survey. The 2011 planetary science decadal survey2 (Planetary2011) provides a good case study. Before concepts were submitted to CATE, significant trades were considered. The initial steps were led by a representative from the survey’s appropriate science panel, who considered the science value or science returned for an initial cost estimate provided by the advocates. A key aspect of the CATE process is that multiple interactions took place between the survey committee and the CATE contractor. In some cases, the Aerospace Corporation was directed to consider alternative solutions identified by the committee that might lower cost and risk, while maintaining science return.
_______________
1 C.W. Freaner, R.E. Bitten, and D.L. Emmons, “Inherent Optimism in Early Conceptual Designs and Its Effect on Cost and Schedule Growth: An Update,” 2010 NASA Program Management Challenge, February 2010.
2 NRC, Vision and Voyages for Planetary Science in the Decade 2013-2022, The National Academies Press, Washington, D.C., 2011.
The major elements of the standard CATE process and interactions with the CATE contractor and survey committee are shown in Figure B.2. The survey committee receives concepts through a variety of mechanisms, including requests for information (RFIs), as discussed in Chapter 2. The survey committee then provides a subset of those concepts to the CATE team for evaluation. The evaluation includes a technical assessment of risk, notional or estimated cost, and schedule for the mission. The CATE team performs its assessment of the concepts in parallel. In some cases, multiple iterations are required based on review and feedback from the panels and/or survey committee.
Each concept is examined in terms of heritage—for example, whether something similar has recently been developed or fielded—as much as possible making use of analogies with already-built systems with demonstrated costs and understood performance. Likewise, schedule evaluation makes use of analogies to determine expected development time. Historical data from previous analogous NASA missions, properly adjusted, are used in the CATE schedule analysis to gauge the realism of the proposed durations of the development phases. The time to critical mission reviews (e.g., preliminary design review and critical design review) and the time required for integration and testing are evaluated for each mission concept based on comparison with appropriate historical experience. In this way, CATE provides an independent estimate of cost and schedule of new concepts anchored by data from previously built hardware.
The CATE Process
A CATE consists of both a technical evaluation and a cost evaluation. The evaluation of technical risk and maturity, summarized in Figure B.3, focuses on the identification of the most important technical risks to achieving
FIGURE B.2 Decadal cost and technical evaluation (CATE) process showing key elements and interactions with the Academies and the survey committee.
FIGURE B.3 The technical evaluation of the cost and technical evaluation (CATE) methodology includes a technology readiness evaluation and a peer-review of technical risks and rankings.
the required mission performance and stated science objectives. For the 2013 solar and space physics (heliophysics) decadal survey3 (Helio2013) and Planetary2011, the technical evaluation consisted of two parts: (1) the proposed concept compared with previously developed systems for which the performance is known and (2) launch-vehicle “mass-to-orbit” capability—if the spacecraft weighs too much, it could exceed the launch-vehicle capability and require either a reduction in mass and/or a more capable, more costly launch vehicle.
Deviations from the current state of the art, as well as system complexity, operational complexity, and integration concerns associated with the use of “heritage components,” are all identified. Technical maturity and the need for mission-specific technology development are evaluated by the CATE technical team to assess readiness levels of key technologies and hardware. During the assessment, the technical team interacts with the cost and schedule teams so that technical risks can be translated into schedule and cost risk. The key output is a technical risk rating: low, medium, or high.
The cost evaluation illustrated in Figure B.4 uses analogies and models to estimate cost for major elements, estimate reserves based on historical growth, and estimate design and schedule growth. The CATE employs multiple methods and databases representing past space systems such as proprietary models (e.g., Aerospace Small Satellite Cost Model) and space-industry standards (e.g., the NASA/Air Force Cost Model [NAFCOM]). The use of multiple methods such as analogies and standard cost models ensures that no one model or database biases the estimate. The use of system-level estimates and arriving at total estimated costs by statistically summing the costs of all individual work breakdown structure (WBS) elements ensures that elements are not omitted and that the system-level complexity is properly represented in the cost estimate. Tools such as the Complexity Based Risk Assessment (CoBRA) are used to cross-check cost and schedule estimates for internal consistency.
In an integrated fashion, the CATE process seeks to quantify the total “threats” to costs from schedule growth,
_______________
3 NRC, Solar and Space Physics: A Science for a Technological Society, The National Academies Press, Washington, D.C., 2013.
FIGURE B.4 The cost and technical evaluation (CATE) cost-estimating approach is a comprehensive, probabilistic assessment of potential mission cost and cost risk.
the costs of maturing technology, and from mass growth that would result in the need for a larger, more costly launch vehicle. The output is a probabilistic view of potential cost that envelopes the potential evolution of the system over time.
CATE Output
For all concepts, the CATE process estimates the cost increases associated with increased contingency mass and power, increased schedule, increased required launch vehicle capability, and other cost threats, depending on the concept maturity and specific risk assessment of a particular concept.
Traditional S-curves of cost probability versus cost are provided for each concept, with the project’s internal and/or external estimate compared to the CATE estimate at the 70th percentile (as set by the survey committee). To aid in the assessment of concept risk, independent schedule estimates are incorporated as part of the CATE cost estimate. This is especially useful for assessment of risk with respect to proposed mission development and execution timelines.
An example of CATE summary output is shown in Figure B.5 for the Jupiter Europa Orbiter (JEO) mission concept from Planetary2011.4 While the science was compelling and prioritized very highly, this particular mission concept was deemed unaffordable.
Summary
After its use in three decadal surveys, CATE has become a recognized and appreciated element of the survey process. CATE is useful for prioritizing elements of the program because it provides a standardized set of credible,
_______________
4 NRC, Vision and Voyages, 2011.
FIGURE B.5 Example of cost and technical evaluation (CATE) output including mission summary, key characteristics, risk assessment and cost-risk distribution. This information was used by panels and committee to assist in the prioritization pro - cess. SOURCE: National Research Council, Vision and Voyages for Planetary Science in the Decade 2014-2022, The National Academies Press, Washington, D.C., 2011.
objective, and independent information about cost, as well as technical and cost risk, for mission concepts. As discussed in Chapter 2, the interface with the CATE process has been customized for each survey. Although the discussion in this report has concerned the utility of CATE for evaluating space-mission concepts, attempts have been made to apply the CATE process ground-based systems and instruments as well, a likely issue for future surveys.5
A key value of CATE for the prioritization process is in understanding the “cost box” a particular concept fits within. The CATE information helps to start the discussion of what can be accomplished relative to a particular science question for a particular cost—even if the eventual implementation strategy turns out to be very different from what is assessed. The CATE process needs to be periodically reassessed and updated to ensure it remains both independent and true to its original goals.
_______________
5 Notably, the 2010 astronomy and astrophysics decadal survey (Astro2010) performed a CATE analysis on the Large Synoptic Survey Telescope and the two “Giant Segmented Mirror Telescope” concepts, the Thirty Meter Telescope (TMT) and the Giant Magellan Telescope (GMT). Despite the Aerospace Corporation’s lack of experience with these types of facilities, their analyses have been born out, including predicting the approximate amount of cost growth for both TMT and GMT.
