O

Independent Technical, Risk, and Cost Evaluation

The Astro2020 decadal survey evaluated the technical, programmatic, and execution risks of large ground- and space-based projects in numerous ways. This evaluation was directed by the relevant program panel, and included multiple layers of review by the panel, as well as an independent Technical, Risk, and Cost Evaluation (TRACE) performed by a contractor (Aerospace Corporation). This full level of independent review was undertaken for all National Science Foundation (NSF) Major Research Equipment and Facilities Construction (MREFC)–scale projects selected by the program panels to be of primary interest, as well as for some version of all of the NASA flagship concepts. (In the case of the Large ultraviolet (UV)/optical/infrared (IR) Surveyor [LUVOIR] and Habitable Exoplanet Observatory [HabEx], multiple implementations were studied, and the Panel on Electromagnetic Observations from Space 1 [EOS-1] selected those of primary interest.) For the NASA flagships, in addition to the project cost estimates, NASA performed a Large Mission Concept Independent Assessment Team (LCIT) evaluation, independent of the projects, which, in addition to the project estimates, was considered as input to the panels. The panels included subject-matter experts in technology, management, and instrumentation, and this expertise guided the process, providing key feedback and iterative steps between both the projects and the independent contractor. Information flow is captured in Figure O.1. The program panel process is documented in their reports (see Appendixes H–M) and summarized in the main text of the report. The purpose of this appendix is to describe the Aerospace Corporation’s TRACE process and how TRACE results were managed and considered by the survey.

The TRACE provided broad guidance to the program panels in the form of a forward-looking assessment of potential technical risks and likely cost and schedule boxes associated with a mission concept, but not definitive cost or schedule estimates.

The decadal independent cost and technical risk assessment process has substantially evolved and improved since its earliest application, as well as its implementation for the New Worlds, New Horizons (NWNH) 2010 astrophysics decadal survey.¹ The current methodology, TRACE, is anchored by successful implementations of its heritage risk analysis methodology, the cost and technical evaluation (CATE) pro-

___________________

¹ National Research Council, 2010, New Worlds, New Horizons in Astronomy and Astrophysics, The National Academies Press, Washington, DC, https://doi.org/10.17226/12951.

cess, for a total of four decadal surveys, including NWNH. It is also anchored in the processes developed in support of the independent estimating of both NASA and Department of Defense (DoD) projects. The details of the Aerospace Corporation process have been captured in multiple reports and presentations, including The Space Science Decadal Surveys: Lessons Learned and Best Practices² and “CATE Overview: Astro2020 Large Mission Concepts.”³

Highlights of the TRACE process included using Aerospace Corporation proprietary models and databases, validated through multiple processes, including comparisons to historical implementations, discussed below. Threats and risks were evaluated using Monte Carlo simulations. Each project was assessed independently, regardless of maturity, and brought to a baseline 70 percent probability of success through application of reserves for both cost and schedule to ensure that equal programmatic evaluation criteria could be applied.

In addition, NASA’s support of the development of high-quality mission concepts significantly assisted the TRACE. The Astrophysics Science and Technology Definition Teams (STDTs), in preparation for this decadal survey, provided well-developed, compelling, and executable mission concepts with high-fidelity estimates.

The Astro2020 decadal survey committee used best practices from NWNH and implemented lessons learned from other completed decadal surveys, as follows:4,5

___________________

² National Academies of Sciences, Engineering, and Medicine, 2015, The Space Science Decadal Surveys: Lessons Learned and Best Practices, The National Academies Press, Washington, DC.

³ D. Emmons, 2016, “Aerospace CATE Overview: Astro2020 Large Mission Concepts,” teleconference presentation to the LUVOIR Science and Technology Definition Team on September 20, 2016, https://asd.gsfc.nasa.gov/luvoir/events/events/telecons/2016-09-20/emmons.pptx.

⁴ National Research Council, 2013, Lessons Learned in Decadal Planning in Space Science: A Summary of a Workshop, The National Academies Press, Washington, DC.

⁵ National Academies of Sciences, Engineering, and Medicine, 2015, The Space Science Decadal Surveys: Lessons Learned and Best Practices, The National Academies Press, Washington, DC.

• Communicating, reviewing, and coordinating directly with program panels to improve the information available to the panels for critical decision-making.
• Using project-provided descoping or rescoping data to assess mission costs and technical risks.
• Scheduling to preclude TRACE from becoming the pacing schedule item and allowing for the TRACE results to further inform the steering committee deliberations and decision-making.
• Using varied budget and funding projections, nominal and optimistic, to frame project execution opportunities and challenges.
• Factoring costs of current and near-available launch vehicles into mission costs.
• Using experts, as needed, to augment the Aerospace Corporation independent assessment process, after gaps were identified.
• Identifying liaisons to serve as go-betweens for the panels, the committee, and the Aerospace Corporation’s TRACE team.
• Early identification of the role of TRACE in panel decisions and deliberations.
• Presentations of final TRACE results directly to the program panels.

The overall process is captured in Figure O.2. Science priorities were provided to the program panels as input to the evaluation process. An early assessment of project science, scope, technical risks, costs, possible descopes, and associated risks were used as input. The program panels used requests for information (RFIs) to gather data on projects using a two-tiered process. Projects of greater maturity were sent RFIs reflecting more detailed data (RFI1). Projects of lesser maturity were sent an alternative, lesser detailed RFI (RFI2).

Outcomes were presented to the program panels using standardized formats. Results included technical risk ratings, power and mass margins, and other risk data. Cost histograms were presented to compare costs developed using a range of independent cost estimating methods, including classical estimating methods, risk estimating programs (e.g., SEER, MICM, NICM, MOCET, FRISK, etc.⁶), and the contractor’s proprietary analytics suite. The key analogies and historical projects used in the analyses for the various design elements, subsystems, instruments, construction elements, and so on were also reviewed by the program panels to assess appropriate application and allow for programs of varied maturities to be brought to a consistent baseline. TRACE products provided to the program panels during the process flow are shown in Figure O.3.

Space concepts benefited from the extensive past program database available to the Aerospace Corporation, allowing for numerical analytical methods to be effectively and consistently applied across a range of concepts. However, the unique aspects of ground observatories continued to provide challenges to the TRACE. The Aerospace Corporation made significant efforts to improve the TRACE process for ground programs, including providing additional historical analogy data and updating their analyses to include current applicable facilities. Ground TRACE evaluation limitations included the exclusion of operations costs, focusing only on design and construction, and the exclusion of funding threats. All gaps were addressed by the program panels, engaging additional unpaid consultants where necessary to augment the database and evaluate program specifics.

___________________

⁶ A subset of classic estimating tools used by TRACE:

• SEER (Systems Evaluation and Estimation of Resources), a knowledge-based estimation cost evaluation and trade-off tool suite by Galorath, Inc.
• MICM (Multi-Variable Instrument Cost Model), by NASA Goddard Space Flight Center.
• NICM (NASA Instrument Cost Model), a probabilistic cost estimation tool developed by NASA containing a system model, subsystem model, and a database search engine.
• MOCET (Mission Operations Cost Estimation Tool), a Phase E estimation tool jointly developed by the Aerospace Corporation and the NASA Science Office for Mission Assessments.
• PCEC (Project Cost Estimating Capability), a parametric cost model developed and maintained by NASA.
• FRISK (Formal Risk Method), an analytical, rather than Monte Carlo–based, risk analysis method. Typical implementation includes establishing total cost distribution for a design based on WBS (work breakdown structure).

The TRACE processes forecasted final project costs, timelines, and critical paths, allowing “unknown unknowns,” or “threats,” to be captured, typically producing more conservative values—for example, manufacturing development and scale-up, increased schedule to reflect demonstrated development timelines, added costs to capture mass contingencies, increased costs, and time to reflect required launch vehicle capacity and availability, increased costs and schedule to capture historic execution and learning curves associated with specific technologies, and so on. Once overall costs and schedules were estimated, probabilistic assessments of cost probability and schedule probability were calculated and provided to the program panels using classic S-curves comparing project estimates to 70th percentile probabilities.

The role of the TRACE data as input to panel decision making is presented in the individual program panel reports.

The projected cost growth range for large space and ground mission concepts in real-year dollars is shown in Figure O.4. It should be emphasized that the “increased projected costs” in the figure reflects the differences between the TRACE total mission cost estimates at 70 percent probability and the project total mission cost estimates. The differences in projected growth assigned to ground versus space concepts were generally attributable to technology maturity gaps and the demonstrated cost and scope growth challenges faced by early-phase projects.

The impact of early-phase programmatic risk on anticipated program cost growth is shown in Figure O.5. (Increased technical risk is associated with higher projected cost growth.) An examination of the aggregate TRACE results, as a percentage of the total project value, were compared to the open technical risks. Every project that received a TRACE evaluation was assessed for technical risk by evaluating both mitigation plans and historical risks. Figures of merit (FOMs) were used to normalize the data and highlight relationships. FOM formulas are provided in Box O.1. In addition to the general trend of higher technical risks driving forecasted cost growth, a distinct difference was apparent between the inherent technical risks of the space mission concepts and the ground concepts.