National Academies Press: OpenBook

Pathways to Discovery in Astronomy and Astrophysics for the 2020s (2021)

Chapter: Appendix O: Independent Technical, Risk, and Cost Evaluation

« Previous: Appendix N: Report of the Panel on the State of the Profession and Societal Impacts
Suggested Citation:"Appendix O: Independent Technical, Risk, and Cost Evaluation." National Academies of Sciences, Engineering, and Medicine. 2021. Pathways to Discovery in Astronomy and Astrophysics for the 2020s. Washington, DC: The National Academies Press. doi: 10.17226/26141.
×
Page 539
Suggested Citation:"Appendix O: Independent Technical, Risk, and Cost Evaluation." National Academies of Sciences, Engineering, and Medicine. 2021. Pathways to Discovery in Astronomy and Astrophysics for the 2020s. Washington, DC: The National Academies Press. doi: 10.17226/26141.
×
Page 540
Suggested Citation:"Appendix O: Independent Technical, Risk, and Cost Evaluation." National Academies of Sciences, Engineering, and Medicine. 2021. Pathways to Discovery in Astronomy and Astrophysics for the 2020s. Washington, DC: The National Academies Press. doi: 10.17226/26141.
×
Page 541
Suggested Citation:"Appendix O: Independent Technical, Risk, and Cost Evaluation." National Academies of Sciences, Engineering, and Medicine. 2021. Pathways to Discovery in Astronomy and Astrophysics for the 2020s. Washington, DC: The National Academies Press. doi: 10.17226/26141.
×
Page 542
Suggested Citation:"Appendix O: Independent Technical, Risk, and Cost Evaluation." National Academies of Sciences, Engineering, and Medicine. 2021. Pathways to Discovery in Astronomy and Astrophysics for the 2020s. Washington, DC: The National Academies Press. doi: 10.17226/26141.
×
Page 543
Suggested Citation:"Appendix O: Independent Technical, Risk, and Cost Evaluation." National Academies of Sciences, Engineering, and Medicine. 2021. Pathways to Discovery in Astronomy and Astrophysics for the 2020s. Washington, DC: The National Academies Press. doi: 10.17226/26141.
×
Page 544

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

O Independent Technical, Risk, and Cost Evaluation The Astro2020 decadal survey evaluated the technical, programmatic and execution risks of large ground and space-based projects in numerous ways. This evaluation was directed by the relevant Program Panel, and included multiple layers of review by the panel, as well as an Independent Technical, Risk and Cost Evaluation (TRACE), performed by a contractor (The Aerospace Corporation). This full level of independent review was undertaken for all National Science Foundation (NSF) Major Research Equipment and Facilities Construction (MREFC)-scale projects selected by the Program Panels to be of primary interest, as well as for some version of all of the NASA flagship concepts (in the case of the Large UV/Optical/IR Surveyor (LUVOIR) and Habitable Exoplanet Observatory (HabEx), multiple implementations were studied, and the EOS-1 program panel selected those of primary interest). For the NASA flagships, in addition to the project cost estimates, NASA performed a Large Mission Concept Independent Assessment Team (LCIT) evaluation, independent of the projects, which, in addition to the project estimates, was considered as input to the panels. The panels included subject-matter experts in technology, management, and instrumentation, and this expertise guided the process, providing key feedback and iterative steps between both the projects and the independent contractor. Information flow is captured in Figure O.1. The Program Panel process is documented in the panel reports and summarized in the main text of the report. The purpose of this appendix is to describe the process through which the Aerospace Corporation’s independent TRACE evaluations were implemented and how they were managed and considered by the survey. The TRACE provided broad guidance to the Program Panels in the form of a forward-looking assessment of potential technical risks and likely cost and schedule boxes associated with a mission concept, but not definitive cost nor schedule estimates. FIGURE O.1 Relationship between projects, program panels, aerospace, and the steering committee. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION O-1

The decadal independent cost and technical risk assessment process has substantially evolved and improved since its earliest application, as well as its implementation for the New Worlds, New Horizons (NWNH) 2010 astrophysics decadal survey. The current methodology, TRACE, is anchored by successful implementations of its heritage risk analysis methodology, the Cost and Technical Evaluation (CATE) process, for a total of four decadal surveys, including NWNH. It is also anchored in the processes developed in support of the independent estimating of both NASA and Department of Defense projects. The details of the Aerospace Corporation process have been captured in multiple reports and presentations, including The Space Science Decadal Surveys-Lessons Learned and Best Practices1 and CATE Overview, Astro2020 Large Mission Concepts.2 Highlights of the TRACE process included using Aerospace Corporation proprietary models and databases, validated through multiple processes, including comparisons to historical implementations, discussed below. Threats and risks were evaluated using Monte Carlo simulations. Each project was assessed independently, regardless of maturity, and brought to a baseline 70 percent probability of success through application of reserves for both cost and schedule to assure equal programmatic evaluation criteria could be applied. In addition, NASA’s support of the development of high-quality mission concepts significantly assisted the TRACE. The Astrophysics Science and Technology Definition Teams (STDT) in preparation for this decadal survey provided well-developed, compelling, and executable mission concepts with high- fidelity estimates. The Astro2020 decadal survey committee used best practices from NWNH and implemented lessons learned from other completed decadal surveys.3,4  Communicating, reviewing, and coordinating directly with Program Panels to improve the information available to the panels for critical decision-making.  Using project-provided descoping or rescoping data to assess mission costs and technical risks.  Scheduling to preclude TRACE from becoming the pacing schedule item and allowing for the TRACE results to further inform the steering committee deliberations and decision-making.  Using varied budget and funding projections, nominal and optimistic, to frame project execution opportunities and challenges.  Factoring costs of current and near-available launch vehicles into mission costs.  Using experts, as needed, to augment the Aerospace Corporation independent assessment process, after gaps were identified.  Identifying liaisons to serve as go-betweens the panels, the committee, and the Aerospace Corporation’s TRACE team.  Early identification of the role of TRACE in panel decisions and deliberations.  Presentations of final TRACE results directly to the Program Panels. The overall process is captured in Figure O.2. Science priorities were provided to the Program Panels as input to the evaluation process. An early assessment of project science, scope, technical risks, costs, possible descopes, and associated risks were used as input. The Program Panels used requests for 1 National Academies of Sciences, Engineering, and Medicine (NASEM), 2015, The Space Science Decadal Surveys: Lessons Learned and Best Practices, The National Academies Press, Washington, D.C. 2  D. Emmons, 2016, “Aerospace CATE Overview: Astro2020 Large Mission Concepts,” teleconference presentation to the LUVOIR Science and Technology Definition Team on September 20, 2016, https://asd.gsfc.nasa.gov/luvoir/events/events/telecons/2016-09-20/emmons.pptx.  3 National Research Council, 2013, Lessons Learned in Decadal Planning in Space Science: A Summary of a Workshop, The National Academies Press, Washington, D.C. 4 NASEM, 2015, The Space Science Decadal Surveys: Lessons Learned and Best Practices, The National Academies Press, Washington, D.C. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION O-2

information (RFI) to gather data on projects using a two-tiered process. Projects of greater maturity were sent RFIs reflecting more detailed data (RFI1). Projects of lesser maturity were sent an alternate, lesser detailed RFI (RFI2). Outcomes were presented to the Program Panels using standardized formats. Results included technical risk ratings, power and mass margins, and other risk data. Cost histograms were presented to compare costs developed using a range of independent cost estimating methods, including classical estimating methods, risk estimating programs (e.g., SEER, MICM, NICM, MOCET, FRISK, etc.5), and the contractor’s proprietary analytics suite. The key analogies and historical projects used in the analyses for the various design elements, subsystems, instruments, construction elements, etc., were also reviewed by the Program Panels to assess appropriate application and allow for programs of varied maturities to be brought to a consistent baseline. TRACE products provided to the Program Panels during the process flow are shown in Figure O.3. FIGURE O.2 Technical Risk and Cost Evaluation (TRACE) process interactions. 5 A subset of classic estimating tools used by TRACE:  SEER (Systems Evaluation and Estimation of Resources), a knowledge-based estimation cost evaluation and trade-off tool suite by Galorath, Inc.  MICM (Multi-variable Instrument Cost Model), by NASA Goddard Space Flight Center.  NICM (NASA Instrument Cost Model), a probabilistic cost estimation tool developed by NASA containing a system model, subsystem model, and a database search engine.  MOCET (Mission Operations Cost Estimation Tool), a Phase E estimation tool jointly developed by The Aerospace Corporation and the NASA Science Office for Mission Assessments.  PCEC (Project Cost Estimating Capability), a parametric cost model developed and maintained by NASA.  FRISK (Formal RISK method), an analytical, rather than Monte Carlo–based, risk analysis method. Typical implementation includes establishing total cost distribution for a design based on WBS (Work Breakdown Structure). PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION O-3

FIGURE O.3 TRACE process and products. Space concepts benefitted from the extensive past program database available to Aerospace Corporation allowing for numerical analytical methods to be effectively and consistently applied across a range of concepts. However, the unique aspects of ground observatories continued to provide challenges to the TRACE. Aerospace made significant efforts to improve the TRACE process for ground programs including adding additional historical analogy data and updating their analyses to include current applicable facilities. Ground TRACE evaluations limitations included the exclusion of operations costs, focusing only on design and construction, and the exclusion of funding threats. All gaps were addressed by the Program Panels, engaging additional unpaid consultants where necessary to augment the database and evaluate program specifics. The TRACE processes forecasted final project costs, timelines, and critical paths, allowing “unknown unknowns,” or “threats,” to be captured, typically producing more conservative values, e.g., manufacturing development and scale-up, increased schedule to reflect demonstrated development timelines, added costs to capture mass contingencies, increased costs, and time to reflect required launch vehicle capacity and availability, increased costs and schedule to capture historic execution and learning curves associated with specific technologies, etc. Once overall costs and schedules were estimated, probabilistic assessments of cost probability and schedule probability were calculated and provided to the Program Panels using classic S-curves comparing project estimates to 70th percentile probabilities. The role of the TRACE data as input to panel decision making is presented in the individual Program Panel reports. The projected cost growth range for large space and ground mission concepts in real year dollars is shown in Figure O.4. It should be emphasized that the “Increased projected cost” in the figure reflects the differences between the TRACE total mission cost estimates at 70 percent probability and the project total mission cost estimates. The differences in projected growth assigned to ground versus space concepts were generally attributable to technology maturity gaps and the demonstrated cost and scope growth challenges faced by early phase projects. The impact of early phase programmatic risk on anticipated program cost growth is shown in Figure O.5. (Increased technical risk is associated with higher projected cost growth.) An examination of the aggregate TRACE results, as a percentage of the total project value, were compared to the open technical risks. Every project that received a TRACE evaluation was assessed for technical risk by evaluating both mitigation plans and historical risks. Figures of merit (FOM) were used to normalize the data and highlight relationships. FOM formulae are provided in Box O.1. In addition to the general trend PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION O-4

of higher technical risks driving forecasted cost growth, a distinct difference was apparent between the inherent technical risks of the space mission concepts and the ground concepts. FIGURE O.4 TRACE projected cost growth (70 percent probability level). FIGURE O.5 Correlation of TRACE “look forward” to technical risk. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION O-5

BOX O.1 Figures of Merit Calculations To assess the extent to which the calculated program execution threats (TRACE) were correlated to open technical risks across all programs, figures of merit (FOM) were developed. TRACE values were normalized as a percentage of project values: TRproject = ((TRcost × 100)/ Pcost) − 100, where  TRproject = TRACE percent increase over total project submitted cost  TRcost = TRACE estimate for complete project in real year dollars  Pcost = Project estimate for complete project in real year dollars As an example, if a project cost totaled $1,000 million in real year dollars and the TRACE 70 percent probability value was $1,100 million in real year dollars, TRproject = 10. Because the TRACE process captures and monetizes risks, all values of TRproject were positive. Technical risks were normalized using a FOM capturing the number of large open technical risk categories, the technology readiness level (TRL) of the open technology, and the time (in years) needed to mature the technology as submitted by the program: TechFOM = (Rhigh × (TRL3 × TRL3time + TRL4 × TRL4time + TRL5 × TRL5time + TRL6) + (Rmedium-high × (TRL3 × TRL3time + TRL4 × TRL4time + TRL5 × TRL5time + TRL6) + (Rmedium × (TRL3 × TRL3time + TRL4 × TRL4time + TRL5 × TRL5time + TRL6) + (Rmedium-low × (TRL3 × TRL3time + TRL4 × TRL4time + TRL5 × TRL5time + TRL6) + (Rlow × (TRL3 × TRL3time + TRL4 × TRL4time + TRL5 × TRL5time + TRL6), where  TechFOM = Technology Risk FOM  Rhigh, Rmedium-high, Rmedium, Rmedium-low, and Rlow = Technology risk rating assigned by Aerospace = Number of major program risks in this risk category × Risk multiple, where — High risk multiple = 5 — Medium-high risk multiple = 4 — Medium risk multiple = 3 — Medium-low risk multiple = 2 — Low risk multiple = 1  TRL3, TRL4, TRL5, and TRL6 = number of technology categories identified by the project at these TRL levels × TRL risk multiplier, where — TRL 3 multiplier = 9 — TRL 4 multiplier = 3 — TRL 5 multiplier = 2 — TRL 6 multiplier = 1  TRL3time, TRL4time, and TRL5time = total length of time, in years as defined by the projects for a technology to mature to the next level. As an example, if a project defined two major development areas, one currently at TRL 6 and rated a medium risk by Aerospace Corporation, and the other at TRL 3 and rated a medium-low risk by Aerospace Corporation, but requiring 2 years each to reach TRL 4, TRL 5, and TRL6, the project TechFOM = 61. PREPUBLICATION COPY – SUBJECT TO FURTHER EDITORIAL CORRECTION O-6

Next: Appendix P: Acronyms »
Pathways to Discovery in Astronomy and Astrophysics for the 2020s Get This Book
×
Buy Prepub | $104.00 Buy Paperback | $95.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

We live in a time of extraordinary discovery and progress in astronomy and astrophysics. The next decade will transform our understanding of the universe and humanity's place in it. Every decade the U.S. agencies that provide primary federal funding for astronomy and astrophysics request a survey to assess the status of, and opportunities for the Nation's efforts to forward our understanding of the cosmos. Pathways to Discovery in Astronomy and Astrophysics for the 2020s identifies the most compelling science goals and presents an ambitious program of ground- and space-based activities for future investment in the next decade and beyond. The decadal survey identifies three important science themes for the next decade aimed at investigating Earth-like extrasolar planets, the most energetic processes in the universe, and the evolution of galaxies. The Astro2020 report also recommends critical near-term actions to support the foundations of the profession as well as the technologies and tools needed to carry out the science.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!