E

Mission Development and Assessment Process

INTRODUCTION

This report follows the astronomy and astrophysics and the planetary science decadal survey reports as the third survey employing the cost and technical evaluation (CATE) process implemented by the Aerospace Corporation. CATE is an independent analytical approach for realistically assessing the expected cost and risk related to recommended initiatives that are at an early stage of formulation (typically pre-Phase-A). A unique attribute of the process is the ability to employ technical methods in combination with statistical methods to develop mission concepts based on their assessed complexity, risk, and lien factors. An essential driver to the CATE process is the requirement that all missions be treated using systematic and consistent methods in order to facilitate a uniform comparison of the relative cost, schedule, and risk of projects proposed for implementation. Secondary, but also essential, to the success of the process is the incorporation of iterative technical feedback from stakeholders, combined with a proven method for independent validation of the results. The successful implementation of these elements as an integrated process was fundamental to achieving confidence in the CATE process and consequent confidence in the assessment results, implementation assumptions, and decision rules applied to the recommended program.

The CATE process is the third leg of a science investigation triad following science mission implementation recommendations and an intermediate stage consisting of mission conceptual design activities. The flow of activities began with white paper submissions by the science community in November 2010, followed by concept evaluation and a mission design effort culminating with initial mission selection in April 2011. The subsequent iterative CATE process began in April and ended in September 2011.

As part of the CATE process, mission descoping options were introduced in recognition of the budget pressures NASA is likely to be facing over the next several years. A unique strength of the heliophysics discipline is the long experience of its investigators in doing exceptional science in a principal-investigator (PI)-led mode using small and medium Explorers. A key finding of the CATE process used in this study was that all of the proposed concepts were technically feasible at a reasonable cost. Furthermore, the descoped missions were of a scope, complexity, and cost that are consistent with successful mid-size PI-led missions executed for NASA’s Discovery program.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 365
E Mission Development and Assessment Process INTRODUCTION This report follows the astronomy and astrophysics and the planetary science decadal survey reports as the third survey employing the cost and technical evaluation (CATE) process implemented by the Aero- space Corporation. CATE is an independent analytical approach for realistically assessing the expected cost and risk related to recommended initiatives that are at an early stage of formulation (typically pre- Phase-A). A unique attribute of the process is the ability to employ technical methods in combination with statistical methods to develop mission concepts based on their assessed complexity, risk, and lien factors. An essential driver to the CATE process is the requirement that all missions be treated using systematic and consistent methods in order to facilitate a uniform comparison of the relative cost, schedule, and risk of projects proposed for implementation. Secondary, but also essential, to the success of the process is the incorporation of iterative technical feedback from stakeholders, combined with a proven method for independent validation of the results. The successful implementation of these elements as an integrated process was fundamental to achieving confidence in the CATE process and consequent confidence in the assessment results, implementation assumptions, and decision rules applied to the recommended program. The CATE process is the third leg of a science investigation triad following science mission imple- mentation recommendations and an intermediate stage consisting of mission conceptual design activities. The flow of activities began with white paper submissions by the science community in November 2010, followed by concept evaluation and a mission design effort culminating with initial mission selection in April 2011. The subsequent iterative CATE process began in April and ended in September 2011. As part of the CATE process, mission descoping options were introduced in recognition of the budget pressures NASA is likely to be facing over the next several years. A unique strength of the heliophysics discipline is the long experience of its investigators in doing exceptional science in a principal-investigator (PI)-led mode using small and medium Explorers. A key finding of the CATE process used in this study was that all of the proposed concepts were technically feasible at a reasonable cost. Furthermore, the descoped missions were of a scope, complexity, and cost that are consistent with successful mid-size PI-led missions executed for NASA’s Discovery program. 365

OCR for page 365
366 SOLAR AND SPACE PHYSICS: A SCIENCE FOR A TECHNOLOGICAL SOCIETY MISSION DESIGN ACTIVITY The initial science assessment and prioritization effort yielded 12 mission concepts, with 4 concepts put forward by each of the three science discipline panels. Each of the concepts were then organized and systematized using a standard design questionnaire intended to provide a relatively uniform set of design parameters as an input to the mission design process. The 12 mission designs were then produced over a 5-week period in February and March of 2011 at the Concept Design Center (CDC) of the Aerospace Corporation in El Segundo, California. The CDC team consisted of experts who were “firewalled” from the CATE process. Each mission was supported by a group of both shared and dedicated CDC team members, one or more members of the survey committee, and a “champion” designated by the science panels to represent each investigation concept. The cornerstone assumption for the mission design process was that cost-effective mission concepts demand a requirements-focused design combined with overall system simplicity. The general system design baseline resulting from these assumptions was a single-string (with selective redundancy in some cases) architecture using commercially available spacecraft bus designs similar in size to a medium Explorer. The bus structure and specific capabilities such as power and pointing were sized to accommodate the required science instrumentation. Mission-unique features were also incorporated to meet requirements for orbit and downlink, radiation hardness, magnetic cleanliness, deployable subsystems, dispensing functions, and de-orbit capability (if determined to be necessary). To constrain the system design and control cost, the system architecture was also scaled such that all missions were both sized and intended for orbits so as to be compatible with a launch on a medium-size launch vehicle or smaller. The largest launch vehicle required for any mission concept was the Falcon 9. The design process was intense but also effective at developing a realistic set of concepts consistent with the above requirements and constraints. The effort was well supported by all stakeholders in the process, with detailed standardized reports prepared for each of the resulting system designs. The quality and consistent nature of the reports allowed for a straightforward comparison of missions leading into the subsequent down-select process. The reports were also designed to be compatible with the input param- eters required by the CATE process. Following distribution of the mission design reports, the science panels performed an initial review and made recommendations to the committee for the CATE down-select from 12 to 6 missions. The rec- ommendations were based on science objectives, complexity factors, and relative cost assumptions. The following recommended concepts were presented by the science panels, endorsed by the committee, and moved forward into the first stage of the CATE process: 1. Dynamical Neutral Atmosphere-Ionosphere Coupling (DYNAMIC) mission, 2. Interstellar Mapping and Acceleration Probe (IMAP), 3. Solar Eruptive Events (SEE), 4. Magnetosphere-Ionosphere Source Term Energetics (MISTE), 5. Magnetosphere Energetics, Dynamics, and Ionospheric Coupling (MEDICI) mission, and 6. Geospace Dynamics Constellation (GDC). CATE PROCESS The CATE process has been discussed at length in previous reports and is based on the following principles, which have remained consistent between recent past surveys and this survey. The CATE flow

OCR for page 365
APPENDIX E 367 chart provided as Figure E.1 diagrams the end-to-end effort as described by these principles and executed for this report. • Use multiple methods and databases regarding past space systems so that no one model or database biases the results. • Use analogy-based estimating that anchors cost and schedule estimates to NASA systems that have already been built with a known cost and schedule. • Use system-level estimates as well as a build-up to system-level estimates by appropriately summing subsystem data so as not to underestimate system cost and complexity. • Use cross-checking tools, such as Complexity Based Risk Assessment (CoBRA), to cross-check cost and schedule estimates for internal consistency and risk assessment. • In an integrated fashion, quantify the total threats to costs from schedule growth, the costs of matur- ing technology, and the threat to costs owing to mass growth resulting in the need for a larger, more costly launch vehicle. The general CATE methodology for cost assessment is presented in Figure E.2. As with most cost assess- ment approaches, the CATE process uses mission analogies, growth metrics, and other relevant criteria applied to each work breakdown structure element that are derived from measured cost and schedule performance on past NASA (and selected Department of Defense) missions. Probabilistic methods are then employed using a triangle distribution to calculate the reserves needed to move the sum of the most Approved C onc epts P arallel E valuation 6 F F our P eptsanels 3 our P anels 2 Present P rovided by l E ac h F2our CP anels E ac h 2 to 7 C onc to 7 onc epts 1 2 to 7 C onc epts E ac h E valuate C onc ept Review Final NR C Multiple Iterations Results Implementation Development R ead Heritage R es pons es Risks Analogies Committee Maturity Technical Analogies Prioritization Models Cost Threats Present Reserve Review Initial Schedule Results Project Phases Critical Path Analogies Committee Feedback FIGURE E.1  Flow chart for the CATE process. SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. Figure E-1

OCR for page 365
368 SOLAR AND SPACE PHYSICS: A SCIENCE FOR A TECHNOLOGICAL SOCIETY Multiple Cost Estimates for Each WBS Element Triangular Distribution of Element Cost $25 Adjusted Analogy Model H Probability $20 Estimate M Cost (FY09$M) $15 L $10 Cost $5 L M H $0 Optimistic Most-Likely Pessimistic Estimate Estimate Estimate (+ DMF) Total Distribution is a Combination of Example Total Cost Distribution 100% Cost Distributions for WBS Elements 90% WBS Element 1 80% Sum of Cumulative Probability + 70% Most-likely 70th Percentile 60% Costs WBS Element 2 50% Initial Cost + 40% Reserves . 30% Estimate . . Total Cost Probability Distribution 20% + 10% 0% $1,000 $1,200 $1,400 $1,600 $1,800 $2,000 WBS Element N Estimated Cost (FY09$M) FIGURE E.2  CATE cost estimating methodology. SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. likely cases up to a 70th percentile point1 on the resulting S-curve describing the statistical distribution Figure E-2 for mission cost. The CATE methodology for schedule assessment shown in Figure E.3 is also analogy based, with a probabilistically derived 70th percentile output. Analogous missions based on mission class, technical simi- larities, and participating organizations are used with historical phase durations for key mission milestones. These durations are used to develop triangular distributions of estimated phase duration. A cumulative probability distribution of the total schedule from Phase-B through launch is then generated by combining the individual schedule distributions. As previously mentioned, a unique attribute of the CATE method is the ability to “grow forward” a program from a very early conceptual stage, where the threats and liens are often much larger than for mis- sions that are at a later stage of development. The dominant factors are most often the natural growth that occurs in mass and power requirements on a mission as its system design matures. Accommodating growth in these areas often requires a larger launch vehicle or other mission changes that drive up the final cost. A major area of cost uncertainty for the heliophysics CATE process was related to launch vehicle cost and availability. Launch vehicles are furnished by NASA and subject to NASA competitive procurement practices. Consequently, specific vehicle costs are unknown or could not be provided except in a broad cost range enveloping numerous vehicle types. As discussed above, smaller vehicles, beginning with the 1  The 70th percentile point represents the total cost (including reserves) within which there is a 70 percent likelihood that the mis- sion can be accomplished.

OCR for page 365
APPENDIX E 369 Multiple Estimates for Each Schedule Phase Triangular Distribution of Phase Duration 30 H Analogies PDR to CDR Duration (months) 25 Probability Aerospace 20 15 M 10 Duration L L M H 5 0 Optimistic Most-Likely Pessimistic Aero GALEX WMAP Kepler Spitzer GLAST Estimate Estimate Estimate Total Distribution is a Combination Example Schedule Distribution of Triangular Distributions for 100% Milestone Durations 90% Phase B start - PDR 80% + Project Cumulative Probability 70% Schedule 60% 70th Percentile PDR - CDR 50% + . 40% . 30% . Total Schedule Probability Distribution + 20% 10% 0% 40 50 60 70 80 90 100 110 120 PSR - Launch Estimated Development Time (months) FIGURE E.3  CATE schedule estimating methodology. SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. Taurus-XL 2XXX, Taurus-XL 3XXX, Taurus-II, and Falcon 9, were baselined and are capable of meeting all Figure E-3 of the mission design cases that were studied. The Falcon 1 and Athena were treated as variants of the Taurus-class vehicle and are considered cost-neutral substitutes for the majority of the applications. The Delta-II was not considered to be available at the time of the mission design activity but is also a viable substitute with respect to launch capability, although at an unknown cost. It is also unclear whether the Delta-II will retain launch capabilities at the Eastern Test Range launch facility. All vehicles (or comparable vehicles at comparable cost) were considered to be available later in the decade when the first missions would approach a state of launch readiness. The CATE process assumed the launch vehicle cost to be a pass-through based on an analysis using multiple publicly available sources. The cost range used in the process started at $60 million for the Taurus-XL 2XXX class and extended up to $125 million for the Falcon 9 in 2012 dollars. For the vehicles baselined in the design studies, the potential for cost growth and associated cost risk is considered low, although there is some risk related to availability of a particular vehicle type. MISSION COST RESULTS The cost comparison between the six mission concepts and associated descoped missions is shown in Figure E.4. For each mission there is a base cost resulting from the probabilistic cost analysis plus the ana- lyzed cost for mission-specific threats and liens. The corresponding schedule results are shown in Figure E.5.

OCR for page 365
370 SOLAR AND SPACE PHYSICS: A SCIENCE FOR A TECHNOLOGICAL SOCIETY Absolute Cost Comparison, $FY12 M $1,200 Threats $993 $1,000 CATE EsƟmate w/o Threats $819 $800 $738 $655 $614 $600 $522 $510 $526 $450 $400 $200 $- IMAP IMAP DYNAMIC SEE MISTE MEDICI MEDICI GDC GDC Descope Descope Descope Notes : 1) C AT E c os t es timates are at the 70 th perc entile FIGURE E.4  CATE 70th percentile cost results for missions, including descoped alternatives. Base cost plus threats and liens shown (in fiscal year (FY) 2012 $ million). SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. Figure E-4 Specific to this survey is the recognition that several large missions are already through formulation or in the late stages of formulation such that they will become operational in stages that envelop most of the coming decade. Given the evident funding challenge early in this decade, a frequent mission cadence via the implementation of small- and medium-class missions over the entire decade is considered more valuable to the community than the implementation of, at most, one additional large mission. This “simplified mission strategy” biased toward targeted executable investigations becomes clearer when examining Figure E.4 and Figure E.6, which represent the CATE assessment of cost and technical risk, respectively, for each of the six baseline missions plus the three descope options. It can be seen that the system approaches chosen for the missions have been successful at limiting the technical risk to the medium-low and medium categories, with a resulting payback in both affordability and cadence. The risk rankings are particularly illuminating when compared to previous surveys, in which most missions were ranked in the medium-high and high categories. A combined assessment of cost and technical risk leads to the conclusion that simplified mission con- cepts similar to IMAP, DYNAMIC, and the descoped MEDICI recommended by the survey committee fall into the range of PI-led missions that can be successfully executed within the scope of the Solar-Terrestrial Probes (STP) mission line. These missions, along with GDC as the recommended Living With a Star (LWS) mission, have their key parameters summarized in Boxes E.1 through E.4 (provided by the Aerospace Corporation under contract with the National Research Council).

OCR for page 365
APPENDIX E 371 Schedule Comparison Referenced to October 2017, months 80 70 CATE B-D Schedule EsƟmate 60 50 40 64 64 67 30 63 54 57 20 10 0 IMAP DYNAMIC SEE MISTE MEDICI GDC Note: GDC descope reduces CATE B-D by 2 months. IMAP and MEDICI unchanged by descopes. FIGURE E.5  CATE 70th percentile Phase-B through Phase-D schedule results. GDC descope reduces schedule by 2 months. IMAP and MEDICI descopes do not affect schedule. SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. Figure E-5 VALIDATION APPROACH The final step of the CATE process consisted of a series of validation reviews intended to ensure that the assumptions and corresponding cost and schedule results are based on defensible assumptions, are fair in their representation of the assessed missions, and are consistent with past missions of similar content and complexity. The reviews consisted of a mix of both internal and external reviews conducted by the Aerospace Corporation and by the committee, respectively. The CATE validation tool used for this survey and all past surveys is the CoBRA tool developed by the Aerospace Corporation. Figures E.7 and E.8 map the mission candidates onto cost and schedule grids for comparison with successful missions of varying complexity plus “anchor missions” considered to be similar in size and complexity. The results show excellent correlation between CATE results for mission concepts compared to suc- cessful missions and anchor missions. Therefore, the CATE results are considered compatible with expe- rience, with respect to both cost and schedule. As one would expect, the CATE missions representing a 70th-percentile distribution are generally above the mean. One interesting result, discussed in the next section, is that the heliospheric anchor missions are generally below the mean trend line.

OCR for page 365
372 SOLAR AND SPACE PHYSICS: A SCIENCE FOR A TECHNOLOGICAL SOCIETY Medium- Medium- Low Medium High Low High IMAP IMAP Des c ope DY NAMIC SEE MIS T E ME DIC I ME DIC I Des c ope G DC Des c ope G DC FIGURE E.6  CATE technical risk estimate. This estimate is provided to represent the level of technical risk resulting from the CATE assessment. Note that in the case of the IMAP and MEDICI investigations the descope resulted in a cost reduc- tion but no change in the technical risk posture. An advantage of using a standardized CATE process is that the metrics for evaluating technical risk are based on technical criteria that are consistent across programs within a single survey as well as across surveys from other disciplines. Note: Red indicates that the concept is likely to be technically infeasible as proposed and/or that required technology development prevents achievement of science/mission objectives. Blue indicates that the Figure E-6 concept has adequate margin and is consistent with a heritage heliophysics mission. SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. PI-MODE IMPLEMENTATION As described in the 2006 National Research Council report Principal-Investigator-Led Missions in the Space Sciences,2 NASA introduced the PI-led mode of implementation with the Discovery program and has since extended it to the Explorer, New Frontiers, and Mars Scout programs. Recent CoBRA analysis of 18 successful PI-led missions demonstrates that such missions with complexity indices of greater than or equal to 40 percent have costs that are typically approximately 30 percent less than non-PI-led missions having the same complexity. Figure E.9 shows the cost versus complexity for the set of successful missions from Figure E.7, with the PI-led missions indicated. The costs of only three of the PI-led missions with complex- ity greater than or equal to 40 percent exceeded the complexity-dependent median cost for all missions. The distributions of individual mission costs relative to the overall median cost are shown in Figure E.10 for PI-led missions and separately for the other missions with 40 to 80 percent complexity. The two cost 2 National Research Council, Principal-Investigator-Led Missions in the Space Sciences, The National Academies Press, Washington, D.C., 2006.

OCR for page 365
APPENDIX E 373 BOX E.1  CATE OUTPUT PRODUCT FOR THE INTERSTELLAR MAPPING AND ACCELERATION PROBE MISSION IMAP — Interstellar Mapping and Acceleration Probe Spinner Spacecraft Key Challenges • Payload Suite – Design uncertainty based on significant differences between instrument concepts and heritage systems as well as long elapsed time since heritage missions • System Mass – Low mass growth allowance – Risk of moving to larger launch vehicle • System Power – Low power growth allowance – Non-rechargeable battery – Liens against power subsystem design Scientific Objectives CATE Cost Estimate • Observations/measurements – High-Resolution & Time Resolved Mapping of Heliosphere Boundary – Physical Properties and Composition of the Interstellar Medium and the Outer Heliosheath – Seed Populations for Energetic Particles and Their Time Variation • Key science themes cited: – What is the nature of Earth’s home in the solar system? – How does variability in Earth’s space environment affect our lives? – What does our space environment teach us about universal, fundamental physical processes? Key Parameters Cost Risk Analysis S-Curve • Payload (10 instruments) – High Sensitivity, High Resolution ENAs: Solar Wind and PUI ENAs, High Energy ENAs – Samples of Interstellar Matter: Interstellar Neutrals, Pickup Ions, Interstellar Dust, Ly-α Photometry – Real Time Solar Wind – Energetic Particle Injection, Acceleration and Propagation 2 • Power: 1 m GaAs array, 270 W End of Life • Launch Mass: 336 kg th • Launch Date: 2018+ (on Taurus XL 3213 w/ 5 stage) • Orbit: Sun-Earth L1 Lissajous Orbit • Mission Duration: 2.5 years

OCR for page 365
374 SOLAR AND SPACE PHYSICS: A SCIENCE FOR A TECHNOLOGICAL SOCIETY BOX E.2  CATE OUTPUT PRODUCT FOR THE DYNAMICAL NEUTRAL ATMOSPHERE-IONOSPHERE COUPLING MISSION DYNAMIC — Dynamical Neutral Atmosphere-Ionosphere Coupling DYNAMIC Two Spacecraft Constellation Key Challenges • Launch Schedule and Spacecraft Reliability – Need to closely schedule 2 launches – Low constellation reliability for 2 out of 2 satellites over full mission duration • WIND Instrument Definition – Preferred design approach not yet selected and options being considered are very different • System Power – Power growth allowance is low Scientific Objectives CATE Cost Estimate • Observations/measurements – Combines in situ and remotely sensed information from two s/c to resolve spatial and temporal structure in the upper atmosphere – Neutral and ion density, auroral inputs, vector wind fields, ionospheric irregularities • Key science themes cited: – How does the lower atmosphere drive geospace? – Do we understand how this driving occurs and what the important spatial and temporal scales are? – How does behavior of the neutral lower atmosphere couple into plasma behavior in the upper atmosphere? Key Parameters Cost Risk Analysis S-Curve • Payload – WIND: Along track vector wind fields, density, temp – FUV: Cross track scanner – NWM: Neutral wind meter – IVM: Ion velocity meter – INMS: Neutral/ion mass spectrometer 2 • Power: 1.5 m GaAs two-axis array, 416 W End of Life • Launch Mass: 286 kg • Launch Date: 2015+ (on two Taurus 2110 vehicles) • Orbit: 600 km circular, 80 deg inclination • Mission Duration: 2 years (dual spacecraft)

OCR for page 365
APPENDIX E 375 BOX E.3  CATE OUTPUT PRODUCT FOR THE DESCOPED MAGNETOSPHERE ENERGETICS, DYNAMICS, AND IONOSPHERIC COUPLING MISSION1 MEDICI — Magnetosphere Energetics, Dynamics, and Ionospheric Coupling Descope NOTE:   he version of the MEDICI concept that included FUV LBH (Lyman-Birge-Hopfield) long- and short-wavelength cameras T on both HEO spacecraft—a configuration strongly advocated by the SWMI panel—was not evaluated; however, the cost of adding a duplicate far-ultraviolet LBH imager to an otherwise identical spacecraft should not add significantly to the overall mission cost, which is estimated to be $526 million. The proposed launch vehicle for MEDICI is a Falcon 9, which has lift capability that easily accommodates the additional imager.

OCR for page 365
376 SOLAR AND SPACE PHYSICS: A SCIENCE FOR A TECHNOLOGICAL SOCIETY BOX E.4  CATE OUTPUT PRODUCT FOR THE GEOSPACE DYNAMICS CONSTELLATION MISSION GDC — Geospace Dynamics Constellation

OCR for page 365
APPENDIX E 377 10000 Successful Missions Anchor Missions CATE SEE MISTE 1000 Successful Missions Trendline RBSP Cost (FY12$M) IMAP SDO GDC GLAST DYNAMIC MEDICI MEO ACE MMS MEDICI 100 HEO GALEX THEMIS Single Satellites Only Plotted 10 0% 20% 40% 60% 80% 100% Complexity Index FIGURE E.7  CATE missions on Complexity Based Risk Assessment (CoBRA) cost versus complexity plot. Missions subjected to the CATE process mapped on to a standard CoBRA plot showing both successful missions and “anchor missions” used for correlation. The trendline represents the mean 50th percentile case. The CATE missions are statistically representative of a 70th percentile case. Results are for a single satellite in cases where multiple satellites are used. MEDICI is a special case since it has different satellites for medium Earth orbit and high Earth orbit. SOURCE: Provided by the Aerospace Corporation Figure E-7 under contract with the National Research Council. distributions are significantly different, with the median cost of PI-led missions 0.7 times that of the other missions. In addition, the cost risk of PI-led missions is also reduced, as indicated by the nearly symmetric distribution of PI-led mission costs about their median, with a small tail of three missions that exceeded the median cost of the other missions. In comparison, the cost distribution for other missions is distinctly asymmetric about its 50th percentile, with a larger fractional variance extending to higher costs. The attributes of the PI-led mode that contributed to the success of these programs were summarized in the Principal-Investigator-Led Missions report and included: • Direct involvement of the PI in shaping the decisions and the mission approach to realizing the science objectives; • Two-step competitive selection: initial selection of two to four missions that receive funding for preparing mission concept studies prior to final selection of one or two missions for implementation; • Standard evaluation process of technical, management, cost, and other factors prior to final selec- tion; and • Capped mission cost with a termination review and threat of cancellation if the mission is projected to exceed its cost cap. The cost estimates from the CATE process do not take into account the potential cost reduction associ- ated with PI-led mode implementation. However, the reduced cost and reduced cost risk associated with PI-led missions offer an important opportunity for optimizing science return in an era of stringent budget constraints.

OCR for page 365
378 SOLAR AND SPACE PHYSICS: A SCIENCE FOR A TECHNOLOGICAL SOCIETY 120 Successful Missions (unconstrained) 110 GLAST Development Time (months) Anchor Missions 100 SEE CATE MEDICI 90 HEO Successful Missions Trendline GDC 80 DYNAMIC 70 SDO MEDICI ACE 60 MEO MMS 50 40 MISTE RBSP IMAP 30 20 THEMIS GALEX 10 Single Satellites Only Plotted 0 0% 20% 40% 60% 80% 100% Com plexity Index FIGURE E.8  CATE missions on Complexity Based Risk Assessment (CoBRA) schedule versus complexity plot. Missions sub- jected to the CATE process mapped on to a standard CoBRA plot showing both successful missions and “anchor missions” used for correlation. Results are for a single satellite in cases where multiple satellites are used. MEDICI is a special case since it has different satellites for medium Earth orbit and high Earth orbit. SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. Figure E-8 10000 Successful Missions Anchor Missions Successful PI- Led Missions 1000 RBSP Cost (FY12$M) Successful Missions Trendline SDO Current Discovery Cost Cap GLAST ACE MMS 100 GALEX THEMIS Single Satellites Only Plotted 10 0% 20% 40% 60% 80% 100% Complexity Index FIGURE E.9  CATE missions on Complexity Based Risk Assessment cost versus complexity plot. The green trendline is the median of the complexity-dependent costs of all successful missions. The costs of principal-investigator (PI)-led missions with >40 percent complexity are mostly below the median cost line. SOURCE: Provided by the Aerospace Corporation under contract with the National Research Council. Figure E-9

OCR for page 365
APPENDIX E 379 100% 90% 80% 70% 60% 50% 40% 30% 20% % Rank for PI-Led Missions 10% % Rank for Other Missions 0% 0.1 1.0 10.0 Actual Cost as a Fraction of CoBRA Green Line FIGURE E.10  Distribution of principal-investigator (PI)-led mission costs (blue) and non-PI-led mission costs (red) as a frac- tion of the median cost of all missions with 40 to 80 percent complexity (green trendline in Figure E.9). The median cost of PI-led missions is 0.7 times that of the other missions, with only 16 percent of the PI-led mission costs exceeding the median of the costs of other missions. The PI-led missions Figure E-10 smaller fractional cost variance. SOURCE: Provided by also have a the Aerospace Corporation under contract with the National Research Council.