Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 365
E
Mission Development and Assessment Process
INTRODUCTION
This report follows the astronomy and astrophysics and the planetary science decadal survey
reports as the third survey employing the Cost Appraisal and Technical Evaluation (CATE) process
implemented by the Aerospace Corporation. CATE is an independent analytical approach for realistically
assessing the expected cost and risk related to recommended initiatives that are at an early stage of
formulation (typically pre- Phase-A). A unique attribute of the process is the ability to employ technical
methods in combination with statistical methods to develop mission concepts based on their assessed
complexity, risk and lien factors. An essential driver to the CATE process is the requirement that all
missions be treated using systematic and consistent methods in order to facilitate a uniform comparison of
the relative cost, schedule, and risk of projects proposed for implementation. Secondary, but also
essential elements to the success of the process, is the incorporation of iterative technical feedback from
stakeholders combined with a proven method for independent validation of the results. The successful
implementation of these elements as an integrated process were fundamental to achieving confidence in
the CATE process and consequent confidence in the assessment results, implementation assumptions, and
decision rules applied to the recommended program.
The CATE process is the third leg of a science investigation triad following in a sequence
beginning with science mission implementation recommendations followed by an intermediate stage
consisting of mission conceptual design activities. The flow of activities began with white paper
submissions by the science community in November 2010 followed by concept evaluation and a mission
design effort culminating with initial mission selection in April 2011. The subsequent iterative CATE
process began in April and ended in September 2011.
As part of the CATE process, mission de-scoping options were introduced in recognition of the
budget pressures NASA is likely to be facing over the next several years. A unique strength of the
heliophysics discipline is the long experience of its investigators doing exceptional science in a principal-
investigator-led mode using Small (SMEX) and Medium (MIDEX) Explorers. A key finding of the
CATE process used in this study was that all of the proposed concepts were technically feasible at a
reasonable cost. Furthermore, the de-scoped missions were of a scope, complexity, and cost that are
consistent with successful mid-size PI-led missions executed for NASA’s Discovery program.
MISSION DESIGN ACTIVITY
The initial science assessment and prioritization effort yielded 12 mission concepts with four
concepts put forward by each of the three science discipline panels. Each of the concepts were then
organized and systematized using a standard design questionnaire intended to provide a relatively uniform
set of design parameters as an input to the mission design process. The 12 mission designs were then
produced over a five week period in February and March of 2011 at the Concept Design Center (CDC) of
the Aerospace Corporation in El Segundo, California. The CDC team consisted of experts that were
“firewalled” from the CATE process. Each mission was supported by a group of both shared and
dedicated CDC team members, one or members of the steering committee, and a “champion” designated
by the science panels to represent each investigation concept.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-1
OCR for page 366
The cornerstone assumption for the mission design process was that cost-effective mission
concepts demand a requirements-focused design combined with overall system simplicity. The general
system design baseline resulting from these assumptions was a single-string (with selective redundancy in
some cases) architecture using commercially available spacecraft bus designs similar in size to a MIDEX.
The bus structure and specific capabilities such as power and pointing were sized to accommodate the
required science instrumentation. Mission-unique features were also incorporated to meet requirements
for orbit and downlink, radiation hardness, magnetic cleanliness, deployable subsystems, dispensing
functions and de-orbit capability (if determined to be necessary). To constrain the system design and
control cost, the system architecture was also scaled such that all missions were both sized and intended
for orbits so as to be compatible with a launch on a medium launch vehicle or smaller. The largest launch
vehicle required for any mission concept was the Falcon-9.
The design process was intense but also effective at developing a realistic set of concepts
consistent with the above requirements and constraints. The effort was well supported by all
stakeholders in the process with detailed standardized reports prepared for each of the resulting system
designs. The quality and consistent nature of the reports allowed for a straightforward comparison of
missions leading into the subsequent down-select process. The reports were also designed to be
compatible with the input parameters required by the CATE process.
Following distribution of the mission design reports, the science panels performed an initial
review and made its recommendations to the committee for the CATE down-select from 12 to 6 missions.
The recommendations were based on science objectives, complexity factors and relative cost
assumptions. The recommended concepts in Table E.1 were presented by the science panels, endorsed by
the committee and moved forward into the first stage of the CATE process.
TABLE E.1 Mission Concepts Selected for Entry into the CATE Process.
CATE PROCESS
The CATE process has been discussed at length in previous reports and is based on the following
principles, which have remained consistent between recent past surveys and this survey. The CATE flow
chart provided as Figure E.1 diagrams the end-to-end effort as described by these principles and executed
for this report.
• Use multiple methods and databases regarding past space systems so that no one model or
database biases the results.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-2
OCR for page 367
• Use analogy based estimating that anchors cost and schedule estimates to NASA systems that
have already been built with a known cost and schedule.
• Use both system level estimates as well as a build-up-to-system level by appropriately
summing subsystem data so as not to underestimate system cost and complexity.
• Use cross checking tools, such as Complexity Based Risk Assessment (CoBRA) to cross
check cost and schedule estimates for internal consistency and risk assessment.
• In an integrated fashion, quantify the total threats to costs from schedule growth, the costs of
maturing technology, and the threat to costs owing to mass growth resulting in the need for a larger, more
costly launch vehicle.
The general CATE methodology for cost assessment is presented in Figure E.2. As with most
cost assessment approaches, the CATE process uses mission analogies, growth metrics, and other
applicable criteria applied to each work breakdown structure element that are derived from measured cost
and schedule performance on past NASA (and selected DoD) missions). Probabilistic methods are then
employed using a triangle distribution to calculate the reserves needed to move the sum of the most likely
cases up to a 70th percentile point1 on the resulting S-curve describing the statistical distribution for
mission cost.
The CATE methodology for schedule assessment shown in Figure E.3 is also analogy based with
a probabilistically derived 70th percentile output. Analogous missions based on mission class, technical
similarities, and participating organizations are used with historical phase durations for key mission
milestones. These durations are used to develop triangular distributions of estimated phase duration. A
cumulative probability distribution of the total schedule from Phase-B through launch is then generated
by combining the individual schedule distributions.
FIGURE E.1 Flow chart for the CATE process. SOURCE: Provided by Aerospace Corporation under
contract with the National Research Council.
1
The 70th percentile point represents the total cost (including reserves) within which there is a 70% likelihood
that the mission can be accomplished.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-3
OCR for page 368
FIGURE E.2 CATE cost estimating methodology. SOURCE: Provided by Aerospace Corporation under
contract with the National Research Council.
FIGURE E.3 CATE schedule estimating methodology. SOURCE: Provided by Aerospace Corporation
under contract with the National Research Council.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-4
OCR for page 369
As previously mentioned, a unique attribute of the CATE method is the ability to “grow forward”
a program from a very early conceptual stage where the threats and liens are often much larger than for
missions that are a later stage of development. The dominant factors are most often the natural growth
that occurs in mass and power requirements on a mission as its system design matures. Growth in these
areas often requires a larger launch vehicle or some other mitigating content that drive up the final cost.
A major area of cost uncertainty for the heliophysics CATE process was related to launch vehicle
cost and availability. Launch vehicles are furnished by NASA and subject to NASA competitive
procurement practices. Consequently, specific vehicle costs are unknown or could not be provided except
in a broad cost range enveloping numerous vehicle types. As discussed above, smaller vehicles beginning
with the Taurus-XL 2XXX, Taurus-XL 3XXX, Taurus-II and Falcon-9 were baselined and are capable of
meeting all of the mission design cases that were studied. The Falcon-1 and Athena were treated as
variants of the Taurus class vehicle and are considered cost-neutral substitutes for the majority of the
applications. The Delta-II was not considered to be available at the time of the mission design activity
but is also a viable substitute with respect to launch capability although at an unknown cost. It is also
unclear whether the Delta-II will retain launch capabilities at the Eastern Test Range launch facility.
All vehicles (or comparable vehicles at comparable cost) were considered to be available later in
the decade when the first missions would approach a state of launch readiness. The CATE process
assumed the launch vehicle cost to be a pass-through based on an analysis using multiple publicly
available sources. The cost range used in the process started at $60M for the Taurus-XL 2XXX class and
extended up to $125M for the Falcon-9 in 2012 dollars. For the vehicles baselined in the design studies
the potential for cost growth and associated cost risk is considered low although there is some risk related
to availability of a particular vehicle type.
MISSION COST RESULTS
The cost comparison between mission the six mission concepts and associated de-scoped
missions are shown in Figure E.4. For each mission there is a base cost resulting from the probabilistic
cost analysis plus the analyzed cost for mission-specific threats and liens. The corresponding schedule
results are shown in Figure E.5.
Specific to this survey is the recognition that several large missions are already through
formulation or in the late stages of formulation such that they will become operational in stages that
envelope most of the coming decade. Given the evident funding challenge early in this decade a frequent
mission cadence via the implementation of small and medium class missions over the entire decade is
considered more valuable to the community than the implementation of, at most, one additional large
mission.
This “simplified mission strategy” biased toward targeted executable investigations becomes
clearer when examining Figure E.4 and Figure E.6 representing the CATE assessment of cost and
technical risk respectively for each of the 6 baseline missions plus the 3 de-scope options. It can be seen
that the system approaches chosen for the missions have been successful at limiting the technical risk to
the medium-low and medium categories with a resulting payback in both affordability and cadence. The
risk rankings are particularly illuminating when compared to previous surveys where most missions were
ranked in the medium-high and high categories.
A combined assessment of cost and technical risk leads to the conclusion that simplified mission
concepts similar to IMAP, DYNAMIC, and MEDICIDescope investigations fall into the range of PI-Led
missions that can be successfully executed within the scope of the STP mission line. These missions
along with GDC, as the recommended LWS mission, have their key parameters summarized in Boxes E.1
through E.4 (provided by Aerospace Corporation under contract with the National Research Council).
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-5
OCR for page 370
FIGURE E.4 CATE 70th percentile cost results for missions including descoped alternatives. Base cost
plus threats and liens shown in FY2012 $ million. SOURCE: Provided by Aerospace Corporation under
contract with the National Research Council.
FIGURE E.5 CATE 70th percentile Phase-B through Phase-D schedule results. GDC de-scope reduces
schedule by 2 months. IMAP and MEDICI de-scopes do not affect schedule. SOURCE: Provided by
Aerospace Corporation under contract with the National Research Council.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-6
OCR for page 371
FIGURE E.6 CATE technical risk estimate. This estimate is provided to represent the level of technical
risk resulting from the CATE assessment. Note that in the case of the IMAP and MEDICI investigations
the de-scope resulted in a cost reduction but no change in the technical risk posture. An advantage of
using a standardized CATE process is the metrics for evaluating technical risk are based on technical
criteria that are consistent across programs within a single survey as well as across surveys from other
disciplines. SOURCE: Provided by Aerospace Corporation under contract with the National Research
Council.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-7
OCR for page 372
BOX E.1 CATE Output Product for the IMAP Mission
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-8
OCR for page 373
BOX E.2 CATE Output Product for the DYNAMIC Mission
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-9
OCR for page 374
BOX E.3 CATE Output Product for the MEDICI Mission2
2
The version of the MEDICI concept that included FUV LBH (Lyman-Birge-Hopfield) long- and short-
wavelength cameras on both HEO spacecraft—a configuration strongly advocated by the SWMI panel—was not
evaluated; however, the cost of adding a duplicate FUV LBH imager to an otherwise identical spacecraft should not
add significantly to the overall mission cost, which is estimated to be $526 million. Note: The proposed launch
vehicle for MEDICI is a Falcon 9, which has lift capability that easily accommodates the additional imager.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-10
OCR for page 375
BOX E.4 CATE Output Product for the GDC Mission
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-11
OCR for page 376
VALIDATION APPROACH
The final step of the CATE process consisted of a series of validation reviews intended to assure
that the assumptions and corresponding cost and schedule results are based on defensible assumptions, are
fair in their representation of the assessed missions and are consistent with past missions of similar
content and complexity. The reviews consisted of a mix of both internal and external reviews conducted
by Aerospace and by the committee, respectively. The CATE validation tool used for this survey and all
past surveys is the Complexity Based Risk Assessment (CoBRA) tool developed by Aerospace
Corporation. Figures E.7 and E.8 map the mission candidates on to cost and schedule grids for
comparison with successful missions of varying complexity plus “anchor missions” considered to be
similar in size and complexity.
The results show excellent correlation between the mission concepts and compared to the
successful missions and anchor missions. Therefore, the CATE results are considered compatible with
experience in respect to both cost and schedule. As one would expect, the CATE missions representing a
70th percentile distribution are generally above the mean. One interesting result, discussed in the next
section, is that the heliospheric anchor missions are generally below the mean trend line.
FIGURE E.7 CATE missions on CoBRA cost versus complexity plot. Missions subjected to the CATE
process mapped on to a standard CoBRA plot showing both successful missions and “anchor missions”
used for correlation. The trend line represents the mean 50th percentile case. The CATE missions are
statistically representative of a 70th percentile case. Results are for a single satellite in cases where
multiple satellites are used. MEDICI is a special case since it has different satellite for the MEO and
HEO orbits. SOURCE: Provided by Aerospace Corporation under contract with the National Research
Council.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-12
OCR for page 377
FIGURE E.8 CATE missions on CoBRA schedule versus complexity plot. Missions subjected to the
CATE process mapped on to a standard CoBRA plot showing both successful missions and “anchor
missions” used for correlation. Results are for a single satellite in cases where multiple satellites are used.
MEDICI is a special case since it has different satellite for the MEO and HEO orbits. SOURCE: Provided
by Aerospace Corporation under contract with the National Research Council.
PI-MODE IMPLEMENTATION
As described in the NRC report titled Principal-Investigator-Led Missions in the Space Sciences,
NASA introduced the PI-mode of implementation with the Discovery Program and has since extended it
to the Explorer, New Frontiers, and Mars Scout Programs. Recent CoBRA analysis of 18 successful PI-
led missions demonstrates that such missions with complexity indices of greater than or equal to 40
percent have costs that are typically ~30 percent less than non-PI-led missions having the same
complexity. Figure E.9 shows the cost vs. complexity for the set of successful missions from Figure E.7
with the PI-led missions indicated. The costs of only three of the PI-led missions with greater than or
equal to 40 percent complexity exceeded the complexity-dependent median cost for all missions.
The distributions of individual mission costs relative to the overall median cost are shown in
Figure E.10 for PI-led missions and separately for the other missions with 40 to 80 percent complexity.
The two cost distributions are significantly different, with the median cost of PI-led missions 0.7 times
that of the other missions. In addition, the cost risk of PI-led missions is also reduced as indicated by the
nearly symmetric distribution of PI-led mission costs about their median, with a small tail of three
missions that exceeded the median cost of the other missions. In comparison, the cost distribution for
other missions is distinctly asymmetric about its 50th percentile with a larger fractional variance extending
to higher costs.
The attributes of the PI-mode that contributed to the success of these programs were summarized
in the NRC report noted above and included:
• Direct involvement of the PI in shaping the decisions and the mission approach to realizing
the scientific objectives.
• Two-step competitive selection: initial selection of 2-4 missions that receive funding for
preparing mission concept studies prior to final selection of 1 or 2 missions for implementation.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-13
OCR for page 378
• Standard evaluation process of technical, management, cost, and other factors prior to final
selection.
• Capped mission cost with a termination review and threat of cancellation if the mission is
projected to exceed its cost cap.
The cost estimates from the CATE process do not take into account the potential cost reduction
associated with PI-mode implementation. However, the reduced cost and reduced cost risk associated
with PI-led missions offers an important opportunity for optimizing scientific return in an era of stringent
budget constraints.
FIGURE E.9 CATE missions on CoBRA cost versus complexity plot. The green trend line is the
median of the complexity-dependent costs of all successful missions. The costs of PI-led missions with
>40% complexity are mostly below the median cost line. SOURCE: Provided by Aerospace Corporation
under contract with the National Research Council.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-14
OCR for page 379
100%
90%
80%
70%
60%
50%
40%
30%
20%
% Rank for PI-Led Missions
10% % Rank for Other Missions
0%
0.1 1.0 10.0
Actual Cost as a Fraction of CoBRA Green Line
FIGURE E.10 Distribution of PI-led mission costs (blue) and non-PI-led mission costs (red) as a fraction
of the median cost of all missions with 40% to 80% complexity (green trendline in Figure E.9). The
median cost of PI-led missions is 0.7 times that of the other missions, with only 16% of the PI-led mission
costs exceeding the median of the costs of other missions. The PI-led missions also have a smaller
fractional cost variance. SOURCE: Provided by Aerospace Corporation under contract with the National
Research Council.
PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION
E-15
OCR for page 380
