The Role of Cost Estimates, Technical Evaluations, and Budget Projections in Prioritizing Missions
Decadal surveys have tried to give some sense of the value and cost of the missions recommended by the surveys. Describing the value of a mission, however, has always been easier than estimating its cost. The most recent round of decadal surveys represent the most serious and thorough attempt to improve the precision of the cost estimation process, but the surveys handled what has come to be known at the cost and technical evaluation (CATE) process differently. The 2007 Earth science and applications from space decadal survey was the first attempt after the previous round of decadal surveys in the late 1990s and early 2000s to improve the cost estimation process, but compared to the surveys that followed, it was a considerably coarser process. Many missions recommended by past surveys are orders of magnitude more expensive than the decadal survey they originated from indicated. This has forced the system toward a more refined, but also more time and resource consuming CATE process, which is exacerbated by an austere government fiscal environment. This workshop session focused on the role of CATE and budget projections in prioritizing missions and how to improve the CATE process.
| Moderator: |
Steven Battel, President, Battel Engineering; Member, Workshop Planning Committee |
| Panelists: |
Randall Friedl, Deputy Director for Research, Engineering and Science Directorate, Jet Propulsion Laboratory, 2007 Earth Science and Applications from Space Decadal Survey Jay Bookbinder, Program Manager, High Energy Division, Smithsonian Astrophysical Observatory, Mission Scientist for the International X-ray Observatory Scott Hubbard, Professor of Aeronautics and Astronautics, Stanford University, 2011 Planetary Science Decadal Survey Harlan E. Spence, Director, Institute for the Study of Earth, Oceans and Space, University of New Hampshire; 2013 Solar and Space Physics Decadal Survey David A. Bearden, General Manager, NASA Programs Office, The Aerospace Corporation |
Panel moderator Steven Battel began his introductory remarks by reviewing the origins of the CATE process used in the three most recent decadal surveys: New Worlds, New Horizons in Astronomy and Astrophysics,1Vision and Voyages for Planetary Science in the Decade 2013-2022,2 and Solar and Space Physics: A Science for a Technological Society.3 The term CATE was devised by Brian Dewhurst,
__________
1 National Research Council (NRC), New Worlds, New Horizons in Astronomy and Astrophysics, The National Academies Press, Washington, D.C., 2010.
2 NRC, Vision and Voyages for Planetary Science in the Decade 2013-2022, The National Academies Press, Washington, D.C., 2011.
3 NRC, Solar and Space Physics: A Science for a Technological Society, The National Academies Press, Washington, D.C., 2013.
Marcia Rieke, and Steven Battel in response to the 2008 congressional language mandating independent assessments of cost and technical readiness of all recommended mission concepts,4 as was implemented for the first time in the statement of task (SOT) for the 2010 astronomy and astrophysics decadal survey.5
Battel then reviewed the evolution of the language relating fiscal and technical realism of the recommended missions as contained in the SOT for the four most recent decadal surveys sponsored by NASA’s Science Mission Directorate (SMD). The SOT for the 2007 Earth science and applications from space decadal study6 predated the 2008 congressional mandate. As a result, the SOT called for the survey committee to “Recommend a prioritized list of measurements, and identify potential new space-based capabilities and supporting activities … to support national needs for research and monitoring of the dynamic Earth system during the decade 2005-2015.”7 Thus, there was no prescription as to how costing would be done. As it turned out (see Chapter 5), the survey committee adopted a very reasonable approach to selecting its recommended missions by figuring out their likely costs.
The SOT for the 2010 astronomy and astrophysics decadal study was drafted after the congressional mandate came into force. So, the instructions to the survey committee were very specific: “the committee will review the technical readiness of the components and the system, it will assess various sources of risk, and it will develop its own estimate of the costs of the activity with help from an independent contractor with expertise in this area. It will not uncritically accept estimates provided by activity proponents or the agencies.”8 Responding to these instructions meant that the survey committee had to draft a work statement for the contractor, evaluate the responses from candidate contractors, select a contractor, and then work with the selected contractor to develop the required processes to undertake the required technical readiness, risk, and cost analyses for both ground- and space-based activities. This very large and complicated process took place at the beginning of the survey and established the basics of the CATE process as it is understood today.
The SOT for the 2011 planetary science decadal study was even more prescriptive in that it required that “the NRC [National Research Council] should include resources for independent and expert cost analysis support to ensure that all flight mission cost estimates can be meaningfully intercompared and are as accurate as possible given the varying maturity of project concepts and other recognized uncertainties.”9 Whereas, the SOT for the 2013 solar and space physics decadal study only noted that “the NRC will include resources for independent, expert cost analysis support, which will be used when appropriate to improve cost estimations, expose and bound uncertainties, and facilitate cost comparisons among missions with varying heritage and technical maturity.”10 Although the CATE process was implemented in both of these surveys, the finer details of how it was applied evolved as required to
__________
4 The term “mission” is used in a generic sense throughout this chapter to indicate not just spacecraft missions but all implementation activities including, ground-based facilities, laboratory analyses, theoretical studies, numerical modeling, and any other research activities necessary to address a specific scientific activity.
5 The National Aeronautics and Space Administration Authorization Act of 2008 (Public Law 110-422, Section 1104b) mandated that the NRC “include independent estimates of the life cycle costs and technical readiness of missions assessed in the decadal survey wherever possible.” However, it should be noted that the Space Studies Board’s first use of independent cost estimates was in the context of the NRC study resulting in the report, NASA’s Beyond Einstein Program: An Architecture for Implementation (The National Academies Press, Washington, D.C., 2007).
6 NRC, Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond, The National Academies Press, Washington, D.C., 2007.
7 NRC, Earth Science and Applications from Space, 2007, p. xxi.
8 NRC, New Worlds, New Horizons in Astronomy and Astrophysics, The National Academies Press, Washington, D.C., 2010, p. 267.
9 NRC, Vision and Voyages for Planetary Science in the Decade 2013-2022, The National Academies Press, Washington, D.C., 2011, p. 321.
10 NRC, Solar and Space Physics: A Science for a Technological Society, The National Academies Press, Washington, D.C., 2013, p. 329.
conform to the varying imperatives contained in the relevant SOT. It is likely that the CATE will continue to evolve to conform to the mandates of subsequent surveys in the decades to come.
Battel concluded his comments with some words of advice for agencies commissioning a decadal survey: Be careful what you ask for, because you will get it. Given the rigor with which NRC committees are required to adhere to their SOTs, an agency drafting an SOT asking for detailed, prescriptive results will get a survey report containing detailed, prescriptive results.
The panel discussion was organized around a series of questions the panel posed to itself. These questions explored different aspects of the common theme: Have the evolving decadal survey SOTs been usefully served by the evolving CATE process?
During their discussion, the moderator and panelists talked about their experience with and opinions on the following topics:
• Tailoring the CATE process to serve future decadal surveys,
• Accommodating evolving science requirements in the CATE process,
• The role of mission advocates in the CATE process,
• Implementing the CATE as an integrated, iterative process, and
• The CATE as a component of an unconstrained scientific process.
Tailoring the CATE Process to Serve Future Decadal Surveys
Steven Battel began the panel discussion by asking Randall Friedl how the CATE process could be tailored to optimally serve future decadal surveys, given the intrinsic differences between the various science disciplines. Friedl began his response by reminding the audience that the SOT for the 2007 Earth science decadal survey made no mention of assessing the cost or technical readiness or recommended missions. The survey committee’s co-chairs approached several members and asked them to develop a process to give a “sense of the cost” of recommended missions. No funds were available to facilitate this costing, and nobody from outside of the decadal survey was involved in this exercise to maintain the integrity of the study process. Although some at NASA were not happy with the results of this costing exercise, the lesson learned is that although a more thorough CATE process is more expensive, it is also a better process and can help reduce mission costs.
Friedl explained that the 2007 Earth science decadal survey was not just pre-CATE, it was “prehistoric” CATE.11 As described in Chapter 5, the survey committee did establish, validate, and implement a primitive costing protocol. The survey was initiated in 2004, but the costing protocol was not distributed to the survey panels until January 2006. Some 30 notional mission concepts were costed by May 2006. The second lesson learned from this exercise was that the “prehistoric” costing protocol was only good for binning the concepts into broadly defined cost boxes. Such an approach is sufficient if, and only if, there is a shared understanding between the survey and its sponsors—communicated via the SOT—as to what the survey is supposed to recommend. The majority of survey participants believed that the goal of the activity was to define a coherent program consisting of a mix of science and mission activities. Thus, the primary goal of the costing exercise was to support the determination of what constituted a viable mix of science and mission activities. Given this goal, the survey committee’s costing protocol was quite helpful.
__________
11 For details concerning the costing methodologies used in the first decadal survey on Earth science and applications from space, see NRC, Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond, 2007, pp. 43-58.
Friedl noted that the initial costing exercises lead to several notable outcomes. First, it helped to establish the minimum, scientifically credible program. Second, the baseline budget provided to the survey committee by NASA was insufficient to support this minimum program. Third, it motivated a “horse-trading” exercise among the proponents of the 30 notional concepts, which resulted in a final suite of 17 merged, interdisciplinary missions. Fourth, the unknown nature of the scientific compromises underlying individual merged missions, combined with their intrinsic complexity, pushed the costing protocol to its limit and beyond. This breakdown did, however, cause the survey committee to focus on the key elements of a balanced program, including a systems approach to key scientific issues, an aversion to flagship missions, and more emphasis on innovation and low-cost missions.
In summary, the “prehistoric” approach to costing was sufficient to sort mission concepts into relative cost bins that were accurate to one-significant figure at best. This enabled the survey committee to focus on fundamental programmatic issues and not get bogged down in the detailed costing of individual missions. However, the failure to provide realistic costs of key concepts caused a lot of trouble.
Friedl concluded his presentation with a slide summarizing his suggested ground rules for future surveys. His suggestions were as follows:
• The maturity of science requirements is the primary factor in determining the optimal CATE approach to be followed;
• A bounding strategy is most suited to new mission concepts, large portfolios of medium-class missions, and smaller, competed mission classes;
• A pricing strategy is most suited to more mature flagship missions that may have been prioritized in earlier decadal surveys;
• Task statements for decadal surveys should give clear direction on CATE approaches to be applied; and
• The next decadal study in Earth science and applications from space should obtain point design estimates for selecting first-, second-, and third-tier missions from the 2007 decadal survey prior to its initiation.
On the basis of these ground rules, Friedl suggested that the SOT for the next decadal survey in Earth science and applications from space include the following language: “The committee will recommend a prioritized list of missions whose costs are bounded by a consistent set of independent and expert cost analysis. Updated and improved cost estimates for a select set of the earlier recommended missions will be provided as a starting point for survey deliberations.”
Accommodating Evolving Science Requirements in the CATE Process
Steven Battel then asked Jay Bookbinder, What is the appropriate feedback loop for the decadal mission selection process and associated CATE process for handling the evolution of science requirements between the early stage of white paper submission and submitted mission configurations? Bookbinder prefaced his response by reminding the audience that he was not a member of any of the recent decadal survey committees. Rather, he was intimately involved with the submission of mission concepts for consideration by two of the surveys. He lead the consortium proposing the International X-ray Observatory (IXO) concept to the 2010 astronomy and astrophysics decadal survey and subsequently proposed a different mission concept to the 2013 solar and space physics decadal survey.
Bookbinder explained that his perception was that the science requirements that mission concepts were attempting to address changed between the time the concepts were submitted to survey committees and the time the relevant decadal reports were issued. Given that one of the goals of this workshop is to come up with actionable suggestions for improvements in the decadal process, he suggested two potential mechanisms by which the proposers of mission concepts could interact more closely with survey committees. One mechanism Bookbinder suggested involves the addition of a feedback loop in the
development of a survey’s science goals. The effect of this feedback would be to inform those submitting mission concepts of the survey’s science goals and, thus, the requirements the proposed missions would need to address. The second mechanism Bookbinder suggested involves a feedback process during the mission-evaluation phase of a survey prior to the CATE process.
Bookbinder noted that previous speakers have emphasized that decadal surveys remain relevant over time—meaning they need to retain flexibility in how they select missions—and not devote too much time and effort engineering mission concepts beyond that needed to give reasonable cost estimates.
To illustrate the utility of his two suggestions, Bookbinder related his experiences with the IXO proposal to the 2010 astronomy and astrophysics decadal survey.12 IXO was a well-developed mission concept designed by a consortium of U.S. and European scientists to meet a comprehensive set of science requirements that predated the initiation of the survey. When the survey report was released, IXO was listed as one of the priority activities for the coming decade (a fourth-priority, large, space-based activity). But close reading of the report revealed that if the IXO concept had been designed to meet the survey’s science requirements, it would have been a far simpler and cheaper mission with fewer instruments and a lesser degree of international cooperation. It was clear to Bookbinder that if the IXO team had known the survey’s ultimate science requirements in advance, it would have been a simple task to reconfigure the concept to be in closer alignment with the survey’s science goals.
Bookbinder suggested that to implement the first feedback mechanism (Figure 6.1), a decadal survey could solicit white papers focusing only on scientific topics. The survey would deliberate on the content of these white papers and derive a set of science goals for the coming decade. These goals would be published and then the community would be asked to submit mission concepts addressing these goals. Although adding this extra step would take time, it would, in Bookbinder’s opinion, result in a closer coupling between a survey’s science goals and recommended set of future missions and allow for greater community innovation.
Bookbinder explained that his second suggested feedback mechanism resulted from his experience with the 2013 solar and space physics decadal survey. Rather than submitting a single mission proposal, he submitted three different concepts addressing the same basic question relating to mechanisms by which the solar-coronal heating takes place. The concepts differed in their instrument complements and had costs ranging from $450 million to $850 million. Bookbinder argued that this approach better meet the needs of the survey because the survey committee could choose the concept most appropriate to the program of missions being recommended and relevant budgetary constraints.
Rather than have all proposers submit multiple mission concepts, Bookbinder proposed the institution of a feedback process during the mission-evaluation phase of the survey. The survey committee should not just evaluate the feasibility of the submitted mission concepts but also evaluate the traceability between the missions and the decadal science goals. The survey should also institute a “sniff test”—a quick turn-around costing exercise—to establish the proposed concepts likely to cost one significant figure (see Figure 6.1). The result of the sniff test would be shared with the mission’s advocates so that they can adjust the concept—for example, to reduce its cost—if necessary. On the completion of this feedback process, the survey committee would conduct its initial prioritization of all the mission concepts still under consideration and then perform a detailed study of the highest-ranking candidates via the CATE process.
The addition of these two feedback mechanisms would maintain close coupling between the survey committee and the community throughout the science and mission evaluation phases of a decadal study.
__________
12 For details concerning the implementation of the CATE process in the context of the most recent astronomy and astrophysics decadal survey, see NRC, New Worlds, New Horizons in Astronomy and Astrophysics, 2010, pp. 253-259.
FIGURE 6.1 A schematic illustrating suggested modifications to the decadal survey process. The first phase is the publication of the survey’s science priorities prior to any mission-formulation process. The second phase is the addition of a quick turn-around costing exercise—the “sniff test”—to provide feedback from the decadal survey committee to mission advocates. The feedback enables advocates to fine-tune their mission concepts so that they are more consistent with the survey’s science goals. NOTE: STM, Science Traceability Matrix. SOURCE: Courtesy of Jay Bookbinder, Smithsonian Astrophysical Observatory.
The Role of Mission Advocates in the CATE Process
Steven Battel then asked Scott Hubbard if the CATE process should be modified to allow advocates to respond with mission design descopes prior to completion of the decadal survey process. Hubbard began his response by reminding the audience that the SOT for the 2011 planetary science decadal survey was quite explicit. The survey committee was to “ensure that all flight mission cost estimates can be meaningfully intercompared and are as accurate as possible.”13 The inclusion of the phrase, “as accurate as possible” motivated the committee to go to fairly extreme lengths to gather all the data necessary to achieve this goal. NASA’s Planetary Science Division spent a lot of money helping the committee gather this data.
__________
13 NRC, Vision and Voyages for Planetary Science in the Decade 2013-2022, 2011, p. 321.
Hubbard continued that he was initially a CATE skeptic.14 He recounted how, at an early committee meeting, he and survey committee members A. Thomas Young and B. Gentry Lee grilled the representatives of the Aerospace Corporation concerning the details of the proposed CATE process. Hubbard recalled that “We drilled, we probed, and we wire brushed” until we were convinced as to what exactly the CATE process involved and how it would be implemented. The CATE was not a standard independent cost estimation process. It wasn’t taking data relating to new mission concepts and comparing it with a database of missions that had already flown. It was doing a cost and technical evaluation that assessed the risks associated with a concept—e.g., schedule risk, mass-margin risk, data-rate risk, etc.— and monetizing them.
The CATE process was not used to assess small (i.e., Discovery-class) missions because they were not prioritized or even identified by the survey, per the SOT. The CATE was used as a quick look to see if medium-size (i.e., New Frontiers) missions identified via the white paper process and prioritized in the report could be implemented within the cost limit of roughly $1 billion. The costs arising for the CATE studies of potential New Frontiers missions were believed to be one significant figure. So if the CATE said that a particular mission concept would cost $1.3 billion, the survey committee argued that a clever principal investigator (PI) could optimize the concept so that it could actually be implemented for a billion dollars or less.
The 2011 planetary science survey committee and its panels focused a lot of time and effort on the flagship, or billion-dollar-plus, missions. The CATE process was particularly useful for these missions. The CATE enabled the survey committee to intercompare the likely costs of well-studied missions—for example, the Jupiter Europa Orbiter (JEO) and the Mars Astrobiology Explorer-Cacher (MAX-C)—and new concepts—for example the Venus Climate Mission and the Uranus Orbiter—that arose through the NASA-funded mission studies conducted in support of the planetary decadal activity. The CATE analyses of individual missions typically led to “sticker shock” and in some cases “Titanic-size” sticker shock. For example, the JEO mission was advertized by its advocates as likely to cost $3.4 billion, but the CATE analysis suggested a cost closer to $4.7 billion.
The origins of these discrepancies provoked a lot of discussion in the survey committee. Closer study usually revealed that the cost discrepancies often arose from an unrealistic assessment of margins on the part of advocates. Hubbard gave a notional example in which the proponents of a mission assumed the use of a particular launch vehicle. But the CATE analysis revealed that if the spacecraft’s mass grew by more than 10 percent—not an unusual occurrence between pre-phase A and launch—then a larger, more expensive rocket would be needed. By looking at all of the relevant margins and by making adjustments to the cost by reference to missions of a similar scope and complexity index, as recorded in the Aerospace Corporation’s databases of flown spacecraft, the CATE team was able to monetize the risk associated with each mission concept studied. The monetization of realistic margins provided a “wake-up call” to mission advocates and the planetary science community. The CATE process forced advocates to acknowledge elements in their concept where margins were overly optimistic.
Assessment of the white papers and the results of the mission studies, the CATE analyses of promising concepts, and subsequent deliberations by the panels led the survey committee to the conclusion that the two highest-priority flagship missions were JEO and MAX-C. But it was clear to the committee members that the scientific potential for these two missions was not worth the CATE costs of $4.7 billion and $3.5 billion, respectively. Therefore, the survey report recommended that the cost of MAX-C be capped at $2.5 billion and that the complex JEO mission be descoped to focus on key Europa science questions.
The community’s reaction to the 2011 planetary science decadal survey was, in Hubbard’s opinion, very positive. NASA Headquarters directed the Jet Propulsion Laboratory (JPL) to look for cheaper alternatives to the full-up JEO mission. One of the three resulting studies identified an approach
__________
14 For details concerning the implementation of the CATE process in the context of the most recent planetary science decadal survey, see NRC, Vision and Voyages for Planetary Science in the Decade 2013-2022, 2011, pp. 331-353.
to achieving a lot of the survey’s Europa science goals for half of the cost of JEO, as determined by a JPL-commissioned CATE undertaken by the Aerospace Corporation. In addition, NASA established the Mars Program Planning Group and chartered it to take an extensive look at MAX-C and other future Mars missions. This study identified a very credible Mars rover mission for about half of the cost of MAX-C.
In Hubbard’s opinion, the CATE process, as applied during the 2011 planetary science decadal survey, had the correct result. The survey process and its aftermath ultimately led to the identification of missions of high science value for half of their advertized decadal cost. His overall conclusion was that the CATE process, or something like it, is essential to understand what you are buying into as a program, at least in planetary science.
Hubbard asked himself if the process could be improved by, for example, allowing iterations to mission concepts during the decadal process—that is, sharing the results of the CATE analyses with mission advocates so that there would be a shared assessment that mission margins, for example, had been correctly assessed. Such iterations were ground-ruled out for various reasons relating to the confidentiality of the NRC study process. But, this option is worth serious discussion.
Implementing the CATE as an Integrated, Iterative Process
Steven Battel began the next phase of the panel discussion by noting that the CATE process was implemented as a stand-alone phase during the three most recent decadal surveys conducted for NASA SMD. How, he asked Harlan Spence, might the CATE be incorporated as an integrated, iterative process that is included in the entire decadal survey from the beginning? Spence commented that the solar and space physics community benefited by being the third to use the CATE process.15 His initial attitude toward the CATE process was skeptical. But by the end of the survey, he “stopped worrying and learned to love CATE.”
Spence continued by explaining that the 2013 solar and space physics survey solicited and eventually received nearly 300 white papers from the community. Some white papers described very mature mission concepts that had been developed in the past decade, but not yet implemented (i.e., items that are easy to subject to CATE). Others discussed very immature mission concepts (i.e., items harder to subject to CATE). Finally, there were white papers that just described interesting scientific topics but had no indication as to how they might be addressed. The survey committee and its panels had a major task sorting out and evaluating the content of the white papers. Spence emphasized that surveys need to focus on notional science topics first and foremost. Focusing on specific mission implementations too early in the study process leads to problems, including the potential exclusion of important scientific questions from consideration.
The community inputs to the solar and space physics survey arrived “fast and furiously,” but they were not always “right sized,” according to Spence. By this he meant that it was not clear what the appropriate scope of individual mission concepts could be until the committee had assembled a complete, notional program in light of realistic budget projections for the coming decade. Ideally, decadal surveys should commence with a clear understanding of likely future budget and a clear articulation to the community of realistic notional mission sizes. Full transparency and community buy-in requires that mission cost boxes should not evolve through a survey.
The survey committee put a lot of emphasis on the drafting of an implementable set of recommendations that would survive the decade. This emphasis caused the survey committee to focus less on flagship missions and pay more attention to intermediate-scale, medium-cost projects. However in some cases these intermediate activities became uncomfortably close in scope to that of likely PI-led
__________
15 For details concerning the implementation of the CATE process in the context of the most recent solar and space physics decadal survey, see NRC, Solar and Space Physics: A Science for a Technological Society, 2013, Appendix E.
missions (e.g., Explorer missions) which decadal surveys do not typically prioritize. This was not, Spence emphasized, a major issue for the survey, but something that needs to be watched in future surveys.
Once the solar and space physics survey report was released in 2012, there was a feeling among some mission advocates that their ideas did not fare as well as they could have if their concepts had been rescoped to lower their likely cost. Another issue, Spence recalled, was that the CATE analyses of some of the community’s “aspirational missions”—for example, constellations of many satellites—resulted in community sticker shock. An issue for the solar and space physics community and other communities using a systems science approach (e.g., Earth science and applications from space) is the following: How do you enable the innovations that will be needed to adequately prepare for the realistic discussion of such missions in future surveys? A related issue is how constellations and other initiatives that have few exemplars in the databases of flown missions are subjected to the CATE process.
Spence concluded by commenting that the multistep process used by the solar and space physics survey to study and assess its notional missions—that is, the pre-CATE mission formulation activity, the CATE study of key missions, and the reCATE of key, descoped concept—worked well to fine-tune and “right-size” notional missions. But given the comparatively short period of time allotted to the survey, the process felt rushed because key study activities that should be separated—for example, the formulation of recommended science questions and the formulation of notional missions—were, in some cases, conducted on an aggressive schedule. Spence argued that future survey committees and their panels should, in his opinion, have sufficient time to undertake the following four activities:
• Develop the best notional missions;
• Get initial indications from an abbreviated mission-formulation activity as to which notional concepts are outside desired cost boxes and why;
• Develop top-rated notional mission concepts for analysis by the CATE process; and
• Refine notional missions in reCATEing prior to final prioritization.
Spence was not sure if sufficient time meant more time or using the same amount of time in a more optimal manner. Spence was clear, however, that the outcome of the 2013 solar and space physics survey would not have been that different if the committee had been allocated another year to complete its tasks.
The CATE as a Component of an Unconstrained Scientific Process
Steven Battel then asked David Bearden how the CATE process could be implemented in a way that allows the scientific process to unfold organically and in an unconstrained manner. Bearden replied that the CATE analyses added a new element to the well-understood decadal survey process. As such, it was not unexpected that some observers regarded the CATE as a “contaminant.” But, in Bearden’s view, the classical, science-driven decadal process has lost credibility with many of its key stakeholders. Congress, the Office of Management and Budget, the Office of Science and Technology Policy, and the Government Accountability Office do not believe that the space and Earth science communities, acting through the decadal survey process, understand the mission concepts they are recommending sufficiently well to determine their ultimate cost. Given the state of the economy and ever increasing pressures on budgets, it is not unexpected that there are calls for more accountability in the decadal surveys.
Thus, in response to Battel’s question, Bearden was clear: the scientific community cannot go back to the “good old days” when it was unencumbered and unconstrained and the decadal survey process could unfold naturally. Today’s environment is such that the space and Earth science communities need to think more carefully about the technical, cost, and schedule risk associated with their priority initiatives and be prepared to back up their assessments with data if they are to regain credibility.
Bearden continued that it is important to keep a clear perspective on what CATE is and what it is not. The main product of the CATE process—the “C” in CATE—is a cost-distribution function. There is
great temptation to pick a point—say, the 70th percentile point—on the distribution function and focus on that. In reality, Bearden explained, the useful information is contained in the cost uncertainty. Moreover, it is better not to focus on a point design for a specific mission concept rather than to understand that there are a range of possible implementations. So, one thing we can probably do better is educating the people looking at the results of a CATE study in terms of point estimates of a point design. In other words, we are not yet ready to look at a specific cost estimate for a specific mission design.
The other part of the CATE is the technical assessment. Understanding technical risk is central to determining whether or not the technology is available to undertake a particular activity. As such, the “T” in CATE is useful for tamping down some of the exuberance of advocates for a certain type of measurement that might not yet be feasible.
Bearden continued his comments on the implementation of the CATE process by noting that in a number of cases the mission concepts under consideration were just not affordable when compared to the budget assumptions supplied by the survey’s sponsors. However, it was clear to the CATE team that the uncertainties introduced by the cost-estimation process were dwarfed by uncertainties in the budgetary projections. As a result, Bearden found Colleen Hartman’s three-world concept (see Chapter 5) attractive because it focuses on bounding possible future budgets rather than trying to predict them in detail. The range of budget scenarios that might unfold in the future is matched by the range of concept costs that might be realized for a particular mission set. So the question to be addressed is as follows: Is there any overlap at all in that trade space? Bearden’s basic message was not to fixate on the cost of one specific implementation and all of the risks associated with that particular approach. Rather, survey committees should become comfortable with uncertainty.
A scenario that played out in multiple surveys was that three or four juggernaut missions arose during the study process, and none of them was affordable. Echoing a comment made by several of the previous speakers, Bearden was opposed to the premature convergence on a point design for a specific mission when, in reality, more cost-effective implementations might exist. He suggested that an iterative process be established prior to the CATE. The goal would be to determine the approximate costs of specific concepts and assess them in light of different budget scenarios. Sharing the results of these rough estimates with their advocates would enable them to fine-tune their concepts before submitting their best and final approach to achieving a specific set of science goals.
In addition to this iterative step early in the mission formulation phase of a decadal survey, Bearden also advocated an analysis of alternatives strategy. This would entail the formulation of two to four alternative approaches for achieving a specific set of desired science goals. These alternative approaches would stretch along a continuum of possible implementations, from one that achieves minimum acceptable science return at minimum cost to one capable of maximizing the science return at the maximum tolerable cost. By analyzing each of the alternatives with CATE, a survey committee could build up a better understanding of the interplay between cost and technical risks associated with achieving a specific set of science goals and, thus, have a firmer basis for their decision-making processes.
The events that occurred following the release of the 2011 planetary science decadal survey, as related by Scott Hubbard, were very useful, in Bearden’s opinion, because they forced the community to think about how they could get priority missions inside a particular cost box. The Mars and Europa teams found some very compelling designs once they understood the cost box within which they had to fit. Bearden also believed that survey committees should define how mission priorities might change in response to changing external circumstances. Such rules are useful because they avoid winner-take-all situations. They provide flexibility by allowing advocates for specific concepts to respond to new information and allow them, potentially, to come up with better missions downstream.
Bearden concluded his comments with a few words about interactions between decadal surveys and the CATE team on one side and the scientific community and mission advocates on the other. There are differences of opinion as to how much interaction can or should take place. The argument being that too much interaction might compromise the independence or objectivity of the survey and/or the CATE process. However, the interactions Bearden witnessed as a participant in the rescoping of the Europa orbiter missions after the release of the 2011 planetary science decadal survey suggested to him that the
integrity of the CATE-advocate interactions could be maintained. The CATE team was unconstrained in terms of its ability to interact with the project team at JPL. These interactions were, in Bearden’s opinion, very useful because everyone learned something, and everyone came to a better understanding of what was being proposed, what was feasible, and what a Europa mission might actually cost. He finished by noting that an open dialog between the NRC and the community should be initiated to determine the right balance between objectivity and arms-length objectivity.
Workshop participants made comments and posed questions to the panelists, as described below. Topics discussed included the following:
• Interface between the CATE process and program formulation,
• Effect of CATE on decadal survey process,
• Feedback in the CATE process and schedule flexibility,
• Credibility of the CATE process,
• CATE for missions lacking precedent, and
• Risk consideration in the CATE process.
Interface Between the CATE Process and Program Formulation
An audience member asked the panelists to describe the interface between the highly technical and time-consuming CATE process with the deliberations of a survey panel. Harlan Spence responded that you need to educate the committee members and the community early in the decadal survey process as to what the CATE is and what it is not. The CATE process is time consuming and sets the pace for all other activities. Moreover, it involves concepts (e.g., point designs and monetization of risk) that are unfamiliar to many survey participants. It is just not possible to learn as you go. Scott Hubbard added that it was clear from the first meeting of the 2011 planetary science survey committee that two cultures were represented at the table—scientists and engineers/program managers. If all survey participants are to understand what is meant by, for example, the 70 percent point on the cost distribution function, then an intentional effort is needed to bridge these two cultures. Finally, Jay Bookbinder noted that an iterative approach to both the mission study and CATEs—for instance, low-fidelity studies followed by higher-fidelity studies of the most promising concepts—might speed up the mission-evaluation process.
Effect of CATE on the Decadal Survey Process
An audience member said that the CATE process distorts the decadal surveys by producing the wrong costs for the wrong missions. History suggests that many of NASA’s major missions—for example, Spitzer and Chandra—are descoped versions of the missions recommended by decadal surveys. Over reliance on CATE-derived costs can distort the prioritization process. For example, a descoped mission achieving 90 percent of a particular set of science goals at 50 percent of the cost (relative to the relevant survey’s mission that was evaluated by CATE) would have a higher science return per dollar than its decadal survey counterpart.
In response, Randall Friedl noted that the full-up costing of point designs may have less utility than using a CATE-like exercise to define the cost boxes into which specific notional missions might be sorted.
Scott Hubbard commented on the subtext he detected behind the question—that the recent surveys devoted too much attention to mission costs and not enough time to science. His experience was
that the planetary science survey went out of its way to keep its focus on the scientific capabilities of the mission concepts under consideration. The descoped Europa and Mars rover concepts that have arisen following the publication of the planetary science survey are consistent with decadal science goals.
Finally, Steven Battel added that his experience on two recent decadal surveys is that the CATE process does perturb the study process, but this perturbation related to the schedule rather than prioritizing the wrong mission concepts. The time-consuming nature of the CATE forces the survey committee to make decisions earlier in the study process than perhaps would be optimal.
Another audience member asked if the CATE process forces low-cost missions, which would typically not be prioritized by a decadal survey, into more expensive cost bins to gain visibility. There was some evidence from the solar and space physics decadal survey that some concepts that might have been implemented as Explorer missions were artificially inflated in scope so that they would gain stature by being subject to the CATE process. David Bearden responded by commenting that there is probably a lower limit below which the CATE process does not have anything to add to the decadal process. The CATE has more utility in analyzing billion-dollar missions that have the greatest potential to break through cost caps. It is most effective in looking at the relative costs of a set of missions.
Feedback in the CATE Process and Schedule Flexibility
An audience member asked the panelists if the decadal survey schedule allows for the scientific capabilities of descoped concepts to be revalidated before being “reCATEd.” There was insufficient time to do this revalidation in the context of the 2013 solar and space physics decadal survey. Bookbinder suggested that when a mission concept is proposed to a decadal survey, its advocate includes a sensitivity analysis, that is, a first-order indication of how science requirements drive cost. So, if, for example, the primary cost driver of a concept is the size of its optics, then the sensitivity analysis would indicate how the use of smaller optics would impact science goals. There also needs to be a reconciliation process if the cost resulting from a particular CATE analysis deviates significantly from that suggested by the concept’s advocates. Without post-CATE reconciliation, there is no way to be sure if a cost discrepancy arises from fundamental errors on the part of the mission’s advocates or on the part of the CATE team or from miscommunication.
Another audience member asked how to optimize the communication between the CATE team and mission advocates while maintaining the independence of the former. Spence noted that in the context of the solar and space physics decadal survey, the communications between the mission design team and CATE team were good. When a CATE analysis suggested that a mission concept was going to cost more than expected, some tweaking was allowed, but a complete redesign of the mission was not permitted. Battel added that the survey committee had to maintain ownership of the whole process. As such, there were numerous interactions between survey committee members and both the mission design teams and the CATE teams in the conduct of the solar and space physics decadal study. These interactions became more efficient and productive as the survey progressed. Similar interactions also occurred during the other recent surveys.
Credibility of the CATE Process
A member of the audience asked if, given the non-robustness of the models underlying the CATE process, the decadal survey is fostering unrealistic expectations among stakeholders concerning our ability to design missions to cost. Bearden responded that the CATE represents the art of the possible of the possible. The CATE is designed to explore whether or not there is a mission concept that can achieve specific science goals within given fiscal and technical constraints. It is not intended to be a design-to-cost tool. Many observers state that a mission’s cost is not really known until the preliminary design review phase of development. However, historical trends indicate that a 30, 40, or 50 percent growth in
cost is possible in the latest stages of mission development, that is, post critical design review. Therefore, missions at their earliest stages of definition, as is the case in a decadal survey, have cost uncertainties of plus or minus 50 percent. The key is: Where does a survey stop trying to do NASA’s job for it? In Bearden’s opinion, NASA did its job after the release of the 2011 planetary science decadal survey. The survey report included decision rules that gave NASA sufficient flexibility to explore possible descopes and find more cost effective approaches to implementing the survey’s top-priority missions. Cost uncertainty is a fact of life, and this needs to be communicated to all stakeholders.
Another audience member asked if the events since the release of the 2011 planetary science decadal survey have demonstrated that the previously expressed concern about the potential distortion of the decadal process by “CATEing” the wrong missions is, for at least that survey, unfounded. Subsequent examination of NASA’s descoped versions of the survey’s high-priority Mars and Europa missions by the relevant community-based analysis groups had indicated that they are capable of addressing most, if not all, of the respective decadal science goals at substantially reduced cost. Hubbard agreed with the questioner’s point and noted that the Space Studies Board’s (SSB’s) Committee on Astrobiology and Planetary Science has also examined the descoped missions and have informally reached the same conclusion.
A different audience member asked if the CATE process could overcome some of the scope-creep problems that occurred during the implementation of missions recommended by the 2007 Earth science decadal survey. Randall Freidl said that one of the missions recommended by that survey grew so much in scope during its implementation that it was ultimately cancelled when its cost came close to $1 billion. Nevertheless, a spacecraft addressing the same science goals with the same basic instrumental approach was subsequently proposed as a $150 million Venture-class mission. Will the application of the CATE process ensure that future Earth science missions are assigned to the correct cost bins? Bearden responded that the “Christmas-tree” effect is a well known pathology. Subjecting such a mission to the CATE process would identify its likely cost. Then the survey committee would assign the implementation to its relevant science goals to a particular cost box and, thus, effectively fence it off from scope creep.
Friedl disagreed, saying that he does not believe that the CATE process would have made a scrap of difference concerning the fate of this particular mission. The 2007 Earth science survey did not include the appropriate language fencing-off this particular mission from the scope creep that contributed to its ultimate cancellation. A CATE alone is not sufficient. The survey committee needs to include the appropriate decision rules in their report.
CATE for Missions Lacking Precedent
An audience member asked how the CATE process addresses missions with few, if any, precedents in the relevant databases, for example, constellations of microsatellites. In response, Bearden commented that the Aerospace Corporation looks at obvious issues, such as learning curves and recurring and non-recurring costs; the nth unit typically costs half of the first unit. There are a few datasets that help in the costing of constellation missions, for example, science missions such as THEMIS and communication-satellite constellations such as Iridium. A key factor for constellations is reliability and acceptance of risk. The survival of a single element is less important that the survival of the constellation. Nevertheless, we have a way to go before we fully understand all of the cost and risk issues associated with constellations of microsatellites.
Battel added that there is another source of risk associated with constellations—the ability of the scientific community to provide multiple copies of the same instruments. There have been examples recently where the demands of several spacecraft for similar systems have saturated the community’s ability to respond. New approaches to manufacturing instruments and risk management may be necessary.
Risk Consideration in the CATE Process
An audience member asked to what extent risk associated with a particular mission concept is assessed during the CATE process. Bookbinder responded that, as someone who has proposed mission concepts to both the 2010 astronomy and astrophysics and 2013 solar and space physics decadal surveys, his submissions always included an assessment of the principal sources of risk. A key issue for the future is a reconciliation of any differences between the risk assessment performed by a mission’s advocate and by the survey’s CATE team. Bearden added additional comments as to how the CATE team assesses risk. The risk assessment begins with an examination of mission requirements, as defined by its advocates. The CATE team then looks at risk-related issues for a subset of missions and/or an instrument selected from the Aerospace Corporation’s database of past missions that are most analogous functionally, organizationally, or via risk posture to what is being proposed.
At this point SSB Director Michael Moloney took the opportunity to publically thank Marcia Rieke and Steven Battel for their role in helping the NRC design the CATE process. He also thanked David Bearden and Russell Persinger and their colleagues at the Aerospace Corporation for going above and beyond the call of duty in their role of implementing the CATE process in the context of three decadal surveys.
