The decadal surveys have many stakeholders and are intended to guide policy decisions for the Earth and space sciences across the legislative and executive branches. Ultimately, though, it is up to the agencies to implement recommended programs as best as they can. The session on agency-specific issues consisted of a discussion between representatives of the four divisions within NASA’s Science Mission Directorate (SMD) for Earth science, astrophysics, planetary science, and heliophysics and a representative from the National Oceanic and Atmospheric Administration (NOAA). Representatives from the National Science Foundation and the Department of Energy, who had varying degrees of involvement in some of the decadal surveys, were unable to attend.
|Moderator:||Mark Abbott, Dean, College of Earth, Ocean and Atmospheric Sciences, Oregon State
University; Chair, Committee on Earth Science and Applications from Space; Member, Space Studies Board
|Panelists:||Michael Freilich, Director, NASA Earth Science Division
Jeffrey Newmark, Strategic Planning Lead, NASA Heliophysics Division
Lisa May, Mars Program Executive, NASA Planetary Science Division
Paul Hertz, Director, NASA Astrophysics Division
John Pereira, Chief, Advanced Satellite Planning Division, NOAA/NESDIS
The moderator, Mark Abbott, opened the session by noting that a major contemporary challenge for surveys is to be useful in today’s constrained budget environment in which “flat is the new normal.” He posed four questions for the panelists who represented federal agency customers for decadal surveys:
• How do you interpret and use decadal surveys?
• What works well and what does not work well?
• What would you like to see in future surveys?
• How do you coordinate within your agency on common issues and recommendations?
Michael Freilich began his response by expressing his enthusiasm for the 2007 inaugural decadal survey for Earth science and applications from space. He said that the report was enormously valuable in view of its focus, clarity, and utility supporting NASA decision making. Freilich emphasized three points regarding the scope of surveys:
• Surveys have a number of roles—including community building, budget advocacy, and management guidance and prescription—but these tend to be mutually incompatible. Therefore, survey committees and their reports need to be very clear about a survey’s objectives. What works well and what does not work well?
• In the Earth sciences, a survey is not just for a single agency, there are important recommendations for both NASA and NOAA. But agencies respond to surveys in different ways that reflect their different priorities and core roles. When a national program is dependent on key roles for each agency, one agency cannot be left “off the hook” so that it can decline to accept its responsibilities.
• Launch vehicle costs have a major impact on overall program costs, and they need to be considered realistically.
Paul Hertz agreed with Freilich that the surveys “are incredibly valuable.” For example, the surveys are used to inform peer review at every level; they define relevance for a program. Noting that the challenge is about how to get good advice in a dynamic and often rapidly changing environment, he offered three recommendations for future surveys:
• Focus on prioritizing science objections more than missions because science priorities can be more enduring and more robust than a recommended mission queue. Therefore, it is better for a survey committee to identify broad scientific and program priorities and then let NASA use those, instead of a committee engaging in extensive scenario analysis and decision-rule crafting.
— Later during the discussion, a member of the audience noted that moving from prioritizing science to prioritizing missions can often resolve competing ideas that are otherwise impossible to compare. Another member of the audience added that if missions were not prioritized, then advocates would push Congress for earmarks; he asked, Do we want that?
• Do not attempt to make the cost and technical evaluation (CATE) process “better;” i.e., don’t try to push the process to provide more accuracy or detail. When a mission concept is only roughly defined, it can only be costed up to a point. Therefore, the CATE for any mission concept design will have a limited lifetime.
• International participation needs to be better incorporated into the survey process. The different planning processes in two sides of a potential international partnership can actually interfere with each other, and that obstacle needs attention.
— Later during the discussion, a member of the audience expressed his agreement that future surveys need to include a careful assessment of potential partners capabilities.
Lisa May commented that surveys provide a framework to explain program structure and priorities to many readers who may not be especially familiar with those key aspects of a program. She added that the 2011 planetary science survey was very responsive to the study charge, and, hence, it succeeded at this task. In looking ahead to future surveys, in agreement with Hertz, May said that having a strong focus on science priorities gives NASA more flexibility to respond to unexpected changes. Furthermore, surveys should not be so overly prescriptive that they compromise the innovation that comes from competition. She also agreed that there is a need to avoid having survey committees get into too much detail with the CATE process. Asking for more detail about a mission concept that is not mature does not, in May’s opinion, make it any better defined. Furthermore, she posited that perhaps CATEs should be conducted after, rather than before, a survey is completed. May observed that there are number of common objectives that cut across all areas and that these could be considered in the next round of surveys. For example:
• Science priorities should take precedence over mission priorities, and
• Mission concept Phase A studies could be funded under the ROSES1 grants program in advance of the survey so that the results would be available to the survey committee. May noted that in the recent survey, NASA made substantial funding available at the request of the survey committee to support mission studies of interest to the committee.
May cautioned, however, that it is also important to respect the differences in different disciplines that present aspects that need not be common across surveys.
Jeffrey Newmark introduced two new points to those already discussed by his colleagues. First, he noted that interaction between NASA and the survey committee before and during the survey is quite important and could be improved in the future. Such interaction is needed, he said, to help the committee understand and deal with nuanced aspects of budgetary constraints. Second, it is very good for the survey report to include recommended decision rules for how to respond to unanticipated budgetary, policy, or technical developments. He applauded the inclusion of such decision rules in the 2013 solar and space physics survey. During the subsequent round-table discussion amongst the panelists, Freilich noted that decision rules can be easier to write than to implement at NASA; perhaps it would be better for a committee to recommend that NASA develop decision rules instead of telling NASA what the rules should be.
John Pereira brought a NOAA perspective to the panel’s discussions. He said that responding to the 2007 Earth science and applications from space survey was difficult because it was released just after the Nunn-McCurdy2 review of the National Polar Orbiting Environmental Satellite System (NPOESS), which led to removal of certain sensors from the NPOESS payload, but the survey report had recommended keeping those sensors. In spite of such difficulties, Pereira said, NOAA is (or already has) acted on some of the recommendations from the survey report. Pereira offered three recommendations for future surveys:
• Determine the principal agency organizations that will receive the recommendations and define the scope of their responsibilities before the survey starts;
• Aim to make survey reports available in time to be used at the beginning of the annual federal budget cycle; and
• Recognize that cost estimates in survey reports generate expectations; the cost data take on lives of their own and can lead to later misunderstanding if they are not properly caveated.
During their discussion, the moderator and panelists talked about their experience with and opinions on the following:
• Managing the decadal survey-NASA relationship,
• Decadal survey ground rules and preparatory activities,
1 Research Opportunities in Space and Earth Sciences.
2 “On September 8, 1982, President Ronald Reagan signed into law the Department of Defense (DOD) Authorization Act, 1983 (P.L. 97-252), which included what has come to be known as the Nunn-McCurdy Act (10 U.S.C. § 2433). The Nunn-McCurdy Act requires DOD to report to Congress whenever a major defense acquisition program experiences cost overruns that exceed certain thresholds. The purpose of the act was to help control cost growth in major defense systems by holding the appropriate Pentagon officials and defense contractors publicly accountable and responsible for managing costs.” M. Schwartz, The Nunn-McCurdy Act: Background, Analysis, and Issues for Congress, Congressional Research Service, 7-5700, June 21, 2010, available at http://fpc.state.gov/documents/organization/145135.pdf.
• Continuity and synergy across mission portfolio,
• Decadal survey stewardship, and
The general discussion began with Paul Hertz noting that the decadal surveys are used to inform peer reviews at NASA at every level. That, he explained, is very important because when NASA runs any kind of competition, NASA references the decadal surveys (in addition to the NASA strategic plan) for people to demonstrate that the research they are proposing is relevant to NASA and the science community at large. Despite the utility of recommending specific missions, he explained, the science detailed in the surveys is used just as much and, in some ways, impacts scientists more than the missions do.
Michael Freilich then asked Lisa May how she made the notional mission architecture studies available to the decadal survey committee and panels and how the Planetary Science Division did it in a way that preserved the survey’s independence. For Earth science, the mechanics of sharing information with the decadal committee and panels could have been smoother, and it will be important, looking forward, to more explicitly determine what is expected from NASA Headquarters and relevant NASA centers, what is expected from the CATE process, and, ultimately, what the decadal survey members need to do their job. May responded that the panels of the planetary decadal survey were used to identify mission concepts they would like brought to a greater degree of maturity prior to being subject to further scrutiny. The panels used community input, including white papers, to make such decisions, and then the decadal survey chair passed those along to his NASA counterparts. NASA developed criteria for developing its mission concepts and worked with the panel contacts to ensure that the missions were aligned with the science goals that were coalescing within the survey committee and panels at the time. Steven Squyres commended NASA for not interfering with the decadal survey process, thus forming a strong bond of trust between NASA and the decadal survey’s membership. May also said that laying out clear expectations from the decadal survey—a theme that will continue throughout the workshop and appears in later chapters—was also very important for maintaining a strong relationship between NASA and the decadal survey’s membership.
May also explained how flexibility is needed on the part of NASA in how it manages its relationship with the decadal survey committees. At the same time, peer-reviewed science is critical to getting what NASA needs because the community does a good job of proposing great ideas when done in a competitive environment. However, she cautions, future decadal surveys being overly prescriptive, which can undermine the community’s—and NASA’s—ability to come up with novel ways of overcoming programmatic challenges during a survey’s implementation, to which Jeffery Newmark agreed.
Moderator Mark Abbott asked the panelists if there could be common ground rules across the four science divisions at NASA regarding the decadal survey. Lisa May began with the commonly cited tension between missions and science in the surveys and also mentioned how the CATE process was different for each decadal survey. It would be good for the agency to not have to continually reinvent procedures for every individual decadal survey on an ad hoc basis, she said. One part of the surveys May believes everyone likes are the decision rules that guide program directors on how to implement the decadal and navigate challenges that arise during implementation. She asked her fellow panelists whether that is one area where common ground rules can be established or if the notion of having decision rules
for every decadal is sufficient; it could then be left to each decadal survey to choose how to craft their rules.
In response to May’s comments about decision rules, Michael Freilich said that the rules are a lot more difficult to parse from a NASA perspective than from the standpoint of the decadal survey committees. In his opinion, the enthusiasm from his fellow division directors for the decision rules in “their” decadal surveys is partly a result of how new those decadal surveys are, saying that the division directors for those disciplines have yet to really attempt to follow them. Perhaps what each community can do when they write a decadal survey, he suggested, is to make it exceedingly clear that NASA—as the managers of the enterprise—needs to create a set of clear processes and decision rules for decadal survey implementation, as opposed to trying to impose a set on NASA management via the decadal surveys.
Paul Hertz said that the different divisions prepared for their respective surveys in different ways. The Astrophysics Division ran ROSES competitions and funded mission studies that were principal investigator (PI)-led and provided very useful input in the form of white papers for the decadal survey. The Astrophysics Division also funded large-mission studies, and having all of these studies already completed obviated some the need for the decadal survey to do that work. He compared that to the planetary science decadal survey, which asked for mission concept studies after the decadal steering committee had decided which concepts merit study by NASA.3 Figuring how and when to do these studies is a process that could be standardized across the surveys. Hertz then said that he is not sure there is a best practice for the decadal surveys because the communities are so different, and the types of missions for each community are so different.
Jeffery Newmark agreed that there are commonalities across the disciplines that can be capitalized on, but he concurred with Hertz’s statement about the disciplines being so different that it is difficult to come up with one-size-fits-all rules in many areas for everyone.
All of the disciplines, particularly Earth science and solar and space physics, have to deal with dependencies on missions and measurements and are always at risk of losing those. Mark Abbott asked if the decadal surveys should look at integrated scenarios that would help NASA craft an integrated Earth and space science portfolio, as well as feasible decision rules.
Paul Hertz said that it is not helpful to ask what would you do in this situation or that situation, and listing all types of scenarios. What the decadal survey can do best is explain what the objectives are for science, technology development, and other areas. Given a clear set of priorities, he explained, it is the job of the division directors and program managers to assess what resources are available and how best to apply the agency’s capabilities to fulfill the objectives of the decadal survey.
Mark Abbott suggested the possibility that decadal survey objectives or priorities might be tweaked after the survey is released, noting that there has to be some kind of corrective mechanism in place to help steward the decadal survey’s implementation.
Michael Freilich replied that in the Earth sciences arena, there will always be another group that changes the priorities articulated by the scientific community. Earth science, he said, is very important and challenging, but he cautioned that administrations have their own priorities when they come into office, and the agency and scientific community at large need to come to terms with that. The NRC’s role
3 The Planetary Science Division had conducted some mission studies prior it its decadal survey for large-class missions.
is to articulate the priorities of the academic research and applications communities, but all stakeholders should assume in the Earth remote sensing and solar and space physics disciplines that the decadal surveys will be subsumed by some kind of initiative or overarching architecture like NASA’s 2010 climate architecture.4
John Pereira said that he and his colleagues on the panel are, in fact, stewards of the decadal survey, but they often lack the resources to implement the surveys to the letter.
Mark Abbott noted that science priorities are heavily influence by technical and fiscal feasibility, so even the highest-priority science can be demoted to a lower priority if it requires a prohibitively expensive mission or non-existent technology. Another issue that arose in past decadal surveys, he said, is that in the mission design process, cost and schedule become features rather than drivers, making it difficult to relax requirements to meet science objectives. Given these challenges, he asked the panelists if there are better ways for the decadal surveys to conduct the cost and technical evaluation portion of the survey.
Paul Hertz asked participants to consider the problems that Congress saw as needing to be solved when it requested a more rigorous approach to what we now call the CATE process. Those problems included missions whose actual costs bore no resemblance to the figures provided in their corresponding decadal survey. In astrophysics, he said, people would likely think that the James Webb Space Telescope (JWST) would be one example of this, and it would probably not have taken a much more detailed CATE process than was used in the 2001 astronomy decadal survey5 to determine that JWST would be more expensive than previously thought. Hertz noted how both JWST and other missions, like those used for the 2007 Earth science decadal survey, came from NASA center mission studies and reiterated how there has to be an independent cost assessment process to capture those missions that are overly expensive. However, he did not think that the process needs to be as onerous, detailed, time-consuming, and expensive as the one that the NRC seems to be heading toward. The question for everyone to consider is whether the decadal surveys need a better CATE process or a simpler, coarser process instead. If the NRC advocates for a CATE process that is very good, which in turn is interpreted by Congress and the administration as very accurate, then it risks damaging its credibility and the fate of the recommended missions if actual mission costs are substantially higher than the estimates.
Michael Freilich concurred with Hertz’s comments and asked participants to think about what the cost answer means and what question the CATE process is trying to answer. Is the question: What does the NRC think that a mission will cost, given a specific set of capabilities? If so, then there is plenty of historical evidence showing that it is very difficult to come up with accurate cost estimates before the formal mission design process begins, because the design is too immature. On the one hand, he said, if the question is, What does the decadal survey panel think is a reasonable amount of money to spend in this general area to have some of these sorts of capabilities?, then it is very clear, and, in fact, the decadal survey committees have the expertise to make that kind of decision, and it is up to NASA management to make the necessary trade-offs and other considerations to fit the recommended mission into the box laid out by the survey. During the 2012 Earth science survey midterm review, it was the latter question that became very clear to that study committee and to Freilich. He also reiterated that the high costs of launch vehicles must be factored into mission costs in the decadal survey to ensure they are accommodating this
4 See NASA, Responding to the Challenges of Climate and Environmental Change: NASA’s Plan for a Climate-Centric Architecture for Earth Observations and Applications from Space, June 2010, available at http://science.nasa.gov/media/medialibrary/2010/07/01/Climate_Architecture_Final.pdf.
5 National Research Council, Astronomy and Astrophysics in the New Millennium, National Academy Press, Washington, D.C., 2001.
resource-intensive aspect of missions, especially since there is little indication that launch costs will depreciate significantly in the future.
Lisa May noted that the planetary science survey made clear what the science in the survey is worth and that if the science could not be accomplished below a certain threshold, then other science opportunities should be pursued instead. She said that “box” is a useful aspect of the decadal survey, but the law of diminishing returns applies to requesting more information on immature mission concepts for increased accuracy in the CATE process.
Jeffery Newmark said that the key part for any CATE-like process is to identify the scale, and he noted that the significant digits are not very important; for instance, there is no meaningful difference in this activity between a $500 million mission and a $520 million mission.
John Pereira pointed out that a decadal surveys’ science objectives and missions can have societal benefits, so there is a value that can be quantified from those objectives and missions that factors into larger considerations of how to implement a survey.
Workshop participants made comments and posed questions to the panelists, as described below. Topics discussed included the following:
• Role of the decadal surveys in interagency planning,
• Tension between science and missions,
• CATE process,
• Intra-/interagency cooperation,
• Decadal survey implementation, and
• Decadal survey preparatory activities.
An audience member asked the panelists what the role of the decadal survey is and what it can do for an agency that is not primarily driven by science priorities like NASA. What is the utility, she asked, for such agencies in having a list of vetted science priorities that are not likely to be a high priority when provided to said agencies? Moreover, she asked how any decadal survey committee can create a reasonably integrated Earth science portfolio in such circumstances.
John Pereira said that linking the science to economic and societal benefits is very helpful and important. He does not believe that a decadal survey committee would disagree with NOAA’s mission for providing measurements or meeting its long-term continuity requirements. Nevertheless, if a decadal survey recommends improvements for those measurements or recommends new types of measurements to be gathered, then NOAA would work to incorporate those enhancements to the best of its ability. Pereira made sure to say that he does not think that the decadal survey recommendations are not important because of the nature of NOAA’s mission, but resource limitations make it difficult for NOAA to carry out the survey’s recommendations.
Michael Freilich said that all other agencies treat the decadal surveys differently than NASA does. NASA SMD, to the extent possible, would like to follow as closely as possible the decadal survey. NASA sees the surveys as their raison d’être from a scientific standpoint. For very good reasons, however, agencies like NOAA and the U.S. Geological Survey do not see the survey that way because they have another set of higher priorities. Freilich said one of the problems with the 2007 Earth science decadal survey is that it parsed out responsibilities for different measurements to different agencies, but if one of the agencies had other priorities that precluded implementing a decadal survey recommendation, then the integrated nature of the recommended science program is jeopardized.
An audience member asked the panelists to comment on the balance between science and mission prioritization. For instance, in astrophysics it is very difficult to say that one area of research is more important than another, but once specific missions are taken into account, there are new options available to the program. One example of this would be the astrophysics community’s use of observatories that can address a number of science objectives recommended in the survey.
Another audience member asked Paul Hertz if he were writing a new statement of task (SOT) for an astronomy and astrophysics decadal survey today, how he would change it with the benefit of hindsight related to the tension.
Paul Hertz replied that he would not change many aspects of the SOT for the 2010 survey, particularly the part about not uncritically accepting the claims of the advocates (even if from NASA headquarters or a NASA center). He could not say what exactly he would change, but he recalls that the SOT did not explicitly ask for prioritized science goals, although the decadal survey did that anyway, which Hertz is using to structure NASA’s astrophysics program.
A different audience member explained that in the debate between prioritizing science objectives or prioritizing missions, when a mission is not prioritized in a decadal survey, advocates for those missions often go out into the community and lobby Congress, which could—and has, he noted—result in an unfunded mandate by Congress to design and build a mission without the necessary funds to do so.
An audience member said that no one thinks that the community needs to rely on cost estimates that are accurate down to the nth-significant digit. He suggested that the next round of decadal surveys take into account what the CATE numbers are actually going to be used for. For example, the 2011 planetary science decadal survey did not exclude mission concepts for the New Frontiers program just because they were slightly above the $1 billion cost-cap of that program and used the CATE process to come up with novel ways of turning a $1.3 billion mission into a $1 billion or less mission.
Paul Hertz said that not only will those numbers be used by the panel, but if the numbers are written down in the decadal survey, then they will be used by others in different ways than the survey committees do.
An audience member noted how heliophysics has profited a great deal from being involved in planetary and Earth science missions. He expressed concern that the current austere fiscal environment will result in a contraction away from this interdivision and interagency cooperation toward more insular, division-only goals.
Lisa May replied that many heliophysics aspects of planetary missions are in fact integral to those missions. Because of this, she said that these partnerships are actually leveraging agency capabilities and are not something they would withdraw from.
Jeffery Newmark highlighted the synergy between astrophysics and heliophysics on NASA’s Kepler mission. He expressed the hope from his division’s standpoint to continue pursuing cross-division collaboration whenever possible.
Michael Freilich said that there are good resources why the divisions are collected together under the aegis of SMD. Accordingly, not only will his and the other divisions be sensitive to cross-division synergies, but SMD management will make the divisions sensitive to such synergies. Where the audience member suggested budget cuts would result in less cross-division or interagency collaboration, Freilich believes that a smaller budget could actually open up synergistic opportunities among the divisions.
An audience member noted that the mission-defining science requirements detailed in a decadal survey can often be immature since they were not written by the people responsible for implementing the decadal surveys. He asked the panelists how much discretion they think they have to implement the decadal surveys and noted that oftentimes the trade-offs that program managers and/or mission designers make can result in cost increases. This, he wondered, might be the result of the decadal survey not explicitly giving NASA the flexibility needed to prevent cost and scope growth when designing decadal survey missions.
Michael Freilich said that is the job for him and his fellow panelists as the program managers for their divisions; that’s what makes their job challenging and fun. In order to do that, the NASA program managers have to work with the decadal survey committee and the wider community to elucidate and justify why they make the decisions they do.
Paul Hertz said that all of the divisions use similar processes, such as community science and technology definition teams, to explore the trade space between capability, science, and cost to identify the “sweet spot” that allows for maximum science return in a reasonable budget envelope.
Lisa May said that, in almost all cases, mission cost estimates will go up once they enter the formal design phase because there are issues that simply cannot be thought of during the more informal and immature design process of the decadal survey. She also explained how science objectives and requirements are further refined along the way up to the preliminary design review of the formal mission design process. Thus, she does not consider a science question from a decadal survey to be a hard requirement.
John Pereira asked participants to consider what the minimum threshold of implementation of a decadal survey is in order for it to be useful and worthwhile in the context of limited financial resources.
A member of the audience asked the panelists—assuming they were in their same position when the next round of decadal surveys begins—if there are any preparatory activities they would be willing to spend money on prior to the commencement of their discipline’s decadal survey.
Paul Hertz said the most important thing is to develop the requisite technology to a reasonable technology readiness level so that the missions that technology can support can be evaluated realistically and prioritized accordingly. He recalled how many missions considered by the 2010 astronomy and astrophysics decadal survey were not prioritized because the necessary technologies were too immature and posed too much cost risk to get the mission ready.
Jeffery Newmark concurred with Hertz’s comments and said another area of preparation to consider for heliophysics is to identify gaps in the heliophysics system, which is important because of the integrated nature of the heliophysics program—much akin to the Earth science program. Identifying those gaps would help the next decadal survey come up with goals or means for closing those gaps; even though the 2013 solar and space physics likely tried to do that, NASA can certainly help that effort by doing this activity in advance of the survey.
John Pereira said that future decadal surveys, at least in Earth science and applications from space, should take into better account commercial industry capabilities for meeting some of the science and/or mission goals of the decadal survey. He cited one example of a recent award by NASA’s Earth Venture program for pollution monitoring that was intended to be a government payload hosted on a commercial geostationary satellite. Pereira said that he and his colleagues receive many questions about the ability of industry to provide services, as opposed to a government-developed solution; it would be helpful for an outside body like a decadal survey committee to assess that landscape.
Michael Freilich said that a modest investment in a critical review of the capabilities and plans of NASA’s international partners (in the Earth sciences) would probably be exceedingly useful to the decadal survey committee.
Lisa May—noting the different ways astronomy and planetary science handled their mission solicitation processes before and during their decadal surveys—said that more investigation is needed to find the balance between having NASA science divisions solicit mission studies from the community at large prior to a decadal survey versus the decadal survey committees sending mission concepts to NASA centers for on-demand studies. Hertz said that the peer review process for his division’s ROSES solicitation that occurred before the 2010 astronomy and astrophysics decadal survey served as a proxy for the community and did not amount to NASA pre-selecting missions on its own. May said that this exchange highlights some ideological differences among her colleagues about how best to handle this process, and she reiterated her call for more investigation into this issue.