Decadal surveys are a signature product of the National Academies of Sciences, Engineering, and Medicine.1 Decadal surveys conducted by the Space Studies Board, singly or in collaboration with other boards of the Academies, provide community-consensus science priorities and recommendations for space and Earth science, principally to NASA and the National Science Foundation (NSF), but also to the Department of Energy (DOE), the National Oceanic and Atmospheric Administration (NOAA), the U.S. Geological Survey (USGS), the White House, and Congress. The Academies have established a reputation for decadal surveys as credible and unbiased science assessments and prioritization across the space sciences.
Decadal surveys are carried out with a cadence of approximately 10 years for each discipline. The four that are the focus of this report are Earth science and applications from space, astronomy and astrophysics, planetary science, and solar and space physics (also known as heliophysics). The Academies have conducted decadal surveys for more than 50 years, since astronomers first developed a strategic plan for ground-based astronomy in the 1964 report Ground-Based Astronomy: A Ten-Year Program.2 The committees and panels that carry out the decadal surveys are drawn from the broad community associated with the discipline in review, and these volunteers comprise some of the nation’s leading scientists and engineers.
The Academies’ decadal surveys are notable in their ability to sample thoroughly the research interests, aspirations, and needs of a scientific community. Through a rigorous process lasting about 2 years, a primary survey committee and “thematic” panels of community members construct a prioritized program of science goals and objectives and define an executable strategy for achieving them. Decadal survey reports to agencies and other government entities play a critical role in defining the nation’s agenda in that science area for the following 10 years, and often beyond.
Eleven decadal surveys have now been completed; the last four have been for Earth science and applications from space (Earth2007), astronomy and astrophysics (Astro2010), planetary science (Planetary2011), and solar and space physics (Helio2013).3 The 2012 Academies’ workshop “Lessons Learned in Decadal Planning in Space
_______________
1 Activities of the National Research Council are now referred to as activities of the National Academies of Sciences, Engineering, and Medicine.
2 National Academy of Sciences, Ground-Based Astronomy: A Ten-Year Program, National Academy of Sciences-National Research Council, Washington, D.C., 1964.
3 The four decadal survey reports discussed are Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond (2007), New Worlds, New Horizons in Astronomy and Astrophysics (2010), Vision and Voyages for Planetary Science in the Decade 2013-2022 (2011), and Solar and Space Physics: A Science for a Technological Society (2013), all published by the National Academies Press, Washington, D.C.
Science,” invited participants from recent surveys and “stakeholders,” such as NASA and NSF division directors, congressional staffers, and representatives of the executive branch. Presentations and moderated panel discussions, with inputs from the gathered attendees, covered all aspects of these recent decadal surveys. The resulting report, Lessons Learned in Decadal Planning in Space Science: Summary of a Workshop,4 captures the breadth and depth of this exceptional, challenging process.
The Committee on Survey of Surveys: Lessons Learned from the Decadal Survey Process (hereinafter “the committee”) was appointed by the Academies with the task of distilling the content of the 2012 workshop, adding the input from presentations to the committee, and providing its own evaluations of the issues. The committee’s goal has been twofold: (1) to provide a handbook to guide the organizers of future surveys, with a moderately detailed discussion of both “tried and true” and novel methods and (2) to identify lessons learned from prior surveys and best practices that have been gleaned from them. Along the way, the committee has identified valuable aspects of decadal surveys that could be taken further, as well as some challenges future surveys are likely to face in searching for the richest areas of scientific endeavor, seeking community consensus of where to go next, and planning how to get there. What decadal surveys are asked to do is no simple task.
The committee’s conclusions are presented in the context of a successful round of recent decadal surveys that faced a few challenges but surprisingly few issues, considering the magnitude of the assignment. In particular, the task of defining the scientific frontier and deciding on a discipline’s future direction is complex and difficult, but this has been done smoothly and reliably through the decadal survey process. The same is true for decadal surveys achieving community consensus on how to advance a field with a 10-year program. Indeed, the committee found no evidence of widespread dissatisfaction about the outcome of a decadal process of prioritizing science activities: no one at the 2012 workshop, or in any other communication to the committee, suggested the outcome was capricious or arbitrary, tied to the composition of the relevant survey committee, or not representative of a community consensus of its highest-priority science goals. On the contrary, the science communities, through individuals and associations, have given strong support in recommending each of the decadal survey reports to its stakeholders.
Likewise, support from the sponsoring agencies for decadal surveys has not wavered over their 50-year history. NASA and NSF officials, in particular, use words like “guidebook” and “blueprint” to describe the role that decadal survey recommendations play in the planning and execution of science programs of government agencies on behalf of the nation. Federal funding has long been an essential component of the entire U.S. science portfolio, but few fields have chosen a democratic process like the decadal survey for deciding how best to direct this resource. Decadal surveys have been praised as a “sword and shield”5 as they work to advance the nation’s science agenda—a sword for winning the approval of the most important programs, and a shield against cancellation when difficulties are encountered and against groups that lobby for certain programs that may not enjoy the consensus support of the community.
This report covers the entire decadal survey process in time order. Chapter 1 provides an overview of decadal surveys, outlines high-level implementation process, and discusses key issues associated with a decadal survey’s statement of task. Chapter 2 reviews the decadal survey process in detail, including mission definition and formulation, prioritization, and the process of cost and technical evaluation (CATE). Chapter 3 covers the decadal survey report itself, including discussion of the importance of clarity of communication of recommendations, particularly with respect to “flagship,” “strategic,” or “high-profile” missions.6Chapter 4 focuses on “stewardship” of the decadal survey after the report is released, including discussion of the midterm assessment process and the vital roles played by international and interagency cooperation. Lessons learned and best practices are included as they arise throughout the report and are also collected in Appendix D. Appendix B provides additional material on the CATE process.
_______________
4 National Research Council, Lessons Learned in Decadal Planning in Space Science: Summary of a Workshop, The National Academies Press, Washington, D.C., 2013.
5 Attributed to Colleen Hartman at the 2012 Workshop; see National Research Council, Lessons Learned in Decadal Planning in Space Science, 2013, p. 39.
6 The terms flagship mission, strategic mission, and high-profile mission are typically used interchangeably to mean large, expensive, technically ambitious, performance-driven activities that are initiated for strategic reasons because they are critical to the advancement of a specific discipline. The committee prefers to call such activities high-profile missions.
As the decadal process first developed for astronomy and astrophysics has been extended to planetary science, solar and space physics, and Earth science, different science themes and unique cultures have been expressed through variations in decadal structure and process, but overall the survey model has proven to be highly adaptable. There is no “one-size-fits-all” approach to a decadal survey: each discipline has heritage and science goals that cannot be directly mapped to any other group. However, there is also much in common—things that every decadal survey needs to do well. Each must draw extensive input from its community and adhere to a process that assures that all ideas are heard—the most important thing is that no good idea is simply missed. All surveys need to demonstrate that science is the prime motivator and develop a methodology of prioritization that identifies the most important science areas where substantial progress can be made, which also means demonstrating to skeptics and partisans that favored activities or highly lobbied missions do not drive the survey’s recommendations.
Crucially, all surveys must put considerable effort into communicating their conclusions, goals, and recommendations to a wide audience of scientists, stakeholders, and the public. The decadal survey report must explain and justify the recommended program and provide clear direction, through priorities and “decision rules” that will help in the implementation of the survey, even as the budget, technology, and in some cases the science, change throughout the decade.
Finally, all survey programs require continued support and nurturing—stewardship, and even advocacy—after they are completed and released. The standing committees of the Space Studies Board play a key role in this stewardship. This committee thinks that this role could be strengthened by allowing the standing committees to continue their work while a decadal survey is in progress, provided they restrict their attention to the current program. In addition, while there are many groups that can speak to the progress in post-survey execution of a decadal program, one lesson learned is that the current advisory structure does not adequately provide for short-term tactical advice on strategic programs.
Although the decadal surveys’ record concerning issues relating to international collaboration and cooperation is good, simple steps can be taken to improve communication before and during a decadal survey. With increasing dependence on international cooperation, activities before a survey begins that facilitate interactions with international groups can be used to better coordinate discussions of shared science goals that can—and should—be pursued through international collaboration.
Differences between the various disciplines are expressed in the organization of each survey. While there is much uniformity in decadal survey committees, the uniqueness of each discipline is reflected in the organization of thematic panels and study groups that are charged with representing the community’s full science interests.
Differences among the disciplines are strongly expressed in the values that inform the survey’s selection of the highest-priority science goals. For example, the discipline of astronomy and astrophysics has two distinct science “imperatives”: “origins” science—how do galaxies, stars, and planets form (and lead to life)—and fundamental physics—the nature of black holes (space-time), cosmology (dark matter and dark energy), and the study of elusive gravity waves and neutrinos. Solar and space physics (also called heliophysics) similarly seeks to further understanding of the fundamental physics of the Sun and its variations in time, the acceleration of particles and the solar wind, Earth’s geospace environment and its links to the Sun, and the Sun’s connection to other bodies in the solar system and to the galaxy beyond. Heliophysics also explores astrophysical processes in the nearby cosmos as well as the impacts of space weather on human activities.
Planetary science has its strong link with the physics of complex matter—condensed matter, chemistry, geology, and biology. In the prioritization of planetary science goals, these disciplines underlie the “hottest topics”: the search for water and life on Mars or within the icy moons of the outer solar system; the history of volcanism on Venus, the Moon, and on icy satellites; and the composition of comets, asteroids, and planetoids that hold clues to the solar system’s formation.
Earth science and applications from space and, to a significant extent, heliophysics are focused on complex natural processes: both fields place a high priority on establishing decades of synoptic data. For Earth science, this entails, for example, measurements of land and sea temperatures and atmospheric composition and their collective effects—weather, climate, and climate change. Long-term heliophysics measurement of levels and characteristics of solar activity, cosmic rays, irradiance, and conditions in geospace can provide critical information about the causes and effects of the solar cycle, extreme events, and “space weather.” These are matters of national interest
and importance. For example, the degree to which weather satellites facilitate “routine” weather prediction is likely to dominate whether they bring fundamental knowledge to meteorology. In short, the variety of natural processes that drive each of these fields is enormous.
In addition, there are substantial differences in the targets of science programs and how science is done: from remote sensing of galaxies a billion light years away to observations of a planet orbiting a distant star; from visiting or roaming on solar system bodies to making continuous, precise, sensitive measurements of conditions on or near Earth over long temporal baselines. Working in the context of such variety of subject and methodology, the decadal process has proven highly adaptable and remained effective in its mission to prioritize science goals and make plans to accomplish them.
This report describes many other aspects of the decadal survey prioritization process, including balance in the science program and across the discipline; balance between the needs of current researchers and the development of the future workforce; and balance in mission scale—smaller, competed programs versus large, strategic missions. While engaging the public is important for all, Earth science and heliophysics have a special focus on societal benefit; outcomes here have unique, real consequences for life on Earth.
There seems little if any doubt that decadal surveys have succeeded in what they set out to achieve; yet, to paraphrase a philosopher, “no fruit of the human tree has ever lacked for improvement.” In its examination of the process, the committee has identified challenges that have made the process of crafting a decadal survey more difficult and affected committees’ ability to do the best possible job.
An important lesson learned has been that budget uncertainty complicates the development of an executable and affordable program. With only a few exceptions, decadal survey programs have been more ambitious than could be accomplished, or at least begun, within the decade ahead. Decadal surveys have been reluctant to adopt the “worst case scenario” budget for fear they will be given it, especially in times of tight budgets. On the other hand, “optimistic” or “aspirational” programs often turn out to be “overly optimistic” or even unreachable. In addition, some uncertainty results from the “blackout period” during which details of the federal budget are embargoed, something that suspends communication between the agencies and surveys on budget expectations. There is also a black-out period lasting several months when the main elements of a decadal survey’s recommended program have been established but cannot be discussed with the agencies until the survey report’s review by the Academies is complete and the report is made public.
Because budget uncertainties seem inevitable, a best practice might be to replace the extrapolations of a current or newly released budget with a baseline that reflects longer-term funding levels for NASA SMD and relevant partner agencies such as NSF and NOAA. Surveys could then build in budget scenarios that “trend-up” and “trend-down” over the decade, as alternatives to the nominal, “baseline” plan they have provided. Greater stability in agency budgets for science would be wonderful, but intentions of the executive branch and congressional priorities seem to guarantee fluctuations as large as 20 percent over a few-year timescale. It seems unwise to base a survey program on a budget run-out for a decade by primarily relying on what has happened only in recent years or on the latest projections of executive or congressional priorities.
Planning within tight budgets has led to increased specificity in the recommended programs of decadal surveys. Implementation plans, in particular, have included detailed descriptions of the facilities, missions, and observing system concepts that have been motivated by the desire to accomplish as much of the science program as possible. However, over-specified programs are a problem for program managers at the agencies for several reasons. One is that implementation of a particular mission architecture is often much more costly than the estimate derived from studying an immature concept (as was the case for the James Webb Space Telescope (JWST) and the Mars Science Laboratory (Curiosity rover). The full cost of ambitious, high-profile missions may not be knowable at the time the survey is conducted.
The lesson learned here is that decadal surveys, in pursuit of ever more accurate cost estimates, may dig too far into implementation details. Implementation descriptions for such missions in the survey report can be easily misconstrued as prescriptive advice. A best practice going forward is that missions described in the survey’s recommendations might best be considered as “reference missions,” except for the concepts that have been studied for many years—where committees explicitly state their intention to recommend a specific implementation approach. A reference mission is intended to serve as a proof of concept that there is a way to do the science within a certain
cost bin, rather than as a detailed recommendation for implementation. After the survey process, the agencies will develop these ideas to take into account other programmatic goals, new technology, and a growing understanding of what it will take to do the mission or build the facility or observing system. The most important thing is for the decadal survey to state clearly the minimum set of requirements underlying a mission’s recommendation and the rationale for its prioritization, including any necessary decision rules to be considered by implementers. After all, it is first and foremost the science that is being prioritized in a decadal survey, not any particular design for a mission or facility.
The committee was asked to consider another way of decreasing the attention given to implementation strategies: a two-phase approach in which decadal survey committees would be asked to prioritize science goals first—independently of the means to carry them out. However, participants at the 2012 workshop, other scientists the committee talked to, and the committee itself judged this is to be undesirable and, in fact, impossible. Fortunately, there is an example of the difficulty in prioritizing science goals first. The five science frontier panels (SFPs) of the Astro2010 produced a list of 20 science questions and six “discovery areas,” all of equal priority; these high-priority questions were distilled from a much larger set of questions covering the field.7 However, the survey committee did not ask the SFPs to go further, to prioritize the questions—nor did the SFPs want to. Consider this: Is answering “Do habitable worlds exist around other stars . . . ?” more important than knowing “How do black holes grow, and radiate . . . ?” Who can say? Anyone. Who can know? No one.
Nevertheless, these SFP questions were the foundation of Astro2010’s recommended program. By stacking “what we want to do” against “what we can do,” another essential dimension is added to judging science priority. Where can the most progress be made with available resources and existing or new technology? It is a matter of fact that, in all previous surveys, the science prioritization process has depended crucially on such mission and facility concepts—what they could do and what they would cost. This non-linear, almost organic, process has been at the very heart of every survey.
The committee was also asked to consider a related proposal: a two-phase decadal survey process where science is prioritized first, as in Astro2010, with a break to communicate the results to the community and the agencies to “tune” the formulation of missions, facilities, and observing systems to these science priorities. The committee is concerned that stretching the decadal process beyond 2 years would prove to be impractical and unaffordable. But, more to the point, the committee has concluded, from looking at the Astro2010 example, that a high-priority but unranked list of science goals would not facilitate the mission formulation process. In fact, participants in the 2012 workshop speaking on behalf of planetary science, Earth science, and heliophysics surveys insisted that their highly interactive (and successful) process of science and mission prioritization would be disabled by attempts to divorce the two. The committee concluded that decisions as to how a decadal survey will prioritize science and recommended programs are best left to the survey committee itself.
Despite, and also because of, these misgivings about the value of a stand-alone process for science prioritization, the committee endorses reviewing the “state of the science” before a new survey begins, as distinct from creating a new process to do “science prioritization.” Fortunately, there are ongoing activities to facilitate that activity, including the midterm decadal review and the Space Studies Board with its discipline-specific standing committees. NASA advisory committees, including NASA’s many assessment and analysis groups (like the Mars Exploration Program Analysis Group, the Cosmic Origins Program Anaysis Group, and the Geospace-Management Operations Working Group), NASA roadmap teams, and the Science Committee of the NASA Advisory Council, can all contribute to this task. White papers and society meetings can also be used to sample the thoughts of the broader community. A best practice to bring this all together would be to initiate processes to collect community input before a new survey begins. This process could include workshops, sessions at meetings of professional societies, white papers, and, perhaps, a process conducted under the aegis of the Academies under the direction of the Space Studies Board. The goal would be to assess how science has evolved from the last survey and call attention to emerging areas of promise. Community ideas for implementation of these science themes could lead to preparatory studies of missions and facilities. This kind of input could give the upcoming survey a running start
_______________
7 National Research Council, New Worlds, New Horizons in Astronomy and Astrophysics, The National Academies Press, Washington, D.C., 2010.
in identifying their key science objectives. A similar activity, on a global scale, is to exploit international scientific meetings and conferences while encouraging communications between decadal surveys and analogous planning exercises abroad, to help lay the groundwork for future international missions.
The committee reviewed the CATE activity that was added to the decadal process in response to the 2008 NASA Authorization Act, which requires an independent cost estimate that can be compared to the budgets provided by mission advocates. The committee concluded that the CATE process has become a best practice of decadal surveys, adding credibility to their implementation plans. Furthermore, the CATE process will likely evolve to become more efficient and more easily adaptable to any particular decadal survey. The committee found little interest in returning to decadal surveys without CATE, but instead found widespread support of CATE and support for improving the CATE process.
This report focuses on whether the CATE process as it has been implemented is overly drawn out and expensive, and whether this puts a strain on its use if very many facilities and missions are under consideration. Worthwhile programs that might have been recommended could have been shut out by missions that—according to a “late CATE”—turn out to be unaffordable. A best practice for future CATEs could be to initially run a much larger number of candidate missions through a faster but coarser “cost-box” analysis, to provide a sense of scale for initial consideration. This extra step would reserve the full CATE process for missions that are likely to become part of the recommended program—that is, those that require more detailed estimates. This “two-step” approach would also help prevent CATE from pacing the survey process.
One rather obvious lesson learned is that a reliable CATE process is crucial for the largest, most ambitious missions—high-profile missions—where cost growth can threaten the health of a wide set of activities over a discipline, and beyond. A best practice for future surveys is to give greater attention and added care in assessing and recommending potentially “discipline-disrupting” programs. A thorough and rigorous CATE process can help, but too often the true cost of such a mission cannot be well established until the program is well under way. Surveys can provide clear decision rules and decision points that will effectively establish cost caps, with the intent of triggering reconsideration of the mission and the possibility, or necessity, of rescoping its science capability.
The committee concludes that the decadal survey process has been very successful. Indeed, decadal surveys set a standard of excellence that encourages the hope that similar processes could be applied more widely across the nation’s science programs. While it has no major flaws, the survey process can, and should, improve and evolve. The remarkable record of decadal surveys makes the committee optimistic that useful changes can and will be made.
