The committee was charged with addressing two questions related to the Planetary Science Division’s (PSD’s) current Research and Analysis (R&A) program elements. The first of two questions posed to the committee asks:
Are the PSD R&A program elements appropriately linked to, and do they encompass the range and scope of activities needed to support the NASA strategic objective for planetary science and the Planetary Science Division science goals, as articulated in the 2014 NASA Science Plan?
As noted in Chapter 1, the current PSD R&A program encompasses core research and core technology program elements, as well as some strategic and focused program elements.1 The committee looked at all these program elements and the range and scope of activities that they support, with a view to assessing their alignment with NASA’s strategic objectives and PSD’s science goals. Particular attention was given to whether the current program elements provide suitable support for all research activities critical to PSD’s mission. The committee also looked at how well PSD balances funding both within and between program elements as it works to optimize its use of scarce resources.
In its deliberations, the committee discussed assertions by several of the chairs of the analysis/assessment groups (Appendix C) highlighting concerns that certain planetary science subdisciplines or areas of research were not adequately funded under the new program elements structured to support PSD’s science goals. There was additional concern that as the new program elements become fully implemented, there will be a loss of both expertise and technical capability that will be needed to support future missions. While the alignment between NASA’s planetary science goals and the PSD R&A program elements prior to the 2014 reorganization is outside the charge of this committee, the committee examined implementation of the reorganization in terms of whether critical capabilities have been lost in the transition to the current program. The loss of critical capabilities would potentially impede PSD’s ability to address its strategic goals.
As noted in Chapter 1, prior studies both by the National Research Council (NRC)2 and the Planetary Science Subcommittee of the NASA Advisory Council3 recommended that PSD reorganize the R&A budget structure to align more closely with NASA’s planetary science goals. When the reorganization was implemented for Research Opportunities in Space and Earth Sciences (ROSES) 2014, the appropriate NASA document was NASA’s 2010 Science Plan.4 This document identified five fundamental science questions that guided NASA’s solar system exploration. These questions were used by PSD to structure a new set of R&A core research program elements, which are described in the section “Planetary Science Division R&A Program Elements” in Chapter 1. While NASA’s 2014 Science Plan5 reworded these science questions as planetary science goals (Box 1.2), they remain fundamentally the same. The mapping of the science questions to the new program elements is illustrated in Figure 3.1. Thus, the reorganization followed the first recommendation of the 1998 NRC report6 on NASA’s research and data analysis activities: “Link NASA research [proposal solicitations] to addressing key scientific questions that can be related to the goals of the strategic plans” (p. 3).
The R&A program elements prior to the ROSES 2014 solicitation included subdisciplinary elements (e.g., Cosmochemistry,7 Planetary Geology, and Geophysics), target-oriented elements (e.g., Lunar Advanced Science
2 National Research Council (NRC), Supporting Research and Data Analysis in NASA’s Science Programs: Engines for Innovation and Synthesis, National Academy Press, Washington, D.C., 1998.
3 Planetary Sciences Subcommittee, Assessment of the NASA Planetary Science Division’s Mission-Enabling Activities, NASA Advisory Council Science Committee, August 29, 2011.
4 NASA, 2010 Science Plan for NASA’s Science Mission Directorate, NASA Headquarters, NP-2010-08-669-HQ, Washington, D.C., July 2010.
5 NASA, 2014 Science Plan, Washington, D.C., 2014.
6 NRC, Supporting Research and Data Analysis in NASA’s Science Programs, 1998.
7 Cosmochemistry is the study of the chemical composition of matter in the universe and the processes that led to these compositions.
and Exploration Research, Outer Planets Research, and Mars Fundamental Research), and elements that have components of both. As illustrated in Figure 3.2, these older program elements do not map simply into the new core research program elements, which are more thematic in character. Proposals appropriate to the new core elements may have been previously submitted to a variety of the older elements. In addition, not all proposals traditionally submitted to one of the older program elements are appropriate to the same new program element. Thus, for example, most Cosmochemistry proposals would now be submitted to either Emerging Worlds or Solar System Workings, depending on the research focus. It was this many-to-many mapping of the previous-to-new program elements (Figure 3.2) that created challenges for PSD in maintaining programmatic balance through the transition and in demonstrating to the planetary science community that no research activities that are critical to NASA’s science mission had been inadvertently disenfranchised.
While the reorganization mostly focused on mapping the old research programs into new core research program elements that are aligned with NASA’s science goals, some restructuring of the strategic and focused program elements also occurred (Figure 3.3), so that all proposals submitted under ROSES would have an appropriate home in the new structure. In particular, the formation of an Exoplanets program element, jointly with the SMD Astrophysics Division, provides a home for studies relevant to both solar system science and exoplanetary research. Planetary Science and Technology Through Analog Research (PSTAR) and Planetary Data Archiving, Restoration, and Tools (PDART) (see the section “Planetary Science Division R&A Program Elements” in Chapter 1) provide funding for some research activities that are integral to understanding our ability to conduct the scientific exploration of other planets by both robotic and human explorers (PSTAR), or to generate higher-order data products, archive and restore data sets, create or consolidate databases, or digitize data and make improved software tools available to researchers (PDART). The committee notes that such research activities did not always fare well in the traditional funding program elements.
ALIGNMENT BETWEEN PSD R&A PROGRAM ELEMENTS AND NASA’S STRATEGIC OBJECTIVE FOR PLANETARY SCIENCE AND PSD SCIENCE GOALS
Based on its assessment of the breadth and scope of the current R&A program elements, the committee finds that these program elements are appropriately linked to and encompass the range and scope of activities needed to support NASA’s strategic objective for planetary science and PSD’s science goals. Presentations to the committee from the chairs of the analysis/assessment groups and the NASA center leads for planetary science largely concurred with this finding.
Finding: The new R&A structure is properly aligned with scientific priorities of the decadal survey and the Planetary Science Division’s 2014 science goals and is consistent with the recommendations of the 2009 National Research Council report An Enabling Foundation for NASA’s Earth and Space Science Missions.
The transition from the old to the new R&A structure has been phased in over the past 3 years (fiscal year [FY] 2015 to FY2017), covering several ROSES proposal cycles to accommodate the budget process and varying grant expiration dates. The process understandably provided some initial confusion among proposers and disruptions in the funding process. Investigators had to determine to which of the new program elements they should submit proposals, and on what deadlines, and how many proposals were needed. Bridging or sustaining funding was needed in some cases to adjust to new schedules. PSD had to decide how to deal with continuing grants and how to optimize the review process for the new thematic program elements. The ensuing confusion produced considerable concern for some investigators. Nonetheless, the committee concludes that NASA has successfully addressed these issues and that the program is now operating well under the new structure.
Finding: There were some start-up issues with program implementation during and following the reorganization. For example, there was uncertainty about the boundaries of the core program elements and about the possibility of funding gaps due to shifting timelines for proposal submission. NASA appears to have resolved these transient problems.
Even as the appropriated budget for NASA PSD has fluctuated significantly in recent years, including a decrease of some 22 percent from FY2012 to FY2013, PSD has protected the funding for R&A awards from such fluctuations. However, because the number of proposals has increased significantly while R&A budgets have received only modest increases, proposal success rates have fallen. Continuous adjustment has been needed to maintain appropriate balance within and between program elements to ensure the health of the PSD program of activities, including spaceflight mission lines, R&A, and technology development. Data analysis tools to measure programmatic balance—for example, analyses of keywords identified for proposals—have thus become increasingly necessary for effective program management.
As the reorganization of the planetary science R&A programs involved many-to-many mapping of old-to-new program elements, it was difficult for the committee to assess whether appropriate support for subdisciplinary activities critical for PSD’s science goals was maintained through the reorganization. Certainly, some changes would be anticipated given the stronger focus on linkages to and alignment with NASA’s strategic goals. In order for the committee to assess balance within and across program elements and alignment of the current program with NASA’s strategic goals, appropriate data were needed on funding distributions by scientific discipline, target body, and type of task.
Over the past several years, PSD has implemented the use of keyword characterization of individual projects as a tool to better assess the program and provide more transparency about where the money is flowing in terms of subject areas. Upon request of the committee, PSD expanded analysis of the keyword data and provided several presentations to the committee on the results of this analysis. NASA’s initial dependence on proposers to assign keywords proved less than ideal (early attempts at using keywords provided by proposers had a disappointingly
small response), so that PSD elected to use program officers to assign keywords for all proposals under their purview. While there remain challenges when changes in PSD staffing result in differing perspectives on meanings of individual keywords, the organization of caucuses of program officers within PSD to provide input on proposal selection has helped to make the keyword use more uniform and more completely utilized.
NASA assigns keywords in four fields, the first three of which are relevant to the committee’s analysis—type of task, target body, and scientific discipline. Program officers can fill in up to five keywords in each of these fields. When the PSD program officer assigns more than one keyword to a particular field for a proposal, the total awarded funding is divided evenly among the assigned keywords. This approach can lead to some distortions and limits the use of the keywords to broad conclusions. Similarly, changes in the personnel assigning the keywords can lead to shifts in the results, also limiting one to rather broad conclusions. Nevertheless, the committee considers that the analysis presented by NASA PSD8 is now sufficiently robust to draw broad conclusions about impacts of the reorganization.
Figure 3.4 shows the distribution of funding awarded by target body/bodies for each fiscal year from 2011 through 2015 (data were not final for FY2016 at the time of report preparation). It is important to remember that data awarded for a given fiscal year are based on proposals to the previous year’s ROSES announcement. Because the reorganization of R&A core research program elements first appeared in ROSES 2014, proposals selected for funding under this NRA were first supported using FY2015 funds. Similarly it is important to recall that a typical grant has a duration of 3 years so that the funding awarded in any given year is a mix of new awards with second- and third-year renewals. Thus, the FY2014 funding data are totally under the old program elements. Because the
8 Presentation to the committee on September 22, 2016, by Meagan Thompson, NASA Headquarters, based on input prepared by Thompson and Jonathan Rall, PSD R&A Lead at NASA Headquarters.
FY2015 data contain renewals from previous years and funding for program elements that remained through the reorganization, only roughly one-third of the funding went to proposals submitted under the new program elements. It is also important to recognize that external factors, such as congressional designation of specific tasks—for example, the near-Earth object and the Europa programs—influence annual distribution of funding. Finally, the declining phases of data analysis program elements that are narrowly targeted to a specific mission can also introduce year-to-year fluctuations in funding allocations.
Absent final data for FY2016, the keyword data have not been collected over a long enough period since reorganization for the committee to make firm conclusions. However, some broad conclusions are possible. In particular, there is no clear indication in the keyword data shown in Figure 3.4 that research support for certain classes of target body has been significantly reduced due the reorganization of the PSD R&A program.
Similarly, Figure 3.5 shows the distribution of awarded funding by scientific discipline. As with the distribution by target, there are year-to-year fluctuations, and the impact of the reorganization is yet to be fully demonstrated. However, there is no indication that funding for any particular science discipline has been dramatically reduced. In particular, the committee heard during presentations by several chairs of the analysis/assessment groups and NASA center planetary leads that the cosmochemistry community had seen severe cuts in funding through the reorganization. The committee is thus surprised to see funding associated with both the cosmochemistry and geochemistry keywords increase from FY2014 to FY2015. In fact, the most dramatic change in Figure 3.5 is the significant decrease in unpopulated keywords as program officers make better use of the keywords to monitor their programs.
Finding: The committee finds that keyword analyses of the type of task, target body, and science discipline revealed no evidence that restructuring is leading to deleterious effects on the planetary science R&A program or on specific segments of the community.
The keyword analyses presented above can also be extremely useful as a management tool in assessing allocations relative to PSD’s science goals, but perhaps more importantly, they can be a very effective way of communicating NASA’s allocation of funds to the community. In the past, various presentations to different communities have seemed to be contradictory because the presentations were made as the reorganization was progressing and data were not complete. As the number of unpopulated keywords is further reduced and the uniformity of assigning keywords by program officers is improved, the principal evolution over the next 2 years will be due to the move from a year in which only one-third of the budget is for proposals under the new organization (FY2015) to a year in which the budget is entirely for proposals under the new organization (FY2017). If these charts are used in the future to communicate with the community about NASA’s R&A program, it is crucially important that they be consistently explained and that the limitations to their interpretability be spelled out. As the committee has noted in this report, it is easy to be led astray by a variety of artifacts in the data.
Finding: In addition to providing a strategic management tool, keyword analyses can be used to promote increased transparency between NASA and the planetary science community with respect to trends in proposal selection decisions and program content.
Prior to the 2014 reorganization, the PSD R&A program already included specific technology development activities (e.g., PICASSO and MatISSE) focused on future instrument technology needs. However, the program did not explicitly encourage or make room for including high-risk/high-payoff advanced technology projects beyond PSTAR-supported field tests of advanced instrumentation. The current program structure remains unchanged in this regard. The same point can be made about other research initiatives beyond technology development. High-risk/high-payoff research proposals submitted to core research program elements face challenges in review that are exacerbated in times of tight budgets and low selection rates.
Many past studies have emphasized the importance of such activities. For example, the 2009 NRC report9 on mission-enabling research commented that
While most of the SMD mission-enabling research budget should be clearly directed at supporting specific goals of the various science divisions, NASA can benefit from separately funded and protected mission-enabling activities that pursue high-risk/high-payoff advanced technologies or other research activities that could produce game-changing results. (p. 40)
The report explained that
Where mission development times can run as long as 7 to 10 years, such an approach is especially important because the phasing-in of new technologies can be 5 to 10 times longer than time to market in the commercial and university research sectors. Thus, one can envision progressively falling behind the state of the art in many technical areas unless technology is purposefully captured and utilized through an active mission-enabling research program. (p. 40)
The 2009 NRC report drew on observations from the highly regarded 2007 report Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future10 to explain why high-risk/high-payoff research often struggles to gain management support.
9 NRC, An Enabling Foundation for NASA’s Earth and Space Science Missions, The National Academies Press, Washington, D.C., 2009.
10 National Academy of Sciences, National Academy of Engineering, National Research Council, Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future, The National Academies Press, Washington, D.C., 2007.
It is not necessarily a matter of providing additional resources for high-risk research, but rather providing incentives for program managers to fund high-risk research out of a discretionary portion of the existing budget. Lack of incentives for (or barriers to) performing high-risk research include (1) a peer review system that tends to favor established investigators using well-known methods; (2) pressure from customers and management for short-term results; and (3) risk averseness, since high-risk projects are prone to failing and increased government and public scrutiny make “projects that fail” increasingly untenable. (p. 41)
Finding: The PSD R&A program elements have not been optimized to support high-risk/high-payoff fundamental research and advanced technology-development activities.
Recommendation: NASA needs to investigate appropriate mechanisms to ensure that high-risk/high-payoff fundamental research and advanced technology-development activities receive appropriate consideration during the review process.
Data presented to the committee by NASA PSD staff shows that the proposal success rates for the five core research program elements are all ~20 percent, consistent with funding allocations among program elements largely driven by proposal pressure, rather than an assessment of strategic importance to NASA PSD science goals. As noted in Box 2.2, PSD does have mechanisms in place to identify proposals that may not have reviewed highly but warrant consideration for funding based on strategic and other criteria. However, these mechanisms operate within a program element and have little impact on the distribution of funds among program elements.
Finding: Funding difference among program elements does not appear to be determined to any significant extent by strategic priorities. Rather, the funding distribution among program elements appears to be defined predominantly by proposal pressure.
The committee notes in Chapter 2 that prior advisory studies of R&A programs have recommended that the programs should not only be structured to match strategic goals but that they should also be managed strategically. PSD’s recently developed keyword system (see the section “Keyword Analysis” earlier in this chapter) for characterizing research activities can serve as a useful strategic management tool. Analysis of keywords is now possible at sufficient depth and breadth to provide the basis for assessing relationships among such factors as program element size and funding levels, proposal demand and success rates, and investigator demographics—all in terms of how well they can support NASA PSD science goals.
The 2011 Planetary Science Subcommittee (PSS) report on supporting research and technology explicitly recommended that PSD should regularly conduct such an across-the-board review of such factors.
The PSD should consider implementing division-wide coordination and evaluation of the mission-enabling activities by NASA and the community by holding a “senior review” at least every 10 years, which could be linked to the NRC surveys. The review should include articulation by the PSD of the current priorities, budget allocations for all mission-enabling activities (including supporting activities), and how the various activities have met their specific program objectives in the past. Such a review should become part of the PSD “portfolio management” process to ensure that resources are apportioned appropriately.11
This committee endorses the 2011 PSS recommendation but is not convinced that a “senior review” every 10 years is sufficiently frequent given the rate of discovery in the field.
Finding: No formal mechanism exists to periodically review funding distribution among program elements based on strategic goals.
11 Planetary Sciences Subcommittee, Assessment of the NASA Planetary Science Division’s Mission-Enabling Activities, 2011, p. 24.
Recommendation: A formal assessment by NASA of how well the program structure and funding are aligned with the Planetary Science Division’s science goals should be conducted at least every 5 years, appropriately phased to the cycle of the decadal surveys and midterm reviews.