3

Linkage Between Planetary R&A and NASA PSD Science Goals

The committee was charged with addressing two questions related to the Planetary Science Division’s (PSD’s) current Research and Analysis (R&A) program elements. The first of two questions posed to the committee asks:

As noted in Chapter 1, the current PSD R&A program encompasses core research and core technology program elements, as well as some strategic and focused program elements.¹ The committee looked at all these program elements and the range and scope of activities that they support, with a view to assessing their alignment with NASA’s strategic objectives and PSD’s science goals. Particular attention was given to whether the current program elements provide suitable support for all research activities critical to PSD’s mission. The committee also looked at how well PSD balances funding both within and between program elements as it works to optimize its use of scarce resources.

In its deliberations, the committee discussed assertions by several of the chairs of the analysis/assessment groups (Appendix C) highlighting concerns that certain planetary science subdisciplines or areas of research were not adequately funded under the new program elements structured to support PSD’s science goals. There was additional concern that as the new program elements become fully implemented, there will be a loss of both expertise and technical capability that will be needed to support future missions. While the alignment between NASA’s planetary science goals and the PSD R&A program elements prior to the 2014 reorganization is outside the charge of this committee, the committee examined implementation of the reorganization in terms of whether critical capabilities have been lost in the transition to the current program. The loss of critical capabilities would potentially impede PSD’s ability to address its strategic goals.

___________________

¹ More detail on the current program elements can be found in the section “Planetary Science Division R&A Program Elements” in Chapter 1.

REORGANIZATION OF THE PSD R&A PROGRAM

As noted in Chapter 1, prior studies both by the National Research Council (NRC)² and the Planetary Science Subcommittee of the NASA Advisory Council³ recommended that PSD reorganize the R&A budget structure to align more closely with NASA’s planetary science goals. When the reorganization was implemented for Research Opportunities in Space and Earth Sciences (ROSES) 2014, the appropriate NASA document was NASA’s 2010 Science Plan.⁴ This document identified five fundamental science questions that guided NASA’s solar system exploration. These questions were used by PSD to structure a new set of R&A core research program elements, which are described in the section “Planetary Science Division R&A Program Elements” in Chapter 1. While NASA’s 2014 Science Plan⁵ reworded these science questions as planetary science goals (Box 1.2), they remain fundamentally the same. The mapping of the science questions to the new program elements is illustrated in Figure 3.1. Thus, the reorganization followed the first recommendation of the 1998 NRC report⁶ on NASA’s research and data analysis activities: “Link NASA research [proposal solicitations] to addressing key scientific questions that can be related to the goals of the strategic plans” (p. 3).

MAPPING OF PRIOR PROGRAM ELEMENTS TO NEW CORE RESEARCH PROGRAM ELEMENTS

The R&A program elements prior to the ROSES 2014 solicitation included subdisciplinary elements (e.g., Cosmochemistry,⁷ Planetary Geology, and Geophysics), target-oriented elements (e.g., Lunar Advanced Science

___________________

² National Research Council (NRC), Supporting Research and Data Analysis in NASA’s Science Programs: Engines for Innovation and Synthesis, National Academy Press, Washington, D.C., 1998.

³ Planetary Sciences Subcommittee, Assessment of the NASA Planetary Science Division’s Mission-Enabling Activities, NASA Advisory Council Science Committee, August 29, 2011.

⁴ NASA, 2010 Science Plan for NASA’s Science Mission Directorate, NASA Headquarters, NP-2010-08-669-HQ, Washington, D.C., July 2010.

⁵ NASA, 2014 Science Plan, Washington, D.C., 2014.

⁶ NRC, Supporting Research and Data Analysis in NASA’s Science Programs, 1998.

⁷ Cosmochemistry is the study of the chemical composition of matter in the universe and the processes that led to these compositions.

and Exploration Research, Outer Planets Research, and Mars Fundamental Research), and elements that have components of both. As illustrated in Figure 3.2, these older program elements do not map simply into the new core research program elements, which are more thematic in character. Proposals appropriate to the new core elements may have been previously submitted to a variety of the older elements. In addition, not all proposals traditionally submitted to one of the older program elements are appropriate to the same new program element. Thus, for example, most Cosmochemistry proposals would now be submitted to either Emerging Worlds or Solar System Workings, depending on the research focus. It was this many-to-many mapping of the previous-to-new program elements (Figure 3.2) that created challenges for PSD in maintaining programmatic balance through the transition and in demonstrating to the planetary science community that no research activities that are critical to NASA’s science mission had been inadvertently disenfranchised.

MAPPING OF PRIOR RESEARCH ACTIVITIES TO NEW STRATEGIC AND FOCUSED PROGRAM ELEMENTS

While the reorganization mostly focused on mapping the old research programs into new core research program elements that are aligned with NASA’s science goals, some restructuring of the strategic and focused program elements also occurred (Figure 3.3), so that all proposals submitted under ROSES would have an appropriate home in the new structure. In particular, the formation of an Exoplanets program element, jointly with the SMD Astrophysics Division, provides a home for studies relevant to both solar system science and exoplanetary research. Planetary Science and Technology Through Analog Research (PSTAR) and Planetary Data Archiving, Restoration, and Tools (PDART) (see the section “Planetary Science Division R&A Program Elements” in Chapter 1) provide funding for some research activities that are integral to understanding our ability to conduct the scientific exploration of other planets by both robotic and human explorers (PSTAR), or to generate higher-order data products, archive and restore data sets, create or consolidate databases, or digitize data and make improved software tools available to researchers (PDART). The committee notes that such research activities did not always fare well in the traditional funding program elements.

ALIGNMENT BETWEEN PSD R&A PROGRAM ELEMENTS AND NASA’S STRATEGIC OBJECTIVE FOR PLANETARY SCIENCE AND PSD SCIENCE GOALS

Based on its assessment of the breadth and scope of the current R&A program elements, the committee finds that these program elements are appropriately linked to and encompass the range and scope of activities needed to support NASA’s strategic objective for planetary science and PSD’s science goals. Presentations to the committee from the chairs of the analysis/assessment groups and the NASA center leads for planetary science largely concurred with this finding.

The transition from the old to the new R&A structure has been phased in over the past 3 years (fiscal year [FY] 2015 to FY2017), covering several ROSES proposal cycles to accommodate the budget process and varying grant expiration dates. The process understandably provided some initial confusion among proposers and disruptions in the funding process. Investigators had to determine to which of the new program elements they should submit proposals, and on what deadlines, and how many proposals were needed. Bridging or sustaining funding was needed in some cases to adjust to new schedules. PSD had to decide how to deal with continuing grants and how to optimize the review process for the new thematic program elements. The ensuing confusion produced considerable concern for some investigators. Nonetheless, the committee concludes that NASA has successfully addressed these issues and that the program is now operating well under the new structure.

KEYWORD ANALYSIS

Even as the appropriated budget for NASA PSD has fluctuated significantly in recent years, including a decrease of some 22 percent from FY2012 to FY2013, PSD has protected the funding for R&A awards from such fluctuations. However, because the number of proposals has increased significantly while R&A budgets have received only modest increases, proposal success rates have fallen. Continuous adjustment has been needed to maintain appropriate balance within and between program elements to ensure the health of the PSD program of activities, including spaceflight mission lines, R&A, and technology development. Data analysis tools to measure programmatic balance—for example, analyses of keywords identified for proposals—have thus become increasingly necessary for effective program management.

As the reorganization of the planetary science R&A programs involved many-to-many mapping of old-to-new program elements, it was difficult for the committee to assess whether appropriate support for subdisciplinary activities critical for PSD’s science goals was maintained through the reorganization. Certainly, some changes would be anticipated given the stronger focus on linkages to and alignment with NASA’s strategic goals. In order for the committee to assess balance within and across program elements and alignment of the current program with NASA’s strategic goals, appropriate data were needed on funding distributions by scientific discipline, target body, and type of task.

Over the past several years, PSD has implemented the use of keyword characterization of individual projects as a tool to better assess the program and provide more transparency about where the money is flowing in terms of subject areas. Upon request of the committee, PSD expanded analysis of the keyword data and provided several presentations to the committee on the results of this analysis. NASA’s initial dependence on proposers to assign keywords proved less than ideal (early attempts at using keywords provided by proposers had a disappointingly

small response), so that PSD elected to use program officers to assign keywords for all proposals under their purview. While there remain challenges when changes in PSD staffing result in differing perspectives on meanings of individual keywords, the organization of caucuses of program officers within PSD to provide input on proposal selection has helped to make the keyword use more uniform and more completely utilized.

NASA assigns keywords in four fields, the first three of which are relevant to the committee’s analysis—type of task, target body, and scientific discipline. Program officers can fill in up to five keywords in each of these fields. When the PSD program officer assigns more than one keyword to a particular field for a proposal, the total awarded funding is divided evenly among the assigned keywords. This approach can lead to some distortions and limits the use of the keywords to broad conclusions. Similarly, changes in the personnel assigning the keywords can lead to shifts in the results, also limiting one to rather broad conclusions. Nevertheless, the committee considers that the analysis presented by NASA PSD⁸ is now sufficiently robust to draw broad conclusions about impacts of the reorganization.

Figure 3.4 shows the distribution of funding awarded by target body/bodies for each fiscal year from 2011 through 2015 (data were not final for FY2016 at the time of report preparation). It is important to remember that data awarded for a given fiscal year are based on proposals to the previous year’s ROSES announcement. Because the reorganization of R&A core research program elements first appeared in ROSES 2014, proposals selected for funding under this NRA were first supported using FY2015 funds. Similarly it is important to recall that a typical grant has a duration of 3 years so that the funding awarded in any given year is a mix of new awards with second- and third-year renewals. Thus, the FY2014 funding data are totally under the old program elements. Because the

___________________

⁸ Presentation to the committee on September 22, 2016, by Meagan Thompson, NASA Headquarters, based on input prepared by Thompson and Jonathan Rall, PSD R&A Lead at NASA Headquarters.

FY2015 data contain renewals from previous years and funding for program elements that remained through the reorganization, only roughly one-third of the funding went to proposals submitted under the new program elements. It is also important to recognize that external factors, such as congressional designation of specific tasks—for example, the near-Earth object and the Europa programs—influence annual distribution of funding. Finally, the declining phases of data analysis program elements that are narrowly targeted to a specific mission can also introduce year-to-year fluctuations in funding allocations.

Absent final data for FY2016, the keyword data have not been collected over a long enough period since reorganization for the committee to make firm conclusions. However, some broad conclusions are possible. In particular, there is no clear indication in the keyword data shown in Figure 3.4 that research support for certain classes of target body has been significantly reduced due the reorganization of the PSD R&A program.

Similarly, Figure 3.5 shows the distribution of awarded funding by scientific discipline. As with the distribution by target, there are year-to-year fluctuations, and the impact of the reorganization is yet to be fully demonstrated. However, there is no indication that funding for any particular science discipline has been dramatically reduced. In particular, the committee heard during presentations by several chairs of the analysis/assessment groups and NASA center planetary leads that the cosmochemistry community had seen severe cuts in funding through the reorganization. The committee is thus surprised to see funding associated with both the cosmochemistry and geochemistry keywords increase from FY2014 to FY2015. In fact, the most dramatic change in Figure 3.5 is the significant decrease in unpopulated keywords as program officers make better use of the keywords to monitor their programs.

The keyword analyses presented above can also be extremely useful as a management tool in assessing allocations relative to PSD’s science goals, but perhaps more importantly, they can be a very effective way of communicating NASA’s allocation of funds to the community. In the past, various presentations to different communities have seemed to be contradictory because the presentations were made as the reorganization was progressing and data were not complete. As the number of unpopulated keywords is further reduced and the uniformity of assigning keywords by program officers is improved, the principal evolution over the next 2 years will be due to the move from a year in which only one-third of the budget is for proposals under the new organization (FY2015) to a year in which the budget is entirely for proposals under the new organization (FY2017). If these charts are used in the future to communicate with the community about NASA’s R&A program, it is crucially important that they be consistently explained and that the limitations to their interpretability be spelled out. As the committee has noted in this report, it is easy to be led astray by a variety of artifacts in the data.

HIGH-RISK/HIGH-PAYOFF RESEARCH ACTIVITIES AND ADVANCED TECHNOLOGY

Prior to the 2014 reorganization, the PSD R&A program already included specific technology development activities (e.g., PICASSO and MatISSE) focused on future instrument technology needs. However, the program did not explicitly encourage or make room for including high-risk/high-payoff advanced technology projects beyond PSTAR-supported field tests of advanced instrumentation. The current program structure remains unchanged in this regard. The same point can be made about other research initiatives beyond technology development. High-risk/high-payoff research proposals submitted to core research program elements face challenges in review that are exacerbated in times of tight budgets and low selection rates.

Many past studies have emphasized the importance of such activities. For example, the 2009 NRC report⁹ on mission-enabling research commented that

The report explained that

The 2009 NRC report drew on observations from the highly regarded 2007 report Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future¹⁰ to explain why high-risk/high-payoff research often struggles to gain management support.

___________________

⁹ NRC, An Enabling Foundation for NASA’s Earth and Space Science Missions, The National Academies Press, Washington, D.C., 2009.

¹⁰ National Academy of Sciences, National Academy of Engineering, National Research Council, Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future, The National Academies Press, Washington, D.C., 2007.

FUNDING DISTRIBUTION AMONG PROGRAM ELEMENTS

Data presented to the committee by NASA PSD staff shows that the proposal success rates for the five core research program elements are all ~20 percent, consistent with funding allocations among program elements largely driven by proposal pressure, rather than an assessment of strategic importance to NASA PSD science goals. As noted in Box 2.2, PSD does have mechanisms in place to identify proposals that may not have reviewed highly but warrant consideration for funding based on strategic and other criteria. However, these mechanisms operate within a program element and have little impact on the distribution of funds among program elements.

The committee notes in Chapter 2 that prior advisory studies of R&A programs have recommended that the programs should not only be structured to match strategic goals but that they should also be managed strategically. PSD’s recently developed keyword system (see the section “Keyword Analysis” earlier in this chapter) for characterizing research activities can serve as a useful strategic management tool. Analysis of keywords is now possible at sufficient depth and breadth to provide the basis for assessing relationships among such factors as program element size and funding levels, proposal demand and success rates, and investigator demographics—all in terms of how well they can support NASA PSD science goals.

The 2011 Planetary Science Subcommittee (PSS) report on supporting research and technology explicitly recommended that PSD should regularly conduct such an across-the-board review of such factors.

This committee endorses the 2011 PSS recommendation but is not convinced that a “senior review” every 10 years is sufficiently frequent given the rate of discovery in the field.

___________________

¹¹ Planetary Sciences Subcommittee, Assessment of the NASA Planetary Science Division’s Mission-Enabling Activities, 2011, p. 24.