The processes by which NASA Planetary Science Division (PSD) selects proposals for funding bear directly on how effectively the current Research and Analysis (R&A) program aligns with NASA’s strategic objectives for planetary science and the PSD science goals, as articulated in NASA’s 2014 Science Plan1 (Figure 2.1). As these processes have been retooled in response to the recent reorganization of the program elements, it is important to review the current approach and how it accommodates the challenges associated with implementing an R&A program structure with new, more encompassing program elements.
The current organization of NASA’s PSD R&A program was designed to map specific core research program elements directly to NASA’s planetary science goals. The presentations by NASA PSD senior staff that characterized the restructuring devoted considerable attention to the much improved ability of NASA to satisfy congressional and administration mandates such as the Government Performance and Results Modernization Act (GPRAMA). However, as one might expect from any such large-scale re-ordering, and despite what appeared to be a good-faith effort on the part of PSD to explain the reorganization, the restructuring also generated a number of concerns in the planetary science community. Many of these concerns reflect on the processes now used by PSD to review and make funding decisions on proposals.
Under the reorganized R&A structure, PSD has modified the processes associated with submission, review, and selection of R&A proposals for funding. Any resultant inconsistencies and inefficiencies in the mechanisms used to submit, review, and select proposals have direct bearing on the linkage among the R&A program elements, PSD goals and objectives (i.e., items directly pertinent to the first question in the committee’s statement of task), and the planetary science community’s ability to interpret and maximize the scientific return from NASA missions (i.e., items directly pertinent the second question in the committee’s statement of task). As such, issues associated with submission, review, and selection of R&A proposals have a direct bearing on the two questions posed to the committee by NASA. Consequently, the committee asked PSD to provide details concerning the R&A proposal process. The material provided by PSD, as informed by the committee’s subsequent discussions and deliberation, forms the basis of the following sections of this chapter. The issues identified by the committee in the following pages are clearly the source of some of the concerns expressed during its data-gathering activities (Appendix C). The committee also notes that the material provided by PSD is not clearly delineated elsewhere.
Although these processes vary somewhat among the various program elements within PSD R&A, all program
1 NASA, 2014 Science Plan, Washington, D.C., 2014.
elements use the same basic approach, following SMD Policy Document SPD-22: Management of ROSES Peer Review and Selection Process2 and the guidelines for peer review from the Guidebook for Proposers.3 To enable consistency of process across program elements, PSD uses a management structure with overall responsibility for R&A residing with PSD’s R&A lead.
A general overview of the current processes, illustrated in Figure 2.2, can be broken down into (1) activities leading up to and including the panel review, (2) activities involving selection of reviewed proposals for funding, and (3) activities associated with principal investigator (PI) requests for debriefings and reconsiderations.
NASA PSD has established approaches to proposal submission and review that were summarized in a presentation to the committee (Box 2.1). PSD currently uses a two-step proposal submission process utilizing the NASA Solicitation and Proposal Integrated Review and Evaluation System (NSPIRES). The Step-1 proposal is required by NASA and has to be submitted by an authorized organizational representative (AOR) of the PI’s institution. It includes only the names of the PIs and co-investigators, their institutional affiliations, and a brief summary of the research to be performed. The Step-1 proposal allows NASA to plan for the panel review process and to provide feedback, as appropriate, to PIs when proposed research appears to be inappropriate to the solicitation. While a Step-1 proposal is required before submission of a Step-2 proposal, it is not a commitment on the part of the PI’s institution to submit a Step-2 proposal, which contains all NASA Research Announcement (NRA)-required elements for peer review.
Under the new organizational structure, the review of proposals submitted to each program element is managed by a caucus of PSD program officers representing the disciplinary breadth of the program element. Additional
2 NASA Science Mission Directorate, SMD Policy Document SPD-22: Management of ROSES Peer Review and Selection Processes, http://www.lpi.usra.edu/PSD-RandA/SPD-22-SMD-Peer-Review-Policy.pdf, accessed April 17, 2017.
3 NASA, “NASA Guidebook for Proposers 2016,” https://www.hq.nasa.gov/office/procurement/nraguidebook/, pp. B6-B7.
support for proposal review management, but not funding decisions, is provided by community members who act as panel chairs and group chiefs. These scientists are recruited from the planetary science community by the appropriate program officer for each program element. The panel chairs provide oversight of the panel review process, while the group chiefs chair individual subpanels that provide panel reviews of subdisciplinary groups of proposals for each program element.
As an example, PSD also provided the committee with the briefing materials that are used in plenary session at the beginning of the Solar System Workings (SSW) panel review, which are largely representative of the materials used across all of the program elements. These presentations provide a uniform approach across all subpanels within the panel review process for each program element and directly mitigate concerns about variability in approach to proposal scoring and preparation of panel summaries. The subpanels are specifically asked to evaluate the merit of each proposal based on the following criteria:4
- Are the stated scientific goals compelling?
- Is the approach (including proposed techniques) appropriate, sound, and likely to succeed?
- Does the proposal acknowledge potential pitfalls and propose alternatives?
- How much will the proposed research program advance the field if successfully executed?
- Does the team have the necessary expertise?
- Can the proposed research program achieve the stated goals on the proposed schedule?
Materials presented to the committee by NASA suggest that the panels were not specifically sensitized to issues of unconscious bias, where proposals by female and/or minority PIs may be reviewed differently from those of their white male colleagues. Nonetheless, PSD program officers do receive training in unconscious bias, and they watch for any such behavior during panel review discussions. While NASA has not traditionally collected demographic information from proposers, and consequently does not know the extent of any bias in proposal reviews, unconscious bias in reviewing has been demonstrated by other funding agencies. The committee is pleased to note, based on the presentation by NASA Chief Scientist Ellen Stofan, that NASA has moved forward with a plan to collect demographic data to assess any disparity in review and selection of proposals. Absent such demographics, it would seem prudent to make the panelists aware of the issue of unconscious bias as part of the introduction to the panel review process.
The committee was impressed by the current implementation of the panel review process. Although the panel reviews for several program elements are larger and more interdisciplinary than for prior program elements, the presentations by NASA and the community representatives suggest that the implementation of the new program elements has been quite effective. In particular, the use of multiple subpanels for each program element, a uniform evaluation process across subpanels, and the use of caucus members having broad disciplinary coverage in both the subpanels and subsequent funding decisions largely alleviated committee concerns with respect to the larger number of proposals in each program element. Nonetheless, the committee retains concerns that the subpanels do not always have sufficient members with appropriate expertise for all proposals and that insufficient use has been made of external reviewers to fill that expertise gap. Some outreach by NASA to the broader reviewer community, to encourage timely and thorough review of proposals, may be warranted. Both NASA and community representatives noted that requests for external reviews were often unheeded, and reviews were frequently late and of insufficient quality to aid in the review process.
The committee is pleased with the addition of an early career planetary scientist as executive secretary for each subpanel. The use of these early career scientists is viewed as an effective way to provide training and experience in proposal preparation and review for the next generation of researchers. These executive secretaries take notes in every panel and subpanel session and are empowered to contact a NASA R&A manager should any untoward or prejudicial behavior be noted. Furthermore, the use of a caucus of program officers for these large program elements also allows the presence of a program officer as an observer in most of the panel discussions. Besides the obvious advantages of being able to provide on-the-spot responses to logistical questions and ensure a fair
4 See also NASA, “NASA Guidebook for Proposers 2016,” p. C2.
and balanced review process, the presence of a caucus member can provide first-hand knowledge of the panel discussion in the event of a later PI request for debriefing and/or reconsideration. With these larger, more thematic program elements, interdisciplinary proposals can be reviewed more effectively, potentially including reviewers from other concurrent review panels to cover expertise shortfalls.
Finding: The committee finds the selection process for the R&A program elements to be reasonable and consistent. The committee finds that by virtue of utilizing multiple subpanels, a rating and selection process was implemented that met the standard of fairness and thoroughness. Nonetheless, challenges remain in maintaining an optimal selection of proposals that meet NASA’s current and future needs.
Finding: Although NASA PSD has a target of one to three external reviewers per proposal, the committee is concerned that this target is not always met and that not all external reviews were of sufficient quality to assist in the review process. Given the broader disciplinary coverage of the new program elements and resultant subpanels, it is critical that there are sufficient qualified external reviews for effective review of all proposals. The critical role of external reviews needs to be communicated to the reviewer community to ensure timely return of high-quality, fair reviews.
Finding: A greater diversity of expertise is now available during the panel review process for the core research program elements due to the broader disciplinary makeup of its panels. Such diversity directly addresses concerns about insufficiently qualified peer reviewers for interdisciplinary research proposals.
Recommendation: In conducting scientific peer reviews of research proposals, the NASA Planetary Science Division should engage the services of several (at least two to three) external (mail) reviewers well in advance of panel reviews. These reviews are critical to a fair and effective proposal evaluation process, particularly when the review panels have a more interdisciplinary character. The panel chair and group chiefs, if recruited early, can take the lead in identification of appropriate external reviewers.
NASA PSD has established approaches to proposal selection for funding that were summarized in a presentation to the committee (Box 2.2). These approaches follow the guidelines in SMD Policy Document SPD-08: Requirements for Selection Decision Documents for NASA Research Announcements including ROSES.5
The committee was impressed by the attempts by NASA PSD to adapt its decision-making process for the new funding program elements. In particular, the use of a caucus of program officers allows greater disciplinary coverage in decision-making and increased oversight of selection of proposals that, for a variety of reasons, may not have ranked highly during panel discussions but still warrant funding. Such proposals should still be responsive to NASA’s 2014 Science Plan but might include high-risk/high-payoff research, research by early career scientists who are less experienced in proposal preparation, interdisciplinary research, mission-enabling research, research that sustains critical functionality for planned missions or anticipated missions, and research that enables future human missions.
Finding: The use of a caucus of R&A program officers at all stages of the proposal review and selection process for the new core research program elements brings greater disciplinary breadth and increased ability to assess alignment with strategic goals, innovative or high-risk endeavors, and mission-enabling character.
5 NASA Science Mission Directorate, SMD Policy Document SPD-08: Requirements for Selection Decision Documents for NASA Research Announcements including ROSES, Washington, D.C., 2007, http://www.lpi.usra.edu/PSD-RandA/SPD-08-NRA-Selection-Documents.pdf.
NASA PSD has established approaches to proposal decision reconsideration that are currently being implemented (Box 2.3).
The committee discussed concerns about non-responsiveness of some NASA program officers to requests for debriefings or reconsiderations, and the apparent lack of awareness in the community about the official procedures for debriefings and reconsiderations. During the presentations by PSD, it became clear that such procedures are still in the process of being implemented. Nonetheless, these need to be communicated clearly to proposers. Because initial requests for debriefing and reconsideration come through email, phone calls, or in person to the lead program officer for each of the program elements, and may later be elevated to the PSD R&A lead, there needs to be some formal mechanism in place to track such requests. Only through such a mechanism can NASA adequately document the numbers of such requests and their outcomes.
Finding: NASA PSD is establishing a uniform, formal process for reconsideration of funding decisions. Once promulgated and fully implemented, this process can provide fairness as well as transparency and improved communication with the planetary science community.
Recommendation: The NASA Planetary Science Division should expeditiously complete establishment of the process for reconsideration of proposal selection decisions, develop and implement a formal
mechanism to track debriefing and reconsideration requests across program elements, and inform the community about the process. More transparency in this area can provide the planetary science community with greater confidence that NASA has appropriate checks and balances in the selection process.