2
Assessment of NASA’s Mission-Enabling Activities
The mission-enabling activities of NASA’s Science Mission Directorate (SMD) have served as the essential foundation for the world’s most diverse and successful space and Earth science program. NASA’s spaceflight program has been highly successful, fundamentally altering our scientific understanding of the Earth, the solar system, and the universe beyond. The committee believes that for progress to continue, similarly visionary science and technology investments must continue and must be recognized as the essential foundation for all of SMD’s activities.
BROAD CONCERNS AND OPPORTUNITIES FOR IMPROVEMENT
During briefings to the committee, NASA officials shared several concerns about mission-enabling programs. First, in a mission-oriented agency like NASA there is a natural tendency to give priority to targeted research investigations rather than to support more broadly based core activities. This preference also creates an inclination toward specific research projects rather than institutional or technical field needs. Second, there is a natural emphasis on spaceflight missions because budget allocation decisions that might lead to a decreasing flight rate are not viewed as consistent with a stable long-term strategy. Third, there are too many proposals being submitted relative to the total funding available to the research community and the resources to review them. More effort in proposal preparation and review means less time devoted to research and a less efficient process.
NASA officials also emphasized that the total research dollars available to the research community are not increasing at a time when the volume and complexity of mission data being returned and the depth and diversity of the science represented in flight missions are increasing rapidly. This leads to a growing gap between the money invested in NASA missions and NASA’s capability to realize a full return on that investment. Indeed, several recent independent studies have concluded that the mismatch between NASA’s assigned responsibilities and available budget is untenable.1 Other participants in the committee’s data-gathering meetings commented that the composition and balance between spaceflight missions and mission-enabling activities have evolved over time, but without quantification and explicit assessment inside or outside NASA.
The committee identified several basic practices for managing mission-enabling activities that do not appear to be widely employed or adequately applied in SMD.
Ensuring Traceability from Strategic Goals to Mission-Enabling Objectives and Activities
The committee believes it is essential to identify those mission-enabling activities that are required to meet the unique strategic goals of each SMD scientific division with respect to past, imminent, and still-to-be-identified future spaceflight missions. The lack of a well-defined process for selecting mission-enabling activities motivated by strategic program goals limits the ability of SMD to justify its mission-enabling activities to senior NASA management, the Office of Managment and Budget, and Congress. Lack of such a process also reduces the transparency of decision making to the scientific community and fails to ensure that the goals of each division are adequately supported by mission-enabling activities.
Establishing Systematic Allocation of Resources and Metrics for Evaluation of Effectiveness
The SMD Strategic Plan discusses how spaceflight missions are assigned priorities within the context of the NRC decadal surveys2 and a prioritized set of science questions, but aside from statements regarding the fundamental importance of mission-enabling activities to the scientific enterprise, the Strategic Plan does not discuss how relative funding allocation decisions are made within mission-enabling programs or between mission-enabling and spaceflight mission programs. Moreover, there is neither a set of criteria for determining the range of mission-enabling programs required to support SMD strategic goals nor an assessment process for determining the directions and effectiveness of the mission-enabling activities in the context of meeting the strategic goals of SMD or its divisions. This lack of strategic integration was noted in the first recommendation of the 1998 NRC report on research and data analysis (R&DA) activities.3
A 1999 NRC report4 recommended ways to evaluate federally sponsored basic and applied research and drew several conclusions that are highly relevant to the committee’s considerations of metrics of effectiveness. It emphasized the importance of using expert reviewers in assessing both types of research. One of the report’s major recommendations was as follows:
For applied research programs, agencies should measure progress toward practical outcomes. For basic research programs, agencies should measure quality, relevance, and leadership. In addition, agencies should conduct periodic reviews of the overall practical outcomes of an agency’s overall past support of applied and basic research. The use of measurements needs to recognize what can and cannot be measured. Misuse of measurement can lead to strongly negative results; for example, measuring basic research on the basis of short-term relevance would be extremely destructive to quality work. (p. 38)
The following set of R&D metrics, which were developed by George Heilmeier,5 former director of the Defense Advanced Research Projects Agency, is widely used by industry and government:
-
What are you trying to do? Articulate your objectives using absolutely no jargon.
-
How is it done today, and what are the limits of current practice?
-
What’s new in your approach, and why do you think it will be successful?
-
Who cares? If you’re successful, what difference will it make?
-
What are the risks and the payoffs?
-
How much will it cost? How long will it take?
-
What are the midterm and final “exams” to check for success?
Most often used for project selection, Heilmeier’s criteria are also frequently used as the basis for reviews and evaluations of project progress. But these criteria can also be adapted at the strategic level—e.g., to measure linkages between customer needs, strategies to satisfy those needs, the objectives and goals associated with each strategy, and finally, the projects and activities that enable the objectives and goals to be realized.
Obtaining Continual Advisory Input
For most of the history of NASA, SMD and its predecessor organizations had a highly effective internal advisory structure. Individual scientific disciplines had informal advisory committees, known as Management Operations Working Groups (MOWGs), which were not chartered under the Federal Advisory Committee Act (FACA). The chairs of these committees served on the advisory committees of the science divisions in SMD; the chairs of the divisional advisory committees served on the advisory committee of the associate administrator (AA) of SMD; and the chair of the AA’s committee served on the NASA Advisory Council. This vertically integrated structure of advisory committees served to link all of the science activities of SMD and gave management at all levels access to expert advice. Conflicts of interest on the formal advisory committees were handled according to the procedures of the FACA. Conflicts of interest on the informal advisory committees were handled in the usual effective way of both revealing and balancing any potential conflict.
Despite the fact that this advisory structure served SMD and NASA very well for more than 40 years, it was effectively eliminated by NASA in 2004 without an adequate replacement. Some MOWGs still exist, but the linkage among advisory committees, and the flow of effective actionable scientific advice at the directorate level, is no longer possible. From the perspective of prioritization and management of SMD mission-enabling activities, this is a serious concern in that there is currently no mechanism in place whereby the scientific community can directly contribute to the design, assessment, or implementation of these programs.
A 2006 NRC report, An Assessment of the Balance in NASA’s Science Programs, also noted the consequences of not having an effective internal advisory structure in SMD:
A past strength of the NASA science programs, in both planning and their execution, has been the intimate involvement of the scientific community…[however] external scientific involvement was absent in the construction of the program that accompanied the FY2007 budget; had an advisory structure existed, it could have warned NASA of the outcry that would accompany cuts to the R&A budget and other decisions. (p. 33)
As a 2004 NRC report noted,6 a scientific community that has some “ownership” in a program creates “constructive tension” that pushes the program to excel.
The effective management of the mission-enabling activities in SMD requires the continual interaction with and assessment by the science community. If it is not possible to reconstitute the advisory structure that previously performed the interaction and assessment functions, it will be necessary to create some equivalent advisory structure, even if it is only on an ad hoc basis and only devoted to mission-enabling activities.
Establishing Budget Transparency
SMD’s funds for mission-enabling activities are carried in many different budget categories. They include, for example, lines for research and analysis, data analysis, data systems, technology development, suborbital programs, and guest investigator programs in some disciplines. Consequently, the structure of the SMD budget
does not readily lend itself to identifying the total mission-enabling level of investment. SMD officials explained to the committee that “research is part of everything we do, and is part of every budget line” and that “isolating research into a single budget line gives the false impression to outside observers that research is separate from flight missions rather than being an integral part of the [nation’s R&D efforts] benefiting from NASA’s flight missions.”7 Thus, while some budget lines clearly relate to mission-enabling activities (e.g., research and analysis lines), other mission-enabling activities appear to be embedded in spaceflight mission budget lines. Without aggregating the total mission-enabling investment levels within SMD divisions and programs, it is difficult if not impossible to conduct analyses and evaluate trade-offs in the allocation of budgets for mission-enabling activities versus spaceflight missions.8
In contrast, the DOD has distinct and separate budgetary program elements for basic research (6.1), applied research (or exploratory development) (6.2), and advanced technology development (6.3).9 Those funds must reside in their stand-alone program elements and cannot be mixed with one another nor embedded in non-research or non-technology budget lines. Thus, for instance, funding for basic research cannot be used for advanced technology development activities, and vice versa, and each of those budget elements must be separately identified and justified with their own respective rationales. Contrary to the idea that DOD research and technology funds should be placed in “one pot” for maximum flexibility and not differentiated from one another in terms of budget construction, such a practice might encourage misuse of basic research funds for advanced technology activities when these latter resource-intensive and highly visible activities run into budget overrun situations. Similarly, failure to clearly identify and differentiate the full magnitude of mission-enabling funds from spaceflight mission funds could lead to a similar situation within SMD. This full transparency of the research and technology budgets has motivated DOD organizations to develop and articulate the rationale for their research and technology budgets. It has also allowed various stakeholders (DOD agencies, Congress, others) to protect those budget elements when overall budgetary shortfalls would tempt some to divert funding away from mission-enabling activities within DOD to mission activities.
As an added consideration, experience within DOD has shown that actively managing basic research types of activities often requires different processes, metrics, and management techniques from those associated with managing advanced technology development and system prototyping activities. Thus, explicit identification and separation of basic research and advanced technology development funding can contribute to more effective portfolio management of the research and technology budget as a whole.
Sustaining a Capable Technical Workforce
In the past some mission-enabling programs have been vague or even inconsistent in their approach to workforce development. For example, the Long-Term Space Astrophysics Program was initially considered as a mechanism for encouraging junior investigators, but in the end supported a substantial number of senior ones.10 The committee found that the lack of a clear definition of the role played by mission-enabling activities in workforce development has led to differences of opinion within SMD on what this role should be. Some divisions (e.g., Earth Science and Planetary Science) have specific programs for early career development, while others do not. All of the divisions have graduate student participation at some level, but the committee found no evidence that workforce development is being considered strategically in any of the divisions, beyond the usual counting
of the number of students supported. Specific selection criteria for proposals include “science merit (from peer review) and programmatic needs (future missions, unique opportunities, portfolio balance, high-risk research).”11 Consideration of workforce development or training of young scientists and engineers is not currently part of the formal selection process. When interviewed, some SMD program managers indicated that workforce development was not a goal of their programs. Although they did find it to be an important side benefit of SMD activities, they did not feel that it should be a consideration when selecting competing proposals. Other program managers did take it into consideration in an informal manner. The diversity in workforce goals among the different divisions is not necessarily undesirable, but the lack of public, clear articulation of the particular workforce goal for each group is a problem.
Providing for Adequate SMD Staffing in Support of Mission-Enabling Activities
Over the last two decades there has been a substantial reduction in staff in SMD at NASA headquarters. (Counting support service contractors, the reduction is by a factor ~3.) The result is that the research program staff no longer has the time to perform the tasks required of them, such as the management of mission-enabling activities, while still overseeing scientific aspects of flight programs, maintaining adequate contact with the scientific communities they serve, and in some instances, participating in both interagency and international programs.
There are several root causes for the committee’s concerns, not the least of which is the need for adequate scientific and support staff at NASA headquarters who have the skills, time, and opportunity to proactively manage the mission-enabling portfolios. The committee encountered program managers who are competent and dedicated but are overworked. Program managers who handle SMD’s mission-enabling activities oversee the solicitation, evaluation, and award selection decisions for more than 4,200 proposals annually. The committee heard, and saw, that program officers have heavy and complex workloads that leave them little time to focus on their mission-enabling program management responsibilities and still meet basic oversight requirements for tracking the progress of funding actions and program developments. For example, a typical SMD science program manager’s responsibilities might now include the following roles:
-
Program scientist for one or more future spaceflight missions and/or instruments in development or going through competitive selection
-
Program scientist for several (e.g., 2-4) operating spaceflight missions
-
Participation in biennial reviews of all operating missions in a discipline area
-
Management of several (e.g., 2-5) different research and analysis grant programs (including oversight for preparation of the proposal solicitations, proposal peer reviews, award selection recommendations, and award paperwork)
-
Program scientist for a suborbital flight program (or other major research facility)
-
Support of internal advisory committees and responses to requests from external advisory bodies (e.g., NRC committees)
-
Participation in meetings of scientific societies to stay abreast of developments in relevant fields
The committee firmly believes that this diversity of roles in which SMD program officers maintain active insight into both mission-enabling and flight mission programs is a strength that should be preserved. Furthermore, responsibility for managing mission-enabling programs is properly a headquarters role; delegating this responsibility to NASA centers would create a conflict of interest because scientists at the centers compete with non-NASA scientists for funding in the programs. But with the current inadequate size of the SMD staff, program officers are not able to devote an appropriate amount of time to these tasks. The committee considers that the staffing shortfall at NASA headquarters is the most important impediment to being able to effectively implement its recommendations.