2
Assessment of NASA’s Mission-Enabling Activities

The mission-enabling activities of NASA’s Science Mission Directorate (SMD) have served as the essential foundation for the world’s most diverse and successful space and Earth science program. NASA’s spaceflight program has been highly successful, fundamentally altering our scientific understanding of the Earth, the solar system, and the universe beyond. The committee believes that for progress to continue, similarly visionary science and technology investments must continue and must be recognized as the essential foundation for all of SMD’s activities.

BROAD CONCERNS AND OPPORTUNITIES FOR IMPROVEMENT

During briefings to the committee, NASA officials shared several concerns about mission-enabling programs. First, in a mission-oriented agency like NASA there is a natural tendency to give priority to targeted research investigations rather than to support more broadly based core activities. This preference also creates an inclination toward specific research projects rather than institutional or technical field needs. Second, there is a natural emphasis on spaceflight missions because budget allocation decisions that might lead to a decreasing flight rate are not viewed as consistent with a stable long-term strategy. Third, there are too many proposals being submitted relative to the total funding available to the research community and the resources to review them. More effort in proposal preparation and review means less time devoted to research and a less efficient process.

NASA officials also emphasized that the total research dollars available to the research community are not increasing at a time when the volume and complexity of mission data being returned and the depth and diversity of the science represented in flight missions are increasing rapidly. This leads to a growing gap between the money invested in NASA missions and NASA’s capability to realize a full return on that investment. Indeed, several recent independent studies have concluded that the mismatch between NASA’s assigned responsibilities and available budget is untenable.1 Other participants in the committee’s data-gathering meetings commented that the composition and balance between spaceflight missions and mission-enabling activities have evolved over time, but without quantification and explicit assessment inside or outside NASA.

1

National Research Council, An Assessment of Balance in NASA’s Science Programs, The National Academies Press, Washington, D.C., 2006; National Research Council, America’s Future in Space: Aligning the Civil Space Program with National Needs, The National Academies Press, Washington, D.C., 2009.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 25
2 Assessment of NASA’s Mission-Enabling Activities The mission-enabling activities of NASA’s Science Mission Directorate (SMD) have served as the essential foundation for the world’s most diverse and successful space and Earth science program. NASA’s spaceflight program has been highly successful, fundamentally altering our scientific understanding of the Earth, the solar system, and the universe beyond. The committee believes that for progress to continue, similarly visionary science and technology investments must continue and must be recognized as the essential foundation for all of SMD’s activities. BROAD CONCERNS AND OPPORTuNITIES FOR IMPROVEMENT During briefings to the committee, NASA officials shared several concerns about mission-enabling programs. First, in a mission-oriented agency like NASA there is a natural tendency to give priority to targeted research investigations rather than to support more broadly based core activities. This preference also creates an inclina - tion toward specific research projects rather than institutional or technical field needs. Second, there is a natural emphasis on spaceflight missions because budget allocation decisions that might lead to a decreasing flight rate are not viewed as consistent with a stable long-term strategy. Third, there are too many proposals being submitted relative to the total funding available to the research community and the resources to review them. More effort in proposal preparation and review means less time devoted to research and a less efficient process. NASA officials also emphasized that the total research dollars available to the research community are not increasing at a time when the volume and complexity of mission data being returned and the depth and diversity of the science represented in flight missions are increasing rapidly. This leads to a growing gap between the money invested in NASA missions and NASA’s capability to realize a full return on that investment. Indeed, several recent independent studies have concluded that the mismatch between NASA’s assigned responsibilities and available budget is untenable.1 Other participants in the committee’s data-gathering meetings commented that the composi - tion and balance between spaceflight missions and mission-enabling activities have evolved over time, but without quantification and explicit assessment inside or outside NASA. 1 National Research Council, An Assessment of Balance in NASA’s Science Programs, The National Academies Press, Washington, D.C., 2006; National Research Council, America’s Future in Space: Aligning the Civil Space Program with National Needs, The National Academies Press, Washington, D.C., 2009. 

OCR for page 25
6 AN ENABLING FOUNDATION FOR NASA’S EARTH AND SPACE SCIENCE MISSIONS The committee identified several basic practices for managing mission-enabling activities that do not appear to be widely employed or adequately applied in SMD. Ensuring Traceability from Strategic Goals to Mission-Enabling Objectives and Activities The committee believes it is essential to identify those mission-enabling activities that are required to meet the unique strategic goals of each SMD scientific division with respect to past, imminent, and still-to-be-identi - fied future spaceflight missions. The lack of a well-defined process for selecting mission-enabling activities motivated by strategic program goals limits the ability of SMD to justify its mission-enabling activities to senior NASA management, the Office of Managment and Budget, and Congress. Lack of such a process also reduces the transparency of decision making to the scientific community and fails to ensure that the goals of each division are adequately supported by mission-enabling activities. Establishing Systematic Allocation of Resources and Metrics for Evaluation of Effectiveness The SMD Strategic Plan discusses how spaceflight missions are assigned priorities within the context of the NRC decadal surveys2 and a prioritized set of science questions, but aside from statements regarding the funda - mental importance of mission-enabling activities to the scientific enterprise, the Strategic Plan does not discuss how relative funding allocation decisions are made within mission-enabling programs or between mission-enabling and spaceflight mission programs. Moreover, there is neither a set of criteria for determining the range of mis - sion-enabling programs required to support SMD strategic goals nor an assessment process for determining the directions and effectiveness of the mission-enabling activities in the context of meeting the strategic goals of SMD or its divisions. This lack of strategic integration was noted in the first recommendation of the 1998 NRC report on research and data analysis (R&DA) activities.3 A 1999 NRC report4 recommended ways to evaluate federally sponsored basic and applied research and drew several conclusions that are highly relevant to the committee’s considerations of metrics of effectiveness. It emphasized the importance of using expert reviewers in assessing both types of research. One of the report’s major recommendations was as follows: For applied research programs, agencies should measure progress toward practical outcomes. For basic research programs, agencies should measure quality, relevance, and leadership. In addition, agencies should conduct periodic reviews of the overall practical outcomes of an agency’s overall past support of applied and basic research. The use of measurements needs to recognize what can and cannot be measured. Misuse of measurement can lead to strongly negative results; for example, measuring basic research on the basis of short-term relevance would be extremely destructive to quality work. (p. 38) The following set of R&D metrics, which were developed by George Heilmeier, 5 former director of the Defense Advanced Research Projects Agency, is widely used by industry and government: • What are you trying to do? Articulate your objectives using absolutely no jargon. • How is it done today, and what are the limits of current practice? • What’s new in your approach, and why do you think it will be successful? 2 The NRC decadal surveys have been widely used by the scientific community and by program decision makers because they (a) present explicit, consensus priorities for the most important, potentially revolutionary science that should be undertaken within the span of a decade; (b) develop priorities for future investments in research facilities, space missions, and/or supporting programs; (c) rank competing opportunities and ideas and clearly indicate which ones are of higher or lower priority in terms of the timing, risk, and cost of their implementation; and (d) make the difficult adverse decisions about other meritorious ideas that cannot be accommodated within realistically available resources. 3 National Research Council, Supporting Research and Data Analysis in NASA’s Science Programs: Engines for Innovation and Synthesis, National Academy Press, Washington, D.C., 1998. 4 National Research Council, Evaluating Federal Research Programs: Research and the Government Performance and Results Act , National Academy Press, Washington, D.C., 1999. 5 G. Heilmeier, Some reflections on innovation and invention, The Bridge 22(4):12-16, 1992.

OCR for page 25
 ASSESSMENT OF NASA’S MISSION-ENABLING ACTIVITIES • Who cares? If you’re successful, what difference will it make? • What are the risks and the payoffs? • How much will it cost? How long will it take? • What are the midterm and final “exams” to check for success? Most often used for project selection, Heilmeier’s criteria are also frequently used as the basis for reviews and evaluations of project progress. But these criteria can also be adapted at the strategic level—e.g., to measure linkages between customer needs, strategies to satisfy those needs, the objectives and goals associated with each strategy, and finally, the projects and activities that enable the objectives and goals to be realized. Obtaining Continual Advisory Input For most of the history of NASA, SMD and its predecessor organizations had a highly effective internal advisory structure. Individual scientific disciplines had informal advisory committees, known as Management Operations Working Groups (MOWGs), which were not chartered under the Federal Advisory Committee Act (FACA). The chairs of these committees served on the advisory committees of the science divisions in SMD; the chairs of the divisional advisory committees served on the advisory committee of the associate administrator (AA) of SMD; and the chair of the AA’s committee served on the NASA Advisory Council. This vertically integrated structure of advisory committees served to link all of the science activities of SMD and gave management at all levels access to expert advice. Conflicts of interest on the formal advisory committees were handled according to the procedures of the FACA. Conflicts of interest on the informal advisory committees were handled in the usual effective way of both revealing and balancing any potential conflict. Despite the fact that this advisory structure served SMD and NASA very well for more than 40 years, it was effectively eliminated by NASA in 2004 without an adequate replacement. Some MOWGs still exist, but the link - age among advisory committees, and the flow of effective actionable scientific advice at the directorate level, is no longer possible. From the perspective of prioritization and management of SMD mission-enabling activities, this is a serious concern in that there is currently no mechanism in place whereby the scientific community can directly contribute to the design, assessment, or implementation of these programs. A 2006 NRC report, An Assessment of the Balance in NASA’s Science Programs, also noted the consequences of not having an effective internal advisory structure in SMD: A past strength of the NASA science programs, in both planning and their execution, has been the intimate involve - ment of the scientific community…[however] external scientific involvement was absent in the construction of the program that accompanied the FY2007 budget; had an advisory structure existed, it could have warned NASA of the outcry that would accompany cuts to the R&A budget and other decisions. (p. 33) As a 2004 NRC report noted,6 a scientific community that has some “ownership” in a program creates “con - structive tension” that pushes the program to excel. The effective management of the mission-enabling activities in SMD requires the continual interaction with and assessment by the science community. If it is not possible to reconstitute the advisory structure that previ - ously performed the interaction and assessment functions, it will be necessary to create some equivalent advisory structure, even if it is only on an ad hoc basis and only devoted to mission-enabling activities. Establishing Budget Transparency SMD’s funds for mission-enabling activities are carried in many different budget categories. They include, for example, lines for research and analysis, data analysis, data systems, technology development, suborbital programs, and guest investigator programs in some disciplines. Consequently, the structure of the SMD budget 6National Research Council, Issues and Opportunities Regarding the U.S. Space Program: A Summary Report of a Workshop on National Space Policy, The National Academies Press, Washington, D.C., 2004.

OCR for page 25
 AN ENABLING FOUNDATION FOR NASA’S EARTH AND SPACE SCIENCE MISSIONS does not readily lend itself to identifying the total mission-enabling level of investment. SMD officials explained to the committee that “research is part of everything we do, and is part of every budget line” and that “isolating research into a single budget line gives the false impression to outside observers that research is separate from flight missions rather than being an integral part of the [nation’s R&D efforts] benefiting from NASA’s flight missions.”7 Thus, while some budget lines clearly relate to mission-enabling activities (e.g., research and analy - sis lines), other mission-enabling activities appear to be embedded in spaceflight mission budget lines. Without aggregating the total mission-enabling investment levels within SMD divisions and programs, it is difficult if not impossible to conduct analyses and evaluate trade-offs in the allocation of budgets for mission-enabling activities versus spaceflight missions.8 In contrast, the DOD has distinct and separate budgetary program elements for basic research (6.1), applied research (or exploratory development) (6.2), and advanced technology development (6.3).9 Those funds must reside in their stand-alone program elements and cannot be mixed with one another nor embedded in non-research or non-technology budget lines. Thus, for instance, funding for basic research cannot be used for advanced technol - ogy development activities, and vice versa, and each of those budget elements must be separately identified and justified with their own respective rationales. Contrary to the idea that DOD research and technology funds should be placed in “one pot” for maximum flexibility and not differentiated from one another in terms of budget con - struction, such a practice might encourage misuse of basic research funds for advanced technology activities when these latter resource-intensive and highly visible activities run into budget overrun situations. Similarly, failure to clearly identify and differentiate the full magnitude of mission-enabling funds from spaceflight mission funds could lead to a similar situation within SMD. This full transparency of the research and technology budgets has motivated DOD organizations to develop and articulate the rationale for their research and technology budgets. It has also allowed various stakeholders (DOD agencies, Congress, others) to protect those budget elements when overall budgetary shortfalls would tempt some to divert funding away from mission-enabling activities within DOD to mission activities. As an added consideration, experience within DOD has shown that actively managing basic research types of activities often requires different processes, metrics, and management techniques from those associated with managing advanced technology development and system prototyping activities. Thus, explicit identification and separation of basic research and advanced technology development funding can contribute to more effective port - folio management of the research and technology budget as a whole. Sustaining a Capable Technical Workforce In the past some mission-enabling programs have been vague or even inconsistent in their approach to workforce development. For example, the Long-Term Space Astrophysics Program was initially considered as a mechanism for encouraging junior investigators, but in the end supported a substantial number of senior ones. 10 The committee found that the lack of a clear definition of the role played by mission-enabling activities in work - force development has led to differences of opinion within SMD on what this role should be. Some divisions (e.g., Earth Science and Planetary Science) have specific programs for early career development, while others do not. All of the divisions have graduate student participation at some level, but the committee found no evidence that workforce development is being considered strategically in any of the divisions, beyond the usual counting 7 Briefing to the committee by SMD Chief Scientist, Paul Hertz, on January 22, 2009. 8 The NRC report, Supporting Research and Data Analysis in NASA’s Science Programs: Engines for Innovation and Synthesis (National Academy Press, Washington, D.C., 1998), made the same points. 9 DOD R&D categories 6.1, 6.2, and 6.3 designate basic research, applied research, and advanced technology development, respectively. Higher categories cover stages of development ranging from technology demonstration and validation (6.4) through preproduction operational system development (6.7). 10 See National Research Council, Astronomy and Astrophysics in the New Millennium (National Academy Press, Washington, D.C., 2001, p. 199) and D. Helfand, Columbia University, “A Longitudinal Study of Selected Astronomers Based on Early Sources of Support,” provided to the Panel on Astronomy Education and Policy of the Astronomy and Astrophysics Survey Committee, National Research Council, Wash - ington, D.C.

OCR for page 25
 ASSESSMENT OF NASA’S MISSION-ENABLING ACTIVITIES of the number of students supported. Specific selection criteria for proposals include “science merit (from peer review) and programmatic needs (future missions, unique opportunities, portfolio balance, high-risk research).” 11 Consideration of workforce development or training of young scientists and engineers is not currently part of the formal selection process. When interviewed, some SMD program managers indicated that workforce development was not a goal of their programs. Although they did find it to be an important side benefit of SMD activities, they did not feel that it should be a consideration when selecting competing proposals. Other program managers did take it into consideration in an informal manner. The diversity in workforce goals among the different divisions is not necessarily undesirable, but the lack of public, clear articulation of the particular workforce goal for each group is a problem. Providing for Adequate SMD Staffing in Support of Mission-Enabling Activities Over the last two decades there has been a substantial reduction in staff in SMD at NASA headquarters. (Count- ing support service contractors, the reduction is by a factor ~3.) The result is that the research program staff no longer has the time to perform the tasks required of them, such as the management of mission-enabling activities, while still overseeing scientific aspects of flight programs, maintaining adequate contact with the scientific com - munities they serve, and in some instances, participating in both interagency and international programs. There are several root causes for the committee’s concerns, not the least of which is the need for adequate scientific and support staff at NASA headquarters who have the skills, time, and opportunity to proactively manage the mission-enabling portfolios. The committee encountered program managers who are competent and dedicated but are overworked. Program managers who handle SMD’s mission-enabling activities oversee the solicitation, evaluation, and award selection decisions for more than 4,200 proposals annually. The committee heard, and saw, that program officers have heavy and complex workloads that leave them little time to focus on their mis - sion-enabling program management responsibilities and still meet basic oversight requirements for tracking the progress of funding actions and program developments. For example, a typical SMD science program manager’s responsibilities might now include the following roles: • Program scientist for one or more future spaceflight missions and/or instruments in development or going through competitive selection • Program scientist for several (e.g., 2-4) operating spaceflight missions • Participation in biennial reviews of all operating missions in a discipline area • Management of several (e.g., 2-5) different research and analysis grant programs (including oversight for preparation of the proposal solicitations, proposal peer reviews, award selection recommendations, and award paperwork) • Program scientist for a suborbital flight program (or other major research facility) • Support of internal advisory committees and responses to requests from external advisory bodies (e.g., NRC committees) • Participation in meetings of scientific societies to stay abreast of developments in relevant fields The committee firmly believes that this diversity of roles in which SMD program officers maintain active insight into both mission-enabling and flight mission programs is a strength that should be preserved. Further- more, responsibility for managing mission-enabling programs is properly a headquarters role; delegating this responsibility to NASA centers would create a conflict of interest because scientists at the centers compete with non-NASA scientists for funding in the programs. But with the current inadequate size of the SMD staff, program officers are not able to devote an appropriate amount of time to these tasks. The committee considers that the staffing shortfall at NASA headquarters is the most important impediment to being able to effectively implement its recommendations. 11 Briefing to the committee by SMD Chief Scientist, Paul Hertz, on January 22, 2009.

OCR for page 25