The statement of task charged the committee to:
Assess the impact of current and recent SMD missions with a range of life-cycle costs. A representative subset of missions within each of SMD’s four science theme areas may be selected for analysis. The committee’s analysis of each representative mission will include a discussion of the relation between mission scientific impact and mission life cycle cost (or cost to date) in order to understand the return on expenditures for various mission classes. In describing the impact of the chosen missions the committee should consider dimensions such as:
- Scientific productivity,
- Impact on the current and future health of the relevant scientific community, and
- Contribution to development and demonstration of technology applicable to future missions.
In 2012 the Jet Propulsion Laboratory (JPL) led a study to try to adequately assess the impact of large strategic missions compared to competed missions for planetary sciences. The study team concluded that there are many rules that need to be followed in order to make an assessment. First, the approach needs to be quantitative and applied in a consistent manner throughout both classes of missions. Second, the information needs to be clearly identifiable and available to the public. Third, the study should be conducted by a group of people with viewpoints that are diverse and across the spectrum of competed and strategic planetary missions. The approach to these evaluations should include (1) the scientific impact, (2) the impact on the number of scientists directly supported, and (3) the impact on the number of mission instruments.
The scientific impact refers to the utilization of existing data, which may include publications, citations, and h-index information taken from the Web of Science on a mission-by-mission basis. The h-index is defined as the number of papers cited at least h times each for a given mission (rather than for a given scientist).1 The JPL study concluded that the citation peak for most publications is reached at the 5- to 6-year mark.
According to the JPL study’s analysis of the h-index and other publication statistics, both large and smaller planetary missions have a significant scientific impact. If larger or smaller missions were eliminated in the future, it would have a negative effect on the program. For instance, eliminating large strategic missions would make
1 J.E. Hirsch, 2005, An index to quantify an individual’s scientific research output, Proceedings of the National Academy of Sciences 102(46):16569-16572.
it difficult to access challenging locations, and eliminating smaller missions would limit the ability to respond quickly to new discoveries. For the scientific community, eliminating either large or smaller missions could lead to major funding gaps that would place strain on the Research and Analysis program. For the engineering community specifically, there could be a loss of opportunity to produce unique classes of instruments required by planetary science, and loss of entry, descent and landing, and deep-space navigation capabilities. The JPL data are retrospective and aggregate over four decades in a variety of political and fiscal environments. The JPL method is not predictive but is informative.
To seek to address the charge regarding comparing the impact of a range of missions, this committee requested data from NASA’s Science Mission Directorate (SMD). The data were sought to inform the committee’s analysis of scientific productivity, impact of missions and mission size on the current and future health of the relevant scientific community, the use of past technology, and the potential feed-forward of technology developed and demonstrated in current missions. Because the committee expected that data on a number of missions might be incomplete or nonexistent due to incomplete historical databases of program costs, it asked for data on more missions than it required for its analysis with the hope of obtaining sufficient information to accomplish the task.
NASA’s SMD expended considerable effort to provide the committee with the requested data. Nevertheless and unfortunately, the data the four science divisions were able to provide are uneven and, in some cases, entirely missing. This affected the committee’s ability to properly and thoroughly address the statement of task. In part, missing data is due to the change in accounting practices at NASA in the early 2000s and the switch to full cost accounting. Missions started before the implementation of full cost accounting, which includes many of the currently operating large strategic missions, have less complete cost data than more recent missions. A greater ability to rapidly provide public data of this kind, however, would have broader implications for the ability of SMD to present their story and successes to Congress, the executive branch (e.g., the Office of Science and Technology Policy and the Office of Management and Budget), the NASA administrator, the scientific community, and the public.
The committee attempted to make some comparisons of the outputs of different-size missions for NASA’s science divisions. The committee cautions, however, that any comparisons of such data are best made within their respective disciplines, not across the disciplines. For example, trying to compare scientific productivity of a heliophysics mission to an astrophysics mission would be highly dubious, because their value cannot be understood without the context of how they fit within their disciplines. Furthermore, publication data alone are not a perfect indicator of importance. The generation of thousands of peer-reviewed scientific papers from a single mission is impressive, but a single observation could significantly change an entire scientific field.
This committee was unable to draw many conclusions from the data on the various space science missions, other than some obvious ones such as larger missions result in funding more full-time equivalent personnel and higher numbers of publications. In general, the data confirmed information and comments that the committee heard from various speakers:
- Large strategic missions are inherently designed to conduct transformative science with order of magnitude increases in discovery space.
- Many large strategic missions are managed to engage a broad segment of the community, ranging from guest observer grants in the case of astrophysics to access to large layered databases in the case of Earth science.
- Many large strategic missions address fundamental questions at the frontiers of the space sciences that are of great interest to the broader scientific community, to policy makers, and to the public.
- Technology development occurs in large strategic missions, as well as dedicated technology development programs. Technology can flow in multiple directions, from large missions to small and from small missions to large.
What several of the missions discussed in the following appendixes demonstrate is that technology development is different for each division, and there is no inherent path for technology flow. Currently, small spacecraft such as CubeSats are generating technology that is migrating to larger missions. Larger missions have developed their own technology, some of which has been adopted by smaller missions and some of which is unique to those
missions that develop it. Finally, separate technology development programs in the divisions, as well as the Space Technology Mission Directorate, have also been important incubators.
FINDING: Large strategic science missions support large teams of scientists and graduate students and therefore support the development and the health of their respective scientific communities in ways that smaller missions cannot.
FINDING: Smaller missions are often necessary to provide new insights, respond to recent discoveries, and refine the scientific goals of less frequent large strategic missions.
FINDING: Technology development occurs at many levels: large strategic missions, medium-size and even small missions, and separate technology development programs.
The limitations of the data across various space science missions are in some cases a result of when these missions were in development. For example, the implementation of full cost accounting at NASA in the early 2000s has made cost comparisons between missions before and after this date—such as Hubble, Chandra, and Cassini—difficult except in broad terms. But now that full cost accounting has become the standard at NASA, it is easier for the agency to acquire and compare these data. In addition, the agency already uses its senior review process to gather data on the scientific productivity of its missions. The senior review data are publicly releasable and NASA has released it, although not consistently across divisions or always easily accessible. The committee concluded that there would be great value to the agency and its programs if the SMD made these data as well as its other programmatic data more publicly available and standardized. If, in the future, the SMD faces questions about the productivity and value of its large strategic missions, then having the data compiled and made public to respond to these questions would be vital.
RECOMMENDATION: In order to demonstrate the role and scientific productivity of large strategic missions in advancing science, technology, and the long-term health of the field, NASA’s Science Mission Directorate (SMD) should develop a publicly accessible database, updated at least annually, that tracks basic data related to all confirmed missions in development as well as operational and past missions from each of the SMD divisions. These data should include development costs; publication numbers and other bibliographic data; outreach data (number of press releases and so on should be tracked); science, engineering, and other full-time equivalents; and other routine data typically sought in senior review proposal submittals once prime missions have been completed. These data should be of sufficient detail and quality to enable basic analyses related to scientific productivity and contributions to the health of the respective fields.
Although the committee acknowledges that establishing such a public database will require effort by NASA, it concluded that this would be useful to the agency in communicating the value and output of its missions as a whole, as opposed to via periodic press releases or at scientific conferences. The agency is in a better position today than it was only a decade ago to accurately report on its mission costs and performance across the entire Science Mission Directorate, and the committee concluded it can and should do so.
This study was initiated in part to evaluate the value and role of large strategic missions and concluded that they are highly valuable, to the scientific endeavor, to NASA, and to the United States. The committee concluded that NASA has substantial information to make that case and can continue to do so provided that it prepares the information in a coherent manner.