NASA’s Earth Science Division (ESD) conducts a wide range of satellite and suborbital missions to observe Earth’s land surface and interior, biosphere, atmosphere, cryosphere, and oceans as part of a program to improve understanding of Earth as an integrated system. Earth observations provide the foundation for critical scientific advances, and environmental data products derived from these observations are used in resource management and for an extraordinary range of societal applications, including weather forecasts, climate projections, sea level change, water management, disease early warning, agricultural production, and the response to natural disasters.
As the complexity of societal infrastructure and its vulnerability to environmental disruption increases, the demands for deeper scientific insights and more actionable information continue to rise. To serve these demands, NASA’s ESD is challenged with optimizing the partitioning of its finite resources among measurements intended for exploring new science frontiers, carefully characterizing long-term changes in the Earth system, and supporting ongoing societal applications. This challenge is most acute in the decisions the division makes between supporting measurement continuity of data streams that are critical components of Earth science research programs (including, but not limited, to climate-related measurements) and the development of new measurement capabilities.
While the distinction between measurements oriented toward “research” and “applications” is somewhat artificial (both types of measurements are typically needed in support of a particular societal application, and both research and application objectives may require continuous or sustained measurements), their requirements are not consistent. In particular, while many applications are associated with a requirement for near real-time data availability, climate change science objectives typically require accurate measurements and long, stable, uninterrupted time-series. Further, within the class of measurements with a science/research focus, the need for new measurements to enable Earth System process studies contrasts with the need to continue well-understood measurements related to key climate change indicators.
Community guidance to NASA ESD from the first National Research Council (NRC)1 Earth science and applications from space decadal survey (NRC, 2007) largely focused on new measurements, owing to assumptions made about the role of other agencies in supporting high-priority climate, weather, and land surface continuity measurements. However, for a variety of reasons, including technical and budgetary challenges, some of these assumptions were not met (NRC, 2012). In response to these changes, as well as to guidance from the Administra-
______________
1 Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council are used in an historic context identifying programs prior to July 1.
tion and Congress, NASA’s Earth science portfolio has expanded to include new responsibilities for the continuation of several previously initiated measurements that were formerly assigned to other agencies.
As decadal survey recommendations are executed and new capabilities and applications are demonstrated, NASA anticipates an increasing number of measurements and associated instruments and missions will be candidates for follow-ons. The agency’s request for the present study (the statement of task is reprinted in Appendix A) recognizes this trend and the importance of establishing a more quantitative understanding of the need for measurement continuity and the consequences of measurement gaps. In addition to requesting a working definition of “continuity,” the task statement asks that a decision framework be provided to help optimize the allocation of resources.
This report, from the Committee on a Framework for Analyzing the Needs for Continuity of NASA-Sustained Remote Sensing Observations of the Earth from Space, is the response to these requests. As detailed in the report, the committee recommends to NASA a decision-making framework, based on key continuity characteristics, that effectively discriminates between competing continuity measurements. The recommended framework carries a strong emphasis on quantitative evaluation methods in order to achieve process objectivity and transparency.
In developing a readily implementable framework, the committee focused on climate change science goals where space-based continuity measurements are expected to make substantial contributions. With this specific focus, the recommended framework is intended as a new method for evaluating science-driven continuity missions and represents a complement to the existing NASA proposal evaluation processes for NASA Research Announcements and Earth Venture Announcements of Opportunity.
This framework should be viewed as an initial step toward a more comprehensive methodology. As discussed in the report, modifications to the framework would allow it to be used to establish priorities among new, first-of-a-kind measurements, as well as to examine operational- or applications-based measurements. Developed appropriately, the committee envisions a single comprehensive evaluation approach for both new and continuity measurements, driven by science and/or application objectives.
ELEMENTS OF THE COMMITTEE’S DECISION FRAMEWORK
The committee’s approach in developing the desired decision-framework begins with a clear definition of measurement continuity in time and space. Ensuring continuity of a geophysical variable2 from a sequence of “improved” instruments, or from copies of the same instrument, requires a careful program of calibration, instrument characterization and comparison, and validation. While the vantage point of space facilitates global and repeatable observations of Earth, the development of long-term measurement time-series having small, combined standard uncertainties on multiple spatial scales is particularly challenging. In operational programs, copies of instruments have been flown multiple times with the goal of simplifying this process. Although copies do not eliminate the need for calibration and characterization studies, such an approach—including carefully chosen group procurements of instruments or spacecraft—will reduce costs and typically reduces the risk in providing a long-term continuous record.
The quality of a measurement is particularly relevant in the context of continuity and is characterized primarily by its combined standard uncertainty, which is the consequence of instrument calibration uncertainty, repeatability; time and space sampling; and data systems and delivery for climate variables (algorithms, reprocessing, and availability)—each of which depends on the scientific objective. Changes in platform observing characteristics (for example, altitude and local observing time) introduce perturbations into the entire system. Development of calibration methods through mission overlaps, in situ validation, and ground-based calibration traceable to National Institute of Standards and Technology standards are necessary to provide repeatable long-term measurements of geophysical variables.
With this in mind, the committee finds that the following is a sufficient, high-level definition of continuity across the Earth science subdisciplines for use in an analysis framework focused on scientific objectives:
______________
2 See Box 2.1 for the committee’s definition of geophysical variable and several other terms used in this report.
Finding: Continuity of an Earth measurement exists when the quality of the measurement for a specific quantified Earth science objective is maintained over the required temporal and spatial domain set by the objective.
The notion of a quantified objective is the starting point for the committee’s recommended decision framework. The characteristics of a well-formulated quantified objective are the following:
- It is directly relevant to achieving an overarching science goal of NASA ESD;
- It is presented in such a way that the required measurement(s) and their resolution (spatial, temporal, and radiometric), calibration uncertainty and repeatability, and other requirements have traceability to the overarching science goal; and
- It is expressed in a way that allows an analytical assessment of the importance of the objective to an Earth science goal and the utility of the targeted geophysical variable(s) for meeting the science objective.
Chapter 3 presents several examples of quantified objectives.
Recommendation: Proposed space-based continuity measurements should be evaluated in the context of the quantified Earth science objectives they address.
The committee envisions NASA ESD establishing a small set of quantified objectives from the same sources that inform the development of its program plan, notably the scientific community’s consensus priorities expressed in NRC decadal surveys and guidance from the executive and congressional branches. Congressionally mandated midterm assessments of the decadal survey afford an additional opportunity for community evaluation of the objectives. Continuity of an established data set will compete with proposed new measurements as well as multi-measurement “intensives,” campaigns that may be mounted to, for example, gain a detailed understanding of a particular climate process. The latter proposals should be defined through a quantified objective that could then be evaluated via the committee’s proposed framework or whatever similar quantitative, open, and objective evaluation ESD establishes for continuity measurements.
Recommendation: NASA, which is anticipated to be a principal sponsor of the next decadal survey in Earth science and applications from space, might task the decadal survey with the identification, and possible prioritization, of the quantified Earth science objectives associated with the recommended science goals.
In addition to their research-oriented objectives, Earth observations and their derived information products support numerous user communities within and outside of the government. Extension of the committee’s decision framework to measurements focused on societal-benefit applications is desirable but will require expertise outside of the Earth science community to formulate analogous quantified objectives in Earth applications. Toward this end, the committee makes the following recommendation:
Recommendation: NASA should initiate studies to identify and assess quantified Earth applications objectives related to high-priority, societal-benefit areas.
Based on lessons from cost-benefit analysis and decision theory, the committee found that a value-centered framework is capable of effectively distinguishing among the relevant Earth measurements; implemented appropriately, it will achieve an improved degree of openness and transparency. The value-centered approach recommended in this report includes both measurement benefit and affordability considerations. The study identified a
relatively small set of characteristics that enable a tractable evaluation of benefit, which along with affordability allow discrimination in value among competing measurement/quantified objective pairs.3 They are:
- The scientific importance (I) of the quantified objective;
- The utility (U) of a geophysical variable record for achieving a quantified objective;
- The quality (Q) of a measurement for providing the desired geophysical variable record; and
- The success probability (S) of achieving the measurement and its associated geophysical variable record.
- The affordability (A) of providing the measurement and its geophysical variable record.
Additional cross-cutting factors are recognized to impact both benefit and affordability, and methods to treat them appropriately within the framework are discussed in the report. Examples of cross-cutting factors include the ability to leverage other measurement opportunities in pursuit of the science objective and the resilience of a geophysical variable record to unexpected degradation (or gaps) in the measurement quality.
As discussed in the report, the committee finds that the quality metric plays a decisive role in determining when a measurement should be collected for durations longer than the typical lifetimes of single satellite missions. The most critical factor is whether (or not) the combined standard uncertainty of the measurement is sufficient for addressing the quantified objective. A related factor is the impact of a data gap (see Section 3.4.2 in Chapter 3), which itself depends on the measurements calibration uncertainty (i.e., traceability to an absolute scale) as well as on the natural variability of the measurand over the gap’s duration. While there are numerous ways to evaluate quality in the context of continuity measurements, a useful quality metric is expected to vary between continuity required for short-term operational use (e.g., weather prediction, hazard warnings, agricultural crop monitoring) versus longer-term science objectives, such as those related to global climate change.4 Examples for assessing quality are given in Chapter 4.
Finding: Assessing the quality of a particular continuity measurement requires knowledge of a measurement’s combined standard uncertainty, which is derived from the instrument calibration uncertainty, repeatability, time and space sampling, and data systems and delivery of climate variables (algorithms, reprocessing, and availability), and the consequences of data gaps on the relevant quantified science objective(s).
Recommendation: The committee recommends that NASA be responsible for refining the assessment approach for the quality characteristic.
Evaluation of a measurement’s affordability and benefit for decision-making purposes can likely be accomplished through a number of equally valid methods, some of which are examined in this report. Regardless of the evaluation methods that NASA and the community adopt, the application of those methods should make consistent use of well-documented and understood tools and studies, as highlighted in the following recommendations.
______________
3 The committee debated at length regarding the choice of framework characteristics; the object was to derive a minimal set of largely independent characteristics (metrics) that would provide meaningful evaluations of proposed continuity measurements. That the factors are not completely independent in a statistical sense is recognized. For example, success probability (S) and affordability (A) are not completely independent; however, the relationship between them is sufficiently complex that it was necessary to retain both in the framework. As an example: NASA’s ability to “buy down” risk (i.e., increase S by decreasing A) is not easily quantified for complex technologies; similarly, accounting for the strategic plans of other national and international partners—a difficult problem—is easier to handle in a framework with separate success and affordability factors. Accordingly, the committee elected to retain both the success probability and affordability characteristics. By retaining success probability, the treatment of uncertainty in the decision process is more readily achieved.
4 The committee notes that the quality requirements for measurements related to climate change objectives will often be most stringent at a global scale and less stringent at zonal or regional scales. (Antarctic ozone, regional aerosol change, and polar ice sheets are exceptions where regional anthropogenic signals can be detected before global average signals.) Instrument accuracy and repeatability will, therefore, often be driven by global average analysis as in many of the examples in this report. However, the committee’s analysis framework can be used at any spatial scale required by the quantified objective.
Recommendation: NASA should foster a consistent methodology to evaluate the utility of geophysical variables for achieving quantified Earth science objectives. The committee notes that such a methodology could also be utilized by the Earth science decadal survey in its priority recommendations.
Recommendation: NASA should extend their current mission cost tools to address continuity measurement-related costs needed for the decision framework.
The ability of ESD officials to make informed decisions requires unbiased and consistent information on benefits and affordability that is re-evaluated regularly and presented on a time frame appropriate for NASA planning. The committee advises that inputs to these evaluations be derived from sources such as submitted proposals and face-to-face interactions with measurement advocates.
Recommendation: NASA’s Earth Science Division should establish a regular process for critical evaluation and modification of quantified objectives in Earth science and applications and their associated measurements. The committee suggests creating an analog to the senior review of current satellite operations, which uses senior researchers from a range of communities and results in consistent recommendations to the ESD director.
In summary, the committee offers the following recommendation:
Recommendation: NASA should establish a value-based decision approach that includes clear evaluation methods for the recommended framework characteristics and well-defined summary methods leading to a value assessment.
REFERENCES
NRC (National Research Council). 2007. Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond. The National Academies Press, Washington, D.C.
NRC. 2012. Earth Science and Applications from Space: A Midterm Assessment of NASA’s Implementation of the Decadal Survey. The National Academies Press, Washington, D.C.
