Implementing the decision framework described in Chapter 3 requires methods to evaluate the affordability and each of the key benefit characteristics. These evaluations are then used to derive an overall value rating. The goal of this chapter is to identify analytic evaluation approaches that are enabled by a quantified objective-driven framework.
Among the five value characteristics, importance and utility most reflect value judgments of the broader science community. As mentioned in Section 3.2.2, observational simulation tools are, increasingly, used to inform subjective evaluations of the utility characteristic. Other sources of information include the Earth science and applications from space decadal survey and, possibly, other appropriate community forums, which could assist in the identification of the quantified objectives of highest importance along with corresponding lists of high utility measurements.
For a numerical cost-benefit approach, and by analogy with the NASA Research Announcement (NRA) scheme shown in Table 3.1, the evaluation of the importance factor would correspond with the top levels1 (e.g., 4 and 5) of a 1 to 5 rating system (Table 4.1). For the evaluation of the utility of a particular geophysical variable, the numerical rating is desired in percentage terms relative to achieving maximum benefit (Table 4.2). A simple approach would be to assign ratings between 0.8 and 1 (complete benefit) for the highest utility measurements. One caveat is that pursuit of some quantified objectives may depend, nearly equally, on many geophysical variable records. For those cases, the ratings will be uniformly low across the important variables. For such cases, it might be more appropriate to consider the set of measurements as an integrated observing system and assess value as discussed in Section 3.7 (Utility Example 2).
As an example scenario, the decadal survey might identify and evaluate the six quantified objectives given in Box 3.2 and the associated geophysical variables given in Table 4.3. Under that scenario, some or all of the
1 Here it is assumed that in practice only quantified Earth science objectives with importance rankings of very high or better will be considered for framework analysis.
TABLE 4.1 Subjective Method Ratings for Importance (I)
|Rating for I||Analytical Method Score (from Chapter 3)|
TABLE 4.2 Subjective Method Ratings for Utility (U)
|Rating for U||Analytical Method Score (from Chapter 3)|
objectives might emerge as most important (rating = 5) with a small number of the relevant geophysical variables achieving a similarly high utility rating (e.g., 1).
In the examples below, it is important to keep in mind the clear distinction between the utility and quality characteristics. Utility is the relevance of an optimal measurement to the objective, while quality is the uncertainty of the measurement relative to the objective requirement.
The recent slowdown in the rate of increase in global mean surface air temperature has raised questions in the public/policy arena about the scientific understanding of climate change, with renewed focus on elements of climate forcing over the 1998-2012 period.2 The effective radiative forcing (ERF) from the well mixed greenhouse gases has increased at a rate of about 0.3 W m−2 per decade over the past 3 decades, while all other forcings (except for stratospheric volcanic aerosols) are estimated to have changed by less than 0.05 W m−2 (Myhre, 2013, Appendix II). While the slowdown was relatively large (about 0.04 °C decade−1 vs. 0.11 °C decade−1 for the previous several decades), it might readily fall within climate variability if the ERF were only 0.2 W m−2, and recent work has suggested small corrections to the ERF over the period of as little as 0.05 W m−2 might explain or contribute to the slowdown (Huber and Knutti, 2014).
These considerations lead us to pose the detection of a change in climate forcing (i.e., ERF) to better than 0.05 W m−2 as a quantified objective (see Box 3.2). While radiative forcing observations are critical to understanding decadal change such as the slowdown in global warming, they are not the only information required. Other data, involving other objectives are required for a full understanding of cause and effect in the climate system, including the role of internal natural variability of the coupled ocean/atmosphere system.
To evaluate the utility of geophysical variables in constraining ERF to levels of 0.05 W m−2, the variables are divided into four levels based on their uncertainty over a decade, namely, <0.003 W m−2; 0.003 to 0.01 W m−2; 0.01 to 0.05 W m−2; >0.05 W m−2. Numbers for this analysis have been taken from the 2013 IPCC Working Group I data tables (IPCC, 2013) and the NOAA TSI workshop report (NOAA, 2013). Note that a number of key indirect effects that might change the CO2 or CH4 abundance are not considered here since changes in the well mixed greenhouse gas abundances are being measured from ground-based networks at the necessary level.
2 See Box 9.2 in Flato et al. (2013). More recently, Trenberth (2014) have published papers that explain the slowdown in surface warming in terms of disposition of energy within the ocean and climate system. See Trenberth (2014). Also see, Meehl (2014).
TABLE 4.3 Key Geophysical Variables and Instrument Data Types Associated with the Measurements Needed to Address the Example Quantified Objectives
|Quantified Objective||Relevant Geophysical Variables||Example Instrument Data Types|
|Equilibrium Climate Sensitivity||
|Earth Radiative Forcing Change||
|Sea Level Rise Acceleration||
|Land Carbon Sink||
|Ocean Heat Storage Change||
|Ice Sheet Mass Balance Change||
NOTE: GNSS, Global Navigation Satellite System; GRACE, Gravity Recovery and Climate Experiment; IR, infrared; MODIS, Moderate-Resolution Imaging Spectroradiometer; SAR, synthetic aperture radar; SMAP, Soil Moisture Active-Passive; TIR, thermal infrared; TOA, top of the atmosphere; UV, ultraviolet; VIIRS, Visible Infrared Imaging Radiometer Suite; VIS, visible.
The geophysical variables that need to be evaluated from space for this objective include: total and spectral solar irradiance, surface and cloud properties and albedo, tropospheric and stratospheric volcanic emissions, tropospheric aerosols, cloud properties (including aerosol indirect effects), and trace gases, including tropospheric and stratospheric O3, and stratospheric H2O. Other aspects of overall radiative forcing can be evaluated using ground based measurements. Ancillary measurements include the radiation budget for a constraint on the total forcing.
From among these variables, those having the highest utility rating are cloud-properties/albedo, stratospheric/volcanic aerosols, tropospheric aerosols, and tropospheric ozone. For clouds, a 1 percent change in cloud albedo corresponds to about 0.8 W m−2, and hence a shift in cloudiness (~1%) that occurs during a decade but is not documented represents a major gap in closing the ERF trends. For volcanic stratospheric aerosols, the change in ERF from the 1990s decade (−0.7 W m−2) to the 2000s decade (−0.1 W m−2) is huge (Boucher et al., 2013). Without satellite observations, the ERF from a Pinatubo-like eruption would not have been well measured, and, thus, the gap error might be a substantial fraction of the change in the volcanic ERF in the 1990s, −0.7 W m−2.
For tropospheric O3, changes are space-time variable like aerosols and clouds and cannot be determined from ground-based observations alone; satellites have the potential capability to detect trends in the quantities that determine ERF. The emissions that create tropospheric O3 are shifting dramatically over the decade (but not year by year) as new areas of industry and pollution arise and as old regions (United States and the European Union) dramatically reduce pollution. Thus, tropospheric O3 ERF appears to be changing very slowly recently; but independent, geographical shifts forced by emissions are driving change. The committee estimates the uncertainty over a decade due to current shifts in emission regions to be at least 20 percent of the total, or 0.08 W m−2 decade−1. For tropospheric aerosols, the ERF from anthropogenic sources is estimated to be −0.9 W m−2. Given the decadal shifts in regional emissions, this has similar uncertainty in the case of satellite gaps as tropospheric O3, ~20 percent per decade or ~0.18 W m−2. Uncertainties in other geophysical variables such as total solar irradiance, surface properties/albedo and stratospheric O3 are seen to be changing more slowly and with less uncertainty.
The above analysis suggests that sustained, multidecadal spaceborne measurements of tropospheric O3, stratospheric aerosols, tropospheric aerosols, and cloud properties are of highest utility for addressing the quantified objective for ERF.
Since 1960, the atmospheric CO2 concentration has risen from 315 ppm to 400 ppm and is growing at the rate of 2 ppm/year (Dlugokencky and Tans, 2015). Atmospheric CO2 concentrations would be even higher if it were not for large carbon uptakes or sinks by both terrestrial vegetation and the oceans, which remove about half of CO2 emissions (Figure 4.1). While the carbon cycle is known globally, regional knowledge is lacking. This is especially true for the land carbon sink that not only has high inter-annual variations, but has a large quantity of stored carbon in plants and soils. Whether land vegetation will continue to absorb half of future CO2 emissions is unknown. Improvements in understanding of the land carbon sink are needed to predict future trajectories of atmospheric carbon: data from satellite observations coupled with numerical process models are critical in this effort and the only way this can be achieved.
The challenge in addressing a quantified objective focused on understanding of the land carbon sink is that all of the components of the carbon cycle—the atmosphere, terrestrial vegetation, soils, fresh water, lakes, and rivers, the ocean, and geological sediments are significant reservoirs of carbon. Capturing the movement of carbon, and hence feedbacks, between these reservoirs requires that individual component fluxes be known to comparable levels of uncertainty (see Appendix G, Figure G.1). Consequently, a number of geophysical parameters are needed to achieve this quantified objective: atmospheric CO2 concentrations; land photosynthesis; ocean photosynthesis vegetation biomass, disturbance, and recovery; biomass burning; land respiration and decomposition; and the air-sea CO2 exchange.
Understanding the carbon cycle feedback to the desired uncertainty level requires utilization and integration of a broad range of satellite and in situ observations, because all of these observations must be made at the same time to capture the movement of carbon accurately (see Table 4.4). In this example, only the geophysical variable “atmospheric CO2 concentration” has a higher utility rating relative to the others, because this observation docu-
FIGURE 4.1 The three sinks of the global carbon budget and their uncertainties with time for (A) the atmosphere, (B) the ocean, and (C) the land. The shaded area is ±1σ. Only about half of total carbon emissions accumulate in the atmosphere with the balance divided between the ocean and land sinks. Note the high year-to-year variation in the land carbon sink while the year-to-year variation in the ocean carbon sink is roughly proportional to atmospheric ρCO2. SOURCE: C. Le Quéré, R. Moriarty, R.M. Andrew, G.P. Peters, P. Ciais, P. Friedlingstein, S.D. Jones, et al., Global carbon budget 2014, Earth System Science Data 7:521-610, 2014.
ments the CO2 concentration forcing in the atmosphere upon the climate system. The other carbon cycle geophysical variables are related to the sources and sinks of carbon that in aggregate result in the measured concentrations of atmospheric CO2. Because of the multiple geophysical variables required, many of the individual observations related to the carbon cycle observations have low utility ratings.
The current system of atmospheric CO2 measurements do not adequately constrain process-based carbon cycle models to allow diagnosis and/or attribution of the land and ocean carbon sinks and sources with any confidence—hence the models yield widely varying patterns of land and ocean sources and sinks. In addition, there must be a significant improvement in quantifying the global methane budget. The inability to reproduce an unambiguous picture of the current pattern of large-scale carbon fluxes and to discriminate the dominant mechanisms driving those patterns compromises scientists’ ability to predict the future trajectory of the planetary carbon sinks and sources. In fact, current data sets are so sparse that systematic failures to capture important processes and their thresholds cannot be adequately diagnosed. Testing and improving the surface and ocean parameterizations in Earth system models that calculate the surface-atmosphere fluxes of energy, water, and carbon, is essential for developing the capability to predict future climate, but this has proved to be a challenging task.
To improve model parameterizations and reduce uncertainty in future projections, regional scale flux estimates of CO2, at monthly time-scales and spatial scales of roughly ~1° × 1°, and with global coverage and over multiple annual cycles, are critical. For major urban areas, and for estimation of anthropogenic emissions, the flux determinations need to be at spatial scales on the order of 10 km.
TABLE 4.4 Current Global Flux Uncertainty Levels for the Land Carbon Cycle and the Total Land Above Ground Carbon Pool for Comparison
|Current Fluxes and Uncertainty|
|Carbon Cycle Component||Pg C||Atmospheric ppm CO2
Equivalent per Year
|Atmospheric CO2 concentration||4.3 ± 2.1||2.0 ± 1|
|Land photosynthesis||123 ± 8||58 ± 3.8|
|Land vegetation biomass disturbance and biomass burning||0.9 ± 0.5||0.5 ± 0.4|
|Land carbon sink||2.9 ± 0.4||1.4 ± 0.2|
|Plant respiration||45 ± 9||21 ± 4.2|
|Soila respiration—decomposition||75 ± 15||35 ± 7|
|Land vegetation above ground biomass—not a flux||450 ± 100||212 ± 47|
a Roots, mycorrhizae, etc.
NOTE: The global land carbon sink is 2.9 ±0.8 Pg C yr−1 with a land cover change flux of 0.9 ±0.5 Pg C yr−1 (see Figure G.1). Currently the largest uncertainties are soil respiration/decomposition followed by plant respiration and land photosynthesis. Pg C is petagrams (1015 grams) of carbon; 1 ppm CO2 (1 part per million by volume) is equal to 2.134 Pg C.
SOURCE: Data from Le Quéré et al. (2014), Schlesinger and Bernhardt (2013), Beer et al. (2010), and van der Werf et al. (2010).
These “top-down” flux products could be directly compared with “bottom-up” estimates of the fluxes generated from carbon cycle models forced by local environmental and remotely sensed data to precisely define the attribution of sink and sources, and thereby resolve model ambiguities. Carbon flux estimates with associated uncertainties will provide rigorous metrics for evaluating land and ocean process models, will help us in refining poorly understood model parameterizations, and will improve our predictive capability for the carbon climate system, supporting both basic geophysical understanding and policy-relevant applications.
As indicated in Table 4.4, soil respiration and soil decomposition are the fluxes whose uncertainty are of most consequence for improved understanding of the carbon cycle. To address these will require the linkage of in situ process studies with a variety of satellite data sources to determine how this important land carbon cycle component can be understood and modeled. A coordinated observing system will be required to execute and sustain a time series of global CO2 atmospheric concentration and flux data products at spatial and temporal resolutions that allow rigorous evaluation and improvement of models needed to reduce uncertainty in future predictions/projections.
The quality characteristic is, along with affordability, the one most amenable to analytic analysis. As described in Section 3.6, quality might typically range from a value of 0 to 1, where 0 indicates that the measurement does not assist in achieving the science objective because quality is very low, and a value of 1, which indicates that the measurement fully meets the quantified objective (Table 4.5). The extremes of 0 and 1 are easy to understand; the challenge for this metric is to define logical partial fulfillment of the objective and provide a meaningful scale to judge the relative benefits of partial fulfillment. As noted in Chapter 2, while there are numerous ways to evaluate quality, a useful metric is expected to vary between continuity required for short-term operational use (weather prediction, agricultural crop monitoring, hazard warnings) versus for longer-term science objectives, such as those related to global change, including climate warming.
In the following subsections, the committee examines three different analytical approaches for evaluating quality.
TABLE 4.5 Subjective Method Ratings for Quality (Q)
|Rating for Q||Analytical Method Score (from Chapter 3)|
|0 - 0.2||Low|
|0.2 - 0.4||Moderate|
|0.4 - 0.6||High|
|0.6 - 0.8||Very high|
|0.8 - 1.0||Highest|
A number of factors determine quality. One is whether the combined standard uncertainty of the measurement meets the quantified objectives. The instrument calibration uncertainty, repeatability, time and space sampling, and data systems and delivery for climate variables (algorithms, reprocessing, and availability) will determine the level of confidence that the measurement meets the objective. Assuming Gaussian statistics for measurement uncertainty, one can derive a metric for uncertainty assessment as follows. Let σ be the 1-sigma measurement uncertainty, r be the accuracy requirement of the objective (closeness of agreement between the result of the measurement and the value of the measurand), then
Q = P(r/σ)
Where P(s) is the probability of a value within ± s for a normal distribution N(0,1). For example, if σ is 1/2 of the objective accuracy requirement, then Q = 0.95. If σ equals the accuracy requirement of the objective, then Q = 0.68.
The strength of this approach to a quality metric is its simplicity. The weakness is that it is not as closely coupled to the impact of changes in quality, such as the delay in time to detect climate trends (Section 4.3.2) or the ability to quantify the effect of a data gap.
Another factor in quality is the impact of a data gap on measurement repeatability and hence the tolerance of a gap in a measurement made with a given calibration uncertainty in addressing the quantified objective. The difference in quality of the climate record with and without continuity of the proposed measurement provides input for continuity decision making: If the difference in quality is small, the continuity observation priority will be low, if the quality difference is large, the continuity observation priority will be high.
The quality metric, Q, reflects the combined standard uncertainties of the existing and proposed continuity measurements in the context of a specified science objective. One approach for quantitatively estimating Q for climate records takes into account the length of a measurement record needed to achieve the required objective. This approach recognizes that higher quality observations (i.e., those with smaller calibration uncertainty and higher repeatability) provide scientific answers sooner, and that delayed knowledge can have corresponding societal consequences in delaying decision making. Leroy et al. (2008) establish equations relating observation uncertainty and time to detect climate change. Wielicki et al. (2013; Figure 3 and sidebars) discuss the uncertainty of a climate change observation relative to a perfect observing system limited by natural internal variability, and show the dependence of uncertainty on the length of measurement record.
Leroy et al. (2008) show that the time needed to achieve a measurement with an acceptable level of scientific uncertainty is:
where s is the signal-to-noise ratio such that s = 2 for a 95 percent confidence bound, m is the desired trend to measure in magnitude/year, σ2 is the variance of natural variability, and τvar is the autocorrelation time scale of natural variability. For the common AR-1 red noise statistical distribution, τvar = (1 + ρ)/(1 – ρ) (Weatherhead et al., 1998), where ρ is the lag-1 autocorrelation coefficient of the time series, commonly determined for global change records from annual mean time series to minimize noise from the seasonal cycle.3
“Perfect” means perfect instrument accuracy (i.e., small uncertainty and high repeatability), perfect sampling, and perfect algorithms (see Leroy et al., 2008, equation 11 with f = 0). ΔtP therefore defines the basic time scale of the scientific problem, and this differs for different problems. It defines the point of diminishing returns for advances that reduce measurement uncertainty and also establishes the minimum time investment for observations to achieve a measurement with the necessary fidelity.
In reality, no observing system is perfect. In actual observing systems, global change trends are subject to multiple uncertainties from factors discussed in the definition of continuity in Chapter 2: instrument calibration uncertainty, repeatability, time and space sampling, and data systems and delivery for climate variables (algorithms, reprocessing, and availability). The consequence of uncertainty in a measurement is that the time to detect a specified change is longer. A straightforward extension of Leroy et al. (2008) and Wielicki et al. (2013) defines the time to detect global change as
where the variance σ2 and autocorrelation time scale τ for observation uncertainties are determined for calibration uncertainty (cal), taking into account both absolute calibration uncertainty and repeatability. If there are gaps in the record, the σ2cal term depends on instrument absolute calibration accuracy (uncertainty), but if continuity of the observation is preserved, σ2cal depends on the ability to prove instrument calibration stability (repeatability) taking into account the overlap intercalibration. In both cases, τcal is usually taken as a typical instrument lifetime (Leroy et al, 2008; Wielicki et al., 2013). The σ2cal and τcal terms, indicating the magnitude of the changing bias and the time scale for such changes, respectively, can also be considered measures of the lack of stationarity of the measurement. The σ2cal and τcal terms can also be considered measures of the lack of stationarity of the measurement: both the magnitude of the changing bias σ2cal as well as the time scale for such changes τcal. In the examples here, the worst case scenario is assumed, as done in Leroy et al. (2008), which accounts for either calibration drifts or gaps between measurements. Other valid approaches to this metric can be found in Weatherhead et al. (1998) that take into account the fraction of the data record with gaps as well as a certain or uncertain calibration change across a gap.
Time and space sampling (sam) uncertainties in satellite observations are dominated by orbital sampling and instrument design (e.g., nadir only versus swath scan), and the σ2sam may be determined using orbital sampling simulations, with τsam determined by the time-averaging interval for the global change time series, typically annual mean values. Algorithm uncertainty (alg) in climate trends is primarily the (in)stability of a retrieval algorithm when applied to climate change observations, including issues of changing instrument design (e.g., spectral bandpass) over time. For natural variability (var), exclusive of the anthropogenic trend, the σ2var and τvar terms are determined using past observations as well as estimated using Earth system model simulations (see Leroy et al., 2008; and Wielicki et al., 2013, for examples).
The increase in the time to detect global change thus depends fundamentally on the magnitude of observational uncertainty and repeatability as compared to natural variability. For example, it can be shown (Wielicki et al., 2013) that the time to detect ratio is
3 For climate records, the time series used to determine natural variability will typically be either deseasonalized monthly or annual average series. For monthly time series, the variance will be higher, but autocorrelation time smaller than for annual mean time series. The final result of the time to detect trends tends to be insensitive to this choice (Phojanamongkolkij et al., 2014).
To quantitatively assess how measurement continuity influences the quality of a measurement record, the time to detect global change (at the corresponding time to detect ratios, relative to a perfect observing system) can be specified for the measurement record with continuity, ΔtC, and without continuity, ΔtNC.
Conceptually, the objective for a continuity decision is to evaluate the societal and scientific impact of measurements with and without continuity. A quality metric that distinguishes between these two options should therefore exhibit:
- Higher ratings for measurements that are more certain and, therefore, achieve the quantified objective in a shorter time, thereby providing climate change societal information in a shorter time; and
- Lower ratings for measurements that are less certain and, therefore, delay achievement of the quantified objective thereby delaying societal benefits. The reduced certainty may arise from poor accuracy, data gaps, poor repeatability, large sampling or algorithm uncertainty.
To be effective in the proposed framework, a quality metric should be simply related to the proposed measurement’s delay in the time to detect trends Δt − ΔtP with respect to a perfect observing system while providing the 0 to 1 quality metric desired for the framework in Chapter 3. An observation with 0 years of delay would therefore receive a Q score of 1.0, while a measurement with a long delay (e.g., 30 years or longer) would receive a Q score of 0.0. A delay of 30 years would be considered a very long delay for both science as well as societal uses. This is especially true for climate data records, given the urgency to narrow uncertainty in climate projections for the 21st century. An example of such a simple Q metric is given by:
|Q2 = 1 − (Δt − ΔtP)/30||for Q > 0,||(3)|
|Q2 = 0||for Q < 0 (i.e., delay longer than 30 years)|
Table 4.6 shows the relationship in Equation 3 between Q2 and the time delay of trend detection. The Q2 metric in Equation 3 has the advantage of both simplicity and the ability to adjust the maximum time delay (30 years in the example) to any value deemed appropriate for the decision process. For use in the decision framework for continuity, Q2 would be evaluated for any of the observation options being considered. Appendix B provides examples of estimations of this Q2 metric both with and without continuity for a range of climate data records.
Because the use of Δt – ΔtP in the Q2 metric depends modestly on the value of the trend m required for the quantified objective, there is a link between a climate record objective and the Q2 quality metric.4 Gaps in a climate record will affect the Q2 metric by increasing the time to detect trends. Examples of the impact of gaps on climate trend detection can be found for TSI (NRC, 2013) and for cloud feedback (Loeb et al., 2009). Reduced time to detect trends is closely related to reduced uncertainty in trends (Weatherhead et al., 1998; Leroy et al., 2008; Wielicki et al., 2013). The later metric is useful for understanding the constraints of uncertainty in climate models (see the example in Appendix C).
A less prescriptive approach might evaluate quality by directly assessing the repeatability and length of the extant measurements record for which continuity is being proposed, relative to the objectives, taking into account the required uncertainty of the detection; for example, to within 1s, 2s, or 3s. In this case, higher Q values reflect
4 If the quantified objective is only capable of detection of very large trends, then its importance may decrease relative to an objective with the capability to detect smaller trends that require higher accuracy to detect. The climate sensitivity example in Appendix C provides such a case: if only very large trends in cloud feedback can be observed, then the uncertainty in cloud feedback is not reduced significantly (see Figure C.1 in Appendix C). If much smaller cloud feedback trends can be observed, then the uncertainty in climate feedback can be reduced significantly. This selection through the objective changes the value of m in (1), which in turn determines the quality of the observation required to meet the more challenging but more important objective. Similar logic would apply to the other objective examples in the appendixes for sea level rise, changing ocean heat storage, changing ice sheet mass balance, and global land carbon sinks.
TABLE 4.6 Example of the Dependence of the Quality Metric Q2 for a Proposed Measurement on the Time Delay for Its Measurement of Climate Trends When Compared to the Time to Detect the Trend for a Perfect Observing System
|Time Delay Δt − ΔtP (years)||Q2|
a measurement with sufficient repeatability to achieve an objective with greater than 95 percent confidence (i.e., within 2s uncertainty), and lower Q values correspond to those with less than 68 percent confidence.
Finding: The quality of a measurement may be quantitatively evaluated by combining metrics arising from instrument calibration uncertainty, repeatability, time and space sampling, and data systems and delivery for climate variables (algorithms, reprocessing, and availability). Measurements that suffer the largest reduction in quality from a gap in observations have higher priority for continuity.
The success probability (S) is defined as the probability that the proposed observation will successfully meet the goal of continuing the extant record, as proposed. This should account for the observation’s resilience to gaps, the possibility of leveraging international partnerships, international redundant observations, or other types of leverage. It also captures issues that would affect the ability of the proposed observation to meet the scientific or societal goal, to the extent that they do not appear via the cost of the measurement. Thus, one definition of the probability of success (S) is
Ps = PaccuPsamPalg (1 – Pgap) Pmgt
where Paccu, Psam, and Palg are treated as independent variables5 that define the success likelihood of achieving the instrument’s long-term accuracy (Paccu; through calibration and repeatability), sampling (Psam), and algorithm (Palg) requirements used in the determination of the quality metric for a proposed observation.
Pgap defines the likelihood of an impactful gap occurring given the proposed observation strategy. This gap probability depends on the extent of required overlap between successive instruments, the number of instruments in orbit, their age, their design life, the spacecraft design life, and the launch date of the next continuity observation (Loeb et al., 2009). If the calibration uncertainty in the quality metric is sufficiently low to avoid the need for data overlap to achieve the quality metric, then Pgap is set equal to 0, because short record gaps have little effect
5 Not independent variables in a strictly statistical sense; instead, these are factors in determining the overall success of achieving a quantified objective. The factors are in fact independent physical concepts: accuracy, sampling, algorithm, gap probability, and management risk. While it is correct that there are statistics involved in the individual components, the reason these probabilities are multiplied is not because they are independent, but instead because any one of them can cause failure to achieve the objective.
on global change detection.6 Finally, Pmgt is the probability that the mission will be successfully managed and carried out as planned. Values for Pmgt will vary according to whether the mission is implemented in a Class B, C, or D modes, with increasing probabilities going from Class D to B.7
Recognizing the challenges with quantifying Paccu, Psam, Palg, Pgap, and Pmgt, a more qualitative evaluation scale can be developed to support a benefit analysis. An example of such a scale is shown in Table 4.7; the committee emphasizes that this scale is presented for illustrative purposes and that NASA may wish to develop its own. As with the other characteristics of the framework, the calculation of success probability in the committee’s notional scheme requires subjective inputs; however, these inputs appear transparently.
Application of the above scoring approach to the quantified objective/measurement examples given in the appendixes yields the success probabilities in Table 4.8.
Affordability should be judged as the cost per year of continuing the prescribed measurement for a specified time period with the required quality, relative to the total budget that NASA has allocated for all its satellite measurements.
To achieve the required quality, the measurement must have the uncertainty, repeatability, time and space coverage, and reduction algorithms to meet the scientific requirements. The cost is, therefore, the full funding needed to make the observations and produce the measurement for a finite length of time: it includes space platform accommodation, launch, validation, algorithm implementation, and science team contributions. Should multiple overlapping observations be prescribed to preclude gaps and achieve the needed repeatability of the measurement over the specified time, they are also part of the cost. Included as well are additional factors that reduce risk, such as advancing the instrument technology readiness level (TRL) and auxiliary measurements, should they be needed, to implement the algorithms that produce geophysical quantities. To the extent that factors such as TRL, gap mitigation, validation and algorithm maturity development are included in the measurement cost, they may enhance the associated success probability of the measurement. Thus cost shares a number of cross cutting characteristics with utility, quality, and success probability that must be quantified and implemented in the value metric.
As discussed in Section 3.6 regarding the quantitative cost-benefit type evaluation, the relative weights given to the affordability and importance factors determine the relative weighting between affordability and benefit. If, for instance, the evaluation scale for importance is comprised of five levels (e.g., Table 3.1), then a similar scale for affordability would be needed to achieve equal weighting. Table 4.9 is an example of such a scale.
As described in Section 3.6, the committee has focused on two approaches for evaluation of the framework measurement characteristics, namely, subjective and analytical methods. For the subjective method, an expert-based adjectival rating for each of five measurement characteristics constitutes the input to the summary evaluation. The final value rating (V) reflects an average, or weighted-average, of the five individual characteristic evaluations. For the analytical approach, value would be calculated according to the formula given in Section 3.6 (i.e., V = I × U × Q × S × A), based on numerical ratings of the characteristics as described in the above subsections. Table 4.10 compares the two methods.
6 In Section 4.4, the committee addresses gaps from the perspective of the success probability characteristic. The effect of gaps on the quality metric is discussed in Section 3.4.2 and in Appendixes B and C. Appendix C also provides references on the analysis of gap risk of occurrence for top of the atmosphere radiation budget observations. The effect of gaps on repeatability is reviewed in Chapter 2 as part of the discussion on the measurement of total solar irradiance (TSI) and the recent report on continuity for TSI measurements (NRC, 2013) also considers gap likelihood of occurrence. Finally, an extensive discussion of methods to calculate the probability of a gap for satellite observations can be found in Loeb et al. (2009).
7 See NASA, NASA Procedural Requirements, NPR 8705.4, Risk Classification for NASA Payloads, Washington, D.C., 2004, http://nodis3.gsfc.nasa.gov/npg_img/N_PR_8705_0004_/N_PR_8705_0004_.pdf; and K.W. Ledbetter, “Science Mission Directorate Implementation of Spacecraft Risk Classifications,” July 6, 2006, http://science.nasa.gov/media/medialibrary/2010/03/31/Gen_Jul06_LedbetterPresentation_.pdf.
TABLE 4.7 Subjective Method Ratings for Success Probability (S)
|Rating for S||Analytical Method Score (from Chapter 3)||Scoring Rationale|
|0 - 0.2||Low||Instrument performance (including instrument calibration uncertainty, repeatability, time and space sampling, and data systems and delivery of climate variables [algorithms, reprocessing, and availability]) has either experienced previous on-orbit degradation or is not well established by traceability to a space-proven design or characterization in a laboratory or airborne environment. Accordingly, degradation of instrument performance and impactful record gaps are likely and would substantially impact record quality. Few or no redundant capabilities exist (or are planned) outside of NASA to support maintenance of the record quality.|
|0.2 - 0.4||Moderate||Instrument performance has either experienced previous on-orbit degradation or is not well established by traceability to a space-proven design or characterization in a laboratory or airborne environment. Accordingly, degradation of instrument performance and impactful record gaps are likely and would substantially degrade record quality. Some redundant/complementary capabilities exist (or are planned) outside of NASA that can support maintenance of the record quality.|
|0.4 - 0.6||High||Some elements of instrument performance have been established by traceability to a space-proven design or characterization in a laboratory or airborne environment. Impacts on data quality due to instrument performance degradation are mitigated by instrument performance margin. Only short record gaps are likely with limited capability to impact the data record quality. Some redundant/complementary capabilities exist (or are planned) outside of NASA that can support maintenance of the record quality.|
|0.6 - 0.8||Very high||Most elements of instrument performance have been established by traceability to a space-proven design or characterization in a laboratory or airborne environment. Impacts on data quality due to instrument performance degradation are mitigated by instrument performance margin. Only short record gaps are likely with limited capability to impact the data record quality. Significant redundant/complementary capabilities exist (or are planned) outside of NASA that can support maintenance of the record quality.|
|0.8 - 1.0||Highest||Long-term instrument performance is well established. Expected degradation of instrument performance is well within required quality thresholds. Impactful record gaps are unlikely given that overlapping instrument records are not required and substantial redundant/complementary capabilities exist (or are planned) outside of NASA that can support maintenance of the record quality.|
To illustrate the application of these methods, the committee evaluated benefit for some example measurements related to the quantified objectives listed in Box 3.2 (see Table 4.11). For these examples, the objectives were not differentiated with respect to importance (all measurements given ratings of “highest” and scores of 5). Also, the examples were confined to evaluation of benefit due to the lack of sufficient, readily available, cost information needed to quantitatively establish affordability. Illustrative examples of the benefit evaluations are given in the appendixes.
TABLE 4.8 Example Calculations of Measurement/Instrument Success Probabilities
|Measurement/ Instrument||Quantified Objective||Ps (S)||Scoring Rationale|
|MODIS/VIIRS||Climate Sensitivity (see Appendix C)||1.0||On-orbit experience|
|CERES+MODIS/VIIRS||0.9||Sensitive to CERES data gaps|
|CLARREO+ CERES+MODIS/ VIIRS||0.8||No on-orbit experience for CLARREO|
|Radar altimetry||Sea Level (see Appendix D)||1.0||On-orbit experience|
|Radar altimetry + gravity||Ocean Heat (see Appendix E)||1.0||On-orbit experience|
|Laser altimetry||Ice Sheet Mass (see Appendix F)||0.8||Novel photon-counting technology for the lidar|
|InSAR||0.8||On-orbit SAR experience|
|CO2 concentration||Land Carbon (see Appendix G)||0.95||Initial OCO-2 experience|
|Land photosynthesis||0.9||On-orbit experience MODIS/VIIRS|
|Land biomass and change||0.8||On-orbit experience Landsat. No on-orbit experience for laser altimetry or radar sensing of volume.|
|Biomass burning||0.95||On-orbit experience MODIS/VIIRS and Landsat|
|Respiration/decomposition||0.8||Requires in situ process studies linked to OCO-2, Landsat, MODIS/VIIRS, and SMAP|
NOTE: CERES, Clouds and Earth’s Radiant Energy System; CLARREO, Climate Absolute Radiance and Refractivity Observatory; InSAR, interferometric synthetic aperture radar; MODIS, Moderate-Resolution Imaging Spectroradiometer; OCO-2, Orbiting Carbon Observatory-2; SMAP, Soil Moisture Active-Passive; VIIRS, Visible Infrared Imaging Radiometer Suite.
TABLE 4.9 Subjective Method Ratings for Affordability (A)
|Rating for A||Analytical Method Score
(from Chapter 3)
|Scoring Rationale—Total Cost as a Percentage of Total Available NASA Funds|
|2||Moderate||Between 60 and 80%|
|3||High||Between 40 and 60%|
|4||Very high||Between 20 and 40%|
a Should international partners or leveraging opportunities exist, they would be reflected in a lower cost to NASA.
TABLE 4.10 Comparison of Summary Evaluation Methods
|Evaluation Metric||Subjective Method Ratings||Analytical Method Scores
(from Chapter 3)
|Importance (I)||Low, Moderate, High, Very High, Highest||1 - 5|
|Utility (U)||Low, Moderate, High, Very High, Highest||0 - 1|
|Quality (Q)||Low, Moderate, High, Very High, Highest||0 - 1|
|Success Probability (S)||Low, Moderate, High, Very High, Highest||0 - 1|
|Affordability (A)||Low, Moderate, High, Very High, Highest||1 - 5|
|Overall Value (V)||Poor, Fair, Good, Very Good, Excellent||0 - 25|
TABLE 4.11 Overall Benefit Evaluations for Example Measurements Related to Quantified Objectives Given in Box 3.1
|Quantified Objective||Measurementa||Subjective Benefit Rating||Analytical Benefit Score (B)|
|Equilibrium Climate Sensitivity||MODIS/VIIRS||Poor||0.5|
|Sea Level Rise Acceleration||Jason||Excellent||5.0|
|Land Photosynthesis||MODIS-VIIRS, OCO-2||Good||4.3|
|Land Biomass & Change Biomass Burning||MODIS-VIIRS, Landsat||Good||3.2|
|Respiration & Decomposition||MODIS-VIIRS, Landsat, OCO-2||Excellent||4.5|
|MODIS-VIIRS, SMAP, OCO-2||Fair||1.0|
|Ocean Heat Storage Change||GRACE + Jason||Good||3.4|
|Ice Sheet Mass Balance Change||ICESat + OIB||Fair||2.4|
a Some of the table entries use mission names as a generic representation of a measurement; thus, for example, the entry “GRACE” indicates current or future measurements of Earth’s time- and spatially varying gravity field; “Jason” refers to current and future precision measurements of ocean surface topography; “ICESat” refers to continuation of benchmark elevation measurements that can be related to ice sheet mass balance; and “Landsat” refers to current (Landsat-8) and future implementations of a moderate resolution, multispectral land imager. In some cases, a mission name indicates an anticipated future measurement capability; for example, CLARREO refers to the development of a suite of climate benchmarking measurements described in the 2007 NRC decadal survey (NRC, 2007).
NOTE: CERES, Clouds and Earth’s Radiant Energy System; CLARREO, Climate Absolute Radiance and Refractivity Observatory; GRACE, Gravity Recovery and Climate Experiment; ICESat, Ice, Cloud, and land Elevation Satellite; InSAR, interferometric synthetic aperture radar; MODIS, Moderate-Resolution Imaging Spectroradiometer; NISAR, NASA-ISRO synthetic aperture radar; OCO-2, Orbiting Carbon Observatory-2; OIB, Operation IceBridge; SMAP, Soil Moisture Active-Passive; VIIRS, Visible Infrared Imaging Radiometer Suite.
Beer, C., M. Reichstein, E. Tomelleri, P. Ciais, M. Jung, N. Carvalhais, C. Rödenbeck, et al. 2010. Terrestrial gross carbon dioxide uptake: Global distribution and covariation with climate. Science 329:834-838.
Boucher, O., D. Randall, P. Artaxo, C. Bretherton, G. Feingold, P. Forster, V.-M. Kerminen, et al. Clouds and Aerosols. Chapter 7 in Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (Stocker, T.F., D. Qin, G.-K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley, eds.). Cambridge University Press, Cambridge, U.K., and New York, N.Y.
Ciais, P., C. Sabine, G. Bala, L. Bopp, V. Brovkin, J. Canadell, A. Chhabra, et al. 2013. Carbon and Other Biogeochemical Cycles. Chapter 6 in Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (Stocker, T.F., D. Qin, G.-K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley, eds.). Cambridge University Press, Cambridge, U.K., and New York, N.Y.
Dlugokencky, E., and P. Tans. 2015. “Trends in Atmospheric Carbon Dioxide.” National Oceanic and Atmospheric Administration, Earth System Research Laboratory. http://www.ersl.noss.gov/gmd/ccgg/trends. Last access May 15, 2015.
Flato, G., J. Marotzke, B. Abiodun, P. Braconnot, S.C. Chou, W. Collins, P. Cox, et al. 2013. Evaluation of Climate Models. Chapter 9 in Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (Stocker, T.F., D. Qin, G.-K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley, eds.). Cambridge University Press, Cambridge, U.K., and New York, N.Y. Box 9.2, “Climate Models and the Hiatus in Global Mean Surface Warming of the Past 15 Years.”
Hansen, M.C., S.V. Stehman, and P.V. Potapov. 2010. Quantification of global gross forest cover loss. Proceedings of the National Academy of Sciences 107(19):8650-8655.
Huber, M., and R. Knutti. 2014. Natural variability, radiative forcing and climate response in the recent hiatus reconciled. Nature Geoscience 7:651-656.
IPCC (Intergovernmental Panel on Climate Change). 2013. Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (Stocker, T.F., D. Qin, G.-K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley, eds.). Cambridge University Press, Cambridge, U.K., and New York, N.Y.
Le Quéré, C., R. Moriarty, R.M. Andrew, G.P. Peters, P. Ciais, P. Friedlingstein, S.D. Jones, et al. 2014. Global carbon budget 2014. Earth System Science Data Discussions 6:1-90.
Leroy, S.S., J.G. Anderson, and G. Ohring. 2008. Climate signal detection times and constraints on climate benchmark accuracy requirements. Journal of Climate 21:841-846.
Loeb, N.G., B.A. Wielicki, T. Wong, and P.A. Parker. 2009. Impact of data gaps on satellite broadband radiation records. Journal of Geophysical Research 114:D11109.
Meehl, G.A., H. Teng, and J.M. Arblaster. 2014. Climate model simulations of the observed early-2000s hiatus of global warming. Nature Climate Change 4:898-902.
Myhre, G., D. Shindell, F.-M. Bréon, W. Collins, J. Fuglestvedt, and J. Huang. 2013. Anthropogenic and Natural Radiative Forcing. Chapter 8 in Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (Stocker, T.F., D. Qin, G.-K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley, eds.). Cambridge University Press, Cambridge, U.K., and New York, N.Y.
NOAA (National Oceanic And Atmospheric Administration). 2013. Maintaining the Continuation of Long-term Satellite Total Solar Irradiance Observation. NOAA Working Group Report. NOAA/National Environmental Satellite, Data, And Information Service/National Climate Data Center and NOAA Joint Polar Satellite System Office. February 27, 2013. Reprinted in National Research Council, 2013, Review of NOAA Working Group Report on Maintaining the Continuation of Long-Term Satellite Total Irradiance Observations, The National Academies Press, Washington, D.C., Appendix B.
NRC (National Research Council). 2013. Review of NOAA Working Group Report on Maintaining the Continuation of Long-Term Satellite Total Irradiance Observations. The National Academies Press, Washington, D.C.
Phojanamongkolkij, N., S. Kato, B. A. Wielicki, P. C. Taylor, and M. G. Mlynczak. 2014. A comparison of climate signal trend detection uncertainty analysis methods. Journal of Climate 27:3363-3376.
Schlesinger, W.H., and E.S. Bernhardt. 2013. The Global Carbon Cycle. Chapter 11 in Biogeochemistry: An Analysis of Global Change (W.H. Schlesinger, ed.). Academic Press.
Trenberth, K. E., J. T. Fasullo and M. A. Balmaseda. 2014. Earth’s energy imbalance. Journal of Climate 37:3129-3144.
Trenberth, K.E., J.T. Fasullo, G. Branstator, and A.S. Phillips. 2014. Seasonal aspects of the recent pause in surface warming. Nature Climate Change 4:911-916.
van der Werf, G.R., J.T. Randerson, L. Giglio, G.J. Collatz, M. Mu, P.S. Kasibhatla, D.C. Morton, R.S. DeFries, Y. Jin, and T.T. van Leeuwen. 2010. Global fire emissions and the contribution of deforestation, savanna, forest, agricultural, and peat fires 1997-2009. Atmospheric Chemistry and Physics 10:11707-11735.
Weatherhead, E.C., G.C. Reinsel, G.C. Tiao, X.-L. Meng, D.Choi, W.-K. Cheang, T. Keller, J. DeLuisi, D.J. Wuebbles, J.B. Kerr, A.J. Miller, S.J. Oltmans, and J.E. Frederick. 1998. Factors affecting the detection of trends: Statistical considerations and applications to environmental data. Journal of Geophysical Research 103(D14):17149-17161.
Wielicki, B.A., D.F. Young, M.G. Mlynczak, K.J. Thome, S. Leroy, J. Corliss, J.G. Anderson, et al. 2013. Achieving climate change absolute accuracy in orbit. Bulletin of the American Meteorological Society 94:1519-1539.