CONTINUITY OF NASA EARTH
OBSERVATIONS FROM SPACE
A VALUE FRAMEWORK
Committee on a Framework for Analyzing the Needs for Continuity of
NASA-Sustained Remote Sensing Observations of the Earth from Space
Space Studies Board
Division on Engineering and Physical Sciences
THE NATIONAL ACADEMIES PRESS
Washington, DC
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This report is based on work supported by Contract NNH11CD57B between the National Academy of Sciences and the National Aeronautics and Space Administration. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of the agency that provided support for the project.
International Standard Book Number-13: 978-0-309-37743-0
International Standard Book Number-10: 0-309-37743-9
DOI: 10.17226/21789
Copies of this report are available free of charge from
Space Studies Board
National Academies of Sciences, Engineering, and Medicine
Keck Center of the National Academies
500 Fifth Street, NW
Washington, DC 20001
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2015 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2015. Continuity of NASA Earth Observations from Space: A Value Framework. Washington, DC: The National Academies Press.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Ralph J. Cicerone is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.
OTHER RECENT REPORTS OF THE SPACE STUDIES BOARD
Sharing the Adventure with the Student: Exploring the Intersections of NASA Space Science and Education: A Workshop Summary (Space Studies Board [SSB], 2015)
The Space Science Decadal Surveys: Lessons Learned and Best Practices (SSB, 2015)
Evaluation of the Implementation of WFIRST/AFTA in the Context of New Worlds, New Horizons in Astronomy and Astrophysics (Board on Physics and Astronomy [BPA] with the SSB, 2014)
Opportunities for High-Power, High-Frequency Transmitters to Advance Ionospheric/Thermospheric Research: Report of a Workshop (SSB, 2014)
Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration (Aeronautics and Space Engineering Board [ASEB] with SSB, 2014)
Landsat and Beyond: Sustaining and Enhancing the Nation’s Land Imaging Program (SSB, 2013)
Lessons Learned in Decadal Planning in Space Science: Summary of a Workshop (SSB, 2013)
Review of the Draft 2014 Science Mission Directorate Science Plan (SSB, 2013)
Solar and Space Physics: A Science for a Technological Society (SSB, 2013)
Assessment of a Plan for U.S. Participation in Euclid (BPA with SSB, 2012)
Assessment of Planetary Protection Requirements for Spacecraft Missions to Icy Solar System Bodies (SSB, 2012)
Earth Science and Applications from Space: A Midterm Assessment of NASA’s Implementation of the Decadal Survey (SSB, 2012)
The Effects of Solar Variability on Earth’s Climate: A Workshop Report (SSB, 2012)
NASA’s Strategic Direction and the Need for a National Consensus (Division on Engineering and Physical Sciences, 2012)
Report of the Panel on Implementing Recommendations from the New Worlds, New Horizons Decadal Survey (BPA and SSB, 2012)
Technical Evaluation of the NASA Model for Cancer Risk to Astronauts Due to Space Radiation (SSB, 2012)
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions (SSB, 2011)
An Assessment of the Science Proposed for the Deep Underground Science and Engineering Laboratory (DUSEL) (BPA, 2011)
Panel Reports New Worlds, New Horizons in Astronomy and Astrophysics (BPA and SSB, 2011)
Recapturing a Future for Space Exploration: Life and Physical Sciences Research for a New Era (SSB and ASEB, 2011)
Sharing the Adventure with the Public The Value and Excitement of “Grand Questions” of Space Science and Exploration: Summary of a Workshop (SSB, 2011)
Vision and Voyages for Planetary Science in the Decade 2013-2022 (SSB, 2011)
Capabilities for the Future: An Assessment of NASA Laboratories for Basic Research (Laboratory Assessments Board with SSB and ASEB, 2010)
Controlling Cost Growth of NASA Earth and Space Science Missions (SSB, 2010)
Defending Planet Earth: Near-Earth-Object Surveys and Hazard Mitigation Strategies: Final Report (SSB with ASEB, 2010)
An Enabling Foundation for NASA’s Space and Earth Science Missions (SSB, 2010)
Forging the Future of Space Science: The Next 50 Years (SSB, 2010)
Life and Physical Sciences Research for a New Era of Space Exploration: An Interim Report (SSB with ASEB, 2010)
New Worlds, New Horizons in Astronomy and Astrophysics (BPA and SSB, 2010)
Revitalizing NASA’s Suborbital Program: Advancing Science, Driving Innovation, and Developing a Workforce (SSB, 2010)
Limited copies of SSB reports are available free of charge from:
Space Studies Board
Keck Center of the National Academies of Sciences, Engineering, and Medicine
500 Fifth Street, NW, Washington, DC 20001
(202) 334-3477/ssb@nas.edu
www.nationalacademies.org/ssb/ssb.html
COMMITTEE ON A FRAMEWORK FOR ANALYZING THE NEEDS FOR CONTINUITY OF
NASA-SUSTAINED REMOTE SENSING OBSERVATIONS OF THE EARTH FROM SPACE
BYRON D. TAPLEY, University of Texas at Austin, Chair
MICHAEL D. KING, University of Colorado, Boulder, Vice Chair
MARK R. ABBOTT, Oregon State University
STEVEN A. ACKERMAN, University of Wisconsin, Madison
JOHN J. BATES, NOAA/NESDIS National Climate Data Center
RAFAEL L. BRAS, Georgia Institute of Technology
ROBERT E. DICKINSON, University of Texas at Austin
RANDALL R. FRIEDL, Jet Propulsion Laboratory
LEE-LUENG FU, Jet Propulsion Laboratory
CHELLE L. GENTEMANN, Remote Sensing Systems
KATHRYN A. KELLY, University of Washington
JUDITH L. LEAN, Naval Research Laboratory
JOYCE E. PENNER, University of Michigan
MICHAEL J. PRATHER, University of California, Irvine
ERIC J. RIGNOT, University of California, Irvine
WILLIAM L. SMITH, Hampton University
COMPTON J. TUCKER, NASA Goddard Space Flight Center
BRUCE A. WIELICKI, NASA Langley Research Center
Staff
ARTHUR A. CHARO, Senior Program Officer, Study Director
LEWIS B. GROSWALD, Associate Program Officer1
KATIE DAUD, Research Associate2
ANESIA WILKS, Senior Project Assistant
ANGELA DAPREMONT, 2014 Fall Lloyd V. Berkner Space Policy Intern3
MICHELLE THOMPSON, 2014 Fall Lloyd V. Berkner Space Policy Intern4
MICHAEL H. MOLONEY, Director, Aeronautics and Space Engineering Board and Space Studies Board
______________
1 Through June 20, 2014.
2 From September 22, 2014.
3 From September 29, 2014 to March 27, 2015.
4 From October 6, 2014 to December 12, 2014.
SPACE STUDIES BOARD
DAVID N. SPERGEL, Princeton University, Chair
ROBERT D. BRAUN, Georgia Institute of Technology Vice Chair
MARK R. ABBOTT, Oregon State University
JAMES G. ANDERSON, Harvard University
JAMES P. BAGIAN, University of Michigan
JEFF M. BINGHAM, Consultant
PENELOPE J. BOSTON, New Mexico Institute of Mining and Technology
JOSEPH FULLER, JR., Futron Corporation
THOMAS R. GAVIN, Jet Propulsion Laboratory
NEIL GEHRELS, NASA Goddard Space Flight Center
SARAH GIBSON, National Center for Atmospheric Research
RODERICK HEELIS, University of Texas
WESLEY HUNTRESS, Carnegie Institution of Washington
ANTHONY C. JANETOS, Boston University
SAUL PERLMUTTER, Lawrence Berkeley National Laboratory
LOUISE M. PROCKTER, Johns Hopkins University, Applied Physics Laboratory
MARCIA J. RIEKE, University of Arizona
MARK THIEMENS, University of California, San Diego
MEENAKSHI WADHWA, Arizona State University
CLIFFORD M. WILL, University of Florida
THOMAS H. ZURBUCHEN, University of Michigan
MICHAEL H. MOLONEY, Director
CARMELA J. CHAMBERLAIN, Administrative Coordinator
TANJA PILZAK, Manager, Program Operations
CELESTE A. NAYLOR, Information Management Associate
MEG A. KNEMEYER, Financial Officer
SANDRA WILSON, Financial Assistant
Preface
In a highly constrained budgetary environment, NASA, like all federal agencies, is faced with difficult choices among competing priorities for investment. Within NASA’s Earth Science Division (ESD), part of the Science Mission Directorate, these choices include whether to invest in the continuation of a particular existing data stream versus another (including, but not limited to, climate-related measurements), or to develop a new measurement capability sought by research and applications communities. None of these choices is straightforward; for example, prioritizing among competing “continuity” measurements requires a uniform valuation method and a rigorous understanding of how that value evolves over time, including the implications of a data gap.
In 2013, at the request of ESD, an ad hoc committee of the National Research Council (NRC)1 was formed with the task of providing a framework to assist in the determination of when a measurement(s) or data set(s) initiated by ESD should be collected for extended periods. In particular, and considering the expected constrained budgets for the NASA Earth science program, the committee was asked to:
- Provide working definitions of, and describe the roles for “continuity” for the measurements and data sets ESD initiates and uses to accomplish Earth system science objectives; and
- Establish methodologies and/or metrics that NASA can use to inform strategic programmatic decisions regarding the scope and design of its observation and processing systems.
In carrying out its task, the committee focused on developing a decision framework that allows prioritization of measurements based on their scientific value. In addition, the committee identified, defined, and evaluated a small set of key measurement characteristics to illustrate the framework concept. In its report, the committee presents two notional evaluation frameworks that may be broadly categorized as qualitative and quantitative. The qualitative framework has an analog in the proposal review process that NASA currently employs while the quantitative framework—a decision approach that is the subject of this report—was developed to provide more rigor to an inherently subjective decision-making process. Though the committee’s quantitative framework also requires inputs that are subjective, they enter the framework in a transparent manner and the sensitivity of the calculated “value” to variations in the inputs is easily seen.
______________
1 Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council are used in an historic context identifying programs prior to July 1.
The committee recognizes an important qualification regarding its treatment of task item 2, above: As explained in the report, the proposed quantitative decision framework can be adapted to include choices between the continuation of an existing measurement and the initiation of a new measurement, or choices among measurements focused on societal-benefit applications. However, the framework it presents is by design directed toward choices among extended missions undertaken for research purposes aimed at quantifying global change. The committee endeavored to provide a more general response to task item 2; however, it found that development of even the simpler “apples-to-apples” decision framework for the measurements highlighted above in italics to be extremely challenging. Finally, the committee acknowledges the limitations of its approach. While the proposed methodology can inform measurement choices based on their value to achieving a quantified science objective, it does not capture non-quantifiable objectives such as increasing the knowledge and experience base to facilitate the development of a new remote sensing capability.
The report from the ad hoc committee is presented here; it is organized as follows:
Chapter 1—Introduction—provides background relevant to the committee’s task;
Chapter 2—Measurement Continuity—includes the committee’s working definition of measurement continuity; a discussion of the four criteria—instrument calibration uncertainty, repeatability, time and space sampling, and data systems and delivery for climate variables (algorithms, reprocessing, and availability)—that are used in a framework to determine whether a data set has the requisite quality for long-term Earth observations and global change research; and the introduction of the “quantified objective” that is central to the committee’s methodology;
Chapter 3—A Decision Framework for NASA Earth Science Continuity Measurements—presents a quantitative framework that can be applied to “value” competing choices for measurement continuity;
Chapter 4—Applying the Framework to Continuity Measurements—provides an overview of the application of the framework; and
Appendixes—Appendixes B-G provide comprehensive illustrations of the framework applied to several representative quantified Earth science objectives. Also in the appendixes are the full task statement (Appendix A), biographical information for committee members (Appendix H), and a list of acronyms (Appendix I).
A note on terminology: When characterizing a measurement, the committee uses terms such as uncertainty, repeatability, accuracy, and precision in a manner consistent with the definitions provided in reference guides published by the National Institute of Standards and Technology (NIST). For example, NIST defines “uncertainty” as a “parameter associated with the result of a measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand [which is the property that is the object of measurement].” Similarly, NIST defines repeatability (of results of measurements) as the “closeness of the agreement between the results of successive measurements of the same measurand carried out under the same conditions of measurement.”2 In this report, the “combined standard uncertainty” is obtained by combining the individual uncertainties, including those evaluated by statistical methods (what the committee terms Type A) and those evaluated by other means (Type B). The committee uses “stability” in the context of the normal dictionary definition—“the quality or state of something that is not easily changed”—whereas repeatability applies to all components that translate a measurement (or measurements) to a geophysical quantity (or qualities) that pertain to a specified quantified science objective. Most often, it refers to the instrument calibration, which carries through all processing levels.
______________
2 See “Measurement Uncertainty,” a publication of the NIST Information Technology Laboratory available online at http://www.nist.gov/itl/sed/gsg/uncertainty.cfm. Also see Appendix D, “Terminology,” in B.N. Taylor and C.E. Kuyatt, Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, NIST Technical Note 1297, 1994 Edition, http://www.nist.gov/pml/pubs/tn1297/index.cfm. Another useful reference is G. Ohring, B. Wielicki, R. Spencer, B. Emery, and R. Datla, eds., Satellite Instrument Calibration for Measuring Global Climate Change, NIST Rep. NISTIR 7047, 2004, http://tinyurl.com/p92bkul.
Acknowledgment of Reviewers
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
George H. Born, University of Colorado, Boulder,
Amy J. Braverman, Jet Propulsion Laboratory,
Inez Y. Fung, University of California, Berkeley,
Kristina B. Katsaros, Northwest Research Associates, Inc. (emeritus),
Ralph F. Milliff, University of Colorado, Boulder,
Steven E. Platnick, NASA Goddard Space Flight Center,
Kevin E. Trenberth, National Center for Atmospheric Research,
Eric F. Wood, Princeton University, and
Howard A. Zebker, Stanford University.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by James O. Berger, Duke University, and Charles F. Kennel, University of California, San Diego, who were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
This page intentionally left blank.
Contents
1.1 The Role of Sustained Observations in NASA and NOAA Research Programs
2.2 Continuity: A Working Definition
2.2.1 Instrument Calibration Uncertainty
2.2.4 Data Systems and Delivery for Climate Variables (Algorithms, Reprocessing, and Availability)
3 A DECISION FRAMEWORK FOR NASA EARTH SCIENCE CONTINUITY MEASUREMENTS
3.1.1 Quantified Earth Science Objectives
3.2 Framework Characteristic: Benefit
3.2.4 Benefit: Success Probability
3.6 Determining Continuity Measurement Value
3.7 Extending the Framework Beyond Single Continuity Measurement/Quantified Objective Pairs
4 APPLYING THE FRAMEWORK TO CONTINUITY MEASUREMENTS
4.2 Evaluating Importance and Utility
4.2.1 Utility Example 1: Earth Radiative Forcing Change
4.2.2 Utility Example 2: Land Carbon Sink
4.4 Evaluating Success Probability
4.6 Summary Evaluation of the Framework Characteristics
B Quality Metric Examples Using Current Climate Data Records
C Full Framework Example: Narrowing Uncertainty in Climate Sensitivity
D Full Framework Example: Determining Sea Level Rise and Its Acceleration
E Full Framework Example: Determining the Change in Ocean Heat Storage
F Full Framework Example: Determining Ice Sheet Mass Balance
G Full Framework Example: Global Land Carbon Sinks