There is no doubt that environmental satellite data have grown to be the most important source of information for daily global weather forecasting. In addition, these data are now also used by innumerable professionals and laypersons in pursuits as varied as oceanic, atmospheric, terrestrial, and climate research; environmental monitoring; aviation safety; precollege science education; and rapid-response decision support for homeland security, to name just a few. Compounding the pressure put on NOAA and NASA by expanding user communities to provide high-quality data products around the clock is the precarious state of the underfunded satellite data utilization program, which is struggling to keep up with demand for currently available data and the rapidly increasing sophistication of user requirements. The planned next-generation operational satellite systems, comprising both polar-orbiting and geostationary platforms, are designed to meet the needs of user communities whose complex applications are rapidly evolving.
Although the focus of this report is the use of satellite data for civilian rather than defense or national security purposes, a dual-use approach is expected as military and civilian satellite systems converge. The new systems will continue the record of climate-quality observations, but the increase in raw data will be unprecedented—perhaps an order-of-magnitude increase every 2 to 3 years. Expected to develop as a result of this expanded Earth-observing capability are novel ways of using satellite data that will have an increasing impact on citizens’ daily lives. Thus satellite data providers will have to continuously evolve, revise, and in some cases radically redefine their role as well as plan for increased research, operations, and infra-
structure. The high-level training required by such personnel and the continuing education of users are equally important and also must be planned and provided for.
Meeting the challenges posed by the imminent and unprecedented exponential increase in the volume of satellite-system data requires an end-to-end review of current practice, including characterization of process weaknesses, an assessment of resources and needs, and identification of critical factors that limit the optimal management of data, plus a strategic analysis of the optimal utilization of environmental satellite data.
In this report, the Committee on Environmental Satellite Data Utilization (CESDU) offers findings and recommendations aimed at defining specific approaches to resolving the potential overload faced by the two agencies—NOAA and NASA—responsible for satellite data (see the preface for the committee’s statement of task). The committee has focused on the end-to-end utilization of environmental satellite data by characterizing the links from the sources of raw data to the end requirements of various user groups, although, given its limited scope, the committee could not thoroughly examine every link in the chain. CESDU’s goal is to characterize and provide sensible recommendations in three areas, namely, (1) the value of and need for environmental satellite data, (2) the distribution of environmental satellite data, and (3) data access and utilization. The committee’s findings are based on its members’ knowledge of trends in technology; past lessons learned; users’ stated requirements; and other supporting information. The committee hopes that this report will help NOAA and NASA identify and avoid impediments to optimal utilization of environmental satellite data.
Over the course of meetings held to collect information for this report, the committee heard presentations from several key agencies and organizations reflecting a broad range of professional perspectives. From these it distilled four consistent and recurring themes that significantly shaped its final findings and recommendations:
A growing and diverse spectrum of individuals, companies, and agencies routinely utilize and depend on environmental satellite data and information;
Products that best serve the public, together with effective use of public funds, create an ongoing evolution of requirements for data imposed on and by operational users;
Improvements in available flight and ground technologies are being made that meet these new requirements—as demonstrated by research satellite missions and aircraft flights; and
NOAA is committed to the collection of data with improved quality, reliability, latency, and information content.
The value of environmental satellite data derives from the unique, near-real-time, continuous global coverage from space of Earth’s land and ocean surfaces and its atmosphere—value that increases significantly as we accumulate satellite records that provide a historical perspective. In addition, the committee believes that, in the near future, environmental satellite data will be employed by a much wider spectrum of users—from individuals with real-time weather data displays in their home, car, truck, boat, plane, business, or campsite, to a wide range of companies with value-added products developed from those data, to farmers, mariners, truckers, and aviators dependent on weather, to numerical modeling centers that provide weather, crop, fire, drought, flood, health, climate, and other predictions and alerts. Indeed, evidence presented to the committee strongly suggests that we should look to and prepare for a future in which cable TV, wireless networks, personal digital assistants, direct satellite broadcast, and the Internet enable continuous, uninterrupted access to environmental satellite data, information, and knowledge as an essential element of commerce, recreation, and the conduct of everyday life for the majority of people.
Thus it will not be sufficient merely to collect greater amounts of environmental satellite data, although the expected orders-of-magnitude increase in the volume of collected data will in itself pose special challenges. The committee heard testimony about increasing requirements to recover more of the information content in the data, and also about an anticipated increase in the number and diversity of environmental satellite data users who will demand instantaneous access to the particular data and information they want. To achieve improved utilization of environmental satellite data will therefore require that as much effort and planning be devoted to the ground systems serving this user community as to the flight systems that originally collect the data. To successfully realize the future outlined above, the agencies responsible for archiving and distributing environmental satellite data must develop the essential visions, plans, and systems. The following findings of the committee and the recommendations based on them are offered to help NASA and NOAA in that process.
THE VALUE OF AND NEED FOR ENVIRONMENTAL SATELLITE DATA
Finding: Improved and continuous access to environmental satellite data is of the highest priority for an increasingly broad and diverse range of users. Their needs include real-time imagery for decision making in response to events such as forest fires, floods, and storms; real-time data for assimilation into numerical weather prediction models; recent imagery for assessment of crops and determination of impacts on the environment resulting from diverse human activities such as marine and land transportation; and data coverage spanning many
years that allows assessment of patterns and long-term trends in variables, such as sea-surface temperature, land use, urbanization, and soil moisture. Users of environmental satellite data include individuals; federal government agencies; state and local managers, planners, and governments; commercial producers of added-value products; and Web, print, and TV/radio broadcasters.
Recommendation 1: To best serve the diverse user communities and to meet growing demand, the committee recommends that, as soon as is practical, agencies providing environmental satellite data and products collaborate, with NASA and NOAA taking the lead, to develop an explicit strategy and implementation plan for data distribution systems, user interfaces, and increased user engagement and education. The goals of this plan should be to facilitate access to current, historical, and future environmental satellite data and products in ways that acknowledge the range of skills and evolving needs of the user communities and to support these users by providing appropriate supporting information and educational material.
Finding: The national and individual user requirements for multiyear climate system data sets from operational environmental satellites, as currently delineated in the Climate Change Science Program strategic plan,1 are placing special demands on current and future data archiving and utilization systems. These demands include more stringent requirements for accurate cross-platform radiometric calibration, new combinations of multiple satellite and instrument data, and algorithms for generating advanced biophysical variables. Detecting climate change trends often involves evaluating data at the limits of measurement precision, and so periodic, absolutely consistent reprocessing of climate data records is a fundamental requirement.
Recommendation 2: Creating climate data records (CDRs),2 which quantify subtle but important global change trends, is not a task that can be accom-
U.S. Climate Change Science Program, Strategic Plan for the U.S. Climate Change Science Program: A Report by the Climate Change Science Program and the Subcommittee on Global Change Research, available at http://www.climatescience.gov/Library/stratplan2003/default.htm, accessed July 12, 2004.
A preliminary report by the NRC’s Board on Atmospheric Sciences and Climate (Climate Data Records from Environmental Satellites: Interim Report, National Academies Press, Washington, D.C., 2004, page 1) defines a climate data record as “a time series of measurements of sufficient length, consistency, and continuity to determine climate variability and change.” The report adds, “In addition we further segment satellite based CDRs into Fundamental CDRs (FCDRs) which are calibrated and quality controlled sensor data that have been improved over time, and Thematic CDRs (TCDRs), which are geophysical variables derived from the FCDRs, such as sea surface cloud temperature and cloud fraction.”
plished solely in routine operational environments (such as with the National Polar-orbiting Operational Environmental Satellite System (NPOESS) and Geostationary Operational Environmental Satellite (GOES)). The committee recommends that NASA, along with NOAA, select multidisciplinary, research-oriented, end-to-end science teams that will select those NPOESS, GOES, and other systems’ data products and variables that are scientifically important and technologically feasible for long-term CDR development. These science teams will design and maintain a proactive strategy for the stewardship and multidecadal production of the selected CDRs.
Finding: NOAA has limited experience with land data sets because historically its mission has focused on the oceans and atmosphere. Major advances in land remote sensing have occurred in the last decade, fostered primarily by the Earth Observing System developed by NASA, that are not reflected in NPOESS planning. The committee found that NOAA has so far not effectively utilized current satellite technologies and data sets for vegetation science, management, or applications. For example, of 58 environmental data records (EDRs) defined for NPOESS, only 6 are specifically for land, and of these only 2 are vegetation oriented. For the 2012 flight of GOES-R, only 20 of the approximately 170 environmental observation requirements (EORs) are land-surface related; of these, only 4 are vegetation related.
Recommendation 3: NOAA should convene an intergovernmental committee with NASA, the U.S. Department of Agriculture, the Department of the Interior, the Environmental Protection Agency, and other interested parties to select the variables for land vegetation data for generation from NPOESS, GOES, and other operational systems that will have high utility for both land management and the hydroecological sciences.
THE DISTRIBUTION OF ENVIRONMENTAL SATELLITE DATA
Finding: The constellations of satellites now in space and planned for the future include platforms launched by several nations, and more complete and comprehensive coverage of environmental data fields can be achieved by combining the data from these different national efforts.
Recommendation 4: The U.S. Environmental Satellite Data Program should work to facilitate user access to data from other nations’ satellites as well as its
own and to facilitate synthesis of data across platforms by providing supporting metadata.
Finding: The Comprehensive Large Array-data Stewardship System is being designed by NOAA to catalog, archive, and disseminate all NOAA environmental satellite data produced after 2006. Given the magnitude of this effort—and considering the growing volume, types, and complexity of environmental satellite data; the increasingly large and diverse user base; and expectations for wider and more effective use of the data—the committee emphasizes the importance of NOAA’s (1) having a comprehensive understanding of the full scope of the technical requirements for data cataloging, archiving, and dissemination and (2) ensuring implementation based on that knowledge. Key to successful implementation of a strong system that will serve operational users and the nation well are detailed planning, proactive follow-through, and NOAA’s incorporation of lessons learned from previously developed, similarly scaled initiatives with similar systems requirements.
NOAA should conduct an immediate review of the entire Comprehensive Large Array-data Stewardship System (CLASS) program. This review should aggressively solicit and incorporate recommendations from the designers, builders, operators, and users of similar systems, particularly those systems comprised by the Earth Observing System Data and Information System.
CLASS should be designated and developed as NOAA’s primary data archive system for environmental satellite data and other related data sets. NOAA should ensure that CLASS is designed to adequately serve the full spectrum of potential environmental satellite data users. In addition to end users, CLASS should be designed to disseminate data to the broadest possible community of data brokers and value-added providers. The CLASS architecture should explicitly include the public programmatic (e.g., Web services) interfaces that these third parties require.
NOAA should plan for and identify resources required for an increased CLASS effort to fulfill the needs outlined in a and b above.
Finding: NOAA does not appear to be effectively leveraging the substantial and growing third-party resources available for creating, archiving, and distributing environmental satellite data products. In particular, the current CLASS effort appears to include end-user services (such as Web ordering, e-commerce, and product customization) that could just as easily be provided by
third parties, while ignoring the lower-level programmatic interfaces that value-added providers require.
NOAA should consider both centralized and decentralized approaches to managing the generation and distribution of environmental satellite data products to ensure cost-effective and efficient utilization of existing human and institutional expertise and resources. Centralized handling should be provided for operationally critical core products and should include the acquisition, processing, distribution, archiving, and management of calibrated, navigated radiances and reflectances at the top and bottom (atmospherically corrected) of the atmosphere, as well as for selected key products and metadata. Specialized higher-level environmental data products could be handled (processed, reprocessed, and distributed) in a physically and organizationally distributed (and diverse) manner.
NOAA should take maximum advantage of the exponentially decreasing costs of computing resources and allow for distributed implementations by third parties.
NOAA should consider mutually beneficial partnerships and partnering models with the private sector (e.g., commercial value-added data and product services providers) that have the twin objectives of ensuring user-oriented open access to the data and providing the best value to end users.
Finding: Over the life of a project the cost of ownership of online (disk) storage is competitive with, and decreasing more rapidly than, that of offline (tape or optical) storage. The ability to store and process large volumes of satellite data online will thus become ubiquitous. More than any physical medium, Internet connections to these online data sources will prove a stable, economical, and widely available mechanism for data transfer.
NOAA’s default policy should be to maintain all public satellite data online, in archives that can be accessed (partitioned) to maximize throughput and replicated (mirrored) to ensure survivability.
NOAA should transition to exclusively online access to satellite data. Distribution on physical media should be provided as a custom service by third parties.
NOAA should plan for and identify resources to support handling of the anticipated increase in archival and dissemination requirements beyond 2010.
DATA ACCESS AND UTILIZATION
Finding: Data from diverse satellite platforms and for different environmental variables must often be retrieved from different sources, and these retrievals often yield data sets in different formats with different resolution and gridding. The multiple steps currently required to retrieve and manipulate environmental satellite data sets are an impediment to their use.
Recommendation 8: Data access and distribution should be designed, and associated products tailored, to be compatible with users’ processing, storage, distribution, and communications resources and their information requirements.
NOAA should improve access to its data by allowing users to focus searches by geographic region, dates, or environmental variables, thus helping provide the means to search from one user interface across all environmental satellite data held by U.S. agencies. Tailored subsets of data products should be made available for routine distribution and/or in response to a specific request.
Further, NOAA’s user interfaces should allow stored environmental satellite data sets and/or images to be retrieved in a common data format and with geolocated gridding selected from a list of options by the user. Subsetting and subsampling should be combined to provide a continuum of data products from broad-area, low/moderate-resolution products to regional, high-resolution products.
NOAA should concentrate on ensuring the commonality, ease, and transparency of access to environmental satellite data and providing no-cost data streams in a few standardized, user-friendly formats selected primarily to maximize ease of translation into community-specific formats.
NOAA should support the development of third-party format translation services and the adaptation of existing community-standard tools to NOAA-standard formats.
The data that NOAA provides to users should be accompanied by metadata that documents data quality, discusses possible sources of error, and includes a complete product “pedigree” (algorithm theoretical basis, sensor and calibration, ancillary data, processing path, and validation status and component uncertainties).
Finding: Some major segments of the user community currently do not have the resources to fully utilize all of the environmental satellite data available to
them. The principal obstacles to expanded use have been inadequate and/or discontinuous funding for applied research as a part of data utilization programs, the lack of support for education and outreach programs, and the lack of trained professional brokers and facilitators available to work with the various bidirectional interfaces between users and providers within the environmental satellite data utilization system.
Recommendation 9: A continuous level of adequate resources, especially for applied research and education of the work force in the use of environmental satellite data, is needed to exploit the huge investments already made in the satellite system. Satellite data providers and the scientific research community should also take a leading role in facilitating collaboration with their end-user partners. These efforts should include outreach, training, and technical assistance for the more sophisticated user communities as well as for the rapidly emerging nonscientific, nongovernmental user groups, with the ultimate goal being to enable straightforward and effortless user access to environmental satellite data and data products.
Finding: Early and ongoing cooperation with dialogue among users, developers of satellite remote sensing hardware and software, and U.S. and international research and operational satellite data providers is essential for the rapid and successful utilization of environmental satellite data. Active research and development is required to achieve operational sustainability—today’s research anticipates and underpins the satisfaction of tomorrow’s operational requirements. Many of the greatest environmental satellite data utilization success stories (see, e.g., the case study on the European Centre for Medium-range Weather Forecasts in Appendix D) have a common theme: the treatment of research and operations as a continuum, with a relentless team focus on excellence with the freedom to continuously improve and evolve.
Recommendation 10: To ensure the ongoing development of future operational environmental satellite data products that have high quality and value requires an ongoing evaluation of the U.S. effort to collect and provide environmental satellite data. An integrated, sustainable basis for the stewardship of future operational systems, sensors, and algorithms should be fostered by establishing close cooperation between the research and operational agencies responsible for the utilization of environmental satellite data (including their development, collection, processing and reprocessing, validation, distribution, and exploitation), with research and operations viewed as a continuum and
not as two independent areas of effort. To meet evolving customer requirements, this cooperation between research and operational agencies should be coordinated in close partnership with the user community. Only a fully funded, end-to-end system, from satellite/sensor design to data assimilation/utilization, can fully optimize the investments that have been made.