Elements in a Dynamic System for Data Utilization
The Committee on Environmental Satellite Data Utilization (CESDU) was formed at the request of NOAA and NASA to provide special input on and a vision for the use of environmental satellite data in 2010 and beyond. Environmental satellite data has been acquired for more than 40 years. Today, the volume of data is increasing dramatically, as is its use.
To successfully achieve comprehensive environmental satellite data utilization in the 2010-2020 era will require, in addition to data continuity, three foundational elements:
Advanced environmental geostationary and polar satellite, airborne, NEXRAD, and in situ sensor systems for excellence in data collection;
Integrated, seamless ground systems whose excellence supports enhanced and tailored data utilization, exploitation, and discovery; and
Knowledgeable utilization brokers who work among the groups involved—essentially, trained practitioners who develop effective linkages, connecting people with people throughout the end-to-end process of satellite data utilization (Figure 1.1).
THE DATA TSUNAMI
The increasing data rates and volumes of global observations from satellites that have presented major challenges over the past four decades will continue to drive planning for environmental data utilization during the coming decades. The number
of environmental satellites (see Box 1.1) will increase—up to 100 are to be launched between 2004 and 2014—as will their sensing capabilities. New user demands for cross-sensor and cross-satellite data products will bring additional new challenges that will have to be addressed not only at the system level, but also often by new national and international partnerships. The development of a large new segment of public and private sector service providers brings special challenges and opportunities. Advances in computing and storage technologies have enabled activities that were only dreamed of a few years ago. Now, NOAA must plan to deal with the ever-increasing wealth of environmental satellite data, as well as the growing number and sophistication of end users, while maintaining current operations. The challenges faced by those planning for environmental satellite data collection and management include not only handling this volume but also achieving the full potential of these data by educating more and increasingly diverse users and by providing for data archiving and retrieval that facilitate user access.
In 1957 the first satellite, Sputnik, was launched. Environmental data acquisition from space began with the launch on April 1, 1960, of the first TIROS (Television Infrared Observations Satellite), a polar-orbiting platform that returned infrared images of clouds. By July 1965, 10 TIROS satellites had been launched. These were followed by 9 ESSA (Environmental Satellite Services Administration) satellites launched between 1966 and 1969, and then 5 ITOS (Improved TIROS) satellites, called NOAA-1 through NOAA-5, that were launched into polar orbits between 1970 and 1976. NASA and NOAA collaborated on the NIMBUS 1 through 7 satellite series (launched from 1964 to 1978). This program provided a test bed for many weather and climate observations. The TIROS-N operational satellites included NOAA-6 and NOAA-7, which carried the AVHRR (Advanced Very High Resolution Radiometer) and TOVS (TIROS Operational Vertical Sounder) and were launched between 1978 and 1981; these platforms provided a significant step forward by returning atmospheric profiles and improved surface temperatures. The current series of polar-orbiting satellites, the ATN (Advanced TIROS-N), began to fly in 1983 with the launch of NOAA-8.
Polar-orbiting satellites also have provided the ability to obtain high-resolution imagery of the land surface; NASA launched the ERTS (Earth Resources Technology Satellite) in 1972 using the pioneering, Earth-oriented NIMBUS-type spacecraft. Its instrumentation provided imagery with a spatial resolution on the ground of 10 meters. The Landsat program employed this technology; with improvements it reached a resolution of several meters. The French developed a similar capability in their SPOT satellite and the Japanese in MOS-1 and JERS-1.
Geostationary satellites were launched beginning in 1966, initially to test the feasibility of maintaining a stationary orbit. The ATS (Applications Technology Satellite) series launched by NASA up through 1974 returned color images of Earth from space and allowed regular cloud observations. Collection of meteorological data to support improved weather forecasts was a goal of the Synchronous Meteorological Satellites (SMS-1 and SMS-2) launched by NASA in 1974 and 1975. NOAA followed with the operational geostationary satellites of the Geostationary Operational Environmental Satellite (GOES) series beginning in 1975. In 1994 new three-axis stabilized satellites came into service in the GOES series that provided full-time coverage of Earth’s surface, allowing visual and infrared images and soundings that looked at severe storms, clouds, winds, ocean currents, fog, snow cover, and other environmental variables as well as providing data relay services. Because of the high value of land and ocean coverage for weather forecasting, the United States operates two GOES satellites, one covering the East Coast and one the West Coast, with an overlap between.
Beginning in the 1980s NOAA began dialogues with other nations for coordination of satellite resources to work toward the best possible global coverage. Partnerships are now in place with EUMETSAT (European Organization for the Exploitation of Meteorological Satellites) and other nations. The international coordination is part of the World Meteorological Organization’s World Weather Watch (WWW) program, and anticipates the International Earth Observation System.
In the mid-1990s the United States embarked on planning for the integration of civilian and defense meteorological satellite systems. This effort by NOAA, NASA, and DOD has led to the development of the National Polar-orbiting Operational Environmental Satellite System (NPOESS). The launch of the first NPOESS satellite is expected in 2009, with a bridge mission—the NPOESS Preparatory Project (NPP) scheduled to fly in late 2006—that will provide spectroradiometric continuity between NPOESS and the research sensors flying on NASA’s Earth Observing System (EOS). To date, EOS has demonstrated the merits of
NPOESS’s multispectral (36-channel) moderate-resolution imagery and hyperspectral temperature and moisture sounding capabilities.
At present, a diverse set of environmental observations are being made, with routine satellite coverage of the land and ocean surfaces, as well as the atmosphere. Included among the phenomena observed are precipitation, flood extent, storm events, dust clouds, fires and smoke, topography, ecosystems, volcanic ash, winds, sea-surface currents, sea-surface temperatures, sea ice, ocean color, atmospheric water vapor and temperature profiles, and so on. The diversity of environmental satellite data is exemplified in two images of Hurricane Isabel: Plate 1 shows Isabel on September 13, 2003, when it was a Category 5 storm threatening the Caribbean and the southern United States; Plate 2 shows Isabel as it approached landfall on the outer banks of North Carolina on September 18. The image is a “true-color” (red-green-blue) NASA image taken by the Moderate-resolution Imaging Spectroradiometer (MODIS)—the research equivalent of the Visible and Infrared Imager/Radiometer Suite (VIIRS) to be flown on NPP and operationally by NPOESS in three orbit planes (with global coverage every 4 hours) beginning in 2009.
Plate 3 anticipates a future capability to be delivered with the Hyperspectral Environmental Suite (HES) on GOES-R in 2012: four-dimensional water vapor structure and wind profiling, in this case water-vapor tracer winds (the tracking of moisture features on constant-altitude surfaces determined by retrieval analyses) for Hurricane Bonnie (August 26, 1998). This simulation demonstrates the power of hyperspectral atmospheric profiling of temperature, moisture, and winds. Compared with the current operational system, the new capability will provide greatly improved spatial resolution, more rapid temporal refresh, and finer vertical resolution through increased spectral information content.
This future, populated by advanced flight instruments yielding higher-quality measurements, will bring further downstream ground system expansion, with the estimated increase in data volume over the next 10 years equivalent to the increase seen over the last 20 to 30 years. The universe of potential operational environmental data beyond 2010 will include data collected by a large portion of the estimated 30 to 40 such satellites being operated each year for research, operations, and technology demonstrations by U.S. agencies as well as by those of other nations. A number of satellite systems and constellations will operate under international partnerships.
As shown in Plate 4, during the 1990s the NOAA archives grew from a little over 100 terabytes to more than 760 terabytes. This growth is not a one-time perturbation. Instead it represents a significant trend that is expected to continue for the next few decades. The second chart in Plate 4 shows the projected growth in archive requirements over the 15-year period from 2000 to 2015. By 2015 the archive requirements are projected to be approximately 15,000 terabytes, an increase of about 20 times the 2000 volume.
Plate 5 shows the archive growth from 2000 to 2015 based on major NOAA systems. Much of the growth is due to new systems such as NPOESS and GOES-R becoming operational. These new systems offer NOAA the opportunity to signifi-
cantly enhance its operational capabilities and provide important data to support NOAA’s mission. However, the resulting data tsunami requires NOAA to plan accordingly for the archiving and distribution of the data.
BIDIRECTIONAL INTERFACES IN AN END-TO-END SYSTEM TO MEET GROWING USER NEEDS
During the last two decades, the greatly increasing volume of environmental satellite data has been accompanied by rapid growth in user demands for environmental information. In response, an array of new environmental data service providers has developed in all the major industrial nations. This environmental satellite data enterprise extends from the instruments, spacecraft, and operating systems of the satellite data providers through the brokers and their data servers that place information into the hands of end users. The system is characterized by a sequence of bidirectional interfaces between the functional units in the end-to-end process of satellite data utilization, as depicted in Figure 1.1, that must be addressed. The entire sequence of events is dynamic and changes at a relatively high frequency (e.g., annually) as a result of healthy push-pull, supply-demand activities.
“Added-value” products are in demand by industry, transportation, agriculture, the military, the science establishment, and many other constituencies. Amid this new world of users, operational environmental satellite data systems of the United States and other countries face not only requirements for meeting a large fraction of the data demand, but also the prospect of being able to draw on some exciting new technologies. A look ahead to 2010 and beyond is required to plan a best fit between needs for environmental satellite data and the opportunities to address those needs. Challenges extend well beyond the exploitation of new technologies to the development of a new system designed and staffed by the human principals of an emerging profession.
Contemplating the multiplicity of user needs for environmental satellite data and information, a number of recent studies by the National Research Council (NRC) (see Appendix C) and other groups have provided detailed information about some well-known segments (e.g., operational weather forecasting, climate science, water management, and others). To learn about user needs from other segments—including land use and water resources management, the agricultural industry, the media, the recreational industry, and others—CESDU held extensive fact-finding sessions. It not only noted numerous new and emerging user requirements but also discerned “chains” or “stages” of users, each drawing on services and products from farther up the chain. The implications range from the need for an active network of user information services to the education and training of career professionals who will facilitate activity and use at the multiple interfaces. Without a trained and
informed system of brokers and data servers at these bidirectional interfaces, full and optimal utilization of environmental satellite data cannot occur.
While there is a focus today on the growth of environmental satellite data in the next 10 years, the longer term also requires special consideration at this time. From many lessons learned it is apparent that future user satisfaction will depend heavily on a sustaining, multidisciplinary, environmental-satellite research program. This research should continue to be led by NASA with contributions from universities, industry, and NOAA. The research must involve both the technology of instrumentation—including calibration—and the technology of data processing. It should include new technology demonstrations as well as NOAA-NASA Pathfinder program experiments with algorithms and analyses.
Furthermore the end-to-end system for utilization of environmental satellite data will have to evolve to satisfy users’ needs in 2010 and beyond. That evolution must be guided by strong and ongoing research and development activities aimed at new and improved capabilities. This effort will require close synergy between research and operational satellite data system groups.
Fortunately some long-term continuity planning by some segments of the user community is available now. For example, the Climate Change Science Program (see National Research Council, Implementing Climate and Global Change Research: A Review of the Final U.S. Climate Change Science Program Strategic Plan, The National Academies Press, Washington, D.C., 2003) is an example of a well-planned federal strategy for which long-term (decades or longer) environmental satellite data are essential.
CHALLENGES POSED BY TECHNOLOGY ENABLERS AND TRENDS
Rapid technology development is occurring today in areas basic to operational environmental satellites—remote sensing, aerospace, communications, and information flow—and will continue at an accelerated rate. In addition to drawing on committee members’ expertise in some of these areas, CESDU obtained special briefings by outside experts on both current and future technologies, including developments in computation and information technology.
Lessons from past planning by NOAA, NASA, and associated academic and private sector partners in environmental satellites have shown a repeated pattern of underestimating the effects of key technology developments. A portion of the growing technology gap—between what is being done by the federally funded environmental satellite data systems and what could be done—may be driven by the federal procurement process and the long cycle time between program changes. Because of the lessons from the past, CESDU in its present study places special emphasis on new technology implementation challenges.
The general trend of semiconductor-based computing capacity doubling every 18 months (known as Moore’s law) will continue for at least the next 10 to 15 years. On a simplistic basis of instruction cycles per data bit, this increase is expected to keep pace with the increased volume and complexity of Earth satellite data streams. The primary challenge will be harnessing the available computing power to address these processing tasks.
For at least a decade, magnetic areal density (the capacity of disk drives to store a given number of bits on a given unit of disk surface area) has been increasing faster than would be predicted by Moore’s law. This pace shows no signs of slowing. Since the volume of satellite observational data, while increasing, is not expected to double every 15 months, it will inevitably become cheaper over the next 10 to 15 years to maintain the cumulative satellite data record online. The committee found that no good alternative to online storage exists that satisfies all the identified needs.
It is difficult to overstate the importance of this simple fact: For the first time, it is technologically feasible to have near-instantaneous (within milliseconds, as opposed to within minutes) access to any satellite data. Such access will enable the assimilative, retrospective, and cross-sensor analyses crucial to developing reliable climatologies and Earth system models.
Inexpensive storage is likely to result in increases in standing orders for and bulk transfers of environmental satellite data and to an increasing number of sites able to function as long-term archives, i.e., having both the physical capacity and the stewardship (capability for security, maintenance, technology migration) needed to reliably maintain records of mission-critical importance.
It is safe to assume that the next 10 to 15 years will see an increasingly networked world, one in which users of satellite data will have ready access to connections at a rate of gigabits per second, and continuous access to megabits-per-second connections (wired and wireless). Distribution of satellite data products on physical media will be relegated to niche applications (e.g., where uninterruptability or physical security are paramount considerations), because of the high cost of transcription and the relatively short market life of media technologies (e.g., several generations of optical disks have outlived any device that can read them).
The explosive growth of commodity computing technology, and especially its penetration into personal electronics such as mobile telephones and personal digital assistants, will transform how people acquire and use satellite data. When there is substantial processing power under the user’s control, it will no longer be incumbent upon the data provider to reduce the information to its least common denominator (e.g., a color-coded picture). Instead, end-user processing will rely on the availability of standardized data streams that can be manipulated by commodity software.
Ensuring Ready Access to High-Quality, Stable Data
NOAA and NASA groups already use some measures to gauge access to environmental satellite data by users, including (1) volume of data transfer, (2) latency from time of observation, (3) response time to meet a new user request, and so on. Such measures are a key aspect of a dynamic system of environmental satellite data utilization. Adjustments and re-planning will always be part of the system. Additional measures of user accessibility should be developed, should be discussed at user interface meetings, and should include those measures suggested or required by various user segments and chains.
Within the environmental satellite data user community, needs for access to data and information cover a wide range—from information for warnings and alerts by community disaster centers through data for providers of value-added products and services for the recreational community to data for use in scholarly research. Obviously a “one-size fits all” system of access will not efficiently meet this spectrum, or matrix, of data type, volume, timing, quality, and cost of access.
Transitioning to Advanced Polar and Geostationary Satellite Architectures
NOAA and NASA are working together to achieve significant improvements in the satellite collection of environmental data. Today’s operational Polar-orbiting Operational Environmental Satellites (POES), Defense Meteorological Satellite Program (DMSP), and GOES provide for limited multispectral sensing of Earth’s atmosphere, oceans, and atmosphere. The converged NPOESS, in three orbit planes, with a first launch in 2009, will deploy advanced replacements for every POES and DMSP sensor flying today. Consider just three examples:
The Advanced Very High Resolution Radiometer (AVHRR), with six spectral bands, and the three-band Operational Linescan System (OLS) will be replaced with the 22-band Visible and Infrared Imager/Radiometer Suite (VIIRS). VIIRS expands upon the operational heritage of AVHRR and OLS with a better signal-to-noise ratio and better absolute accuracy, bringing the advanced spectroradiometry of the NASA Moderate-resolution Imaging Spectroradiometer (MODIS). At the same time, VIIRS brings the improved spatial performance and low-light/moonlight sensing capability of the OLS.
NOAA’s infrared sounder component of the Advanced TIROS Operational Vertical Sounder (ATOVS)—the High-resolution Infrared Sounder (HIRS) with 22 channels sensing broad bands of temperature, moisture, and ozone—will be replaced by the Cross-track Infrared Sounder (CrIS), a 1305-band hyperspectral sounder with sharper bands and 1-K-uncertainty temperature retrievals with direct heritage to NASA’s hyperspectral Atmospheric InfraRed Sounder (AIRS).
NOAA’s Spectral Backscatter UltraViolet (SBUV/2), a 22-channel operational nadir sounder, and NASA’s Total Ozone Mapping Spectrometer (TOMS), an eight-channel scanning radiometer, will be replaced by the NPOESS Ozone Monitor and Profiler Suite (OMPS). OMPS brings hyperspectral nadir and limb sounding capability to the heritage of the Space Shuttle Ozone Limb Scattering Experiment (SOLSE)/Limb Ozone Retrieval Experiment (LORE).1
In every case, the advanced sensors extend today’s measurement sets, while providing significant evolved capability to meet the tightened requirements of the Integrated Operational Requirements Document (IORD/2). To reduce the risk of these advances in sensors and the operational conversion of the research algorithms, NASA will fly a bridge mission, the NPOESS Preparatory Project (NPP), in 2007. NPP will carry the VIIRS, CrIS, and OMPS.
Advances in POES are not limited to sensors and algorithms. NPOESS will replace the traditional once-per-orbit downlink with a 15-station “SafetyNet” (Figure 1.2). This improved communications topology will yield greatly improved timeliness, benefiting all operational users interested in near-real-time data.
With the NPOESS flight and ground segments under development, NASA and NOAA are now working to achieve similar advances in geostationary orbit. The advanced imaging and hyperspectral infrared sounding capabilities of the next generation of GOES (GOES-R) will be comparable to those of NPOESS. The Advanced Baseline Imager (ABI) will provide multispectral reflective and emissive
imagery comparable to that of VIIRS; rapid temporal refresh; simultaneous mesoscale, continental United States, and full-disk coverage; and reduced sensitivity to solar impingement on the sensor near local midnight. The Hyperspectral Environmental Suite (HES) will bring comparable infrared sounding capability to CrIS, while adding high-resolution reflective ocean-color coverage of U.S. coastal oceans and littoral zones (see also Box 1.1).
The added spectral bands, improved radiometric bit depth, sharpened ground sample distance, and accelerated temporal refresh have a consequence in data rate. With increases of 10X to 100X, NPOESS and GOES data requirements will demand improved downlinks and ground processing, storage, and dissemination systems capable of keeping up with the higher data volumes and more-complex data product algorithms.
Reconciling Stability and Change
The NOAA-NASA operational environmental satellite systems operate amid the opposing forces of (1) technology-driven change, (2) changing user requirements, and (3) requirements for an efficient, dependable system of access for operational users. Furthermore, the multidecadal lifetime of the satellite systems requires sustainability through numerous federal budget cycles and prioritizations. In partnerships with the aerospace industry, satellite data providers can employ certain systems engineering principles to bring flexibility and adaptability for the optimal utilization of satellite data. However, beyond the initial ground processing of a suite of satellite remote sensing output, each of the many user pathways requires individual attention for the reconciliation of stability and change. Some user segments or pathways are more change-averse than others.
Algorithm Development: Spiral Model—The “Virtuous Cycle”
Today’s algorithms for reducing data acquired by increasingly sophisticated imagers, sounders, spectrometers, radiometers, and other instruments belong to a long and varied heritage of science and operational algorithms, based on multiple previous missions over the past two to three decades. For example, the Defense Meteorological Satellite Program (DMSP)/Optical Linescan System (OLS), NESDIS Polar Operational Environmental Satellite (POES)/Advanced Very High Resolution Radiometer (AVHRR), NASA Sea-viewing Wide Field Sensor (SeaWiFS), and EOS/ MODIS form the principal electro-optical spectroradiometric heritage for future imagers, while the Nimbus-6 and Nimbus-7 Earth Radiation Budget (ERB) experiments, the NOAA-9, NOAA-10, and Earth Radiation Budget Satellite (ERBS) Earth Radiation Budget Experiment (ERBE), and the EOS Cloud and the Earth’s Radiant
Energy System (CERES) form the principal Earth radiation budget heritage. Each of these missions, with varying levels of formalism and an integrated multimission science thread, involved algorithm development according to a spiral development approach (Figure 1.3).
Consider radiometry—moving from 8-, to 10-, to 12-, to 14-bit data reveals an increased quantity of new information with every generation. While driven by requirements for precision, higher radiometric precision does not translate directly to improved quality for geophysical data products. Every additional two bits are equivalent to a 4X-deeper look into the data, but undesired “noise” is also unmasked in these deeper looks, requiring that the retrieval algorithms be significantly revised so as to uncover the desired information that the sensor now provides—separating the new information from the “noise” present in other, competing phenomenological signals. Extracting new information from higher-bit data also drives the need for additional spectral data—in the form of cleaner specialized spectral bands—to fuel the emerging algorithm requirements. Coordinated validation activities, in the form
of in situ networks or targeted field campaigns for ground truth confirmation, guide the fine-tuning and quality-improvement process.
Traditional “waterfall” program management approaches often bring high risk when significant new capabilities are being developed and implemented. Schedule pressures may freeze milestones too early, often while the algorithm and hardware technologies are still maturing. The spiral development process, however, is designed to permit system development, yet not restrict the activities required to ensure this increasing maturity. Every turn around the spiral includes more detailed activities, including generation and inspection of test data sets, further algorithm development, more thorough testing and validation strategies, and more test cases leading to a deeper understanding of the stratified algorithm performance over a wider range of environmental conditions, e.g.:
Across the measurement range,
For all surface types and air masses,
For diverse combinations of solar and viewing angles, and
For cloud states ranging from subvisible to broken multilayer overcast.
The collection of test data sets, theoretical basis documents, architecture and design documentation, algorithm software, and test plans and reports is enlarged and enhanced by each turn of the spiral. With each revision, limitations and risks are identified and removed. The approach is derived from experience gained during heritage programs (previous turns around the spiral).
CHARACTERIZATION OF AN END-TO-END SYSTEM FOR OPTIMAL USE OF DATA
It may not be possible to devise a well-defined and complete end-to-end system for use of satellite data that addresses all concerns. While working on this report, the committee considered certain key aspects, which are summarized here. Usually an end-to-end process requires a full range of functions that, at a minimum, include enabling capabilities to design, assemble, execute, process, integrate, distribute, make decisions, and carry out other complicated and ill-defined tasks such as education, training, and outreach. As shown in Figure 1.1, satellite data utilization is a complex, end-to-end process with many opportunities for feedback between the processing steps. The end-to-end system may occasionally lack either end point, but specific functions and links in the sequence can be identified. Figure 1.1 also illustrates the circular relationship of each function and the links with many bidirectional interfaces supporting the chain of events. A satellite mission commonly starts with the mission concept, which is heavily influenced by detailed knowledge of users’ needs, the available technology, and the requirements for successfully achiev-
ing the project. The end-to-end diagram in Figure 1.1 starts with the mission concept and ends with the study and assessment of the impact of the mission. The study and assessment of impact also provide significant inputs and lessons learned to the planning of any future missions and to the execution of current missions. The circular end-to-end system for the process of satellite data utilization can be summarized as follows:
A vision of a mission by an expert, farsighted individual, team, or program office that demonstrates new ways of making unique and enhanced measurements or of meeting ongoing operational requirements;
Incorporation of users’ requirements and lessons learned into the guidelines for the configuration of the design and building of a new sensor system;
Instrument characterization, including efforts in calibration and navigation, which enables consistent measurements that allow certainty and generation of quality data products;
Product development and demonstration, and processing research, to ensure that the data products meet the requirements identified in the mission concept;
Validation of data and data products to characterize the accuracy and long-term stability of the measurements and products, with feedback for algorithm development and product generation before distribution and use recognized as vital;
Distribution and storage for downstream data access to allow indirect users and specific agencies to tailor the data for their own needs;
Other final links in the chain, including weather forecast applications, data assimilation, application for decision support systems, impact studies, and mission assessment;
Indirect use and applications utilizing a variety of satellite data product types and formats; and
Training and education across all functions in the process and links in the chain to ensure proactive and iterative collection, understanding, and embracing of operational users’ requirements and feedback.