EXECUTIVE SUMMARY OF CLIMATE DATA RECORDS FROM ENVIRONMENTAL SATELLITES*
At the dawn of the twenty-first century, NOAA’s mission includes a bold new mandate to “understand climate variability and change to enhance society’s ability to plan and respond.” An integral component of NOAA’s emphasis on climate involves creating a stewardship plan to generate, analyze, and archive long-term satellite climate data records (CDRs) for assessing the state of the environment. Although the concept of a “climate data record” has surfaced numerous times in recent literature (e.g. NRC, 2000c,e), the climate community has yet to settle on a consistent definition. In this report the committee defines a climate data record as a time series of measurements of sufficient length, consistency, and continuity to determine climate variability and change. We further segment satellite-based CDRs into fundamental CDRs (FCDRs), which are calibrated and quality-controlled sensor data that have been improved over time, and thematic CDRs (TCDRs), which are geophysical variables derived from the FCDRs, such as sea surface temperature and cloud fraction.
To generate the best possible plan for creating satellite CDRs, NOAA asked the National Academies to conduct a two-phase study to provide advice on creating CDRs. In phase 1, the committee is providing an interim report with advice on the key elements of a satellite-based CDR program, including lessons learned from previous attempts, important considerations for identifying an appropriate organizational framework for long-term success and sustainability, suggested steps for generating and archiving CDRs, and the importance of partnerships. The objective of the interim report is to provide NOAA with general guidance about what needs to be included in its plan. More specific comments will be provided once NOAA writes the plan, expected to be completed in late summer of 2004.
NOAA’s new climate mandate is fundamentally different from its traditional weather forecasting mandate and raises a new set of challenges owing to the varied uses of climate data, the complexities of data generation, and the difficulties in sustaining the program indefinitely. The task and structures being proposed for NOAA in this report are considerably more complex, costly, and demanding than those currently in place. A high level of commitment and a number of changes at multiple levels within the agency will be needed to institute and fund the various components of CDR stewardship. NOAA will not, however, be the first entity to generate climate-quality data and NOAA can learn many lessons from previous efforts; looking back on historical programs, some commonalities for success include science advisory panels, regular calibration and validation of data, adequate resources for reprocessing, user workshops to solicit advice on the future of the program, clear data storage and dissemination policies, and a willingness to form partnerships. Based on these historical lessons, community surveys, a workshop, and committee expertise, the committee identified 14 key elements for creating a climate data record program based mainly on satellites (Box ES-1). Adherence to these elements would help NOAA to create CDRs that are accepted as community standards, while ensuring that they remain responsive to user needs.
Underlying many of these elements of success is early attention to data stewardship, management, access and dissemination policies, and the actual practices implemented. Because a successful CDR program will ultimately require reprocessing, datasets and information used in their creation, such as metadata, should be preserved indefinitely in formats that promote easy access. The ultimate legacy of long-term CDR programs is the data left to the next generation, and the cost of data management and archiving must be considered as an integral part of every CDR program.
CDR Organizational Elements
CDR Generation Elements
Sustaining CDR Elements
The new emphasis and importance of climate within NOAA’s mission requires an increased focus on partnerships and new approaches as it relates to supporting extramural research. Many agencies and groups are interested and involved in creating, analyzing, and storing CDRs. By partnering with other government agencies, academia, and the private sector in development, analysis, and reprocessing of CDRs, NOAA can create and sustain a successful CDR effort; a high degree of interagency coordination on the requirements, definition, and implementation of CDRs is essential for satisfying the broad user communities of today and providing climate data stewardship for future generations.
OVERARCHING RECOMMENDATION: NOAA should embrace its new mandate to understand climate variability and change by asserting national leadership for satellite-based Climate Data Record generation, applying new approaches to generate and manage satellite Climate Data Records, developing new community relationships, and ensuring long-term consistency and continuity for a satellite Climate Data Record generation program.
NOAA is recognized as a national leader in weather information, including the management of a weather satellite program and creation of weather products. However, success in establishing and sustaining a CDR program requires a long-term commitment and a level of effort that goes beyond NOAA’s weather program. A particularly key component of NOAA’s success will be defining steps for creating FCDRs and TCDRs. NOAA’s plan also needs to account for all of the data and metadata that must be stored in easily accessible, self-describing formats. Fortunately, NOAA should not feel obligated to generate all of the nation’s CDRs, and by enhancing and expanding community involvement in the CDR program, NOAA can help to ensure community acceptance and creation of high-quality CDRs.
Supporting Recommendation 1: NOAA should utilize an organizational structure where a high-level leadership council within NOAA receives advice from an advisory council that provides input to the process on behalf of the climate research community and other stakeholders. The advisory council should be supported by instrument and science teams responsible for overseeing the generation of Climate Data Records.
An important step in maintaining a successful program is developing or utilizing an appropriate organizational framework that incorporates feedback and advice from user communities. The committee believes that NOAA will help to ensure success if it includes scientists interested in CDRs, assigns committed people to generate the CDRs, develops technical and science support for users, and creates science teams that are renewed regularly. In particular, NOAA should utilize an advisory council of internationally recognized climate experts to:
Recommend and prioritize the variables that are developed into TCDRs;
Oversee the calibration of FCDRs and validation of TCDRs;
Evaluate proposed new TCDRs as measurement capabilities improve or scientific insights change over time;
Review the utility and acceptance of TCDRs and recommend the elimination of those that are not successful; and
Review and oversee NOAA’s stewardship of the CDR program.
The actual creation of FCDRs should be carried out by a team of engineers and scientists, who should monitor satellite characteristics and document their work extensively so that future generations can assess and understand their work. Additionally, TCDR science teams with broad interdisciplinary representation should define algorithms for TCDR development and oversee TCDR generation. These teams should include research scientists funded by or employed by NOAA and scientists from other agencies, academia, or private industry who use the data, and they should be competitively selected, with limited (but renewable) terms.
Supporting Recommendation 2: NOAA should base its satellite-based Climate Data Record generation program on lessons learned from previous attempts, which point out several unique characteristics of satellite Climate Data Records, including the need for continuing calibration, validation, and algorithm refinements, all leading to periodic reprocessing and reanalysis to improve error quantification and reduce uncertainties.
Because most of NOAA’s operational satellites were created as weather rather than climate platforms, the committee stresses that NOAA should include nominal calibration, vicarious calibration monitoring, and satellite-to-satellite cross-calibration as part of the operational satellite system; this is important because orbital drift, sensor degradation, and instrument biases will affect the creation of consistent CDRs. Nominal calibration involves determining the calibration of a single sensor on a single platform, and while this is standard prelaunch practice, it is important to calibrate the sensor in orbit as well. Vicarious calibration monitoring involves measuring a known target or comparing the satellite signal with simultaneous in situ, balloon, radiosonde, or aircraft measurements; all instruments should undergo vicarious calibration monitoring at regular intervals,
regardless of on-board nominal calibration, to prevent drifting of the data over time due to orbital drift and drift in the observation time, which aliases the diurnal cycle onto the record. Satellite-to-satellite cross-calibration involves adjusting several same-generation instruments to a common baseline, and this is particularly important for long-term studies, as each sensor will have slightly different baselines even if they are built to the same specifications.
An ongoing program of validation also should be carried out to determine the uncertainty associated with TCDRs. This is based on establishing rigorously derived uncertainties for the TCDR using independent correlative measurements conducted throughout the data record and over global scales, which in turn determines whether a trend can be detected. NOAA should establish a two-track generation program, including an upgradeable baseline CDR track and a second (mostly extramural) funded research program to validate, analyze, assess, and reduce uncertainties in future base versions. The two-track approach encourages a culture where scientists and users know that future improvements will be available over time.
Supporting Recommendation 3: NOAA should define satellite Climate Data Record stewardship policies and procedures to ensure that data records and documentation are inexpensive and easily accessible for the current generation and permanently preserved for future generations.
History reveals that programs are more successful when the data management system provides free and open access to data, facilitates the reprocessing of CDRs, allows for new satellite TCDRs to be created, and has an easy problem-reporting procedure. A clear data policy can ensure continuity in the data record, including the ancillary data used to reprocess CDRs, project and dataset documentation, and the science production software. NOAA also should ensure that the data management infrastructure can accommodate user requests and provide different data formats, given the large satellite data volumes that a CDR program will create. This system should include the capability for temporal searches and subsetting. NOAA also can ensure a more robust program if the data are available in self-describing formats appropriate for a variety of uses, including geospatial and socio-economic applications. NOAA should establish a process for scientifically assessing the long-term potential of data and data products. Scientific assessments of the data can help NOAA to organize its archive so that data dissemination is efficient and cost-effective.
Supporting Recommendation 4: NOAA should develop new community relationships by engaging a broader academic community, other government agencies, and the private sector in the development and continuing stewardship of satellite Climate Data Records.
One of the best methods NOAA can institute for gathering community input is to convene regular open science meetings where users share their research and discuss limitations and recommendations for improving the CDRs. It is important to hold these meetings regularly because research will improve data quality over time and the meetings will help to foster community support. These meetings could be held in conjunction with conferences held by such organizations as the American Meteorological Society or the American Geophysical Union, with benefits being cost savings and broader attendance. NOAA should actively encourage other agencies and user communities to assist in development, analysis, and reprocessing of CDRs because expertise for CDRs lies within many sectors. NOAA can create a more successful CDR program by developing these partnerships.
Supporting Recommendation 5: NOAA should consider existing U.S. multi-agency organizations for implementation of the Climate Data Record program, rather than devising a new structure. The most appropriate organization is the Climate Change Science Program.
Stewardship of CDRs is complex, costly, and demanding, and NOAA should aggressively seek partnerships to help to ensure a successful program. The committee does not believe that NOAA needs to invent and implement a new management structure for generating, analyzing, and archiving CDRs; for instance, the goals and management structure of the Climate Change Science
Program (CCSP) are similar to NOAA’s climate goals, and NOAA may therefore be able to implement part of the CDR program under the CCSP. If NOAA were to volunteer to be the lead or executive agency (or delegate leadership to a partner) responsible for satellite CDRs under the CCSP umbrella, NOAA could advance its climate mandate and assert national leadership. Because the CCSP structure already has built-in interagency interactions, NOAA could also leverage them for the CDR program.
Supporting Recommendation 6: NOAA should pursue appropriate financial and human resources to sustain a multidecadal program focused on satellite Climate Data Records.
Developing a CDR program is fundamentally important to the nation, and it is imperative that the effort not be inhibited by a lack of human or financial resources. Even if NOAA leverages funds and personnel from other agencies, academia, and private industry, and even if it integrates the CDR program into CCSP, it will still have to be aggressive in seeking additional funds. This program will require a long-term vision and commitment, and it will be important to account for inflationary increases when outlining the human and infrastructure needs for successfully generating, analyzing, reprocessing, storing, and disseminating CDRs.