Elements of a Successful Satellite CDR Generation Program
Developing a successful Climate Data Record (CDR) generation program poses many challenges because of the varied uses of climate data, the complexities of data generation, and the difficulties in sustaining the program over extended periods of time. Many of these challenges are not unique to a climate data record program, but are common elements faced in creating most long-term science programs (e.g., NRC, 2002). The previous chapter described the experiences from programs where satellite data are the primary global data records. These represent only a fraction of such experiences and yet the investment to date in just these programs alone has been enormous. This chapter outlines 14 key elements that the committee believes are important for the successful generation of CDRs. The first section discusses four elements related to organization, emphasizing the importance of having a high-level coordinating body within NOAA, broad involvement of stakeholders, and mechanisms for review (and redirection) by the science community. The second section presents elements related to the generation and stewardship of the fundamental CDRs (FCDRs) and the thematic CDRs (TCDRs). The final section discusses elements related to sustaining a CDR generation program.
In devising a program for generating CDRs, NOAA would benefit greatly from developing an organizational framework that includes mechanisms for providing scientific oversight and advice, encouraging feedback from user communities, and allowing opportunities to redirect the program based on advice and feedback. The task of generating CDRs is ambitious, but NOAA does not need to accomplish everything on its own because CDR expertise lies within other agencies, academia, and the private sector as well. Therefore, in developing a management and administration component, NOAA
can help to ensure success if it involves scientists with a vested interest in CDRs, finds committed people to generate the CDRs, develops technical and science support for broad involvement, and creates teams that are reviewed and renewed regularly, consisting of NOAA personnel, outside scientists, industry, and others. The following elements lay out a framework of responsibilities that should be accomplished and the kinds of groups needed for each role. The committee believes that the following 14 elements will help NOAA to create a successful CDR program.
1: A high-level leadership council within NOAA is needed to oversee the process of creating climate data records from satellite data.
A leadership council of NOAA management personnel would receive input from the climate research community and other stakeholders through an advisory council of internationally recognized climate experts and would have the authority to approve plans and commit resources to generate the CDRs. The leadership council would adopt responsibility for overall stewardship of the CDR program, determining whether the FDCRs and TCDRs are effective and if not, working with partners to correct problems.
2: An advisory council is needed to provide input to the process on behalf of the climate research community and other stakeholders.
An advisory council would advise the leadership council concerning the generation of CDRs. The function of the advisory council would be to
recommend and prioritize the variables that are developed into TCDRs;
oversee the calibration of FCDRs and validation of TCDRs;
evaluate proposed new TCDRs and refinements of existing TCDRs as measurement capabilities improve or scientific insights change over time;
review the utility and acceptance of TCDRs and recommend the elimination of those that are not successful; and
review and oversee NOAA’s stewardship of the CDR program.
Members of the advisory council should include participants from within NOAA, other federal agencies, academia, and industry. Given the importance of the advisory council, compensation for their services would be appropriate. Respondents to the community survey and attendees at the
workshop were all very clear in stating that without some form of compensation, NOAA will not be able to attract the best scientists for the advisory council, which in turn could limit the success of the program.
3: Each FCDR should be created by an appointed team of CDR experts.
The expertise needed to create and validate FCDRs is different from that needed to produce TCDRs. The FCDR teams should involve engineers and spacecraft specialists, as well as representatives from the thematic science teams, because feedback from the generation of TCDRs is essential. NOAA already has in-house staff (either within NOAA centers or in NOAA-funded cooperative institutes) to create FCDRs from satellite data, but history indicates that staff not familiar with the product may not be aware of problems with the data. Therefore, NOAA should involve scientists from government, academia, and the private sector who have expertise in how the FCDRs are to be used, and are familiar enough with the variables to know what values are reasonable. The main functions of the FCDR teams include instrument monitoring and production of FCDRs, and they should document their work extensively so that future generations can easily assess and understand what they have done. Proper documentation is also important for users, because all FCDRs will have limitations and errors, and these must be well documented.
The size of an FCDR team will be based on financial resources, but guidance from studies examining why projects succeed (Standish Group, 1999) indicates that smaller teams are more likely to achieve their goals.
4: Science teams should be formed within broad disciplinary theme areas to prescribe algorithms for the TCDRs and oversee their generation.
While the FCDR generation teams are focused on creating accurate and precise radiance measurements, the program will not be successful unless scientists are actively utilizing the FCDRs to create TCDRs. The committee recommends the formation of thematic science teams within broad disciplinary theme areas (Table 3-1) to oversee the generation of TCDRs and evaluate TCDRs created by outside groups. These teams historically have formed around specific technologies, but this may not be the best approach in the long term; for example, users of active microwave remotely sensed data have not typically interacted with users of passive microwave, visible, and infrared remotely sensed data. An alternative approach would be to form teams around science themes as previously recommended by the NRC
TABLE 3-1 NASA and NOAA Thematic Groupings of Space-based Observations
24 NASA/Earth Observing Systems (EOS) measurements are divided into 5 groups.
5. Solar radiation
National Polar-orbiting Operational Satellite System (NPOESS) divides its 61 Environmental Data Records (EDRs) into 7 “parameter” groups.
1. Key parameters
2. Atmospheric parameters
3. Cloud parameters
4. Earth radiation budget parameters
5. Land parameters
6. Ocean and water parameters
7. Space environmental parameters
(2000b). The committee believes that establishing CDR science theme teams, not instrument teams, will be essential to the goal of generating successful CDRs.
These teams should be led by recognized scientists who are actively engaged in research generating, utilizing, or validating the TCDRs. The team leaders should be ultimately responsible for the quality of the TCDRs, and they should provide the advisory council with periodic updates. The thematic teams should include research scientists funded by or employed by NOAA and other agencies, organizations, or private sector companies who use the data, and they should have some representation from the FCDR teams. The TCDR science teams should be competitively selected, with limited (but renewable) terms. Team members should also be compensated for their work, similar to the advisory council. Funding for the thematic teams should be broadly based and could be orchestrated by the Climiate Change Science Program (CCSP) or a partnership of federal agencies (see Chapter 5).
CREATING FUNDAMENTAL CLIMATE DATA RECORDS
The distinction between FCDRs and TCDRs1 is important and to a large extent unique to the generation of CDRs from satellite data. The heart of NOAA’s efforts to create a successful CDR program lies in the creation of the FCDRs. It is vitally important that NOAA appreciate the steps required to create the FCDRs, as the success of the CDR program hinges on creating reliable, consistent, and stable FCDRs.
5: FCDRs must be generated with the highest possible accuracy and stability.
As explained in Chapter 1, the FCDRs are the time series of calibrated signals (e.g., top-of-atmosphere radiances, brightness temperatures, radar backscatter) for a family of sensors (e.g., Advanced Very High Resolution Radiometer [AVHRRs]), together with the ancillary data used to calibrate them. In some cases, extensive in situ datasets might need to be included as part of the FCDRs if these ancillary data are needed to regenerate the time series at a future date. These in situ datasets also are occasionally improved, so efforts are needed to ensure that the most up-to-date in situ dataset is utilized for the FCDR analysis. The FCDRs will be used to create a variety of TCDRs for various disciplines. Where possible, NOAA should adhere to the Global Climate Observing System (GCOS) climate-monitoring principles (Box 3-1). In some cases it will not be possible to meet all the requirements. For instance, Requirement 1, “Complete sampling within the diurnal cycle (minimizing the effects of orbital decay and orbit drift) should be maintained,” will not always be met when using historic polar orbiter data to create CDRs. Clearly TCDRs should be developed, to the extent possible, to account for diurnal effects, but in some instances it will not be possible. A TCDR record tied to a narrow segment of the diurnal range might still prove valuable decades hence, when the physical understanding of some processes become better understood.
6: Sensors must be thoroughly characterized before and after launch, and their performance should be continuously monitored throughout their lifetime.
Verification of instrument performance requires a comprehensive understanding of the physics behind the measurement. Satellite sensors must have a thorough prelaunch characterization and the ability to measure important instrument properties on orbit, including the ability to calibrate the sensor after launch. Procedures should be in place to monitor sensor performance in near real time.
The satellite and sensor engineers, working with the FCDR team, should do the best job possible to ensure the accuracy of the calibrated data used to create the FCDRs. An integral part of this process is a full characterization of instrument performance and stability, and continuous monitoring of the observing system for changes in the sensors. Since most of NOAA’s operational satellites were created as weather rather than climate platforms, this
The international GCOS Panel (GCOS, 2003) developed the following principles for monitoring climate variables from satellites, which the committee endorses and suggests that NOAA implement:
step is particularly relevant for NOAA to address. Changes in satellite characteristics, such as orbital drift, system calibration, sensor degradation, and instrument failure compromise the ability to create high-quality, consistent CDRs over time. NOAA should assure that procedures are in place to monitor the observing system for irregularities that could corrupt the long-term value of the FCDRs. Such a diagnostic scheme will allow the FCDR team to distinguish between artificial changes related to the observing system and real changes due to climate.
It is essential that a period of overlap between successive sensors be incorporated into launch schedules to assess and correct for differences between sensors, thus assuring continuity of the long-term climate records. Over time as new systems are developed the FCDR team also should be charged with determining what impact the new sensor has on observations. NOAA must ensure that introduction of new observing techniques does not result in artificial changes in the FCDR. The committee recommends that NOAA continue the Polar Operational Environmental Satellite (POES) performance-monitoring activities with some modifications (in italics), namely,
inventorying data, filling in missing periods with other data in NOAA archives provided that the other data is fully compatible with the FCDR (that is, other data cannot cause spurious trends or variability);
converting internal satellite quality data into useful information for end users; and
providing easy-to-use Web-based tools that link end user quality control information to more detailed instrument information.
7: Sensors should be thoroughly calibrated, including nominal calibration of sensors in-orbit, vicarious calibration with in situ data, and satellite-to-satellite cross-calibration.
For the FCDRs to be useful for future applications and to maintain a consistent record based on multiple satellites over several decades, the sensors must be well calibrated. Sporadic efforts have been mounted by various groups to correct existing radiance biases, but this function should be centralized, standardized, and routinely performed by NOAA. The GCOS panel recommended that steps be taken to make radiance calibration, calibration monitoring, and satellite-to-satellite cross-calibration of the full operational constellation a part of the operational satellite system. The International Satellite Cloud Climatology Project recognized that calibration is an iterative process, and they defined three types of calibration: (1) nominal calibration; (2) vicarious calibration; and (3) satellite intercalibration.
Nominal calibration involves determining the calibration of a single sensor on a single platform. Prelaunch calibration is a standard procedure, but the calibration of the instrument must be monitored in orbit, and if necessary adjusted. Depending on the instrument, this may involve onboard calibration lamps or other means of onboard calibration, but often there is uncertainty associated with the “known source” onboard the spacecraft.
Alternative methods involve maneuvers to view deep space, or the Sun or Moon as a constant radiation source by which to monitor the stability of an instrument during its orbital lifetime. This does not necessarily replace the need to calibrate the instrument, but it does provide a method of monitoring instrument stability over time (Figure 3-1).
Vicarious calibration is accomplished by measuring a known target
(e.g., desert) or comparing the satellite signal with simultaneous in situ balloon, radiosonde, or aircraft measurements. The satellite calibration is then adjusted, after correcting for atmospheric effects, to agree with the target. All CDRs should be vicariously calibrated at regular intervals, regardless of onboard calibration.2 Without vicarious calibration, sensor data can drift over time (Figure 3-2). Vicarious calibration also can serve as an additional means for monitoring instrument stability. In addition, the future experience gained in the vicarious calibration of well-calibrated satellite
The GCOS Climate Monitoring Principles (see Box 3-1) do not recommend vicarious calibration, but this committee believes it is an important calibration-monitoring technique.
instruments will undoubtedly prove to be useful in extending CDRs backward in time.
The third form of calibration is satellite intercalibration. This involves adjusting several same-generation instruments to a common baseline, such as calibrating all SSM/I sensors to one baseline. This is particularly important for long-term studies, as each sensor will have slightly different baselines even if built to the same specifications. Processes that lead to slight differences between sensors include uncertainties in prelaunch instrument response characterization, deposition of contaminants on sensors, drift in the satellite orbit and altitude, electronic noise, and cross-talk. Each bias source must be addressed, preferably with reference to known physical mechanisms. The best assurance of sensor intercalibration is to guarantee a period of overlap between successive sensors. Even though we are in the fifth decade of satellite studies, we have more to learn about radiometric calibration and data analysis. Overlap is a crutch to help us to continue to produce useful results while we are still learning how to use the data. Without intercalibration, long-term trends in CDRs likely will lead to spurious trends (Figure 3-3).
CREATING THEMATIC CLIMATE DATA RECORDS
Although the FCDRs represent NOAA’s long-term legacy, the majority of users will use the thematic CDRs (Box 3-2). Since NOAA cannot create all possible TCDRs, mechanisms must be in place to select an appropriate number of TCDRs to generate. Once created these TCDRs must have rigorous validation and estimated uncertainty levels.
8: TCDRs should be selected based on well-defined criteria established by the advisory council.
The advisory council should begin by identifying thematic areas and establish science teams to recommend generation or acceptance of existing TCDRs in each discipline area. A number of possible “themes” can be defined (e.g., Table 3-1). One simple method that NOAA could employ to select the TCDRs is to select team members based on their ability to generate data records; the selection of the science teams and TCDRs would automatically determine which variables are produced. An alternative approach would be for NOAA to issue a Request for Information (RFI) to formally solicit ideas rather than proposals as a first step. NASA did this a few years ago to determine which future satellite missions should be in-
With an increased appreciation for the impact of weather and climate on daily activities (e.g., NRC, 2003a), society’s need for climate data grows rapidly. Many of the major users and uses of climate information have been illustrated in the recent NRC reports Making Climate Forecasts Matter (NRC, 1999c) and Fair Weather: Effective Partnerships in Weather and Climate Services (NRC, 2003b). In brief, some of the sectors using CDRs include education, research, water resources, energy, agriculture, forestry, transportation, defense, health, insurance, recreation and tourism, manufacturing, and retail. It is likely that CDRs will become more valuable and more used over time, requiring NOAA to do the best possible job to create reliable and consistent CDRs.
cluded in its Earth science strategic plan. The RFI was administered very much like a request for proposals. Mission concepts were submitted and panels were assembled to review the concepts. An RFI might be issued to solicit recommendations for TCDRs. The council should subsequently work with thematic team leaders to arrive at a prioritized list of TCDRs. Variables chosen for TCDR development should address key questions about the climate system, leading to clear improvements in (1) scientific understanding of the climate system, enabling climate variability and change to be better documented; (2) projections for future climate states; (3) regional, national, and international climate assessments; and/or (4) the nation’s ability to respond to climate variations. The benefits of the TCDRs should address applications of climate information as well as the scientific understanding of the climate system. The criteria for selection of TCDRs should also include the technological readiness of the record. In most instances NOAA likely will select one TCDR for a given parameter, but in certain situations NOAA may fund a baseline TCDR and encourage the creation of other TCDRs for comparison. In exceptional situations NOAA may even fund several TCDR efforts for the same parameter (recall the lessons learned from the MSU temperatures).
9: A mechanism should be established whereby scientists, decision makers, and other stakeholders can propose TCDRs and provide feedback that is considered in the selection of TCDRs.
Since the TCDRs will be used by scientists, decision makers, and other stakeholders, the process of selecting TCDRs will provide the greatest long-term utility if it is fully documented and open to input from user communities. The science community ultimately will submit proposals to NOAA and other funding agencies for generating TCDRs, and to be truly successful NOAA needs a buy-in from the science community from the start. As noted by Weisberg et al. (2000), “The most successful programs have been those with clearly defined users for the data they produce, which requires early interaction between the scientists responsible for designing the program and targeted data users.” When these communities are brought together early to identify their needs, rather than just being asked to approve or comment on what the science community has planned, there is a much greater chance of incorporating the needs and concerns of user communities into planning the TCDRs, and thus the opportunity to design a more satisfactory program that enjoys long-term user support.
Open science meetings have been used with great success to assist in
determining programmatic priorities, and NOAA could convene such conferences on the creation of TCDRs. These meetings could be held in conjunction with conferences held by other organizations, such as the American Meteorological Society or the American Geophysical Union, which would open the conference to broader attendance and reduce costs. Community surveys are also useful in generating a list of needs from various decision makers and user groups.
10: Validated TCDRs must have well-defined levels of uncertainty.
The process of validating a TCDR derived from satellite measurements is not simply a matter of “ground truthing” a satellite-derived product. It is the process of establishing uncertainty levels for the TCDR based on principles of error propagation and comparisons with independent correlative measurements. The identification, quantification, and minimization of biases and errors helps users understand how much confidence they should have in the TCDRs and whether the data are appropriate for their applications. More specifically, the uncertainty associated with a TCDR determines how much of a trend can be detected with the record.
An understanding of the measurement error structure is critical. Error propagation from raw data to the final derived product must be understood. Instruments and data processing will introduce systematic artifacts in the data. Error sources are not necessarily Gaussian, nor are errors uncorrelated spatially and temporally. The steps involved in deriving a geophysical variable (e.g., geolocation, calibration, and correction for atmospheric effects) must be identified. Each of these requires algorithms specific to the task and will introduce uncertainties in the geophysical product. Knowing the magnitude of the processing task will help to choose the right algorithms and ultimately quantify the uncertainty in the final geophysical product.
In discussing CDR uncertainty it is important to distinguish between geophysical quantities and indices. Geophysical quantities, the products of the TCDRs, have in principle an uncertainty that can be determined for a given time and space scale. Estimates of the quantity will lie within the uncertainty if measured by any observing system capable of measurements for the particular time and space scale. Indices, on the other hand, are defined by the instrumentation and the algorithm used to construct the index. Normalized Difference Vegetation Index (NDVI) is a typical example. There may not be correlative measurements for comparison, and thus uncertainty estimates are strictly based on instrument and algorithm properties. Nevertheless, indices provide valuable insight into the workings
of the climate system that as yet defy predictive capabilities through modeling. Ultimately, understanding will form links between indices and physical quantities. The significance of the latter is that with such understanding, historic data can be reanalyzed, given the new insight, to extend CDRs backward in time.
11: An ongoing program of correlative in situ measurements is required to validate TCDRs.
The process of validating a TCDR is an ongoing activity. There should be a program of in situ measurements established and maintained for this purpose. In some cases assessments to determine the amount and location of in situ data needed to estimate uncertainty may have to be performed. NOAA should also consider incorporating other satellite data as another source of correlative measurements. Data from geostationary satellites could provide information not available from the polar-orbiting satellites, in turn reducing uncertainty in the data. In particular, data from geostationary satellites can help with sampling the diurnal cycle. NOAA should also consider examining several algorithms and techniques to determine whether there is agreement among multiple routes to obtaining the same results. Blended products may be used to develop TCDRs, although this may make it more difficult to understand how to reconstruct datasets and how to account for errors.
SUSTAINING A CDR PROGRAM
Lessons from the past suggest that sustaining a long-term research program is difficult and is rarely achieved. In the short term NOAA will create successful FCDRs and TCDRs by following the steps discussed in this chapter. Since the CDR program is conceived with a long-term vision, it is important to recognize several elements related to sustaining the CDR program. Producing ongoing climate products (such as CDRs) is contingent on having stable funding both for obtaining data and for conducting the sustained scientific research that will necessarily underlie the production of CDRs. NOAA has traditionally been a mission agency with responsibility for weather monitoring and prediction and support of this mission through satellite operations, and data archiving and management. In comparison, a CDR generation program requires efforts above and beyond NOAA’s traditional role in weather forecasting.
NOAA has long supported climate research, but much of the research it
supports is done in-house or through research programs that are generally located in research units (e.g., Office of Atmospheric Research Laboratories) in the agency rather than operational units like the National Environmental Satellite, data, and Information Service (NESDIS). Research on CDRs must be directly linked with the production of CDRs and must be an integral part of the CDR program. Climate, unlike weather prediction, is a subject with widely distributed expertise across disciplines within academic, government, and private industry sectors. CDRs will inevitably serve the needs of multiple highly disparate communities ranging from scientists to individuals with responsibility for public policy and management and will draw on a changing array of satellites and sensors, requiring overlap and intercomparison of measurements. To succeed in the CDR mission NOAA will need to obtain a new higher level and probably new sources of sustained financial support for this valuable mission that will extend over many decades. If long-term funding is not sustained, history suggests that this the program will fail.
12: Resources should be made available for reprocessing the CDRs as new information and improved algorithms become available, while also maintaining the forward processing of data in near real time.
Over time, errors will be uncovered in the FCDRs and TCDRs, new algorithms will be developed for the TCDRs, and new technologies will be available. To ensure the success of any long-term program, mechanisms are needed to address deficiencies, correct problems, and ensure continuity, and these actions require adequate resources. As noted by NOAA’s Climate and Global Change Working Group report, “Given the continuing improvement in our understanding of climate observations and the need for long time series, reprocessing is a hallmark of every climate observing system.” As our understanding of the climate and satellite instruments improves, reprocessing could also be useful for extending CDRs back in time; for instance, CDRs created from NPOESS could be extended back to the Television Infrared Operational Satellite-Next Generation (TIROS-N) series, with knowledge that uncertainties in the earlier records may be larger (emphasizing the need for stating uncertainly levels).
In determining the resources needed to generate CDRs it is therefore essential to include the capability (e.g., computer processors, storage devices, personnel) to reprocess the data at periodic intervals. The FCDRs will need to be reprocessed as new information is acquired or better calibrations are made, but these records will eventually become stable. On the other
hand, the TCDRs will continue to change indefinitely as new or improved algorithms are developed and improved applications are made of the FCDRs. NOAA should consider development and maintenance of a two-track approach: (1) a commitment from an organization to implement, document, and disseminate (free and open) a Version X of a CDR under the guidance of advisory and science teams; and (2) a funded extramural research program to validate, assess, and provide improvements (upgraded algorithms or blending procedures) on which to base future versions and reanalyses. Mechanisms also should be in place for active data users, on Track 1 to be fully informed about basic findings, progress, and tentative plans for Track 2.
13: Provisions to receive feedback from the scientific community should be included.
Community feedback is important in developing a successful program. There should be a continual dialog about utility, quality, and problems between those who make observations and those who use them. This dialog will improve the quality of the data and foster their continuation. Users are the ones exercising the data, and as people use the data, problems are uncovered. Systematic metrics should be collected for all TCDR data streams to determine the utility of the TCDR. If the metric suggests that a TCDR is not being utilized at a high enough level to warrant continuation, it should be scuttled and funds directed to another TCDR. Since the FCDRs will still be generated, the scuttled TCDR could be recomputed at a future date. NOAA should publish acceptable levels of use so that users know the cutoff point (decommissioning) of TCDR production.
Regular user workshops are a meaningful way to convert user comments into new policies and procedures. NOAA could convene a workshop of the investigators who are using TCDRs in a particular theme area. This would be an “open science meeting” in which scientists would share their findings (e.g., give science talks) and also discuss limitations and recommendations for improving the TCDRs. Regular opportunities for dialog between the scientists and decision makers or other users of the data are also valuable.
14: A long-term commitment of resources should be made to the generation and archival of CDRs and associated documentation and metadata.
NOAA cannot afford to create all the desired CDRs; and thus, careful consideration should be given to the process of prioritizing the list of TCDRs. Once priorities are established, the resources needed to process them should be determined as realistically as possible, and then a long-term commitment must be made to provide the resources to sustain them and archive them, which is covered in more detail in the next chapter.
Stable support is an essential characteristic of a successful CDR generation program; thus inflationary increases should be programmed into budget planning. Operating cost increases or other factors often require flexibility and adjustments by the system operators to maintain data flow while long-term solutions are sought.
There should be a commitment to support research that utilizes the TCDRs. We do not believe that NESDIS should be responsible for supporting the research of all or even most science team members. There must be a commitment made by other agencies (e.g., NASA, NSF) and other NOAA line offices (e.g., National Marine Fisheries Service) to support the research community. The Climate Change Science Program could serve as the organizing entity for ensuring this commitment. Chapter 5 outlines the further details of partnerships that NOAA should explore.
How does a CDR become a community standard? How can NOAA ensure that the CDRs are responsive to user needs? The elements of a successful CDR generation program described above would assure this over time, because the steps outlined involve a community of people who use the records and allow them real input into their creation. The involvement of the climate research community will not happen simply by producing CDRs and making them available. It must be fostered by support for research specifically involving the CDRs and organized meetings of funded investigators. It also requires a long-term commitment on the part of NOAA to the generation of the CDRs and to their integrity and validity.