Ensuring the Climate Record from the NPP and NPOESS Meteorological Satellites (2000)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Ensuring the Climate Record from the NPP and NPOESS Meteorological Satellites 4 Conclusions and Recommendations CONCLUSIONS While encouraged by NASA's and NOAA's recent attention to preserving the climate record of NPP and NPOESS (and also EOS), the committee believes that an enormous investment in Earth observations is at serious risk. In briefings to the committee, NASA and NOAA officials acknowledged that there is no operational ground system infrastructure for U.S. climate data and services. (NOAA has requested funding for a prototype activity, the National Environmental Data Archive System—NEDAAS.) The climate research community requires that satellite data from NPP and NPOESS be able to generate climate data records, data whose quality is known quantitatively and whose temporal and spatial biases are minimized (or at least quantified). CDR production will require considerable scientific insight, including the blending of multiple data sources, error analyses, and access to raw data sets. Moreover, information on sensor design, operation, and calibration will also be necessary to develop a consistent CDR across multiple sensors. Production and refinement of CDRs by reprocessing data may be difficult (or unaffordable) under present plans. Archives often attempt to maximize the number of users to justify continuing financial support, but this may not be necessary for supporting the development of CDRs. Moreover, efforts to facilitate broad use may divert attention from CDR issues. For example, archives may try to provide a broad range of data services, ranging from data visualization to data analysis. The EOS Data and Information System (EOSDIS) was driven in its early development by the concept of one-stop shopping, whereby users could locate and retrieve all of the data they needed from a single site, although the data were actually stored in many distributed archives. Thorough analysis and design of the data system are essential, especially if a key objective is the provision of long-term data sets to guide public policy. Design of archive services must be firmly based on actual user requirements in order to control costs. A look at the recent history of NASA and NOAA data systems reveals both successes and failures. Much can be learned from previous experience, but the fundamental objective of establishing a set of data systems and services to meet the needs of climate research will require more than MOUs and larger magnetic tape silos. New services must be supported that are not available in the present mix of NASA research mission data systems and NOAA long-term archives. New scientific and policy demands are being placed on these systems, and new management and technical approaches must be established. Although increasing funding is a necessary condition, it alone is not sufficient. RECOMMENDATIONS 

The committee's study of issues related to preservation of the climate record from future NASA and NOAA satellites was necessarily brief and drew heavily on previous work and on the 2-day workshop. Underlying the committee's recommendations are strongly held views about the critical need and unique potential for data from NPP and NPOESS to contribute to the work of the climate research community. In particular, the committee believes that prudent planning and modest investments early in the program will allow the NPOESS system to generate and continue essential climate-quality data records and to develop new records based on the rich blend of planned instruments. In putting forward its recommendations, the committee also drew on the lessons learned from the data system for the EOS Data and Information System (EOSDIS). Climate research will require a variety of services, ranging from careful long-term stewardship of the basic data sets to intensive data analysis and algorithm refinement. This complex set of scientific and information system activities should be broken into manageable pieces rather than used to form a comprehensive, single-system solution. Two critical functions are the long- term archive (LTA) and active archive functions. Although there are strong similarities, there are also fundamental differences in the nature of the interactions between the data system and its users and the degree of flexibility and stability. Climate research requires an integration of the more stable, long-term functionality of the LTA and the flexibility to pursue and develop new capabilities in the active archives. The first four recommendations are presented in order of priority. They can be summarized as follows: 1. Preserve the basic data sets. 2. Provide access to these data sets. 3. Develop CDRs based on these data sets. 4. Provide basic user services and tools to the science research community. The next four recommendations, which are not listed in order of priority, focus on programmatic and management structures to meet the essential requirements for a climate data system. They can be summarized as follows: 5a. and 5b. Include the NPOESS Preparatory Project (NPP) as part an emerging climate system. 6. Invest in early prototypes of LTA and active archive functions. 7. Develop LTA and active archives as a set of complementary services. Recommendation 1. NOAA should begin now to develop and implement the capability to preserve in perpetuity the basic satellite measurements (radiances and brightness temperatures). The development of long-term, consistent time series based on CDRs requires access to the lowest level of data available. In general, this means the raw data records (RDRs), or Level 1A data. The low-level data can be used to develop refined CDRs as scientific and technical understanding of Earth processes and sensor performance improves over time. The committee recommends that NOAA do the following: Archive both current and future data sets, including those from both research and operational satellite missions, in an LTA. Archive information on sensor development, calibration, operation product validation, and appropriate metadata along with the basic radiances. Migrate data sets to new, computer-compatible media on a regular basis, such that data sets are refreshed every 2 to 3 years consistent with the pace of technology evolution. 

Organize data in the LTA based on user access patterns to optimize data retrieval. Recommendation 2. NOAA should guarantee climate researchers affordable access to all RDRs in the long-term archive, with an emphasis on large-volume data access. Development of CDRs requires access to enormous data volumes, but it is likely that only a small number of researchers will need such extensive access to the raw data. Thus, a well-designed set of basic services would meet this basic function without being too costly. The committee recommends that NOAA act on the following: Award the LTA functions on a competitive basis to both government and private organizations to promote innovation. Start the development of the LTA immediately with a simple set of end-to-end capabilities to gain experience and modify the plans and implementation accordingly. (End-to-end is defined as being from sensor aperture to the desktop of the climate information user.) Recommendation 3. NASA, in cooperation with NOAA, should support the development and evaluation of CDRs, as well as their refinement through data reprocessing. Because the CDR process is driven by science understanding, there will be a continuing need for the involvement of researchers. The NOAA/NASA Pathfinder (see Box 4.1) shows that the agencies can generate critical data sets for transitioning research products into operational data products. Over the next decades, the committee expects that a few experimental CDRs may become effectively "operational" products and will be produced by NOAA. The committee recommends that NASA, in cooperation with NOAA, take action as follows: Periodically select science investigations and provide adequate support to develop and evaluate new CDRs. Preserve sensor calibration and operating information, as well as metadata and ancillary data fields, in a manner that allows reprocessing the CDRs. Evaluate on a regular basis the organization of data sets in the LTA in light of actual data usage patterns to improve reprocessing and access efficiency. Recommendation 4. NOAA and NASA should define and develop a basic set of user services and tools to meet specific functions for the science community, with NOAA assuming increasing responsibility for this activity as data migrates to the long-term archive. NASA's Distributed Active Archives Centers, as well as components of NASA's Earth Science Information Partners, are gaining experience with responding to data requests and setting up user services. Although the focus is on the order entry process (catalog, data location, browse, etc.), more attention needs to be given to quality assurance and the order fulfillment process (metadata, subsetting, electronic data delivery, etc.). Emphasis should be given to reducing cost through automation. It is essential that the large-volume data sets from the archive be affordable for the science user community. The committee recommends that NOAA and NASA do the following: Select teams on a competitive basis that will identify and provide specific user services and tools (see Appendix D). As part of an ongoing process of system evaluation and 

improvement, these teams would assist in identifying and providing essential user services. Based on a rigorous analysis of a user model for climate research, they would make recommendations on characteristics such as data subsetting and browse capabilities. BOX 4.1 NOAA/NASA Pathfinder Experience In the early 1990s, NASA and NOAA developed a joint initiative to provide critical data sets for the global change community. This initiative was known as the NOAA/NASA Pathfinder and was supported largely by the EOS Program Office at NASA. Pathfinder projects involved the global change research community in identifying their most pressing data needs. Pathfinder projects aimed at providing improved access to and processing of large time-series data sets from different satellite systems (for example, Advanced Very High Resolution Radiometer (AVHRR), TIROS Operational Vertical Sounder (TOVS), Geostationary Operational Environmental Satellite (GOES), Special Sensor Microwave Imager (SSM/I), and Landsat). Priority data sets were identified and a science working group was formed for each product to advise on product specifications, consensus algorithms, and data processing requirements. Emphasis was given to reprocessing decadal data records with raw data amenable to calibration, allowing calibration between multiple instruments. Joint Announcements of Opportunity from NASA and NOAA provided open competition for the generation of new Pathfinder products. A Pathfinder benchmark period (April 1987 to November 1988) was established for all the projects to facilitate complementary analyses and intercomparison studies. The Pathfinder projects substantially improved the availability of time-series data for the science community and demonstrated how NASA and NOAA could collaborate on joint data initiatives to serve the global change research community. For example, the AVHRR GAC (Global Area Cover) Land Pathfinder undertook the first major reprocessing of the entire AVHRR land data record and the associated transfer of the data record to a new media. Future joint initiatives between NASA and NOAA in the context of the NPP and NPOESS data system development would benefit from the lessons learned from the various Pathfinder data activities. Support and maintain a balance between internal and external expertise at the government data centers. Examine the feasibility of providing open electronic access to a rolling archive of RDRs and EDRs through the NESDIS Central that is planned for NPOESS and NPP. Recommendation 5a. NASA, in cooperation with the Integrated Program Office, should develop the NPOESS Preparatory Project as an integral component of a climate data system. NPP represents a unique opportunity to test both scientific and programmatic interfaces related to an integrated data systems strategy. It will bridge the gap between the NASA research missions and the NPOESS operational missions. There is potential to begin the development of long-term, high-quality CDRs and an associated data system for climate research, but it is an opportunity that could be missed. The committee recommends that NASA, in cooperation with the NPOESS IPO, proceed as follows: 

Develop and implement a prototyping activity to link the NPP Science Data Segment with the NOAA LTA. This activity should start with NASA EOS data sets, including Atmospheric Infrared Sounder (AIRS)/Advanced Microwave Sounding Unit (AMSU)/MODIS in anticipation of the NPP data sets, including CriS/ATMS/VIIRS. Put aside reserve funds from data system development to support evolutionary development activities as the program matures to ensure that the system is not locked in with no resources for subsequent enhancement. Develop prototype user services for NPP climate data records. Recommendation 5b. Select, on a competitive basis, and then support an NPP science team as soon as possible. The team should consist of sensor experts, algorithm developers, and science data users. Because the functions will require different levels of involvement in the sensor development and operation process, they will require different levels of support. The team would advise on the NPP data system needs, including scientific data processing, archiving, and distribution requirements. Recommendation 6. NOAA, in cooperation with NASA, should invest in early, limited capability prototypes for both long-term archiving and the NPP data system. Data systems that do not develop, test, and evaluate on a frequent, regular basis are nearly always late and over budget. System development costs generally increase as the cube of the number of years in development. A climate data system will build on existing components and existing capabilities, but new functions and new interfaces must be developed and implemented to meet the requirements for climate research. The committee recommends that NOAA, in cooperation with NASA, take action as follows: Competitively select and support a science data team to assist a NOAA long-term archiving program on the following: —Archive requirements for long-term data sets, including RDRs, metadata, and ancillary data fields; —Archival of CDRs, algorithms, and processing environments; —Data structure and organization to facilitate access and reprocessing; —Flexible, open standards to facilitate data access and refinement; —Data reprocessing priorities; and —Minimal user services and tools. Require the NPOESS total system performance requirements (TSPR) contractor to work with the science data team to facilitate CDR production and archiving from both NPP and NPOESS. Develop flexible standards and formats that allow new services to be developed in the future. Begin to develop a small number of CDRs using the LTA services. Recommendation 7. NASA and NOAA should develop and support activities that will enable a blend of distributed and centralized data and information services for climate research. 

NASA and NOAA should consider a hybrid mode of operation rather than building a rigid, centralized system or relying on structure to emerge from an uncoordinated set of data systems. The government should ensure and manage the activities it does best, while fostering innovation and flexibility in those parts of the overall system that do not need to be closely managed. The committee recommends that NASA and NOAA proceed as follows: Implement the NPP data system as a federation of linked activities, such as that proposed in the NewDISS framework. Where appropriate, build on existing and planned capabilities, including EOSDIS, the Earth Science Information Partners, NASA's Distributed Active Archive Centers, and NOAA's Data Centers, and develop new capabilities as user experience is gained.