-
The NCDC wants to acquire and process more data, but they are already overwhelmed by the rescue and analysis of their current holdings. The physical deterioration of the manuscripts requires a greater rescue effort.
-
The panel applauds the goal of increasing customer services by increasing on-line access and by designating dataset “experts” who can effectively interact with the user community, particularly the science user community. The panel is concerned, however, that current staffing levels seem insufficient to fill orders from the scientific community in a timely fashion, especially as the number of these customers is likely to grow with increasing demand for global climate datasets.
-
The panel strongly endorses the NCDC's efforts to create, document, and archive the high-quality baseline climate datasets that are critical to climate change research.
-
The panel agrees that an ongoing analysis and upgrade of computing facilities is essential, but believes that the NCDC's computing capabilities are not being upgraded quickly enough to keep up with new demands in data access, processing, and storage.
-
The panel agrees that coordination within NOAA or with other agencies who are planning or operating climate observation systems is vitally important. Progress must be made not only to forestall data continuity problems, but also to determine which data are important enough to be stored at the NCDC.
Recommendations
As a result of its deliberations based on the input it has received, the panel offers the following recommendations to guide the NCDC in developing priorities to achieve their goals:
Recommendation 1: Higher priority should be given to recalibrating retrospective datasets that are critical to global change research.
Climate quality records are not simply aggregations of operational weather data. For purposes of studying climate, high standards of data
continuity and consistency are required, and upgrading present operational data to such standards will involve additional resources. Debate regarding the future impact of human activities on global climate hinges in part on understanding what that impact has been in the past. Under such circumstances, if the existing collection of radiosonde data over the past 30 40 years, and surface data since their beginnings, could be brought to climate standards, it would allow more solid conclusions about the resilience of the global climate system to be reached now. High priority should be given, therefore, to retrospective recalibration. Wherever possible, existing records should be reexamined to determine if and when changes in instrumentation and methodology occurred so that the data can be adjusted. As an example, the monthly Historical Climatology Network (HCN) dataset, prepared at the NCDC and widely used for U.S. climate studies, consists of a subset of (originally) 1,219 stations in the cooperative network. Old data were back-punched, detailed station records were prepared, measures of temporal non-homogeneity were prepared and used to calculate time-varying adjustment factors, and network managers were notified to give priority to maintaining continuity at these stations.
Similar efforts are particularly urgent for observations of humidity throughout the depth of the troposphere, where significant observational difficulties have long been identified, and where the importance of water vapor as the earth's primary greenhouse gas is clear. Upper level temperature and wind measurements also need reexamining. At the very least, one should be able to prepare climate datasets from the past 30 40 years where the uncertainties that remain from uncorrected problems are estimated. Moreover, regardless of our ability to retrieve past data, it is important to establish practices for acquiring future operational weather data that are consistent with the needs of climate studies so as to assure future quality.
In the context of global change studies, optimal improvement of existing records is probably the most cost-effective investment one can make. In this regard, plans are being made to reanalyze existing datasets with numerical prediction models in order to provide a global gridded dataset rather than the irregularly spaced sounding data. Without a good basic set of observations, however, such programs as the proposed reanalysis of existing records will be severely limited.
Recommendation 2: Rather than duplicating databases that exist at other data centers, resources may be better spent in improving access to the data, regardless of where they physically reside.
In the face of limited resources and major new data sources flooding the center, the NCDC should examine on a case-by-case basis whether it needs to physically acquire additional datasets, or if simply acquiring access to such datasets is sufficient. In the latter case, the NCDC should utilize its experience and expertise to work with the data's physical custodians to ensure that the requirements of NARA are met, that adequate metadata are available, and that the data are accessible to personnel in the User Services section at the NCDC. In recognition of the very rapid increase in networking, this would seem an opportune time to examine whether other approaches, involving distributed data storage and dissemination capabilities, might emerge as preferred modes of operation in the future. As a wide variety of publicly funded weather and climate databases become more available through user-friendly interfaces, the needs of a client may be served by access to either local or remote computers, at national, regional, or state data centers. The User Services Branch might benefit by utilizing such remote connections and by working with the remote managers to ensure that the remote systems will meet their needs. Although the future is difficult to predict, the networking revolution seems to be proceeding with inexorable inevitability. The NCDC should actively attempt to position itself to play a leading role in the coordination that will be required for a distributed system to function effectively.
Recommendation 3: The NCDC should expand its research program by hiring additional permanent or visiting scientific staff. These scientists should establish stronger links to the research user community through increased scientific collaborations.
Certain members of the NCDC's research staff have acquired a well-earned reputation for high quality work that is of great usefulness for a variety of important problems in basic and applied climatology. A large portion of this success is derived from close proximity to the data and
ready access to expertise on data interpretation that arise from being housed in the data facility.
The NCDC should expand its research program to increase its credibility within NOAA and the research community. This will also help it to generate resources. Increased scientific expertise would enable the NCDC to optimize archival strategies, to anticipate user needs, to recognize problems in datasets earlier, and to exploit the scientific information contained in the data more fully. In this last regard, the data streams resulting from the new observing systems offer an opportunity for the NCDC to create a new line of value-added products, providing that the necessary scientific expertise can be built in-house.
Recommendation 4: More effective use of standing advisory panels should be made to review strategic plans and to help set priorities, including how resources should be allocated among competing data archival projects.
To provide researchers with the data that they need, choices must be made about which datasets will be available on-line, and the data migrated to the appropriate storage media. As the first part of a data migration plan, the NCDC should work with its advisory panel to set priorities for its existing datasets as a function of demand, scientific merit, level of effort required to support the dataset, and danger of loss if not migrated from current form. From this prioritized list, the NCDC should identify the media for each dataset that is appropriate to the user community. This could include producing a CD-ROM (where the volume is relatively small), publishing in hard copy form, providing on-line access, providing near-line access, or providing off-line storage and preservation of the dataset for potential future migration and use. Another part of this activity is to assess and set priorities for datasets that the NCDC anticipates supporting in the future. Before accepting responsibility for future data, it would be in the NCDC's best interest to perform the above-mentioned analysis so that sufficient resources could be identified and provided to the NCDC along with the data.
The NCDC should also seek input from a broader user community that includes economic sectors (agriculture, engineering, tourism, recre-
ation, and utilities), private meteorologists, non-NOAA government agencies, professional societies, and the state and regional climate centers. The advice and support of this large, nationwide constituency is an invaluable asset to the NCDC, and should help promote the NCDC 's position within NOAA.
Recommendation 5: The NCDC should place higher priority on making their data more generally accessible, including increasing the production rate of CD-ROMs with high-profile datasets.
Some users within the scientific and commercial communities have had difficulties in obtaining raw data or quality-controlled datasets from the NCDC. It is crucial, therefore, for the NCDC to implement its goal of improving user services to avoid further alienating its user communities. One way of increasing access to data is to greatly increase the production of CD-ROMs. The use of CD-ROMs to distribute limited datasets of potential widespread interest has been growing in popularity. Although somewhat slow to respond to this trend, the NCDC has by now produced more than a half-dozen discs and has plans for many more. A hallmark of the NCDC discs is the software that is included to help manipulate and visualize the data; the development and customization of this software for each disc help explain delays in producing the disc series, and the desire to make the series as user-friendly as possible is commended. In this regard, however, the NCDC ought to be aware that its software packages can be executed only on IBM PC compatible systems and that many of its customers own alternative systems. Hence, the NCDC ought to place a high priority on making the use of its CD-ROMs more independent of computer platform or, alternatively, providing software support for additional operating systems. More effort ought also to be given to making this disc series better known; the advisory panel may be able to provide guidance about such publicity, in addition to helping set priorities for datasets to be included in the series, as noted above. Finally, the NCDC ought to exchange insights and expertise in producing CD-ROMs with the other national data centers to maximize efficiencies in this process.
Recommendation 6: The planned computer systems upgrade, including a hierarchical storage system and expanded networking capability, should have high priority. As much data as possible should be stored in a form that allows users rapid access.
Modern storage and retrieval technology allows rapid inventory, access, and transfer of large volumes of useful data. From the panel's brief introduction to the NCDC's capabilities, it is apparent that the center has not kept up with available technology (e.g., the networking and data storage revolution) and that its facilities are inadequate to provide useful access to the tremendous amount of raw or processed data. This deficiency inhibits any plan to place even modest amounts of data on-line for remote access by research oriented users.
Recommendation 7: The NCDC should have a role in planning NOAA's data collection efforts for purposes of archival and data continuity.
As the end of the data pipeline, the NCDC historically had to accept what poured out, in whatever form it was in. The real source of many of the problems that the NCDC faced, and still does, is at a higher organizational level and at an earlier time. There should be better coordination among all NOAA units that are involved in the activities that result in data being archived by the NCDC. This includes activities that are already underway as well as planned new activities. As things now stand, the needs of the archival function and data continuity through sufficient overlap of observation systems are not adequately considered during planning. There has been some improvement in recent years, but these areas still need attention at the NOAA level.
Recommendation 8: The NCDC should strive for good communication with state, regional, and national partners to ensure that the nation's climate data are adequately stored and available.
The center is to be commended for trying to keep abreast of the recent rapidly developing changes in the national climate service sector. To refer requests to the most appropriate information source(s), the
