1993 Review of the World Center-A for Meteorology and the National Climatic Data Center (1994)

fraction of the total cost of the ASOS program. Moreover, this approach is susceptible to power outages and other interruptions. This undue dependence on so fragile a data collection system could have been avoided had the need to archive these data been considered earlier in the planning of ASOS.

The panel's concerns cover a wide spectrum ranging from the inaccessibility of nineteenth and early twentieth century data through inadequacies in the present service and the impending deluge of data from new meteorological observing systems. The NCDC is being overwhelmed by simultaneous problems: 1) rescuing deteriorating paper records, 2) facing increasing demand for access to global and national data, and 3) explosive growth in automated digital data that must be received, processed, and archived. Our perception is that funding has not kept up with the demand for services as manifested by the observed problems described below.

The data rescue efforts are of two types: 1) transfer of data on microfilmed punched cards to magnetic tape, and 2) reduction of old paper records to microfiche or microfilm. Although funding has been inadequate to ensure the timely rescue of the microfilmed data, the technical problems that had bedeviled the film-to-tape conversion process for years may now be solved by a film reader recently acquired from the Bureau of the Census. Of greater concern is the more than three decades of very slow and irregular progress in microfilming the vast collection of pre-World War II manuscript weather records. This sad history leads to the perception of the panel that NOAA generally does not recognize the value of retrospective data. Significantly greater priority and funding must be assigned to this work if loss of some of the deteriorating paper records is to be avoided. At the current rate, filming and other conversion methods

of the manuscript records will not be completed until well into the next century.

At present, the NCDC receives more than 400 data and information requests per day. Nearly three-quarters of these requests come from lawyers, teachers, consultants, and other groups that do not routinely use the center's data for research purposes. These customers have relatively simple demands, and the NCDC has been largely successful in filling their orders in a timely fashion. More specialized and complicated requests tend to come from the scientific research community, and to service these needs the NCDC has established a Research Customer Service Group. Despite the accomplishments of this group, the panel is of the opinion that existing facilities at the NCDC are inadequate to satisfy much of the climate research community interested in obtaining and manipulating pertinent datasets. Because the number of such requests and the volume of the datasets are both expected to increase, it is evident that user service will deteriorate further unless remedial action is taken.

The panel believes that the inadequacy of service to the science user community drives researchers to use other data centers. Global gridded datasets have become invaluable sources of information about the dynamics of the climate system, and researchers are more likely to use a data center whose scientists understand such data and can provide it to others in a timely, convenient, and inexpensive fashion. These needs are better met at the NCAR than the NCDC. Station-based datasets will continue to be needed for climate research, but requests for information derived from these are also apt to become more complex and demanding. Improvements in database and computer technologies at the NCDC will help in satisfying such requests, but this can be only part of the solution. The scientific staff at the NCDC also needs to be expanded, so that the center can acquire a working knowledge of the full range of information relevant to climate research and, armed with this knowledge, can properly design computer systems to serve the growing needs of the science community. Unless upgrades to both computer facilities and scientific staff are accomplished, these needs will remain unmet by the NCDC.

At the time of the review, data at the NCDC were archived on the following systems: 1) a UNISYS 2200/400, which has 16 MB of memory and 19 GB of on-line storage capacity; 2) a Hewlett Packard optical storage system, which holds 85 GB; and 3) a VAX 11/780 with 6.5 MB of memory and 1.4 GB of disk space. In addition, the center has a file server with 4.7 GB of disk space. For simple data analysis, the center has a local area network of 120 “scientific” PCs, 17 data entry PCs, and 100 office automation PCs. The research group at the NCDC has 10 SUN SPARC-2 workstations that support CARDS (Comprehensive Aerological Reference Data Set), the OASIS system, and other research projects. Finally, data processing and system development are supported by 13 IBM RS-6000s.

The NCDC currently stores 181 TB (1 TB = 1000 GB) of data, 145 TB of which reside at the University of Wisconsin. The holdings in Asheville, which are mainly stored on model 3480 cartridge and exabyte tapes, are growing at a rate of 10-12 TB/year. As the NEXRAD sites become fully operational by 1996, however, the holdings will grow by more than 86 TB/year. It is clear that the NCDC's current computing capability is insufficient to accommodate the large-volume data streams associated with ASOS, NEXRAD, and the Geostationary Operational Environmental Satellite (GOES) system. The NCDC is acquiring a hierarchical mass storage system with 128 MB of memory, 18 GB of local disk space, 23 GB of RAID (Redundant Array of Inexpensive Disks) storage, and the possibility of attaching up to 10 TB of high speed tape. While the NCDC is to be commended for trying to address their digital data storage problem, it is clear that even these planned archival storage systems will be overwhelmed by the future growth in their holdings.

Current developments in mass data storage technologies offer data centers like the NCDC the opportunity to provide researchers access to datasets in a timely and efficient fashion. Existing systems using either optical disc or high density tape cartridges offer near-line storage for tens to hundreds of terabytes. If the NCDC is to provide the kind of service to users that they anticipate, then it is absolutely essential that they devel-

op a data migration plan and the corresponding hardware procurement and upgrade plan.

Of concern to the panel is the generic limitation imposed on the NCDC by the basic organization of NOAA. Data collection, processing, and archival are separate functions within NOAA, with little consistent opportunity for feedback. As discussed in connection with ASOS, the NCDC has had only a limited role in determining which datasets will be archived, and it must accept datasets without the corresponding understanding of the user demand for these datasets and the costs associated with their support.

NOAA's National Weather Service (NWS) is the largest source of data archived at the NCDC. Appropriate roles for the NWS relating to data collection have been identified in the National Research Council (NRC) report Toward a New National Weather Service -- Second Report (Commission on Engineering and Technical Systems, 1992). This publication includes a number of recommendations that stress the importance of collecting high quality data for both weather forecasting and climate purposes. These recommendations, which continue to be relevant to the NCDC, address the need for the following: 1) funding and infrastructure to collect quality climate data, 2) quality criteria that are useful for both the needs of the NWS and the climate record, and 3) overlap of instruments and simultaneous operation of old and new instrument sites to establish data continuity. The report concludes that NOAA 's response to similar recommendations from the first report (NRC, 1991) was inadequate. Recently, however, NOAA appears to be giving the issue of data and information management renewed attention (NOAA, 1994), offering hope that this subject will now receive the priority it deserves across the agency.