Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
5. A Vision of a National Information System for Global Change In the previous chapters of this report we listed some weaknesses of the current data management system and gave a partial list of the elements that a new system must include. In this chapter we sketch one possible realization of a data system that appears to satisfy all the concerns listed in the previous sections. The advantage in presenting such a vision is that in so doing contradictions, redun- dancies, or missing elements are more readily identified. However, we stress that other realizations are possible. We imagine a national information system (NIS) for global change built on existing systems, so that it can take full advantage of the strengths of existing centers, datasets, software, and expertise. Our approach does not require extensive reformatting or refiling of data nor the replacement of existing systems; rather we describe enhance- ments, methods for increasing the scientific content of the datasets and the overview of the system's operation, development of coordination and data exchange procedures, and improved user services. The approach given here should be consistent with a manageable and affordable development effort . Hierarchical Structure We envision a "virtual system," which, as described below, implies that the user sees a uniform and coherent structure. In actuality the system will be built by many participating organizations out of many different parts on different computers using different software. An additional layer of software and/or hardware will be used to tie together the pieces, so that the user can navigate through the entire system virtually as if it were a single entity. 38
A VISION OF A NATIONAL INFORMATION SYSTEM 39 National Coordination Uniform user view Research efforts to use & improve datasets Preservation Quality & Distribution Control & Analysis Data. Acquisition Figure 1 The National Information System (NIS) for global change as it might be organized in a four-level hierarchy: 1. Data acquisition level. 2. Quality control and analysis level. 3. Preservation and distribution level. 4. National coordination level.
40 A VISION OF A NATIONAL INFORMATION SYSTEM Conceptually, then, the user sees the NIS as organized in a four-level hierarchy, sketched in Figure 1: 1. Data acquisition level. 2. Quality control and analysis level. 3. Preservation and distribution level. 4. National coordination level. The previous sections of this report indicate that many ele- ments in the diagram are currently missing or do not function reliably. Many critical datasets are not gathered in a way that they can be synthesized into a global dataset. Many such datasets may not be avail- able to the scientific community at large. The external quality control level may be missing or unsatisfactory, or it may not be possible to control the quality of the data adequately. The preservation and distri- bution levels may not receive all the required data and may fail to preserve it. The system as a whole may not adequately serve the scien- tific community, especially users who are not experts in the particular dataset. Often missing are the connections between scientists active in research, modelling, or data assimilation and the centers. This loop is essential to improve the datasets. Finally, the national coordination level is missing. Data and information about data flow upward to the preservation and distribution level as shown in Figure 1. The feedback loops involve active research efforts and serve either to ensure quality or to identify errors or needed improvements for later uses of the data. The levels in Figure 1 are not directly mapped to physical locations; some centers may carry out the functions of several levels, or one function may be parcelled out to several groups. An example of a multifunction center is the Carbon Dioxide Information Analysis Center (CDIAC), which includes data gathering, quality control, documentation, preservation, and distribution of data. Some data, such as marine expendable bathythermograph data from the Pacific Ocean, are collected through an operational program or by individual scientists. These data are telemetered to the National Oceanographic Data Center (NODC), where the quality of the data is controlled for valid ranges. Then the data are sent to scientists at the Scripps Institution of Oceanography, where the data quality is again
A VISION OF A NATIONAL INFORMATION SYSTEM 41 considered and the data are documented. The cleaned-up products and documents are returned to NODC for preservation and distribution. The National Geophysical Data Center (NGDC) has similar arrange- ments with external scientists for some of its holdings. The planned Planetary Data System (PDS) has a different separation of functions: the National Space Science Data Center (NSSDC) preserves the data, while a central NASA/PDS node handles the distribution service, passing requests to lower nodes. Functions of Each Level Each level within the hierarchy has distinct functions as outlined below: 1. Data Acquisition Level: ⢠Gather data with suitable instrumentation. ⢠Process and calibrate data to an agreed-upon level. ⢠Supply data and metadata to the quality control level. ⢠Respond to queries and requests for more information or more detailed data either from the higher nodes or from users. 2. Quality Control and Analysis Level: ⢠Gather data from one or more suppliers. ⢠Check quality and consistency. ⢠Prepare products such as data summaries, figures, and browse files. ⢠Pass products to the distribution/preservation level. ⢠Gather metadata. ⢠Prepare catalog and inventory information and pass to higher nodes. ⢠Work with scientists using the datasets for modelling, synthesis, and assimilation. 3. Preservation and Distribution Level: ⢠Acquire catalog information, browse information, and validated datasets.
42 A VISION OF A NATIONAL INFORMATION SYSTEM ⢠Prepare directory information for local holdings and subnode holdings. ⢠Make information available to higher nodes. ⢠Work with users to supply metadata and browse information. ⢠Arrange delivery of datasets or facilitate delivery from any higher or lower node. ⢠Serve as the primary focus for user requests. ⢠Arrange for accounting of data costs and security. ⢠Aid in the development of user interfaces and browse capabilities. ⢠Establish feedback loops with scientists to ensure quality of the datasets. ⢠Arrange for long-term data preservation. 4. National Coordination Level: ⢠Arrange for transfer of information and user requests between various preservation/distribution centers. ⢠Define standards to allow heterogeneous elements to act as cooperating parts of a system. ⢠Ensure that lower nodes are functioning properly and effectively. ⢠Maintain an overall directory and lexicon (possibly distributed among the discipline nodes). User View An NIS user should be able to enter the system at any one of the distribution nodes and thereafter have transparent access to any information in the system. Requests would be transferred down to subnodes or up to the coordination node to be referred to another node. The user should see one user interface and deal with all the information as if it were stored in one coherent database. By referring to "information" rather than "data," we emphasize the necessity that metadata accompany the data. Metadata allow a contemporary user in another discipline to find and intelligently use
A VISION OF A NATIONAL INFORMATION SYSTEM 43 the data, help future users to understand limitations of the data, and simplify the search and retrieval of data. Another essential element of the NIS will be ongoing scientific research efforts connected with the global change datasets, both individually and in concert. A great increase in this kind of activity is needed, and the connections between researchers and data manage- ment activities must be greatly improved, recognized, funded, and (when appropriate) formalized. Advice and products from these groups must be incorporated into the information system. A Virtual National Information System Rationale By a "virtual national system" we mean that the aggregate behaves as though it were a single system. The scientific and opera- tional groups that collect the data, carry out quality control, preserve and distribute the data, and use the data for other scientific studies must have well defined interfaces with responsibilities toward the overall system. This must hold even though the elements involved may belong to different agencies and disciplines, may use different hardware and software, and may have existed for years or have been created only for global change research. Since each group can exchange information according to the specified protocols, the whole acts as a system that streamlines the flow of data from the collector to an interdisciplinary scientist or a global modeller. A virtual system has many advantages. By incorporating existing components directly, the system gains immediate access to a tremen- dous amount of data and expertise. This in turn builds stronger ties between researchers and existing data centers, develops a national center for coordination, and strengthens mechanisms for acquiring and assembling the required datasets. Software and hardware will be essential to automate searches and transfers of data and requests. This is the binder that holds the disparate elements together and, as such, must function smoothly and accurately. Agreements among agencies, organizations, and research groups on what information to exchange and how to do it will be
44 A VISION OF A NATIONAL INFORMATION SYSTEM essential to avoid interruptions in the data flow. Only motivated people will make it all happen. High-level software is needed in this virtual system. It should permit a query of the existing and newly developed database management systems to retrieve requested information and pass requests and information from one part of the system to another. An NIS user should not have to know about the different individual databases, data centers, agencies, and computers. Software and hardware developments can concentrate on defining protocols and standards (being careful not to make them limiting) and implementing the communications process. Individuals familiar with the databases can be responsible for integrating them into the system. Since the databases used in the individual components are not altered, they retain optimization for local use. They simultaneously serve a larger community. Changes in local systems and the addition of new centers will not affect the NIS except perhaps for making some information temporarily unavailable. Likewise, new technology, both hardware and software, can be integrated into the system, in effect by replacing one of the modular parts. Personnel able to handle both the database and the scientific aspects of the work at each center can be identified and brought into the system. Current understanding is not enough to design and build such a system in one centrally directed giant leap, even though the tech- nology for the separate components appears to be available. The approach must be stepwise, using prototypes that are tested both for utility and for the potential to become operational in a cost-effective manner and that help define the next step. Necessary Steps To implement the system imagined here starting from our current situation, we would take the following steps, which are grouped into three areas:
A VISION OF A NATIONAL INFORMATION SYSTEM 45 1. Flow of Information: ⢠Establish a plan for developing priorities for the datasets needed for the development of an earth system science. ⢠Ensure reliable sources of documented data (this is what the U.S. Global Change Research Program initiative has emphasized). ⢠Ensure quality control of the data and generation of products usable by researchers who may be unfamiliar with the details of the data collection equipment. ⢠Ensure that data and data products reach a data center where they can be distributed and preserved in a timely manner. ⢠Establish new data centers and groups as needed. ⢠Ensure that data centers, old and new, preserve data, distribute them, and provide feedback mechanisms to assess the scientific quality of the data they hold. ⢠Establish a structure to coordinate centers and overall activities and define responsibilities. 2. Scientific Involvement: ⢠Establish procedures, funding, and incentives for researchers to work with data centers in exercising datasets, generating new products, updating quality assessments, and influencing the data center. ⢠Set up scientific advisory and control mechanisms at each level. 3. Unimpeded Access: ⢠Develop protocols and methods for obtaining a uniform user view of all of the information in the NTS. ⢠Minimize paperwork and user costs. ⢠Set up effective user support personnel and procedures. ⢠Support research in scientific data management.
46 A VISION OF A NATIONAL INFORMATION SYSTEM Flow of Information One objective of the NIS development is to disrupt operations at existing data centers to the least extent possible. However, many current datasets do not supply full metadata; efforts in cooperation with interested and concerned scientists will be required to enhance current databases with the necessary metadata, calibration and validation information, references, and documentation. In the section Scientific Involvement, suggestions are given for specific recommen- dations that would help. Each center must ensure that the operations at each level are carried out efficiently and accurately. The suggested structure requires a national center for coordinating die centers, which would allow data to flow from one center to another and then to the user. Such a coordinating center must be able to work effectively across national agencies, a tricky issue that we bring up but leave open. Scientific Involvement The NIS must establish links to and cooperative efforts with research groups. Data management must be perceived as an activity critical to a successful research program; incentives for participating in NIS prototyping and construction must be created. Experts on methods and calibration of datasets will be needed to annotate extant datasets. Expert data analysts are needed who can take advantage of the growing database and produce worthwhile products which can themselves become part of the system. Assimilation modellers are needed who can produce dynamically consistent and complete datasets. Examples of specific steps that can be taken are as follows: 1. Establish postdoctoral programs and visiting scientist fellowships. These programs would bring a researcher to a data center for a specific data-intensive project, from which the data center would receive a specific product.
A VISION OF A NATIONAL INFORMATION SYSTEM 47 2. Fund academic groups through the data centers to perform quality control and documentation of selected data at their own institutions or to generate products based on data from the data centers. 3. Establish staff scientist positions at the data centers. This must be handled carefully, since a scientist remains a research scientist by publishing useful results in the open literature, not by performing administrative tasks and technical consulting. 4. Collocate data centers with research groups. The National Center for Atmospheric Research (NCAR) data support section is a good example: while it is well regarded for various reasons, the fact that data managers share building and computing facilities with a representative sample of their users, whose voice can be heard by the hierarchy at NCAR, provides a strong and immediate feedback loop to improve performance. Scientific papers and publications from these groups will help add to a wider perception of the value of the NIS. While there are many journals where science can be published, a high-quality journal where the data products are identified and thereby published should be considered. Advisory groups and users who find the system valuable in their own research will provide essential feedback and impetus for changes as well as inducement for more scientists to use the system. A system can be most elegant and beautiful yet fail if the users are not there. If NIS is used successfully, scientists involved in global change research will also be convinced of the value of expert management of and ready access to interdisciplinary information. Unimpeded Access Each subsystem (e.g., a solar-terrestrial data center) must be melded into the virtual system in the best possible way. This will require work at each site. To do this, the terminal interfaces currently supported on each system will have to be supplemented with an
48 A VISION OF A NATIONAL INFORMATION SYSTEM interface to the NIS that can receive and transmit queries and information according to the protocols established through the coordinating center. Internal query and response standards or protocols to be used in the NIS must be developed. Communications and control programs for various systems will need to be developed and installed on the hardware used by the different centers. Networks that can accommodate the work must be selected, and the systems must be hooked into them. Preliminary user interface(s) must be developed, tested, and used in demonstrations for the system. The NIS must develop a strong and viable support system. This will require ample professional documentation of the standards and first-generation software. New technical experts will be needed to aid in defining specifications and to develop the prototype. These will include programmers and system designers, as well as data managers and scientists. Knowledgeable user support people who are retained for long periods will be vital. Too many users have been discouraged in the past by difficulties in obtaining rapid and knowledgeable answers to queries, a job best done when the support personnel also appreciate the scientific content and meaning of a dataset. Here again, joint projects involving data center personnel and scientific users (rather than a simple service from the data center to the user) can ensure interested, motivated, and knowledgeable support personnel.
