9

Processing and Distributing Earth Observations and Information

INTRODUCTION

All scientific endeavors based on observations, including those envisioned for the U.S. Global Change Research Program (USGCRP), must acquire relevant and accurate data, transform the data into scientifically useful information, and distribute the information to scientists and others who will use it to advance understanding, applications, or education. Data operations for the USGCRP and the National Aeronautic and Space Administration's (NASA) Mission to Planet Earth are particularly challenging. First, the Earth Observing System (EOS) and the spacecraft of other nations will produce an unprecedented variety and amount of data. Second, the integration of data from other agencies and sources, including operational and in situ observations, will be necessary for a wide range of scientific purposes. Finally, the higher-level and summary data products will be sought and used by a broad constituency through global electronic networks.

THE EOS DATA AND INFORMATION SYSTEM: IMPLICATIONS FOR THE USGCRP

This chapter concentrates on the data and information issues associated with EOS: they are urgent in themselves, and they are also a prologue for the issues that must be resolved for the USGCRP as it develops early in the twenty-first century.

From the beginning, the space observation and data-handling components of EOS were considered to be essentially independent of one another. The instruments are to acquire the data, and the data are then to be processed and distributed by the EOS Data and Information System (EOSDIS). However, a number of



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade 9 Processing and Distributing Earth Observations and Information INTRODUCTION All scientific endeavors based on observations, including those envisioned for the U.S. Global Change Research Program (USGCRP), must acquire relevant and accurate data, transform the data into scientifically useful information, and distribute the information to scientists and others who will use it to advance understanding, applications, or education. Data operations for the USGCRP and the National Aeronautic and Space Administration's (NASA) Mission to Planet Earth are particularly challenging. First, the Earth Observing System (EOS) and the spacecraft of other nations will produce an unprecedented variety and amount of data. Second, the integration of data from other agencies and sources, including operational and in situ observations, will be necessary for a wide range of scientific purposes. Finally, the higher-level and summary data products will be sought and used by a broad constituency through global electronic networks. THE EOS DATA AND INFORMATION SYSTEM: IMPLICATIONS FOR THE USGCRP This chapter concentrates on the data and information issues associated with EOS: they are urgent in themselves, and they are also a prologue for the issues that must be resolved for the USGCRP as it develops early in the twenty-first century. From the beginning, the space observation and data-handling components of EOS were considered to be essentially independent of one another. The instruments are to acquire the data, and the data are then to be processed and distributed by the EOS Data and Information System (EOSDIS). However, a number of

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade significant issues, discussed in the sections below and covered in greater depth in Annex 1 (which provides a short history of the EOSDIS), have swirled around EOSDIS from the beginning. Attention over the past decade has focused on the design and implementation of a data and information system—with emphasis on the “system.” Today, in view of the issues, it may be advantageous to focus less on a system and more on the capabilities and attributes that will enable and encourage the most effective use of the data by scientists and others. Issues and Tensions The motivation for public investment in EOS is the enhancement in scientific understanding, applications, and education that will result from the observations. But for these benefits to be realized, the data must be readily available to all who would use them. Over the past decade the advance of computer and information systems has dramatically altered the possibilities of taking advantage of data streams with large data rates. Nevertheless, many issues remain. In fact, many of these issues are true for any large data system and hence also bear on the so-called Global Change Data and Information System. The only issue unique to EOS is the fact that it is space borne with specific instruments. There are three different areas of concern: Data stream processing and operations. For Earth observations such as those of EOS, there will be strong interdependencies among instruments that must be resolved during data processing, and thus algorithmic complexities and sequencing issues complicate the problems associated with large data rates. The instrument teams must find ways to interact effectively to produce accurate data and to document carefully and rigorously what has been done to the observations. These are management not technology issues, and it is important to seek a solution that is no more complex than the particular circumstances require. Archiving. Technological and conceptual advances together have dramatically improved communication lines between the sources and potential users of data. Individual scientists can have capabilities at their workstations that rival those of computer centers of a few years ago; and the power of contemporary communications capabilities such as the World Wide Web, which have revolutionized interactions among scientists as well as access to data sources. How much we should depend on this rapidly evolving environment in developing EOS data capabilities has been controversial. Questions about how to store and eventually archive data are also pressing, with modern technologies obviating old assumptions and offering wide replication of datasets as a potentially more secure approach than formal archiving.

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade Decisions on priorities. Developing any data management capability generates questions about what needs to be done and how do it. Clearly, the data and information capabilities associated with EOS should be responsive to the needs of science and its applications and should stimulate continuing advance. Most scientists will not want to be concerned with operations of the data system, but they know that decisions about data processing, access, and archiving will affect the quality and quantity of their work. The most important information system issues concern control, participation, and responsibility. As in managing any scientific program, these issues include setting priorities among known tasks, openness to new opportunities, and accountability for the use of public funds, all while facilitating the work of users. The broader the user community and the more complex the system becomes, the more likely it is that competing demands and conflicting requirements will lead to chaos rather than consensus. It has been assumed for a decade or more that the processes of producing, using, and archiving EOS observations could be described by a set of requirements specifying data rates and paths, specific capabilities and facilities needed by users, interoperability mandates, and archiving methods. The presumption has been that such a collection of requirements would lead, through the system design process, to a system that would resolve the issues discussed above. Today, we understand that this approach often leads to failure in creating large information systems, as has been demonstrated by a number of well-publicized efforts at the federal and other levels. The failures become inevitable because user needs are not well understood, are difficult to describe quantitatively, and lead to serious conflicts. Moreover, user needs will inevitably evolve rapidly as soon as users begin to work with the data. Hence, a system constructed on the basis of requirements derived from stated user needs has little resiliency, in part because the requirements often reflect an implicitly specified architecture. The problem is exacerbated by the federal procurement process, which often specifies a fairly detailed architecture along with a plethora of “requirements.”a Under such conditions there is little room for innovation, and the probabilities of failure and extraordinary cost, as has been amply demonstrated, are high. The astounding growth, first, of the Internet and, second, of the World Wide Web, illustrates another development path. In both cases a relatively simple set of rules or interface standards specifies how interactions will proceed. Users, through the creative power unleashed by highly distributed but interactive efforts, have developed the rest using the capability provided by the standards. a The Request for Proposals issued by NASA for the EOSDIS core system in 1991 specified the architecture of the system and listed more than 1,000 specific requirements for it to meet.

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade The long-standing tension and controversy over the basic nature of the EOS information system thus lead to a number of questions: Do contemporary information system concepts and technology suggest entirely new models? What actions will increase the probability of success of the information system and the activities it supports? Should the responsibilities of the scientific and other user communities be different than in the past? Should the responsibilities of NASA and other components of the federal government be different than in the past? Finding appropriate answers to these questions is critical to the future of the USGCRP and, in fact, to the future of all Earth observations. Again, we note that these issues are not unique to NASA or EOSDIS. The Original EOSDIS Federation Proposal The National Research Council (NRC) committee authoring the present report—the Committee on Global Change Research—examined NASA's plans for EOSDIS as part of its 1995 review of the USGCRP and made recommendations for significant changes in direction.1 The committee observed that the long-term functional requirements for EOS and other Earth information systems remain valid and must serve as the criteria for design and for judging success. It concluded that any concept for an EOS information system should ensure that several goals are met: Users can readily locate datasets with real and valuable scientific content. Users can access and utilize such datasets readily and in a timely fashion. Collaborative analysis and research are stimulated and encouraged. Demonstrable progress in scientific endeavors and applications to other activities is evident. Scientists and scientific teams can use the system for interaction as well as a form of electronic publication and dissemination of results. At that time the committee argued that the commitment to the “right model” for governance, management, and operation of the system would ensure that “the details related to design and technology would readily fall into place.”b Arguing that expectations for the information system mandated that the scientific community accept full partnership and shared responsibility, the committee identified two key requirements for the system model: that it use an open management approach in which consequential decisions are made with community leadership and with assignment of responsibilities based on peer review, and that it encour- b Note that this view of the model or conceptual framework for information system governance and management is quite different from the process described earlier, in which the implicit assumption is that a “right system” can be designed.

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade age innovation and creativity through wide participation of the scientific, public, and private sectors, and the committee then recommended that: Responsibility for product generation and publication and for user services should be transferred to a federation of partners selected through a competitive process and open to all.2 In this context a federation entails an association of autonomous partners creating a central structure to provide leadership and administration and agreeing to abide by certain interface standards, business practices, and expectations of conduct to achieve a common purpose (further basic information about this arrangement is presented in Annex 2, on an EOS information federation). To assist in implementing the recommendation, the 1995 report suggested that NASA explore key issues with a number of prototype efforts. The 1995 recommendations generated considerable discussion and examination of alternatives, especially by NASA advisory committees involved with the EOS project. As this discussion evolved, the differences between the original EOSDIS model and various federation models became clearer. The original concept might be described as a “data push” model, in which the flow of data from instruments would be processed in data centers and converted into a hierarchy of predetermined data products that would be available to users through a request mechanism. The federation concept retains “data push” processing of instrument data into physical variables but then expects “data demand and pull” to shape the higher-level products in response to actual needs, including data for scientific studies sponsored by the government, information summaries for business and policy considerations, and a wide range of purposes developed in a valued-added market. A panel of the NRC Committee of Geophysical and Environmental Data held a workshop on “federated” structure (i.e., the federation of partners recommended in the NRC 1995a report) for the Earth Science Enterprise (ESE). The report examines federated structures from a variety of organizations—libraries, international organizations, industry, government, and academia—and discussed objectives, governance, potential costs and benefits, measures of success, and lessons for an ESE federation. 3 Revisiting EOS Information Federation Models The EOS information federation model recommended by the NRC in 1995 envisioned that NASA would convert the EOS data it received from the spacecraft into geophysical units and would couple this information with spatial and temporal coordinates to produce a basic data stream. Further processing, distribution, analysis, and combination of the data would occur in two distinct efforts. The first would involve various federation partners supported by NASA through a competitive peer review process; the second would involve federation partners

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade TABLE 9.1 Three Possible Models for the EOS Information System Model Original EOSDIS Model A New Federation Model A Free Enterprise Model Description EOS instrument data processed by NASA, with algorithms developed by instrument teams. Instrument teams produce basic data streams with federal support and make them available to all others. All NASA and EOSDIS data funds flow to grants for research or applications to buy data, thus creating a market and a distributed capability.   Defined datasets and products prepared and distributed to all users by NASA or its contractors. All further data processing done by federation members, some with NASA support to ensure a representative menu of EOS products.     Government archives basic data and datasets. Government archives basic datasets.   Main advantages Data streams under NASA control; a specified set of products can be expected with confidence. Principal investigators responsible for initial processing; innovation likely in development of EOS products and information management capabilities; federation engages scientific and other communities. Market-driven approach should respond well to highest-priority efforts in science and applications; multiple capabilities and pathways will develop. Main disadvantages Lack of flexibility and innovation; conflicting requirements and high costs likely. Federal support not available for all potentially useful products; market may be thin for important products. Some data will never be processed unless NASA manipulates market with research grants. Comments System subject to unilateral NASA control. Appears desirable for global change researchers, but some science areas may be neglected. Both archiving and international collaboration present serious issues.

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade or others who developed value-added products at their own expense and made their own arrangements for distributing them to customers. A wider range of models for an EOS information system have been explored. Table 9.1 describes and examines three such models, including the original EOSDIS, a free enterprise model, and a combination of the two, denoted as a New Federation Model. In the NRC 1995 federation model the functions of the original EOS model would be performed by federation partners under contracts secured through a competitive process. Thus, the government would be responsible for determining a set of products to be produced under contract. That model was thus intermediate between the original EOS model and the particular new federation model described in Table 9.1, which allows more innovation in the federation but provides less assurance that a specific set of products will be available. The new federation model described in Table 9.1 provides a new approach to space-based observations for global change and other Earth system research. At the present time, as typified by EOS, the federal government contracts for the construction of an instrument to fly in space and for algorithms to process data from the instrument on a government computer for subsequent distribution to other scientists. In the new federation model the government would contract for delivery of a stream of data in geophysical units, along with navigation and other descriptive information, to a network that would make it available for further processing. Thus, rather than the data being processed on a government computer with algorithms provided by the instrument team, the data would be processed directly by the instrument team on its own computers or by a contractor under the team's control. Except for this change in responsibility for basic data processing, the new federation model is similar to the EOS model until the basic data processing is complete. Then it becomes similar to the NRC 1995 federation and, in some respects, to the free market model. A significant advantage is thus that the basic data processing is performed by the principal investigators under government sponsorship and management through the contract, presumably ensuring that the data streams from space will indeed be made available to both science and applications in a timely manner. Another advantage of the new federation model is that it provides a mechanism for NASA to continue to support experimentation and innovation in information access, management concepts, and technology that will advance global change research and applications, thus enhancing the value of the EOS data. MOVING TOWARD A NEW EOS INFORMATION SYSTEM With the EOS it has been recognized from the very first that making the data available for scientific research and other applications is as important as acquiring the basic observations from space. Indeed, if the data do not serve public

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade purposes of enhancing, understanding, and supporting other important activities, then there is no point in building the instruments and no point in launching and operating the spacecraft that carry them. Thus, NASA faces the dilemma that it must create a data system in the public interest but that it cannot create a satisfactory system by itself. A number of critical issues will arise for any choice of model for an Earth information system: The community must decide what it really wants and what responsibilities it is willing to accept. NASA and other senior federal officials must think through their attitudes about a federation or partnership, including what elements will be funded and how incentives are to be created and made effective. NASA must manage a complex relationship between itself and the parts of the EOS and global change research community that it is funding; it must provide incentives so that the associated institutions perform for the benefit of the entire community. The government must adopt some model and then be prepared to implement it. Many members of the EOS community have been thinking about EOSDIS for years and about federated models at least since this committee 's endorsement in 1995.4 A wide range of models has been considered, as shown by Table 9.1. The committee believes that the new federated model offers advantages over its 1995 proposal and that: The new EOS Federated Information System, including processing by the instrument teams to create the basic geophysical data, should be thoroughly explored to be sure it meets the needs of the four segments of the community: data producers; scientific assimilators and data consumers, including global change scientists; other scientific users; and the private sector. The new model should then either be adopted for EOS and the USGCRP or modified appropriately. Federations seem to be successful if their defining architecture concentrates on the highest-level goals and the interfaces between components. For the EOS Federated Information System the implication is that at first we try to envision only the capabilities that will facilitate scientific progress, the basic structure of the federation, and perhaps the inviolable rules. A candidate set of such rules is the following: The low-level data from all EOS sensors (the telemetry from space) must be preserved for at least several decades in a manner that enables access to, and reprocessing of, the entire dataset.

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade All data products from EOS sensors, including calibration and validation information, must be readily accessible and available to any user or entity that wishes to provide a new service. Access interfaces to the EOS Federated Information System should be clearly defined and sufficiently open to allow a broad range of services to be developed now and in the future. No data, product, or service produced from EOS sensors shall be restricted to any user based on proprietary rights. Data and service providers shall be able to charge for their services, as allowed by the federation. The formation of such a federation will require initiative and action on the part of the science community and the private sector. The governance structure must provide an adequate legal status for financial operations and an adequate executive structure to maintain operations and ensure that the federation is fulfilling its purpose. Presumably there will be both a formal structure with a governing board representing substantial partners and a wide range of volunteer activities. For such an organization to be successful, the incentives, as perceived by the participants, must be aligned with the purpose of the federation. Thus, NASA, as the principal initial source of funding, must devise contract award and management procedures that attract federation partners committed to high-quality science and service to their community. Mechanisms will be needed to ensure institutional credit for the complex task of publishing reliable and trustworthy datasets and supporting users, as well as for the usual regard for well-founded scientific conclusions (see Annex 3, on producing trustworthy scientific information in the EOS Information Federation). The federation must also resolve the apparent conflict between charges restricted to the costs of filling user requests (COFUR) and the aim of stimulating for-profit and not-for-profit organizations to develop value-added products. One possibility is to allow amortization of the costs of developing and producing value-added products to be charged to all users, with profit restricted by agreement or obtained only from for-profit use of the product. Another is to apply COFUR rules only to researchers seeking data for projects directly supported by federal funds and to allow the market to shape the pricing structure for all other public and private purposes. A third is to apply the COFUR policy only to those products that were produced entirely with government funding. Regardless of the specific model, the federation concept includes an assumption that funds to acquire EOS data, whether for basic products or value-added datasets, will be made available in federal research grants, thus creating part of the market for EOS data and supporting the federation. The relatively intense discussion of the 1995a NRC recommendation —namely that NASA adopt a federated competitive approach to EOSDIS —has sharpened understanding of the advantages and disadvantages of a range of data

OCR for page 435
GLOBAL ENVIRONMENTAL CHANGE: Research Pathways for the Next Decade and information system models. It has also intensified awareness of two unavoidable conclusions: NASA is accountable for what happens to EOS data, no matter what information system is implemented, and the scientific community can recommend a structure and an approach but cannot decide what to do. For the future we envision a federation that will be effective in transcending lists of requirements by providing enabling capabilities and by focusing on the value of EOS data for science and applications rather than on system mechanics. The operational details of the EOS information system will then fade into the background of an existing supportive, but largely invisible, global information infrastructure, and scientists will be able to concentrate on science. NOTES 1. NRC (1995a). 2. Ibid. 3. NRC (1998). 4. NRC (1995a). REFERENCES AND BIBLIOGRAPHY Dopplick, T. 1995. A Science User's Guide to the EOSDIS Core System (ECS) Development Process. Science Office, EOSDIS Core System Project, Technical Paper 160-TP-003-001 . Dutton, J.A. 1989. The EOS data and information system: Concepts for design. IEEE Transactions on Geoscience and Remote Sensing 27:109-116. Handy, C. 1992. Balancing corporate power: A new federalist paper. Harvard Business Review, (Nov.-Dec.):59-72. National Aeronautics and Space Administration. 1984. Earth Observing Science: Science and Mission Requirements Working Group Report. Technical Memorandum 86129. NASA, Washington, D.C. National Aeronautics and Space Administration. 1986. Report of the EOS Data Panel. Technical Memorandum 87777. NASA, Washington, D.C. National Research Council. 1982. Data Management and Computation, Volume 1, Issues and Recommendations. National Academy Press, Washington, D.C. National Research Council. 1995a. A Review of the U.S. Global Change Research Program and NASA's Mission to Planet Earth/Earth Observing System. National Academy Press, Washington, D.C. National Research Council. 1995b. Earth Observations from Space: History, Promise, Reality. National Academy Press, Washington, D.C. National Research Council. 1997. Bits of Power: Issues in Global Access to Scientific Data. National Academy Press, Washington, D.C. National Research Council. 1998. Toward an Earth Science Enterprise Federation: Results from a Workshop. National Academy Press, Washington, D.C. Weibel S., and C. Lagoze. 1997. An element set to support resource discovery—the state of the Dublin Core: January 1997. International Journal on Digital Libraries 1(2):176-186.